Ceph backports

Ceph backports

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 1486 bytes --]

Hi Ceph,

I'm going to spend time to care for the Ceph backports (i.e. help reduce the time they stay in pull requests or redmine tickets). It should roughly go as follows:

0. Developer follows normal process to land PR to master. Once complete and ticket is marked Pending Backport this process initiates.
1. I periodically polls Redmine to look for tickets in Pending Backport state
2. I find commit associated with Redmine ticket and Cherry Picks it to backport integration branch off of desired maintenance branch (Dumping, Firefly, etc). (Note - patch may require backport to multiple branches)
3. I resolve any merge conflicts with the cherry-picked commit
4. Once satisfied with group of backported commits to integration branch, I notifies QE.
5. QE tests backport integration branch against appropriate suites
6a. If QE is satisfied with test results, they merge backport integration branch.
6b. If QE is NOT satisfied with the test results, they indicate backport integration branch is NOT ready to merge and return to me to work with original Developer to resolve issue and return to steps 2/3
7. Ticket is moved to Resolved once backport integration branch containing cherry-picked backport is merged to the desired mainteance branch(es)

I'll first try to implement this semi manually and document / script when convenient. If anyone has ideas to improve this tentative process, now is the time :-)

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ceph backports

[John Spray]

Sounds sane -- is the new plan to always do backports via this
process?  i.e. if I see a backport PR which has not been through
integration testing, should I refrain from merging it?

John

On Mon, Jan 5, 2015 at 11:53 AM, Loic Dachary <loic@dachary.org> wrote:
> Hi Ceph,
>
> I'm going to spend time to care for the Ceph backports (i.e. help reduce the time they stay in pull requests or redmine tickets). It should roughly go as follows:
>
> 0. Developer follows normal process to land PR to master. Once complete and ticket is marked Pending Backport this process initiates.
> 1. I periodically polls Redmine to look for tickets in Pending Backport state
> 2. I find commit associated with Redmine ticket and Cherry Picks it to backport integration branch off of desired maintenance branch (Dumping, Firefly, etc). (Note - patch may require backport to multiple branches)
> 3. I resolve any merge conflicts with the cherry-picked commit
> 4. Once satisfied with group of backported commits to integration branch, I notifies QE.
> 5. QE tests backport integration branch against appropriate suites
> 6a. If QE is satisfied with test results, they merge backport integration branch.
> 6b. If QE is NOT satisfied with the test results, they indicate backport integration branch is NOT ready to merge and return to me to work with original Developer to resolve issue and return to steps 2/3
> 7. Ticket is moved to Resolved once backport integration branch containing cherry-picked backport is merged to the desired mainteance branch(es)
>
> I'll first try to implement this semi manually and document / script when convenient. If anyone has ideas to improve this tentative process, now is the time :-)
>
> Cheers
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph backports

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 2003 bytes --]



On 05/01/2015 13:03, John Spray wrote:
> Sounds sane -- is the new plan to always do backports via this
> process?  i.e. if I see a backport PR which has not been through
> integration testing, should I refrain from merging it?

I think that's the idea, indeed. QE does the merge, when and if tests are green.

> 
> John
> 
> On Mon, Jan 5, 2015 at 11:53 AM, Loic Dachary <loic@dachary.org> wrote:
>> Hi Ceph,
>>
>> I'm going to spend time to care for the Ceph backports (i.e. help reduce the time they stay in pull requests or redmine tickets). It should roughly go as follows:
>>
>> 0. Developer follows normal process to land PR to master. Once complete and ticket is marked Pending Backport this process initiates.
>> 1. I periodically polls Redmine to look for tickets in Pending Backport state
>> 2. I find commit associated with Redmine ticket and Cherry Picks it to backport integration branch off of desired maintenance branch (Dumping, Firefly, etc). (Note - patch may require backport to multiple branches)
>> 3. I resolve any merge conflicts with the cherry-picked commit
>> 4. Once satisfied with group of backported commits to integration branch, I notifies QE.
>> 5. QE tests backport integration branch against appropriate suites
>> 6a. If QE is satisfied with test results, they merge backport integration branch.
>> 6b. If QE is NOT satisfied with the test results, they indicate backport integration branch is NOT ready to merge and return to me to work with original Developer to resolve issue and return to steps 2/3
>> 7. Ticket is moved to Resolved once backport integration branch containing cherry-picked backport is merged to the desired mainteance branch(es)
>>
>> I'll first try to implement this semi manually and document / script when convenient. If anyone has ideas to improve this tentative process, now is the time :-)
>>
>> Cheers
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
>>

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ceph backports

[Gregory Farnum]

On Mon, Jan 5, 2015 at 4:16 AM, Loic Dachary <loic@dachary.org> wrote:
>
>
> On 05/01/2015 13:03, John Spray wrote:
>> Sounds sane -- is the new plan to always do backports via this
>> process?  i.e. if I see a backport PR which has not been through
>> integration testing, should I refrain from merging it?
>
> I think that's the idea, indeed. QE does the merge, when and if tests are green.
>
>>
>> John
>>
>> On Mon, Jan 5, 2015 at 11:53 AM, Loic Dachary <loic@dachary.org> wrote:
>>> Hi Ceph,
>>>
>>> I'm going to spend time to care for the Ceph backports (i.e. help reduce the time they stay in pull requests or redmine tickets). It should roughly go as follows:
>>>
>>> 0. Developer follows normal process to land PR to master. Once complete and ticket is marked Pending Backport this process initiates.
>>> 1. I periodically polls Redmine to look for tickets in Pending Backport state
>>> 2. I find commit associated with Redmine ticket and Cherry Picks it to backport integration branch off of desired maintenance branch (Dumping, Firefly, etc). (Note - patch may require backport to multiple branches)
>>> 3. I resolve any merge conflicts with the cherry-picked commit
>>> 4. Once satisfied with group of backported commits to integration branch, I notifies QE.
>>> 5. QE tests backport integration branch against appropriate suites
>>> 6a. If QE is satisfied with test results, they merge backport integration branch.
>>> 6b. If QE is NOT satisfied with the test results, they indicate backport integration branch is NOT ready to merge and return to me to work with original Developer to resolve issue and return to steps 2/3
>>> 7. Ticket is moved to Resolved once backport integration branch containing cherry-picked backport is merged to the desired mainteance branch(es)
>>>
>>> I'll first try to implement this semi manually and document / script when convenient. If anyone has ideas to improve this tentative process, now is the time :-)


Okay, I've got a bunch of questions. In your first email it sounds
like you're saying this is how you're going to voluntarily handle some
backports. But in response to John's email it sounds like you want to
gate all backports through yourself and this process. Is this a
request that everybody else stops performing backports, and have you
checked with enough usual suspects to make sure they'll respect the
process? ;)

I am 100% on board with making QE responsible for gating backports, so
thank you for starting down that path. :) But I'm not at all sure how
this scales for you. Right now backports are nominally run through two
important checks:
1) Is it suitable for backport (decided by author or tech lead, marked
via the Pending Backport tag)
2) Has it been through sufficient validation in master to be safe to
backport (not marked in the system anywhere, just by somebody actually
doing the backport).

Knowing if something has been through sufficient validation to
backport requires a fair bit of attention to the details of the ticket
and the patches involved. How do you plan to keep up on that?

Similarly, while point releases are largely ad-hoc, we are trying to
involve all the leads in the time-to-go decision. A lot of those
decisions rest on whether specific backports have been performed yet,
whether there are very new backports we want to run through testing
for a little longer, etc. That sounds like a lot of communications
overhead between the backport gates and the leads when making these
kinds of decisions and I'm not sure how that should happen; is there a
plan? (We can look at ticket status for things which are pending
backport, but that doesn't facilitate prioritizing their backports;
and in the opposite direction there's not a good way to say "this
relatively large backport needs to go through at least three test runs
before a release".)
-Greg

Re: Ceph backports

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 6155 bytes --]



On 06/01/2015 00:24, Gregory Farnum wrote:
> On Mon, Jan 5, 2015 at 4:16 AM, Loic Dachary <loic@dachary.org> wrote:
>>
>>
>> On 05/01/2015 13:03, John Spray wrote:
>>> Sounds sane -- is the new plan to always do backports via this
>>> process?  i.e. if I see a backport PR which has not been through
>>> integration testing, should I refrain from merging it?
>>
>> I think that's the idea, indeed. QE does the merge, when and if tests are green.
>>
>>>
>>> John
>>>
>>> On Mon, Jan 5, 2015 at 11:53 AM, Loic Dachary <loic@dachary.org> wrote:
>>>> Hi Ceph,
>>>>
>>>> I'm going to spend time to care for the Ceph backports (i.e. help reduce the time they stay in pull requests or redmine tickets). It should roughly go as follows:
>>>>
>>>> 0. Developer follows normal process to land PR to master. Once complete and ticket is marked Pending Backport this process initiates.
>>>> 1. I periodically polls Redmine to look for tickets in Pending Backport state
>>>> 2. I find commit associated with Redmine ticket and Cherry Picks it to backport integration branch off of desired maintenance branch (Dumping, Firefly, etc). (Note - patch may require backport to multiple branches)
>>>> 3. I resolve any merge conflicts with the cherry-picked commit
>>>> 4. Once satisfied with group of backported commits to integration branch, I notifies QE.
>>>> 5. QE tests backport integration branch against appropriate suites
>>>> 6a. If QE is satisfied with test results, they merge backport integration branch.
>>>> 6b. If QE is NOT satisfied with the test results, they indicate backport integration branch is NOT ready to merge and return to me to work with original Developer to resolve issue and return to steps 2/3
>>>> 7. Ticket is moved to Resolved once backport integration branch containing cherry-picked backport is merged to the desired mainteance branch(es)
>>>>
>>>> I'll first try to implement this semi manually and document / script when convenient. If anyone has ideas to improve this tentative process, now is the time :-)
> 
> 
> Okay, I've got a bunch of questions. In your first email it sounds
> like you're saying this is how you're going to voluntarily handle some
> backports. But in response to John's email it sounds like you want to
> gate all backports through yourself and this process. Is this a
> request that everybody else stops performing backports, and have you
> checked with enough usual suspects to make sure they'll respect the
> process? ;)

:-) This process is helpful if it allows me to help a little more than I currently do with the backport process. It would be a loss if the end result is that everyone cares less about backports. My primary incentive for sending this mail is to start the conversation and avoid that kind of unintended drawback.

> I am 100% on board with making QE responsible for gating backports, so
> thank you for starting down that path. :) But I'm not at all sure how
> this scales for you. Right now backports are nominally run through two
> important checks:
> 1) Is it suitable for backport (decided by author or tech lead, marked
> via the Pending Backport tag)
> 2) Has it been through sufficient validation in master to be safe to
> backport (not marked in the system anywhere, just by somebody actually
> doing the backport).
> 
> Knowing if something has been through sufficient validation to
> backport requires a fair bit of attention to the details of the ticket
> and the patches involved. How do you plan to keep up on that?

I can't do that all by myself.

> Similarly, while point releases are largely ad-hoc, we are trying to
> involve all the leads in the time-to-go decision. A lot of those
> decisions rest on whether specific backports have been performed yet,
> whether there are very new backports we want to run through testing
> for a little longer, etc. That sounds like a lot of communications
> overhead between the backport gates and the leads when making these
> kinds of decisions and I'm not sure how that should happen; is there a
> plan? (We can look at ticket status for things which are pending
> backport, but that doesn't facilitate prioritizing their backports;
> and in the opposite direction there's not a good way to say "this
> relatively large backport needs to go through at least three test runs
> before a release".)

Could you point me to a mail thread / IRC conversation that is representative of this process ?

Here is a revised process which is hopefully more realistic:

0. Developer follows normal process to land PR to master. Once complete and ticket is marked Pending Backport this process initiates.
1. I periodically polls Redmine to look for tickets in Pending Backport state and focus on the ones that are left unattended for too long
1a. Under the supervision of the author of the original patch, I find the commits associated with the Redmine ticket and Cherry Pick to the backport integration branch off of the desired maintenance branch (Dumping, Firefly, etc).
1b. I resolve any merge conflicts with the cherry-picked commit
2. I merge all backports for a given branch in an integration branch
3. I ask the leads of each project to review the integration
4. Once satisfied with group of backported commits to integration branch, I notify QE.
5. QE tests backport integration branch against appropriate suites
6a. If QE is satisfied with test results, they merge backport integration branch.
6b. If QE is NOT satisfied with the test results, they indicate backport integration branch is NOT ready to merge and return to me to work with original Developer to resolve issue and return to steps 2/3
7. Ticket is moved to Resolved once backport integration branch containing cherry-picked backport is merged to the desired mainteance branch(es)

What do you think ?

Cheers

> -Greg
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ceph backports

[Gregory Farnum]

On Mon, Jan 5, 2015 at 4:12 PM, Loic Dachary <loic@dachary.org> wrote:
> :-) This process is helpful if it allows me to help a little more than I currently do with the backport process. It would be a loss if the end result is that everyone cares less about backports. My primary incentive for sending this mail is to start the conversation and avoid that kind of unintended drawback.

Why do you want to get involved with other people's backports at all?
I don't mean that to sound possessive, but having the patch's primary
author responsible for getting backports done at least has the
singular merit of sharding the work up into manageable pieces. ;)

>
>> I am 100% on board with making QE responsible for gating backports, so
>> thank you for starting down that path. :) But I'm not at all sure how
>> this scales for you. Right now backports are nominally run through two
>> important checks:
>> 1) Is it suitable for backport (decided by author or tech lead, marked
>> via the Pending Backport tag)
>> 2) Has it been through sufficient validation in master to be safe to
>> backport (not marked in the system anywhere, just by somebody actually
>> doing the backport).
>>
>> Knowing if something has been through sufficient validation to
>> backport requires a fair bit of attention to the details of the ticket
>> and the patches involved. How do you plan to keep up on that?
>
> I can't do that all by myself.
>
>> Similarly, while point releases are largely ad-hoc, we are trying to
>> involve all the leads in the time-to-go decision. A lot of those
>> decisions rest on whether specific backports have been performed yet,
>> whether there are very new backports we want to run through testing
>> for a little longer, etc. That sounds like a lot of communications
>> overhead between the backport gates and the leads when making these
>> kinds of decisions and I'm not sure how that should happen; is there a
>> plan? (We can look at ticket status for things which are pending
>> backport, but that doesn't facilitate prioritizing their backports;
>> and in the opposite direction there's not a good way to say "this
>> relatively large backport needs to go through at least three test runs
>> before a release".)
>
> Could you point me to a mail thread / IRC conversation that is representative of this process ?

No; that's pretty much all done in video chats. :/

>
> Here is a revised process which is hopefully more realistic:
>
> 0. Developer follows normal process to land PR to master. Once complete and ticket is marked Pending Backport this process initiates.
> 1. I periodically polls Redmine to look for tickets in Pending Backport state and focus on the ones that are left unattended for too long
> 1a. Under the supervision of the author of the original patch, I find the commits associated with the Redmine ticket and Cherry Pick to the backport integration branch off of the desired maintenance branch (Dumping, Firefly, etc).
> 1b. I resolve any merge conflicts with the cherry-picked commit
> 2. I merge all backports for a given branch in an integration branch
> 3. I ask the leads of each project to review the integration
> 4. Once satisfied with group of backported commits to integration branch, I notify QE.
> 5. QE tests backport integration branch against appropriate suites
> 6a. If QE is satisfied with test results, they merge backport integration branch.
> 6b. If QE is NOT satisfied with the test results, they indicate backport integration branch is NOT ready to merge and return to me to work with original Developer to resolve issue and return to steps 2/3
> 7. Ticket is moved to Resolved once backport integration branch containing cherry-picked backport is merged to the desired mainteance branch(es)
>
> What do you think ?

I think if we're going to add a process to anything it should be
followed by everybody involved. I really would love for everything to
be gated by QE before it goes into a backport branch, but if you're
going off and building integration branches and QE is testing them, I
think other people are going to keep backporting as we have been and
trip all over each other. We've periodically used "firefly-next"
branches and related things, but it's always been ad-hoc.

Something more realistic might involve locking down the stable
branches so they can only be merged into by QE or some approved group,
and then letting people do their own backports onto a
<stable-branch>-next that is periodically taken up and
integration-tested prior to merge into the LTS proper. That ensures
that only patches which have all been tested together get into a
stable branch without forcing each individual backport into a lot of
process.
-Greg

Re: Ceph backports

[Sage Weil]

On Mon, 5 Jan 2015, Gregory Farnum wrote:
> On Mon, Jan 5, 2015 at 4:12 PM, Loic Dachary <loic@dachary.org> wrote:
> > :-) This process is helpful if it allows me to help a little more than I currently do with the backport process. It would be a loss if the end result is that everyone cares less about backports. My primary incentive for sending this mail is to start the conversation and avoid that kind of unintended drawback.
> 
> Why do you want to get involved with other people's backports at all?

Because Ian and I asked.  :)

> I don't mean that to sound possessive, but having the patch's primary
> author responsible for getting backports done at least has the
> singular merit of sharding the work up into manageable pieces. ;)

We have a huge backlog of backports and need some process to move them 
forward.  The goals are to

 - have some process that prevents backports from lingering
   indefinitely
 - test everything in an integration branch before pushing
   to the $stable branch
 - let QE handle the actual integration branch testing
 - limit time investment by leads where possible

> > Here is a revised process which is hopefully more realistic:
> >
> > 0. Developer follows normal process to land PR to master. Once complete and ticket is marked Pending Backport this process initiates.
> > 1. I periodically polls Redmine to look for tickets in Pending Backport state and focus on the ones that are left unattended for too long
> > 1a. Under the supervision of the author of the original patch, I find the commits associated with the Redmine ticket and Cherry Pick to the backport integration branch off of the desired maintenance branch (Dumping, Firefly, etc).
> > 1b. I resolve any merge conflicts with the cherry-picked commit
> > 2. I merge all backports for a given branch in an integration branch
> > 3. I ask the leads of each project to review the integration
> > 4. Once satisfied with group of backported commits to integration branch, I notify QE.
> > 5. QE tests backport integration branch against appropriate suites
> > 6a. If QE is satisfied with test results, they merge backport integration branch.
> > 6b. If QE is NOT satisfied with the test results, they indicate backport integration branch is NOT ready to merge and return to me to work with original Developer to resolve issue and return to steps 2/3
> > 7. Ticket is moved to Resolved once backport integration branch containing cherry-picked backport is merged to the desired mainteance branch(es)
> >
> > What do you think ?
> 
> I think if we're going to add a process to anything it should be
> followed by everybody involved. I really would love for everything to
> be gated by QE before it goes into a backport branch, but if you're
> going off and building integration branches and QE is testing them, I
> think other people are going to keep backporting as we have been and
> trip all over each other. We've periodically used "firefly-next"
> branches and related things, but it's always been ad-hoc.
> 
> Something more realistic might involve locking down the stable
> branches so they can only be merged into by QE or some approved group,
> and then letting people do their own backports onto a
> <stable-branch>-next that is periodically taken up and
> integration-tested prior to merge into the LTS proper. That ensures
> that only patches which have all been tested together get into a
> stable branch without forcing each individual backport into a lot of
> process.

That sounds more or less like what Loic proposed, except that leads are 
always doing the backports.  I'm not sure that will scale, and lots of 
fixes are simple and don't need a huge amount of attention.  I agree that 
it'd be great to have a common workflow, though.

I think the idea is as much to have someone driving the process ("should 
this get backported now?  oh, you want to do it?  ok, ping me when you 
have a PR ready") as to dictate that they do the actual backports.

Maybe we augment Loic's list by changing (1) so that anybody can do 
the backport and open a pull request to merge to the $stable branch.  
The original patch author can do it (ideally at the same time as 
the original fix, espeically when non-trivial), the lead can do 
it, or Loic (or anyone) can do it.

The key piece is that the redmine tickets are regularly polled to ensure 
the backports get done.  We can note things like "wait a few weeks before 
backport" in the ticket or the backport pull request, or possibly in an 
explicit redmine field, depending on how formal we want to be.  (I'd say 
start simple.)

Then Loic (or whoever) prepares an integration branch and passes it to QE 
to test.  If the tests go well, the pull requests can be merged.  
Conflicts here are generally pretty rare, but when present and non-trivial 
the obviously others can review as needed.

?

sage

Re: Ceph backports

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 6470 bytes --]



On 06/01/2015 01:22, Gregory Farnum wrote:
> On Mon, Jan 5, 2015 at 4:12 PM, Loic Dachary <loic@dachary.org> wrote:
>> :-) This process is helpful if it allows me to help a little more than I currently do with the backport process. It would be a loss if the end result is that everyone cares less about backports. My primary incentive for sending this mail is to start the conversation and avoid that kind of unintended drawback.
> 
> Why do you want to get involved with other people's backports at all?

Just in case there is a need for more workforce.

> I don't mean that to sound possessive, but having the patch's primary
> author responsible for getting backports done at least has the
> singular merit of sharding the work up into manageable pieces. ;)

Absolutely. 
> 
>>
>>> I am 100% on board with making QE responsible for gating backports, so
>>> thank you for starting down that path. :) But I'm not at all sure how
>>> this scales for you. Right now backports are nominally run through two
>>> important checks:
>>> 1) Is it suitable for backport (decided by author or tech lead, marked
>>> via the Pending Backport tag)
>>> 2) Has it been through sufficient validation in master to be safe to
>>> backport (not marked in the system anywhere, just by somebody actually
>>> doing the backport).
>>>
>>> Knowing if something has been through sufficient validation to
>>> backport requires a fair bit of attention to the details of the ticket
>>> and the patches involved. How do you plan to keep up on that?
>>
>> I can't do that all by myself.
>>
>>> Similarly, while point releases are largely ad-hoc, we are trying to
>>> involve all the leads in the time-to-go decision. A lot of those
>>> decisions rest on whether specific backports have been performed yet,
>>> whether there are very new backports we want to run through testing
>>> for a little longer, etc. That sounds like a lot of communications
>>> overhead between the backport gates and the leads when making these
>>> kinds of decisions and I'm not sure how that should happen; is there a
>>> plan? (We can look at ticket status for things which are pending
>>> backport, but that doesn't facilitate prioritizing their backports;
>>> and in the opposite direction there's not a good way to say "this
>>> relatively large backport needs to go through at least three test runs
>>> before a release".)
>>
>> Could you point me to a mail thread / IRC conversation that is representative of this process ?
> 
> No; that's pretty much all done in video chats. :/
> 
>>
>> Here is a revised process which is hopefully more realistic:
>>
>> 0. Developer follows normal process to land PR to master. Once complete and ticket is marked Pending Backport this process initiates.
>> 1. I periodically polls Redmine to look for tickets in Pending Backport state and focus on the ones that are left unattended for too long
>> 1a. Under the supervision of the author of the original patch, I find the commits associated with the Redmine ticket and Cherry Pick to the backport integration branch off of the desired maintenance branch (Dumping, Firefly, etc).
>> 1b. I resolve any merge conflicts with the cherry-picked commit
>> 2. I merge all backports for a given branch in an integration branch
>> 3. I ask the leads of each project to review the integration
>> 4. Once satisfied with group of backported commits to integration branch, I notify QE.
>> 5. QE tests backport integration branch against appropriate suites
>> 6a. If QE is satisfied with test results, they merge backport integration branch.
>> 6b. If QE is NOT satisfied with the test results, they indicate backport integration branch is NOT ready to merge and return to me to work with original Developer to resolve issue and return to steps 2/3
>> 7. Ticket is moved to Resolved once backport integration branch containing cherry-picked backport is merged to the desired mainteance branch(es)
>>
>> What do you think ?
> 
> I think if we're going to add a process to anything it should be
> followed by everybody involved. I really would love for everything to
> be gated by QE before it goes into a backport branch, but if you're
> going off and building integration branches and QE is testing them, I
> think other people are going to keep backporting as we have been and
> trip all over each other. We've periodically used "firefly-next"
> branches and related things, but it's always been ad-hoc.
> 
> Something more realistic might involve locking down the stable
> branches so they can only be merged into by QE or some approved group,
> and then letting people do their own backports onto a
> <stable-branch>-next that is periodically taken up and
> integration-tested prior to merge into the LTS proper. That ensures
> that only patches which have all been tested together get into a
> stable branch without forcing each individual backport into a lot of
> process.

Let me rephrase to make sure I understand what you're suggesting. 

At the moment, as far as I can tell, developers do the backport of their patches if / when necessary and make sure they are green / yellow in gitbuilder. The integration itself happens on the stable branch, when such backports are merged: there is no integration branch (with the exception of an occasional XXX-next) nor someone focusing on integration. At some point in time the leads of each component get together and check if the current set of patches in the not-yet-released stable branch would make a sensible point release. The teuthology test suites are run, the results are analysed, the errors fixed and the release published. 

My past experience is that once the backport is merged in the stable branch my task is done as a developer. I'm not required when the release time comes and integration is something I'm mostly unaware of.

You propose that developers do some of the integration work. Instead of merging into the stable branch one backport at a time, they would first merge their backports into their own integration branch. These individual integration branches would then be taken (by me for instance or someone else willing to do that), put together, and sent to QE for testing. If it turns out that a developer did not create an integration branch, the pending backports would be merged as they currently are.

Am I following ?

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Ceph backports

[Gregory Farnum]

On Tue, Jan 6, 2015 at 12:39 AM, Loic Dachary <loic@dachary.org> wrote:
>
>
> On 06/01/2015 01:22, Gregory Farnum wrote:
>> On Mon, Jan 5, 2015 at 4:12 PM, Loic Dachary <loic@dachary.org> wrote:
>>> :-) This process is helpful if it allows me to help a little more than I currently do with the backport process. It would be a loss if the end result is that everyone cares less about backports. My primary incentive for sending this mail is to start the conversation and avoid that kind of unintended drawback.
>>
>> Why do you want to get involved with other people's backports at all?
>
> Just in case there is a need for more workforce.
>
>> I don't mean that to sound possessive, but having the patch's primary
>> author responsible for getting backports done at least has the
>> singular merit of sharding the work up into manageable pieces. ;)
>
> Absolutely.
>>
>>>
>>>> I am 100% on board with making QE responsible for gating backports, so
>>>> thank you for starting down that path. :) But I'm not at all sure how
>>>> this scales for you. Right now backports are nominally run through two
>>>> important checks:
>>>> 1) Is it suitable for backport (decided by author or tech lead, marked
>>>> via the Pending Backport tag)
>>>> 2) Has it been through sufficient validation in master to be safe to
>>>> backport (not marked in the system anywhere, just by somebody actually
>>>> doing the backport).
>>>>
>>>> Knowing if something has been through sufficient validation to
>>>> backport requires a fair bit of attention to the details of the ticket
>>>> and the patches involved. How do you plan to keep up on that?
>>>
>>> I can't do that all by myself.
>>>
>>>> Similarly, while point releases are largely ad-hoc, we are trying to
>>>> involve all the leads in the time-to-go decision. A lot of those
>>>> decisions rest on whether specific backports have been performed yet,
>>>> whether there are very new backports we want to run through testing
>>>> for a little longer, etc. That sounds like a lot of communications
>>>> overhead between the backport gates and the leads when making these
>>>> kinds of decisions and I'm not sure how that should happen; is there a
>>>> plan? (We can look at ticket status for things which are pending
>>>> backport, but that doesn't facilitate prioritizing their backports;
>>>> and in the opposite direction there's not a good way to say "this
>>>> relatively large backport needs to go through at least three test runs
>>>> before a release".)
>>>
>>> Could you point me to a mail thread / IRC conversation that is representative of this process ?
>>
>> No; that's pretty much all done in video chats. :/
>>
>>>
>>> Here is a revised process which is hopefully more realistic:
>>>
>>> 0. Developer follows normal process to land PR to master. Once complete and ticket is marked Pending Backport this process initiates.
>>> 1. I periodically polls Redmine to look for tickets in Pending Backport state and focus on the ones that are left unattended for too long
>>> 1a. Under the supervision of the author of the original patch, I find the commits associated with the Redmine ticket and Cherry Pick to the backport integration branch off of the desired maintenance branch (Dumping, Firefly, etc).
>>> 1b. I resolve any merge conflicts with the cherry-picked commit
>>> 2. I merge all backports for a given branch in an integration branch
>>> 3. I ask the leads of each project to review the integration
>>> 4. Once satisfied with group of backported commits to integration branch, I notify QE.
>>> 5. QE tests backport integration branch against appropriate suites
>>> 6a. If QE is satisfied with test results, they merge backport integration branch.
>>> 6b. If QE is NOT satisfied with the test results, they indicate backport integration branch is NOT ready to merge and return to me to work with original Developer to resolve issue and return to steps 2/3
>>> 7. Ticket is moved to Resolved once backport integration branch containing cherry-picked backport is merged to the desired mainteance branch(es)
>>>
>>> What do you think ?
>>
>> I think if we're going to add a process to anything it should be
>> followed by everybody involved. I really would love for everything to
>> be gated by QE before it goes into a backport branch, but if you're
>> going off and building integration branches and QE is testing them, I
>> think other people are going to keep backporting as we have been and
>> trip all over each other. We've periodically used "firefly-next"
>> branches and related things, but it's always been ad-hoc.
>>
>> Something more realistic might involve locking down the stable
>> branches so they can only be merged into by QE or some approved group,
>> and then letting people do their own backports onto a
>> <stable-branch>-next that is periodically taken up and
>> integration-tested prior to merge into the LTS proper. That ensures
>> that only patches which have all been tested together get into a
>> stable branch without forcing each individual backport into a lot of
>> process.
>
> Let me rephrase to make sure I understand what you're suggesting.
>
> At the moment, as far as I can tell, developers do the backport of their patches if / when necessary and make sure they are green / yellow in gitbuilder. The integration itself happens on the stable branch, when such backports are merged: there is no integration branch (with the exception of an occasional XXX-next) nor someone focusing on integration. At some point in time the leads of each component get together and check if the current set of patches in the not-yet-released stable branch would make a sensible point release. The teuthology test suites are run, the results are analysed, the errors fixed and the release published.
>
> My past experience is that once the backport is merged in the stable branch my task is done as a developer. I'm not required when the release time comes and integration is something I'm mostly unaware of.
>
> You propose that developers do some of the integration work. Instead of merging into the stable branch one backport at a time, they would first merge their backports into their own integration branch. These individual integration branches would then be taken (by me for instance or someone else willing to do that), put together, and sent to QE for testing. If it turns out that a developer did not create an integration branch, the pending backports would be merged as they currently are.

I'm just trying to understand how things scale past one developer. I
gather that Sam and Sage do a lot more backports than I do and are
already not getting them done, so simply having somebody poke at
backports is an improvement? If that's all you're after then this is
sensible — I just don't want to have backports of my own and do the
wrong thing with them. If you want to pick a process and tell me where
I stick my nose in I'm happy to try it out. :)
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph backports

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 7494 bytes --]



On 06/01/2015 19:21, Gregory Farnum wrote:
> On Tue, Jan 6, 2015 at 12:39 AM, Loic Dachary <loic@dachary.org> wrote:
>>
>>
>> On 06/01/2015 01:22, Gregory Farnum wrote:
>>> On Mon, Jan 5, 2015 at 4:12 PM, Loic Dachary <loic@dachary.org> wrote:
>>>> :-) This process is helpful if it allows me to help a little more than I currently do with the backport process. It would be a loss if the end result is that everyone cares less about backports. My primary incentive for sending this mail is to start the conversation and avoid that kind of unintended drawback.
>>>
>>> Why do you want to get involved with other people's backports at all?
>>
>> Just in case there is a need for more workforce.
>>
>>> I don't mean that to sound possessive, but having the patch's primary
>>> author responsible for getting backports done at least has the
>>> singular merit of sharding the work up into manageable pieces. ;)
>>
>> Absolutely.
>>>
>>>>
>>>>> I am 100% on board with making QE responsible for gating backports, so
>>>>> thank you for starting down that path. :) But I'm not at all sure how
>>>>> this scales for you. Right now backports are nominally run through two
>>>>> important checks:
>>>>> 1) Is it suitable for backport (decided by author or tech lead, marked
>>>>> via the Pending Backport tag)
>>>>> 2) Has it been through sufficient validation in master to be safe to
>>>>> backport (not marked in the system anywhere, just by somebody actually
>>>>> doing the backport).
>>>>>
>>>>> Knowing if something has been through sufficient validation to
>>>>> backport requires a fair bit of attention to the details of the ticket
>>>>> and the patches involved. How do you plan to keep up on that?
>>>>
>>>> I can't do that all by myself.
>>>>
>>>>> Similarly, while point releases are largely ad-hoc, we are trying to
>>>>> involve all the leads in the time-to-go decision. A lot of those
>>>>> decisions rest on whether specific backports have been performed yet,
>>>>> whether there are very new backports we want to run through testing
>>>>> for a little longer, etc. That sounds like a lot of communications
>>>>> overhead between the backport gates and the leads when making these
>>>>> kinds of decisions and I'm not sure how that should happen; is there a
>>>>> plan? (We can look at ticket status for things which are pending
>>>>> backport, but that doesn't facilitate prioritizing their backports;
>>>>> and in the opposite direction there's not a good way to say "this
>>>>> relatively large backport needs to go through at least three test runs
>>>>> before a release".)
>>>>
>>>> Could you point me to a mail thread / IRC conversation that is representative of this process ?
>>>
>>> No; that's pretty much all done in video chats. :/
>>>
>>>>
>>>> Here is a revised process which is hopefully more realistic:
>>>>
>>>> 0. Developer follows normal process to land PR to master. Once complete and ticket is marked Pending Backport this process initiates.
>>>> 1. I periodically polls Redmine to look for tickets in Pending Backport state and focus on the ones that are left unattended for too long
>>>> 1a. Under the supervision of the author of the original patch, I find the commits associated with the Redmine ticket and Cherry Pick to the backport integration branch off of the desired maintenance branch (Dumping, Firefly, etc).
>>>> 1b. I resolve any merge conflicts with the cherry-picked commit
>>>> 2. I merge all backports for a given branch in an integration branch
>>>> 3. I ask the leads of each project to review the integration
>>>> 4. Once satisfied with group of backported commits to integration branch, I notify QE.
>>>> 5. QE tests backport integration branch against appropriate suites
>>>> 6a. If QE is satisfied with test results, they merge backport integration branch.
>>>> 6b. If QE is NOT satisfied with the test results, they indicate backport integration branch is NOT ready to merge and return to me to work with original Developer to resolve issue and return to steps 2/3
>>>> 7. Ticket is moved to Resolved once backport integration branch containing cherry-picked backport is merged to the desired mainteance branch(es)
>>>>
>>>> What do you think ?
>>>
>>> I think if we're going to add a process to anything it should be
>>> followed by everybody involved. I really would love for everything to
>>> be gated by QE before it goes into a backport branch, but if you're
>>> going off and building integration branches and QE is testing them, I
>>> think other people are going to keep backporting as we have been and
>>> trip all over each other. We've periodically used "firefly-next"
>>> branches and related things, but it's always been ad-hoc.
>>>
>>> Something more realistic might involve locking down the stable
>>> branches so they can only be merged into by QE or some approved group,
>>> and then letting people do their own backports onto a
>>> <stable-branch>-next that is periodically taken up and
>>> integration-tested prior to merge into the LTS proper. That ensures
>>> that only patches which have all been tested together get into a
>>> stable branch without forcing each individual backport into a lot of
>>> process.
>>
>> Let me rephrase to make sure I understand what you're suggesting.
>>
>> At the moment, as far as I can tell, developers do the backport of their patches if / when necessary and make sure they are green / yellow in gitbuilder. The integration itself happens on the stable branch, when such backports are merged: there is no integration branch (with the exception of an occasional XXX-next) nor someone focusing on integration. At some point in time the leads of each component get together and check if the current set of patches in the not-yet-released stable branch would make a sensible point release. The teuthology test suites are run, the results are analysed, the errors fixed and the release published.
>>
>> My past experience is that once the backport is merged in the stable branch my task is done as a developer. I'm not required when the release time comes and integration is something I'm mostly unaware of.
>>
>> You propose that developers do some of the integration work. Instead of merging into the stable branch one backport at a time, they would first merge their backports into their own integration branch. These individual integration branches would then be taken (by me for instance or someone else willing to do that), put together, and sent to QE for testing. If it turns out that a developer did not create an integration branch, the pending backports would be merged as they currently are.
> 
> I'm just trying to understand how things scale past one developer. I
> gather that Sam and Sage do a lot more backports than I do and are
> already not getting them done, so simply having somebody poke at
> backports is an improvement? If that's all you're after then this is
> sensible — I just don't want to have backports of my own and do the
> wrong thing with them. If you want to pick a process and tell me where
> I stick my nose in I'm happy to try it out. :)

Cool :-). Things are a little vague at the moment so I just went ahead and try to semi-manually collect backports and cross check the informations I found here and there. I guess making it clear where we're at is already a help. 

Cheers

> -Greg
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]