Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi

Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 2068 bytes --]

Hi Greg,

I'm curious to know how you handle the flow of mails from QA runs. Here is a wild guess:

* from time to time check that the nightlies run the suites that should be run
* read the ceph-qa reports daily
* for each failed job, either relate it to an issue or create one or declare it noise
* if a job fails on an existing ticket store a link to the job if it's rare occurrence and the cause is not yet known
* bi-weekly bug scrub makes sure no issue, old or new, is forgotten
* at release time you decide that it is ready based on:
** the list of urgent/immediate issues that you can browse to ensure no issue is a blocker
** the last run of each suite to ensure they are recent enough and environmental noise did not permanently shadow anything

Cheers


On 13/02/2015 20:29, Gregory Farnum wrote:
> On Thu, Feb 12, 2015 at 7:58 PM,
> <teuthworker@teuthology.front.sepia.ceph.com> wrote:
>> Test Run: teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi
>> =================================================================
>> info:   http://pulpito.ceph.com/teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi/
>> logs:   http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi/
>> failed: 0
>> hung:   1
>> passed: 11
>>
>> Hung
>> =================================================================
>> [752393] samba/{clusters/samba-basic.yaml debug/mds_client.yaml fs/btrfs.yaml install/install.yaml mount/noceph.yaml workload/smbtorture.yaml}
>> info:   http://pulpito.ceph.com/teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi/752393/
> 
> Looks like http://tracker.ceph.com/issues/10413, the upstream fix for
> which should be trickling down soon.
> 
>>
>> Passed
>> =================================================================
> _______________________________________________
> Ceph-qa mailing list
> Ceph-qa@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-qa-ceph.com
> 
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi

[Gregory Farnum]

On Fri, Feb 13, 2015 at 10:34 PM, Loic Dachary <loic@dachary.org> wrote:
> Hi Greg,
>
> I'm curious to know how you handle the flow of mails from QA runs. Here is a wild guess:
>
> * from time to time check that the nightlies run the suites that should be run

Uh, I guess?

> * read the ceph-qa reports daily

Yeah

> * for each failed job, either relate it to an issue or create one or declare it noise

Yeah

> * if a job fails on an existing ticket store a link to the job if it's rare occurrence and the cause is not yet known

Yeah, or just to make clear it's still happening or whatever

> * bi-weekly bug scrub makes sure no issue, old or new, is forgotten

Hopefully!

> * at release time you decide that it is ready based on:
> ** the list of urgent/immediate issues that you can browse to ensure no issue is a blocker
> ** the last run of each suite to ensure they are recent enough and environmental noise did not permanently shadow anything

Yeah. Well, the last run alone isn't so important; we want to see a
string of clean runs because a lot of issues aren't reproduced in
every run.
-Greg

Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 1468 bytes --]



On 14/02/2015 08:56, Gregory Farnum wrote:
> On Fri, Feb 13, 2015 at 10:34 PM, Loic Dachary <loic@dachary.org> wrote:
>> Hi Greg,
>>
>> I'm curious to know how you handle the flow of mails from QA runs. Here is a wild guess:
>>
>> * from time to time check that the nightlies run the suites that should be run
> 
> Uh, I guess?
> 
>> * read the ceph-qa reports daily
> 
> Yeah
> 
>> * for each failed job, either relate it to an issue or create one or declare it noise
> 
> Yeah
> 
>> * if a job fails on an existing ticket store a link to the job if it's rare occurrence and the cause is not yet known
> 
> Yeah, or just to make clear it's still happening or whatever
> 
>> * bi-weekly bug scrub makes sure no issue, old or new, is forgotten
> 
> Hopefully!
> 
>> * at release time you decide that it is ready based on:
>> ** the list of urgent/immediate issues that you can browse to ensure no issue is a blocker
>> ** the last run of each suite to ensure they are recent enough and environmental noise did not permanently shadow anything
> 
> Yeah. Well, the last run alone isn't so important; we want to see a
> string of clean runs because a lot of issues aren't reproduced in
> every run.

Do you check the string of clean runs with http://pulpito.ceph.com/?suite=fs&branch=giant for instance ? Or do you have a command line that does the same ?

Cheers

> -Greg
> 
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi

[Yuri Weinstein]

> Yeah. Well, the last run alone isn't so important; we want to see a
> string of clean runs because a lot of issues aren't reproduced in
> every run.

My hope was that we can see all "green" results for say this giant release/backport, but I agree that we would need to make our go/no-go decision based on multiple run results, as I am not sure if we can get them all "green" due to complexity, time needed to execute, environment state etc..

We could thou modify our process a bit:
1. after backport-branch is ready for QE, merge it to the named branch (say 'giant' in this example) - that what we did now
2. cut a release numbered brach (maybe it's tag, not sure), say "v0.87.1"
3. run all QE suites on "v0.87.1" and get it to "all passed" state
4. make sure that commits to "v0.87.1" are committed to the named branch ('giant') 

#2 is that we have not done this time.

Thx
YuriW

----- Original Message -----
From: "Gregory Farnum" <greg@gregs42.com>
To: "Loic Dachary" <loic@dachary.org>
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Friday, February 13, 2015 11:56:18 PM
Subject: Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi

On Fri, Feb 13, 2015 at 10:34 PM, Loic Dachary <loic@dachary.org> wrote:
> Hi Greg,
>
> I'm curious to know how you handle the flow of mails from QA runs. Here is a wild guess:
>
> * from time to time check that the nightlies run the suites that should be run

Uh, I guess?

> * read the ceph-qa reports daily

Yeah

> * for each failed job, either relate it to an issue or create one or declare it noise

Yeah

> * if a job fails on an existing ticket store a link to the job if it's rare occurrence and the cause is not yet known

Yeah, or just to make clear it's still happening or whatever

> * bi-weekly bug scrub makes sure no issue, old or new, is forgotten

Hopefully!

> * at release time you decide that it is ready based on:
> ** the list of urgent/immediate issues that you can browse to ensure no issue is a blocker
> ** the last run of each suite to ensure they are recent enough and environmental noise did not permanently shadow anything

Yeah. Well, the last run alone isn't so important; we want to see a
string of clean runs because a lot of issues aren't reproduced in
every run.
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 3985 bytes --]

Hi Yuri,

On 14/02/2015 17:22, Yuri Weinstein wrote:
>> Yeah. Well, the last run alone isn't so important; we want to see a
>> string of clean runs because a lot of issues aren't reproduced in
>> every run.
> 
> My hope was that we can see all "green" results for say this giant release/backport, but I agree that we would need to make our go/no-go decision based on multiple run results, as I am not sure if we can get them all "green" due to complexity, time needed to execute, environment state etc..
> 
> We could thou modify our process a bit:
> 1. after backport-branch is ready for QE, merge it to the named branch (say 'giant' in this example) - that what we did now
> 2. cut a release numbered brach (maybe it's tag, not sure), say "v0.87.1"
> 3. run all QE suites on "v0.87.1" and get it to "all passed" state
> 4. make sure that commits to "v0.87.1" are committed to the named branch ('giant') 

That makes sense to me, only with s/v0.87.1/78c71b9200da5e7d832ec58765478404d31ae6b5/.

> #2 is that we have not done this time.

We have not done #2 but we have cut the branch at given SHA ( 78c71b9200da5e7d832ec58765478404d31ae6b5 ) instead, which is can be referenced by a tag if and when it is released. In the mail "Re: giant integration branch for v0.87.1 ready for QE" dated 11th february 2015 I wrote:

> The giant-backports pull requests were merged into https://github.com/ceph/ceph/tree/giant which is not ready for testing.

> For the record, the head is https://github.com/ceph/ceph/commit/78c71b9200da5e7d832ec58765478404d31ae6b5

We cannot add a v0.87.1 tag to the branch before the release process is complete because we won't be able to change it afterwards (people rely on the fact that the history of the giant branch is not rewritten and that tags references are not changed). If during the QE test process we discover that a backport must be included (I'm thinking about https://github.com/ceph/ceph/pull/3731 for instance), 78c71b9200da5e7d832ec58765478404d31ae6b5 won't be v0.87.1 after all.

In a nutshell I think we're having the same view of the process, modulo the timing of the tagging of the release.

Cheers

> 
> Thx
> YuriW
> 
> ----- Original Message -----
> From: "Gregory Farnum" <greg@gregs42.com>
> To: "Loic Dachary" <loic@dachary.org>
> Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
> Sent: Friday, February 13, 2015 11:56:18 PM
> Subject: Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi
> 
> On Fri, Feb 13, 2015 at 10:34 PM, Loic Dachary <loic@dachary.org> wrote:
>> Hi Greg,
>>
>> I'm curious to know how you handle the flow of mails from QA runs. Here is a wild guess:
>>
>> * from time to time check that the nightlies run the suites that should be run
> 
> Uh, I guess?
> 
>> * read the ceph-qa reports daily
> 
> Yeah
> 
>> * for each failed job, either relate it to an issue or create one or declare it noise
> 
> Yeah
> 
>> * if a job fails on an existing ticket store a link to the job if it's rare occurrence and the cause is not yet known
> 
> Yeah, or just to make clear it's still happening or whatever
> 
>> * bi-weekly bug scrub makes sure no issue, old or new, is forgotten
> 
> Hopefully!
> 
>> * at release time you decide that it is ready based on:
>> ** the list of urgent/immediate issues that you can browse to ensure no issue is a blocker
>> ** the last run of each suite to ensure they are recent enough and environmental noise did not permanently shadow anything
> 
> Yeah. Well, the last run alone isn't so important; we want to see a
> string of clean runs because a lot of issues aren't reproduced in
> every run.
> -Greg
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 4531 bytes --]



On 14/02/2015 22:53, Loic Dachary wrote:
> Hi Yuri,
> 
> On 14/02/2015 17:22, Yuri Weinstein wrote:
>>> Yeah. Well, the last run alone isn't so important; we want to see a
>>> string of clean runs because a lot of issues aren't reproduced in
>>> every run.
>>
>> My hope was that we can see all "green" results for say this giant release/backport, but I agree that we would need to make our go/no-go decision based on multiple run results, as I am not sure if we can get them all "green" due to complexity, time needed to execute, environment state etc..
>>
>> We could thou modify our process a bit:
>> 1. after backport-branch is ready for QE, merge it to the named branch (say 'giant' in this example) - that what we did now
>> 2. cut a release numbered brach (maybe it's tag, not sure), say "v0.87.1"
>> 3. run all QE suites on "v0.87.1" and get it to "all passed" state
>> 4. make sure that commits to "v0.87.1" are committed to the named branch ('giant') 
> 
> That makes sense to me, only with s/v0.87.1/78c71b9200da5e7d832ec58765478404d31ae6b5/.
> 
>> #2 is that we have not done this time.
> 
> We have not done #2 but we have cut the branch at given SHA ( 78c71b9200da5e7d832ec58765478404d31ae6b5 ) instead, which is can be referenced by a tag if and when it is released. In the mail "Re: giant integration branch for v0.87.1 ready for QE" dated 11th february 2015 I wrote:
> 
>> The giant-backports pull requests were merged into https://github.com/ceph/ceph/tree/giant which is not ready for testing.
> 
>> For the record, the head is https://github.com/ceph/ceph/commit/78c71b9200da5e7d832ec58765478404d31ae6b5
> 
> We cannot add a v0.87.1 tag to the branch before the release process is complete because we won't be able to change it afterwards (people rely on the fact that the history of the giant branch is not rewritten and that tags references are not changed). If during the QE test process we discover that a backport must be included (I'm thinking about https://github.com/ceph/ceph/pull/3731 for instance), 78c71b9200da5e7d832ec58765478404d31ae6b5 won't be v0.87.1 after all.
> 
> In a nutshell I think we're having the same view of the process, modulo the timing of the tagging of the release.

We could also have tags like:

v0.87.1-rc1 => 78c71b9200da5e7d832ec58765478404d31ae6b5
v0.87.1-rc2 => whatever SHA includes more backports

and if v0.87.1-rc2 turns out to be good the release notes could be committed and other non code changes. This naming scheme common, is there a downside to it ? It's easier to talk about v0.87.1-rc1 rather than 78c71b9200da5e7d832ec58765478404d31ae6b5 ;-) 

Cheers

> 
> Cheers
> 
>>
>> Thx
>> YuriW
>>
>> ----- Original Message -----
>> From: "Gregory Farnum" <greg@gregs42.com>
>> To: "Loic Dachary" <loic@dachary.org>
>> Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
>> Sent: Friday, February 13, 2015 11:56:18 PM
>> Subject: Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi
>>
>> On Fri, Feb 13, 2015 at 10:34 PM, Loic Dachary <loic@dachary.org> wrote:
>>> Hi Greg,
>>>
>>> I'm curious to know how you handle the flow of mails from QA runs. Here is a wild guess:
>>>
>>> * from time to time check that the nightlies run the suites that should be run
>>
>> Uh, I guess?
>>
>>> * read the ceph-qa reports daily
>>
>> Yeah
>>
>>> * for each failed job, either relate it to an issue or create one or declare it noise
>>
>> Yeah
>>
>>> * if a job fails on an existing ticket store a link to the job if it's rare occurrence and the cause is not yet known
>>
>> Yeah, or just to make clear it's still happening or whatever
>>
>>> * bi-weekly bug scrub makes sure no issue, old or new, is forgotten
>>
>> Hopefully!
>>
>>> * at release time you decide that it is ready based on:
>>> ** the list of urgent/immediate issues that you can browse to ensure no issue is a blocker
>>> ** the last run of each suite to ensure they are recent enough and environmental noise did not permanently shadow anything
>>
>> Yeah. Well, the last run alone isn't so important; we want to see a
>> string of clean runs because a lot of issues aren't reproduced in
>> every run.
>> -Greg
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi

[Yuri Weinstein]

Loic,

+1 - I like the way you're discussing:

v0.87.1-rc2
v0.87.1-rcX => v0.87.1 - is it easy to make this look like this after the validation is completed?

BTW:  When I re-run suites now for validation I use "-s <named_branch>" arg in the command line.   Maybe I should be using SHA ref instead?  I never tried this way, but guessing it should work, what do you think?

Thx
YuriW

----- Original Message -----
From: "Loic Dachary" <loic@dachary.org>
To: "Yuri Weinstein" <yweinste@redhat.com>
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Saturday, February 14, 2015 2:12:05 PM
Subject: Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi



On 14/02/2015 22:53, Loic Dachary wrote:
> Hi Yuri,
> 
> On 14/02/2015 17:22, Yuri Weinstein wrote:
>>> Yeah. Well, the last run alone isn't so important; we want to see a
>>> string of clean runs because a lot of issues aren't reproduced in
>>> every run.
>>
>> My hope was that we can see all "green" results for say this giant release/backport, but I agree that we would need to make our go/no-go decision based on multiple run results, as I am not sure if we can get them all "green" due to complexity, time needed to execute, environment state etc..
>>
>> We could thou modify our process a bit:
>> 1. after backport-branch is ready for QE, merge it to the named branch (say 'giant' in this example) - that what we did now
>> 2. cut a release numbered brach (maybe it's tag, not sure), say "v0.87.1"
>> 3. run all QE suites on "v0.87.1" and get it to "all passed" state
>> 4. make sure that commits to "v0.87.1" are committed to the named branch ('giant') 
> 
> That makes sense to me, only with s/v0.87.1/78c71b9200da5e7d832ec58765478404d31ae6b5/.
> 
>> #2 is that we have not done this time.
> 
> We have not done #2 but we have cut the branch at given SHA ( 78c71b9200da5e7d832ec58765478404d31ae6b5 ) instead, which is can be referenced by a tag if and when it is released. In the mail "Re: giant integration branch for v0.87.1 ready for QE" dated 11th february 2015 I wrote:
> 
>> The giant-backports pull requests were merged into https://github.com/ceph/ceph/tree/giant which is not ready for testing.
> 
>> For the record, the head is https://github.com/ceph/ceph/commit/78c71b9200da5e7d832ec58765478404d31ae6b5
> 
> We cannot add a v0.87.1 tag to the branch before the release process is complete because we won't be able to change it afterwards (people rely on the fact that the history of the giant branch is not rewritten and that tags references are not changed). If during the QE test process we discover that a backport must be included (I'm thinking about https://github.com/ceph/ceph/pull/3731 for instance), 78c71b9200da5e7d832ec58765478404d31ae6b5 won't be v0.87.1 after all.
> 
> In a nutshell I think we're having the same view of the process, modulo the timing of the tagging of the release.

We could also have tags like:

v0.87.1-rc1 => 78c71b9200da5e7d832ec58765478404d31ae6b5
v0.87.1-rc2 => whatever SHA includes more backports

and if v0.87.1-rc2 turns out to be good the release notes could be committed and other non code changes. This naming scheme common, is there a downside to it ? It's easier to talk about v0.87.1-rc1 rather than 78c71b9200da5e7d832ec58765478404d31ae6b5 ;-) 

Cheers

> 
> Cheers
> 
>>
>> Thx
>> YuriW
>>
>> ----- Original Message -----
>> From: "Gregory Farnum" <greg@gregs42.com>
>> To: "Loic Dachary" <loic@dachary.org>
>> Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
>> Sent: Friday, February 13, 2015 11:56:18 PM
>> Subject: Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi
>>
>> On Fri, Feb 13, 2015 at 10:34 PM, Loic Dachary <loic@dachary.org> wrote:
>>> Hi Greg,
>>>
>>> I'm curious to know how you handle the flow of mails from QA runs. Here is a wild guess:
>>>
>>> * from time to time check that the nightlies run the suites that should be run
>>
>> Uh, I guess?
>>
>>> * read the ceph-qa reports daily
>>
>> Yeah
>>
>>> * for each failed job, either relate it to an issue or create one or declare it noise
>>
>> Yeah
>>
>>> * if a job fails on an existing ticket store a link to the job if it's rare occurrence and the cause is not yet known
>>
>> Yeah, or just to make clear it's still happening or whatever
>>
>>> * bi-weekly bug scrub makes sure no issue, old or new, is forgotten
>>
>> Hopefully!
>>
>>> * at release time you decide that it is ready based on:
>>> ** the list of urgent/immediate issues that you can browse to ensure no issue is a blocker
>>> ** the last run of each suite to ensure they are recent enough and environmental noise did not permanently shadow anything
>>
>> Yeah. Well, the last run alone isn't so important; we want to see a
>> string of clean runs because a lot of issues aren't reproduced in
>> every run.
>> -Greg
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 5830 bytes --]



On 14/02/2015 23:20, Yuri Weinstein wrote:
> Loic,
> 
> +1 - I like the way you're discussing:
> 
> v0.87.1-rc2
> v0.87.1-rcX => v0.87.1 - is it easy to make this look like this after the validation is completed?

Actually (Alfredo can confirm) it is the release process that sets the tag. From a higher level the process could be described as:

* All developer leads agree that v0.87.1-rcX is ready for release
* QE tests v0.87.1-rcX and
** return it to the developers if issues are found
** gives v0.87.1-rcX to the release team if no issues are found
* the release team publishes v0.87.1

Cheers

> BTW:  When I re-run suites now for validation I use "-s <named_branch>" arg in the command line.   Maybe I should be using SHA ref instead?  I never tried this way, but guessing it should work, what do you think?
> 
> Thx
> YuriW
> 
> ----- Original Message -----
> From: "Loic Dachary" <loic@dachary.org>
> To: "Yuri Weinstein" <yweinste@redhat.com>
> Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
> Sent: Saturday, February 14, 2015 2:12:05 PM
> Subject: Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi
> 
> 
> 
> On 14/02/2015 22:53, Loic Dachary wrote:
>> Hi Yuri,
>>
>> On 14/02/2015 17:22, Yuri Weinstein wrote:
>>>> Yeah. Well, the last run alone isn't so important; we want to see a
>>>> string of clean runs because a lot of issues aren't reproduced in
>>>> every run.
>>>
>>> My hope was that we can see all "green" results for say this giant release/backport, but I agree that we would need to make our go/no-go decision based on multiple run results, as I am not sure if we can get them all "green" due to complexity, time needed to execute, environment state etc..
>>>
>>> We could thou modify our process a bit:
>>> 1. after backport-branch is ready for QE, merge it to the named branch (say 'giant' in this example) - that what we did now
>>> 2. cut a release numbered brach (maybe it's tag, not sure), say "v0.87.1"
>>> 3. run all QE suites on "v0.87.1" and get it to "all passed" state
>>> 4. make sure that commits to "v0.87.1" are committed to the named branch ('giant') 
>>
>> That makes sense to me, only with s/v0.87.1/78c71b9200da5e7d832ec58765478404d31ae6b5/.
>>
>>> #2 is that we have not done this time.
>>
>> We have not done #2 but we have cut the branch at given SHA ( 78c71b9200da5e7d832ec58765478404d31ae6b5 ) instead, which is can be referenced by a tag if and when it is released. In the mail "Re: giant integration branch for v0.87.1 ready for QE" dated 11th february 2015 I wrote:
>>
>>> The giant-backports pull requests were merged into https://github.com/ceph/ceph/tree/giant which is not ready for testing.
>>
>>> For the record, the head is https://github.com/ceph/ceph/commit/78c71b9200da5e7d832ec58765478404d31ae6b5
>>
>> We cannot add a v0.87.1 tag to the branch before the release process is complete because we won't be able to change it afterwards (people rely on the fact that the history of the giant branch is not rewritten and that tags references are not changed). If during the QE test process we discover that a backport must be included (I'm thinking about https://github.com/ceph/ceph/pull/3731 for instance), 78c71b9200da5e7d832ec58765478404d31ae6b5 won't be v0.87.1 after all.
>>
>> In a nutshell I think we're having the same view of the process, modulo the timing of the tagging of the release.
> 
> We could also have tags like:
> 
> v0.87.1-rc1 => 78c71b9200da5e7d832ec58765478404d31ae6b5
> v0.87.1-rc2 => whatever SHA includes more backports
> 
> and if v0.87.1-rc2 turns out to be good the release notes could be committed and other non code changes. This naming scheme common, is there a downside to it ? It's easier to talk about v0.87.1-rc1 rather than 78c71b9200da5e7d832ec58765478404d31ae6b5 ;-) 
> 
> Cheers
> 
>>
>> Cheers
>>
>>>
>>> Thx
>>> YuriW
>>>
>>> ----- Original Message -----
>>> From: "Gregory Farnum" <greg@gregs42.com>
>>> To: "Loic Dachary" <loic@dachary.org>
>>> Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
>>> Sent: Friday, February 13, 2015 11:56:18 PM
>>> Subject: Re: [Ceph-qa] 1 hung, 11 passed in teuthology-2015-02-11_16:13:01-samba-giant-distro-basic-multi
>>>
>>> On Fri, Feb 13, 2015 at 10:34 PM, Loic Dachary <loic@dachary.org> wrote:
>>>> Hi Greg,
>>>>
>>>> I'm curious to know how you handle the flow of mails from QA runs. Here is a wild guess:
>>>>
>>>> * from time to time check that the nightlies run the suites that should be run
>>>
>>> Uh, I guess?
>>>
>>>> * read the ceph-qa reports daily
>>>
>>> Yeah
>>>
>>>> * for each failed job, either relate it to an issue or create one or declare it noise
>>>
>>> Yeah
>>>
>>>> * if a job fails on an existing ticket store a link to the job if it's rare occurrence and the cause is not yet known
>>>
>>> Yeah, or just to make clear it's still happening or whatever
>>>
>>>> * bi-weekly bug scrub makes sure no issue, old or new, is forgotten
>>>
>>> Hopefully!
>>>
>>>> * at release time you decide that it is ready based on:
>>>> ** the list of urgent/immediate issues that you can browse to ensure no issue is a blocker
>>>> ** the last run of each suite to ensure they are recent enough and environmental noise did not permanently shadow anything
>>>
>>> Yeah. Well, the last run alone isn't so important; we want to see a
>>> string of clean runs because a lot of issues aren't reproduced in
>>> every run.
>>> -Greg
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]