[xen 4.6 retrospective] Kicking off a retrospective for Xen 4.6 (deadline August 28th)

[xen 4.6 retrospective] Kicking off a retrospective for Xen 4.6 (deadline August 28th)

[Lars Kurth]

[-- Attachment #1.1: Type: text/plain, Size: 4459 bytes --]


Hi all,

I have been asked by a number of people to kick off a retrospective for
* what worked well
* didn't work well
in the Xen 4.6 release cycle.

As most of the stress of the last few weeks is now out of the way, I
thought it would be appropriate to start this process now, and let it run
for a few weeks. We have not done this type of retrospective over e-mail
before (we have done so at developer meetings as well as Hackathons). 
As we have a global and distributed community, I think it is worthwhile
trying this by email. To make this work, you need to follow a few ground
rules.

= Ground Rules =
We would like to hear 
a) what worked well and 
b) what didn't work well 
in this release cycle. 

Please provide feedback before August 28th.

== What to discuss / what not to discuss ==

* Do NOT send issues which are personal to the mailing list. This means,
do NOT send anything related to individuals or companies that may have
occurred in this release cycle. We do not want to have divisive flame 
wars in our community: this does not fit our values and will help 
no-one. If you believe that we do have an issue, that requires referring
to individuals/companies, please send me a private email following the 
same formatting conventions as outlined below. I can then work with you 
to figure out how to best approach this issue or piece of feedback. 

* If you are not sure, contact me beforehand and we can discuss the best
way forward.

* It is entirely acceptable to discuss process issues in public. Examples
of topics that are suitable are: "the hard freeze did not work", "the hard 
freeze did work well", "we should have longer freeze exception windows", 
"we should have no freeze exceptions at all", "we should open master to 
contributions after RC2", "we should rename freeze to something else as 
it encourages misunderstandings", "we should have a shorter release cycle", 
etc. - these are just examples.  

* If you agree with an issue/solution you can reply to it with a comment
or a "+1" in the normal way

* If you disagree or have some concerns about what has been proposed,
feel free to reply. You can use the "-1 comment" format.

Please be MINDFUL when you reply to one of the ideas and suggestions that
were raised. The ground rule about NOT becoming personal also applies to
replies.

== Formatting of Mails ==

* Please only cover only *one* topic per feedback e-mail. In other 
words please do not mix topics. This makes it easier to collate feedback
and for everyone to follow the threads.

* Use the [xen 4.6 retrospective] tag in the subject line to xen-devel@

* You can use the [good] or [bad] tag in the subject line to xen-devel@

* Make sure you CC community.manager@xenproject.org <mailto:community.manager@xenproject.org>

* Use [private] and the tags above, and do NOT send the mail to xen-devel@
if this is a question/issue ONLY directed at me. This will help me manage
my inbox

* Use [urgent], if your issue should be discussed at the upcoming
Developer Summit Aug 18-19, OR if it is something which affects the current
release cycle (e.g. "we should open master to contributions after RC2")

* Use a descriptive subject line describing the issue/feedback, e.g. 
"release cadence was too long", ...

* If you can, reply to this e-mail thread and change the title as
described above. That way, most of the feedback will be nicely threaded

== Content ==
Ideally, you would follow the following template

---
  Subject: [xen 4.6 retrospective] ...

  = Issue / Observation =
  Describe what you have noticed
  Described whether this was positive or negative and how
  ...

  = Possible Solution / Improvement =
  Describe how you think we could improve the situation 
  Explain why your proposal would make a positive impact
  ...
---

== Closing the Discussion ==

I will monitor the post-mortem and I will act as facilitator: for example
I may call out improper behaviour in public or private to keep the
retrospective focussed. 

At some point after August 28th, we will look at the issues and
suggestions raised and see which ones we can address and how. For the 
ones we can address, we will condense feedback into concrete proposals 
which we may put forward for voting (unless everyone agrees that something 
is a great idea, or strictly speaking no vote is necessary).

Best Regards
Lars

[-- Attachment #1.2: Type: text/html, Size: 10774 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[xen 4.6 retrospective] [urgent] rename "freeze" window and make release branch as soon as possible after RC1

[Lars Kurth]

[-- Attachment #1.1: Type: text/plain, Size: 2405 bytes --]

This is one item of feedback, which I believe is a quick win for us. This is one piece of feedback from a list of items that have during the last few weeks been raised with me personally, either during face-2-face conversations in a private e-mail thread. See http://lists.xenproject.org/archives/html/xen-devel/2015-08/msg00173.html <http://lists.xenproject.org/archives/html/xen-devel/2015-08/msg00173.html> for information on the retrospective


= Issue / Observation =
The name "freeze" window/period - aka the time period from when we "feature freeze" until we branch master and/or make the release leads some contributors to mistakenly assume that development for the next release stops. I saw a few mails on xen-devel@ recently, pointing out to contributors that development does not stop during "freeze".  Chatting to Ian Campbell, he mentioned that he replied to several different people who said they were waiting for the tree to reopen. Maybe choosing a better name will help.

In addition, we used to branch master a lot earlier I believe up to Xen 4.1 (around RC2 or RC3). At some point we started branching master when we release. I do not know why we changed, but it seems we did not have any issues branching master around RC2 or RC3. Branching earlier, would mean that contributors do not have to carry patches for as long as they do now, and the risk of having to rebase patches several times is lower.

= Possible Solution / Improvement =
Change Terminology:
* Keep "Feature Freeze" as is
* Rename "Freeze Window/Period" to "Stabilisation Window/Period" or something similar
* Make clear that "Stabilisation Window/Period" != no development in the "Development Update x.y mail template"

Release Process improvements:
* Reopen the tree development tree as soon as possible after RC1 (I will let you guys figure out the best RC - let's call it RCx)
* In other words, create the release  branch at RCx

There could be some optimisations and additional things that may make sense:
* Encourage maintainers/committers to refrain from committing big refactoring changes during RCx and the final RC for a release to avoid complications if we want to cherry port bug fixes, etc. from master to the release branch
* Committers should be permitted to apply backports to the release branch until the actual release rather than putting all the burden on the stable maintainer(s)

[-- Attachment #1.2: Type: text/html, Size: 3055 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [xen 4.6 retrospective] [urgent] rename "freeze" window and make release branch as soon as possible after RC1

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 2935 bytes --]

On Wed, 5 Aug 2015, Lars Kurth wrote:
> This is one item of feedback, which I believe is a quick win for us. This is one piece of feedback from a list of
> items that have during the last few weeks been raised with me personally, either during face-2-face conversations
> in a private e-mail thread. See http://lists.xenproject.org/archives/html/xen-devel/2015-08/msg00173.html for
> information on the retrospective
>
> = Issue / Observation =
> The name "freeze" window/period - aka the time period from when we "feature freeze" until we branch master and/or
> make the release leads some contributors to mistakenly assume that development for the next release stops. I saw a
> few mails on xen-devel@ recently, pointing out to contributors that development does not stop during "freeze".
>  Chatting to Ian Campbell, he mentioned that he replied to several different people who said they were waiting for
> the tree to reopen. Maybe choosing a better name will help.

+1

Maybe "RC window" ?


> In addition, we used to branch master a lot earlier I believe up to Xen 4.1 (around RC2 or RC3). At some point we
> started branching master when we release. I do not know why we changed, but it seems we did not have any issues
> branching master around RC2 or RC3. Branching earlier, would mean that contributors do not have to carry patches
> for as long as they do now, and the risk of having to rebase patches several times is lower.

+1

I would even go as far as recommending maintainers to take patches in
their personal trees while xen-unstable is "frozen". Personally I
already do that for QEMU.

In fact I think that if a tree was always open to check stuff in (it
doesn't matter if the tree is xen-unstable, the maintainer's own git
tree, or a temporary branch somewhere), everybody would be much more
relaxed about releases and code freezes.


> = Possible Solution / Improvement =
> Change Terminology:
> * Keep "Feature Freeze" as is
> * Rename "Freeze Window/Period" to "Stabilisation Window/Period" or something similar

+1


> * Make clear that "Stabilisation Window/Period" != no development in the "Development Update x.y mail template"

+1


> Release Process improvements:
> * Reopen the tree development tree as soon as possible after RC1 (I will let you guys figure out the best RC -
> let's call it RCx)

+1


> * In other words, create the release  branch at RCx
>
> There could be some optimisations and additional things that may make sense:
> * Encourage maintainers/committers to refrain from committing big refactoring changes during RCx and the final RC
> for a release to avoid complications if we want to cherry port bug fixes, etc. from master to the release branch
> * Committers should be permitted to apply backports to the release branch until the actual release rather than
> putting all the burden on the stable maintainer(s)

+1

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [xen 4.6 retrospective] [urgent] rename "freeze" window and make release branch as soon as possible after RC1

[Jan Beulich]

>>> On 06.08.15 at 12:52, <stefano.stabellini@eu.citrix.com> wrote:
> I would even go as far as recommending maintainers to take patches in
> their personal trees while xen-unstable is "frozen". Personally I
> already do that for QEMU.

For small patches that might work. For large series perhaps repeatedly
needing non-trivial re-basing I don't see me as a maintainer do that
work over perhaps an extended period of time on behalf of contributors.

> In fact I think that if a tree was always open to check stuff in (it
> doesn't matter if the tree is xen-unstable, the maintainer's own git
> tree, or a temporary branch somewhere), everybody would be much more
> relaxed about releases and code freezes.

Still I agree that such a model would be the easier one for submitters.
What I'm not convinced is that making their lives easier is the higher
priority.

Jan

Re: [xen 4.6 retrospective] [urgent] rename "freeze" window and make release branch as soon as possible after RC1

[Lars Kurth]

Hi all,
I would like to pick this up again. We are now getting close to RC3 and do not seem to have consensus. Would it make sense to summarise and just go for a vote?
Lars

> On 12 Aug 2015, at 08:35, Jan Beulich <jbeulich@suse.com> wrote:
> 
>>>> On 06.08.15 at 12:52, <stefano.stabellini@eu.citrix.com> wrote:
>> I would even go as far as recommending maintainers to take patches in
>> their personal trees while xen-unstable is "frozen". Personally I
>> already do that for QEMU.
> 
> For small patches that might work. For large series perhaps repeatedly
> needing non-trivial re-basing I don't see me as a maintainer do that
> work over perhaps an extended period of time on behalf of contributors.
> 
>> In fact I think that if a tree was always open to check stuff in (it
>> doesn't matter if the tree is xen-unstable, the maintainer's own git
>> tree, or a temporary branch somewhere), everybody would be much more
>> relaxed about releases and code freezes.
> 
> Still I agree that such a model would be the easier one for submitters.
> What I'm not convinced is that making their lives easier is the higher
> priority.
> 
> Jan
> 

Re: [xen 4.6 retrospective] [urgent] rename "freeze" window and make release branch as soon as possible after RC1

[Jan Beulich]

>>> On 02.09.15 at 13:50, <lars.kurth.xen@gmail.com> wrote:
> I would like to pick this up again. We are now getting close to RC3 and do 
> not seem to have consensus. Would it make sense to summarise and just go for 
> a vote?

Feel free to go ahead.

Jan

Re: [xen 4.6 retrospective] [urgent] rename "freeze" window and make release branch as soon as possible after RC1

[Stefano Stabellini]

On Wed, 2 Sep 2015, Jan Beulich wrote:
> >>> On 02.09.15 at 13:50, <lars.kurth.xen@gmail.com> wrote:
> > I would like to pick this up again. We are now getting close to RC3 and do 
> > not seem to have consensus. Would it make sense to summarise and just go for 
> > a vote?
> 
> Feel free to go ahead.
 
Agreed

Re: [xen 4.6 retrospective] [urgent] rename "freeze" window and make release branch as soon as possible after RC1

[Lars Kurth]

> On 2 Sep 2015, at 14:12, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
> 
> On Wed, 2 Sep 2015, Jan Beulich wrote:
>>>>> On 02.09.15 at 13:50, <lars.kurth.xen@gmail.com> wrote:
>>> I would like to pick this up again. We are now getting close to RC3 and do 
>>> not seem to have consensus. Would it make sense to summarise and just go for 
>>> a vote?
>> 
>> Feel free to go ahead.
> 
> Agreed

See http://lists.xenproject.org/archives/html/xen-devel/2015-09/msg00282.html

Re: [xen 4.6 retrospective] [urgent] rename "freeze" window and make release branch as soon as possible after RC1

[Wei Liu]

On Wed, Aug 05, 2015 at 10:22:13AM +0100, Lars Kurth wrote:
> This is one item of feedback, which I believe is a quick win for us.
> This is one piece of feedback from a list of items that have during
> the last few weeks been raised with me personally, either during
> face-2-face conversations in a private e-mail thread. See
> http://lists.xenproject.org/archives/html/xen-devel/2015-08/msg00173.html
> <http://lists.xenproject.org/archives/html/xen-devel/2015-08/msg00173.html>
> for information on the retrospective
> 
> 
> = Issue / Observation =
> The name "freeze" window/period - aka the time period from when we
> "feature freeze" until we branch master and/or make the release leads
> some contributors to mistakenly assume that development for the next
> release stops. I saw a few mails on xen-devel@ recently, pointing out
> to contributors that development does not stop during "freeze".
> Chatting to Ian Campbell, he mentioned that he replied to several
> different people who said they were waiting for the tree to reopen.
> Maybe choosing a better name will help.
> 
> In addition, we used to branch master a lot earlier I believe up to
> Xen 4.1 (around RC2 or RC3). At some point we started branching master
> when we release. I do not know why we changed, but it seems we did not
> have any issues branching master around RC2 or RC3. Branching earlier,
> would mean that contributors do not have to carry patches for as long
> as they do now, and the risk of having to rebase patches several times
> is lower.
> 
> = Possible Solution / Improvement =
> Change Terminology:
> * Keep "Feature Freeze" as is
> * Rename "Freeze Window/Period" to "Stabilisation Window/Period" or

Don't really have preference on this.

> * Make clear that "Stabilisation Window/Period" != no development in
> the "Development Update x.y mail template"

+1

> 
> Release Process improvements:
> * Reopen the tree development tree as soon as possible after RC1 (I
> will let you guys figure out the best RC - let's call it RCx)

+1

> * In other words, create the release  branch at RCx
> 
> There could be some optimisations and additional things that may make
> sense:

> * Encourage maintainers/committers to refrain from committing big
> refactoring changes during RCx and the final RC for a release to avoid
> complications if we want to cherry port bug fixes, etc. from master to
> the release branch

+1

> * Committers should be permitted to apply backports to the release
> branch until the actual release rather than putting all the burden on
> the stable maintainer(s)

+1

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [xen 4.6 retrospective] [urgent] rename "freeze" window and make release branch as soon as possible after RC1

[Roger Pau Monné]

El 05/08/15 a les 11.22, Lars Kurth ha escrit:
> This is one item of feedback, which I believe is a quick win for us. This is one piece of feedback from a list of items that have during the last few weeks been raised with me personally, either during face-2-face conversations in a private e-mail thread. See http://lists.xenproject.org/archives/html/xen-devel/2015-08/msg00173.html <http://lists.xenproject.org/archives/html/xen-devel/2015-08/msg00173.html> for information on the retrospective
> 
> 
> = Issue / Observation =
> The name "freeze" window/period - aka the time period from when we "feature freeze" until we branch master and/or make the release leads some contributors to mistakenly assume that development for the next release stops. I saw a few mails on xen-devel@ recently, pointing out to contributors that development does not stop during "freeze".  Chatting to Ian Campbell, he mentioned that he replied to several different people who said they were waiting for the tree to reopen. Maybe choosing a better name will help.
> 
> In addition, we used to branch master a lot earlier I believe up to Xen 4.1 (around RC2 or RC3). At some point we started branching master when we release. I do not know why we changed, but it seems we did not have any issues branching master around RC2 or RC3. Branching earlier, would mean that contributors do not have to carry patches for as long as they do now, and the risk of having to rebase patches several times is lower.
> 
> = Possible Solution / Improvement =
> Change Terminology:
> * Keep "Feature Freeze" as is

+1

> * Rename "Freeze Window/Period" to "Stabilisation Window/Period" or something similar

-1. IMHO all projects I work with use the "freeze" terminology, changing
it to something else is just going to confuse people.

> * Make clear that "Stabilisation Window/Period" != no development in the "Development Update x.y mail template"

+1

Roger.

Re: [xen 4.6 retrospective] [urgent] rename "freeze" window and make release branch as soon as possible after RC1

[Jan Beulich]

>>> On 05.08.15 at 11:22, <lars.kurth.xen@gmail.com> wrote:
> Release Process improvements:
> * Reopen the tree development tree as soon as possible after RC1 (I will let 
> you guys figure out the best RC - let's call it RCx)
> * In other words, create the release  branch at RCx

Assuming this is what the majority wants (I'm not particularly convinced
this is the best model, but I'm also not particularly convinced branching
as late as possible is - both have up and down sides) ...

> There could be some optimisations and additional things that may make sense:
> * Encourage maintainers/committers to refrain from committing big 
> refactoring changes during RCx and the final RC for a release to avoid 
> complications if we want to cherry port bug fixes, etc. from master to the 
> release branch

... I dislike this, i.e. I don't see a "half open" tree to be very helpful.
Especially large refactoring patches tend to repeatedly need manual
adjustment when retained outside of the main tree.

> * Committers should be permitted to apply backports to the release branch 
> until the actual release rather than putting all the burden on the stable 
> maintainer(s)

Yet a clear +1 here.

Jan

Re: [xen 4.6 retrospective] More public/easy to find information about the release schedule

[Roger Pau Monné]

= Issue / Observation =

The information about the release schedule is not clearly published
anywhere apart from the mailing lists, which makes it hard for
non-developers (or even for developers) given that the mailing list
traffic for xen-devel is high.

= Possible Solution / Improvement =

Publish the release schedule in a web page with a concrete schedule,
like the FreeBSD Release Engineering Team does:

https://www.freebsd.org/releng/

They even have the schedule for the 11.0-RELEASE, which is not expected
until a year from now. Also each step/date contains an explanation of
what's happening and what it means from a developer point of view.

Roger.

Re: [xen 4.6 retrospective] More public/easy to find information about the release schedule

[Wei Liu]

On Fri, Aug 07, 2015 at 05:36:57PM +0200, Roger Pau Monné wrote:
> = Issue / Observation =
> 
> The information about the release schedule is not clearly published
> anywhere apart from the mailing lists, which makes it hard for
> non-developers (or even for developers) given that the mailing list
> traffic for xen-devel is high.
> 
> = Possible Solution / Improvement =
> 
> Publish the release schedule in a web page with a concrete schedule,
> like the FreeBSD Release Engineering Team does:
> 
> https://www.freebsd.org/releng/
> 
> They even have the schedule for the 11.0-RELEASE, which is not expected
> until a year from now. Also each step/date contains an explanation of
> what's happening and what it means from a developer point of view.
> 

This is a good idea.

> Roger.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [xen 4.6 retrospective] More public/easy to find information about the release schedule

[Lars Kurth]

> On 10 Aug 2015, at 09:33, Wei Liu <wei.liu2@citrix.com> wrote:
> 
> On Fri, Aug 07, 2015 at 05:36:57PM +0200, Roger Pau Monné wrote:
>> = Issue / Observation =
>> 
>> The information about the release schedule is not clearly published
>> anywhere apart from the mailing lists, which makes it hard for
>> non-developers (or even for developers) given that the mailing list
>> traffic for xen-devel is high.

This is not entirely true: see http://wiki.xenproject.org/wiki/Xen_Project_Hypervisor_Roadmap/4.6
However, I think https://www.freebsd.org/releng/ and also the odd mail on announce@ would make sense

Lars

Re: [xen 4.6 retrospective] More public/easy to find information about the release schedule

[Fabio Fantoni]

Il 10/08/2015 11:06, Lars Kurth ha scritto:
>> On 10 Aug 2015, at 09:33, Wei Liu <wei.liu2@citrix.com> wrote:
>>
>> On Fri, Aug 07, 2015 at 05:36:57PM +0200, Roger Pau Monné wrote:
>>> = Issue / Observation =
>>>
>>> The information about the release schedule is not clearly published
>>> anywhere apart from the mailing lists, which makes it hard for
>>> non-developers (or even for developers) given that the mailing list
>>> traffic for xen-devel is high.
> This is not entirely true: see http://wiki.xenproject.org/wiki/Xen_Project_Hypervisor_Roadmap/4.6
Hi, I take a look to the wiki page, can be good mention also the "add of 
ahci disk controller support for hvm domUs"? I saw that is missed in 
features list.
> However, I think https://www.freebsd.org/releng/ and also the odd mail on announce@ would make sense
>
> Lars
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [xen 4.6 retrospective] More public/easy to find information about the release schedule

[Lars Kurth]

> On 10 Aug 2015, at 10:40, Fabio Fantoni <fabio.fantoni@m2r.biz> wrote:
> 
> Il 10/08/2015 11:06, Lars Kurth ha scritto:
>>> On 10 Aug 2015, at 09:33, Wei Liu <wei.liu2@citrix.com> wrote:
>>> 
>>> On Fri, Aug 07, 2015 at 05:36:57PM +0200, Roger Pau Monné wrote:
>>>> = Issue / Observation =
>>>> 
>>>> The information about the release schedule is not clearly published
>>>> anywhere apart from the mailing lists, which makes it hard for
>>>> non-developers (or even for developers) given that the mailing list
>>>> traffic for xen-devel is high.
>> This is not entirely true: see http://wiki.xenproject.org/wiki/Xen_Project_Hypervisor_Roadmap/4.6
> Hi, I take a look to the wiki page, can be good mention also the "add of ahci disk controller support for hvm domUs"? I saw that is missed in features list.

Will add it. I scraped the information from the last 4.6 email. But there were some omissions, which have not yet been addressed. Was waiting for Wei's Xen Dev Summit presentation before I update. 

>> However, I think https://www.freebsd.org/releng/ and also the odd mail on announce@ would make sense
>> 
>> Lars
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
> 

Re: [xen 4.6 retrospective] More public/easy to find information about the release schedule

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 997 bytes --]

On Fri, 2015-08-07 at 17:36 +0200, Roger Pau Monné wrote:
> = Issue / Observation =
> 
> The information about the release schedule is not clearly published
> anywhere apart from the mailing lists, which makes it hard for
> non-developers (or even for developers) given that the mailing list
> traffic for xen-devel is high.
> 
> = Possible Solution / Improvement =
> 
> Publish the release schedule in a web page with a concrete schedule,
> like the FreeBSD Release Engineering Team does:
> 
> https://www.freebsd.org/releng/
> 
+1

At Fedora, they do something similar:

 https://fedoraproject.org/wiki/Releases/23/Schedule

 https://fedoraproject.org/wiki/Releases/22/Schedule?rd=Releases/22

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[xen 4.6 retrospective] [bad] review load near freeze point

[Jan Beulich]

>>> On 04.08.15 at 14:52, <lars.kurth.xen@gmail.com> wrote:
   = Issue / Observation =

Maybe my memory regarding the 4.5 release has faded, but I'm
having the impression that 4.6 was quite a bit worse. This was at
parts due to the sheer number of patch series and their sizes,
but also because frequently they took quite a bit more than just a
couple of iterations.

Some of this is (as mentioned on past discussions) caused by too
little review involvement of non-maintainers (the need of which
exists because of limited bandwidth maintainers have for doing
reviews), but I think there are also quality issues here (see below).

   = Possible Solution / Improvement =

While I agree that some aspects (like coding style issues) could be
taken care of by requiring submitters to have their patches be
checked by (not yet existing) tools/scripts, I think much of this is a
problem of enforcing discipline on oneself: Sloppiness in style,
according to my observation, frequently is accompanied by
sloppiness in the actual coding. While I think that suggestion may
be considered harsh, I'd like to throw up for discussion a cut off
on the number of iterations that may be submitted for a particular
series within a certain time (with a high threshold on exceeding
the limit for really minor adjustments). I would hope that making
people aware that the number of tries they have is limited, they
would start self-reviewing their changes more consciously.

And yes (to preempt respective responses), I'm aware that such a
rule may raise the bar for first time contributors. But those often
don't contribute large patches/series, and I don't think such a rule
would need to be enforced as strictly on small contributions (plus
newcomers could be granted some [limited] slack).

And of course, people should submit their code early, and follow
up diligently to review comments. Submitting a v1 half a year
before the freeze, and being at say v4 around freeze time (and
then perhaps even demanding the code to go in because "it was
submitted early") is about as bad as starting work on a big series
a month ahead of freeze.

Jan

Re: [xen 4.6 retrospective] [bad] review load near freeze point

[Lars Kurth]

[-- Attachment #1: Type: text/plain, Size: 1099 bytes --]

Jan,

> On 12 Aug 2015, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
> 
>>>> On 04.08.15 at 14:52, <lars.kurth.xen@gmail.com> wrote:
>   = Issue / Observation =
> 
> Maybe my memory regarding the 4.5 release has faded, but I'm
> having the impression that 4.6 was quite a bit worse. This was at
> parts due to the sheer number of patch series and their sizes,
> but also because frequently they took quite a bit more than just a
> couple of iterations.

it definitely feels as if 4.6 was worse than 4.5 (and other releases). You may remember that at last year's summit I highlighted that the average from first post on xen-devel@ to commit has increased. Moving from the soft freeze we used to have to the hard freeze, IMHO exacerbated the "stress" for both reviewers and contributors, because of the shorter time window.

Anyway, Stefano run some of the same tools that we ran last year on this release cycle. This does show that the situation has become worse. Note that there is a "gap" from Sept - Dec 14. Didn't have time to run the scripts: they take forever to complete


[-- Attachment #2: PastedGraphic-1.pdf --]
[-- Type: application/pdf, Size: 65845 bytes --]

[-- Attachment #3: Type: text/plain, Size: 2880 bytes --]



If you look at the average time from first post on xen-devel@ to commit, the data looks even worse

July 2013 - Jan 2014, Average to Commit = 28.8 days
Feb 2014 - Aug 2014, Average to Commit =  31.5 days 
Jan 2015 - Jul 2015, Average to Commit =  70 days

Note that the 70 days are caused by about 40 patches which took more than 1, 2 or 3 years to commit. There are a number of reasons why a series may take > 1 year: 
* One possible reason is excessively long review times, maybe combined with long wait times in between reviews
* Another one is that a series may have been parked by the contributor and picked up later, sometimes several times

If you remove the 1+ year outliers from the stats, you get...

July 2013 - Jan 2014, Average to Commit Adjusted = 26.4 days
Feb 2014 - Aug 2014, Average to Commit Adjusted =  26.7 days 
Jan 2015 - Jul 2015, Average to Commit Adjusted =  34.7 days

This does show that we still do have a significant issue

> Some of this is (as mentioned on past discussions) caused by too
> little review involvement of non-maintainers (the need of which
> exists because of limited bandwidth maintainers have for doing
> reviews), but I think there are also quality issues here (see below).

I definitely agree that we need more review involvement of non-maintainers (see later)

I did a bit more number crunching to try and get a clearer handle on the root causes of what we have seen in this release cycle. Unfortunately, with existing open source data mining tools it is near-impossible to model an e-mail based review process. With the tools I have today I can't identify whether there are clear patterns which contribute to the issues we are seeing with reviews. Of course individual reviewers and contributors will have their own experiences and get a sense of what is going on: but at least I would like to understand more clearly what is going on, backed up by data, before we make any drastic changes which could be having unintended consequences.

I asked to Advisory Board to fund Bitergia to help us develop some tools/database support to model our review process. This was approved 3 weeks ago. I a started working with them, and it appears that they can model our e-mail based review process (it assumes that we all use "git send-email" / patch bomb, which I think is almost always true). This will take until end of Sept/beginning of October to complete because of summer holidays in Spain.

In any case, I did some work to try and understand what is happening, presented it to the Advisory Board and refined it after a couple of conversations with Bitergia. 

First I looked at the number of patches we typically accept per year: in the last 5 years we have accepted between 2000 - 2200 patches and this year looks similar. 

Then I looked at who does reviews (the 2015 figures are partial)


[-- Attachment #4: PastedGraphic-3.pdf --]
[-- Type: application/pdf, Size: 39280 bytes --]

[-- Attachment #5: Type: text/plain, Size: 642 bytes --]


 
This shows that we have a small number of developers (about 14) who do almost all of the reviews. This clearly is an issue. The table shows the number of developers doing more/less than x% of reviews. This would be become a problem, if we saw something that upsets the status quo (e.g. more contributions, more complex contributions, more communication issues, ...).

So I started looking at patches posted and comments on patches posted (without being able to tie patch series together, and tie versions of series together). Note that the 2015 figures count Jan to Aug 11th - I added a projection column based on past performance.


[-- Attachment #6: PastedGraphic-4.pdf --]
[-- Type: application/pdf, Size: 40114 bytes --]

[-- Attachment #7: Type: text/plain, Size: 1304 bytes --]



This is actually staggering, but also interesting
* The first observation is that on average for the past 3 year each patch version had on average 2.1 replies and that this ratio is fairly stable
* The number of patches posted (and/or re-posted) has crept up slowly over the last few years
* This year: the number of patches posted (and/or re-posted) will almost be twice that of 2014. 

So it appears that either 
a) we have seen *a lot* of new contributions, OR 
b) on average the number of iterations to get patches into xen.git has almost doubled for a number of reasons (somehow this doesn't feel right) 

Most likely both is happening. In any case the effect of both of these would be an increased back-log, while our capacity to review code has remained static, which is causing the issues we are seeing with reviews.

Given that the average number of replies to patches stayed fairly constant, I thought I'd look at the ratio of patches (including re-posts) on the list divided by patches that made it into xen.git - and I also thought I'd do this for a similar project such as QEMU. We had a go at Linux as well, but it appears that Linux is a lot more de-centralized today using lots of different mailing lists: to cut things short, we didn't know whether the data was sound.


[-- Attachment #8: PastedGraphic-5.pdf --]
[-- Type: application/pdf, Size: 30697 bytes --]

[-- Attachment #9: Type: text/plain, Size: 6452 bytes --]



In any case, this appears to show that a ratio of 2-3 was normal for us and QEMU in the last few year (note that there is no research which shows an optimum), when we had no problems. It also shows that both Xen and Qemu have started to see similar issues in 2015 (albeit ours is more severe). Again, this doesn't allow us to pinpoint where exactly our problem is.

What we are seeing could be due to ...

A) Increasing number of contributions 
Not enough review capacity to support growth; we know that review capacity has remained stable 
=> Increasing back-log (aka ongoing reviews on xendevel@) 

B) Increasing number of review cycles per patch/patch series 

Note: we know that the average number of review comments per patch version is stable. Also the data could be *significantly* skewed when large patch series are reposted frequently 

Reasons for this could be:
- More disagreements amongst maintainers, reviewers & contributors 
- Lower quality contributions, requiring more review cycles 
- More complex and larger contributions, requiring more review cycles 
- Increasing standards required to get code up-streamed (aka quality) 
- ...

=> Increasing back-log (aka ongoing reviews on xendevel@)

For the latter, with appropriate tooling, we should be able to see patterns in review process data. This would allow us for example to correlate size of patch with review times/number of iterations, identify whether individuals struggle in some form - and then help them, correlate areas of code with ... review times/number of iterations, ... Right now I can't do any of this: I can only ever be reactive towards issues and crises.

C) A combination of all/some of the he above

If the issue is mostly caused by A), we can only fix this with more reviewers. If the issue is B) there may actions we can come up with to buy us some time to take off the pressure. Also in some cases, more reviewers could be counteractive

In any case, I have been working with a number of vendors who say they are committed to helping out with code reviews. But we all know that training reviewers (and ultimately maintainers) takes time and commitment. It is also not something everyone is comfortable doing.
 
>   = Possible Solution / Improvement =
> 
> While I agree that some aspects (like coding style issues) could be
> taken care of by requiring submitters to have their patches be
> checked by (not yet existing) tools/scripts,

This would be one of the measures which may buy us some time

> I think much of this is a
> problem of enforcing discipline on oneself: Sloppiness in style,
> according to my observation, frequently is accompanied by
> sloppiness in the actual coding.

I am not sure whether this is actually the primary issue. Everyone will most likely have a different take on the situation based on their personal experience as well as their personal level of expectations and experience. When I talk to reviewers and maintainers, there does not appear to be a single issue that sticks out. 

> While I think that suggestion may
> be considered harsh, I'd like to throw up for discussion a cut off
> on the number of iterations that may be submitted for a particular
> series within a certain time (with a high threshold on exceeding
> the limit for really minor adjustments). I would hope that making
> people aware that the number of tries they have is limited, they
> would start self-reviewing their changes more consciously.

Can you elaborate? I am not sure I fully understand what you propose. Also, there is a risk that this becomes complex. 

I would feel much more inclined and comfortable to
* get better tooling to help us understand the review process
* then actively work with people that are struggling 
** IMHO we have done this fairly successfully with design discussions

> And yes (to preempt respective responses), I'm aware that such a
> rule may raise the bar for first time contributors. But those often
> don't contribute large patches/series, and I don't think such a rule
> would need to be enforced as strictly on small contributions (plus
> newcomers could be granted some [limited] slack).

One of the challenges that newcomers often have is tracking review comments. Maybe we can put together a number of "recipes" which work - such as http://article.gmane.org/gmane.comp.emulators.xen.devel/254965 from Ian C - and include them into the training material at http://wiki.xenproject.org/wiki/Category:Developers

Maybe we should also start pointing people to training material, if the same issues come up with the same person. 

> And of course, people should submit their code early, and follow
> up diligently to review comments. Submitting a v1 half a year
> before the freeze, and being at say v4 around freeze time (and
> then perhaps even demanding the code to go in because "it was
> submitted early") is about as bad as starting work on a big series
> a month ahead of freeze.

One thing which we have seen this year, and which I believe you refer to, is an increase where we saw designs + implementations in the same release cycle, with the hope of getting it all done. Necessarily the implementation can't start until after the the design is agreed. 

Part of the issue here IMHO is that a 9-10 months release cadence is too long. The whole industry is moving at a very fast pace, which creates significant business pressure to get features into a specific release. If we were going too achieve say 6 months, then that pressure would mostly go away, as the cost of missing a release is less than now.

In other words
a) the pressure would be less, and we would see fewer requests for exceptions - we could even consider not to allow freeze exceptions
b) the development window would be too short to do a design & implementation of a larger/complex feature - so that problem would go away
c) we would also need more predictable release dates (let's say March and September) - right now, release schedules are only visibility up to the next release. That means that vendors/contributors can't plan for the longer term. This will lead most managers and project managers who work with developers to push them to get stuff into the release time-frame that is known.

However that should probably be a separate discussion/thread.

Sorry: that mail was a lot longer than I thought it would be

Regards
Lars


[-- Attachment #10: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [xen 4.6 retrospective] [bad] review load near freeze point

[Lars Kurth]

Jan,
I wanted to pick this one up again, as this also came up in the developer meeting (where we did a face-2-face retrospective). A few other possible solutions were bounced regarding this problem.
Regards
Lars

> On 12 Aug 2015, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
> 
>>>> On 04.08.15 at 14:52, <lars.kurth.xen@gmail.com> wrote:
>   = Issue / Observation =
> 
> Maybe my memory regarding the 4.5 release has faded, but I'm
> having the impression that 4.6 was quite a bit worse. This was at
> parts due to the sheer number of patch series and their sizes,
> but also because frequently they took quite a bit more than just a
> couple of iterations.
> 
> Some of this is (as mentioned on past discussions) caused by too
> little review involvement of non-maintainers (the need of which
> exists because of limited bandwidth maintainers have for doing
> reviews), but I think there are also quality issues here (see below).

This also came in the retrospective at the developer meeting, on Wed the 19th. I believe that the following items contributed to this (I want to cover the "not enough review capacity" and "quality issues" separately).

A) Too many things happening around the freeze
B) Not enough coordination amongst committers
C) Poor communication: the community didn't read Wei's emails on the change of the release process and was surprised by it

>   = Possible Solution / Improvement =
> 
> While I agree that some aspects (like coding style issues) could be
> taken care of by requiring submitters to have their patches be
> checked by (not yet existing) tools/scripts, I think much of this is a
> problem of enforcing discipline on oneself: Sloppiness in style,
> according to my observation, frequently is accompanied by
> sloppiness in the actual coding. While I think that suggestion may
> be considered harsh, I'd like to throw up for discussion a cut off
> on the number of iterations that may be submitted for a particular
> series within a certain time (with a high threshold on exceeding
> the limit for really minor adjustments). I would hope that making
> people aware that the number of tries they have is limited, they
> would start self-reviewing their changes more consciously.
> 
> And yes (to preempt respective responses), I'm aware that such a
> rule may raise the bar for first time contributors. But those often
> don't contribute large patches/series, and I don't think such a rule
> would need to be enforced as strictly on small contributions (plus
> newcomers could be granted some [limited] slack).
> 
> And of course, people should submit their code early, and follow
> up diligently to review comments. Submitting a v1 half a year
> before the freeze, and being at say v4 around freeze time (and
> then perhaps even demanding the code to go in because "it was
> submitted early") is about as bad as starting work on a big series
> a month ahead of freeze.

A few other possible solutions, which may help alleviate the freeze point crunch may be:

== 1) Highlight any changes to the process in the release manager's boilerplate in every release mail ==

This would fix C)

== 2) Get rid of freeze extensions altogether ==

This should shorten the stressful period a little; albeit the effect may be that the stress may start somewhat earlier and help with A)
Even though in the short term, this may not immediately lead to contributors planning to get patches in at the beginning of the release cycle, in the long run, this should do it. 

== 3) Earlier hard freeze for large and risky patches ==

Another thing we could do, is have an earlier freeze point for large and risky patches. Aka, anything of a certain size or high risk has to go in earlier than the hard freeze, allowing for some re-work or tidy up between that early freeze point and the final one. It could would act as a filter and create some focus. As such it should help with A) and B). Whether we allow exceptions, between those two "freeze" points is an open question.

== 4) Better upfront planning of what is desired to be in a release. In particular for larger patches. ==

Part of the reason why 4.6 was so stressful, had to do with sheer numbers: Xen 4.5 had 1812 commits, and 4.6 had 1974 commits after the push gate cleared. We had a number of large patch series (at least 50% of the ones which were close and caused some of the stress) where we started designs in the 4.5 release cycle, with contributors hoping they would go in.

That number of features was unusually high. If in Feb we had known that there way say 10 large/complex series coming towards the end of the cycle, and say 2 were from vendor X, 4 from vendor Y, ... we could ask the community and each vendor, to sort these larger ones by priority. 

Not quite sure how the community prioritisation would work in practice. Asking vendors to prioritise their own planned patches is obviously easier.

We probably can get some data that tells us how many of these patch series we can on average into good shape in a release cycle. And build in some slack.

The release manager and reviewers could then track the subset more closely. I think related to this is that the fair/good/... tagging is maybe not that useful. It may be worth thinking about different ways of tagging these.

Playing devils advocate:
Let's just assume we had known upfront, which series were more "important" than others. Warning flags to contributors could have been raised earlier along the lines of: "You highlighted this patch as important, but the patch is only at v2 now and should be much further. Unless, we there is significant progress in X weeks, we would take that series out of the 'important' bucket.". It would also create more focus earlier and encourage maintainers to review those "important" patches earlier. So this should help with A) and B).

Of course this may not work that well in practice. However, any software development team does that type of prioritisation and then tightens what they do closer to the release.    

Regards
Lars

Re: [xen 4.6 retrospective] [bad] review load near freeze point

[Jan Beulich]

>>> On 28.08.15 at 17:05, <lars.kurth.xen@gmail.com> wrote:
>> On 12 Aug 2015, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
>>>>> On 04.08.15 at 14:52, <lars.kurth.xen@gmail.com> wrote:
>>   = Issue / Observation =
>> 
>> Maybe my memory regarding the 4.5 release has faded, but I'm
>> having the impression that 4.6 was quite a bit worse. This was at
>> parts due to the sheer number of patch series and their sizes,
>> but also because frequently they took quite a bit more than just a
>> couple of iterations.
>> 
>> Some of this is (as mentioned on past discussions) caused by too
>> little review involvement of non-maintainers (the need of which
>> exists because of limited bandwidth maintainers have for doing
>> reviews), but I think there are also quality issues here (see below).
> 
> This also came in the retrospective at the developer meeting, on Wed the 
> 19th. I believe that the following items contributed to this (I want to cover 
> the "not enough review capacity" and "quality issues" separately).
> 
> A) Too many things happening around the freeze
> B) Not enough coordination amongst committers

Can you be more specific (perhaps with examples) about this one?

> C) Poor communication: the community didn't read Wei's emails on the change 
> of the release process and was surprised by it

How is people not reading mails poor communication?

>>   = Possible Solution / Improvement =
>> [...]
> A few other possible solutions, which may help alleviate the freeze point 
> crunch may be:
> 
> == 1) Highlight any changes to the process in the release manager's 
> boilerplate in every release mail ==
> 
> This would fix C)

Would it? People not reading mails means people not reading mails.

> == 2) Get rid of freeze extensions altogether ==
> 
> This should shorten the stressful period a little; albeit the effect may be 
> that the stress may start somewhat earlier and help with A)
> Even though in the short term, this may not immediately lead to contributors 
> planning to get patches in at the beginning of the release cycle, in the long 
> run, this should do it. 

I think there's always going to be reasons for making exceptions.
We may be able to further limit their amount, but I don't think it's
realistic to exclude them altogether.

> == 3) Earlier hard freeze for large and risky patches ==
> 
> Another thing we could do, is have an earlier freeze point for large and 
> risky patches. Aka, anything of a certain size or high risk has to go in 
> earlier than the hard freeze, allowing for some re-work or tidy up between 
> that early freeze point and the final one. It could would act as a filter and 
> create some focus. As such it should help with A) and B). Whether we allow 
> exceptions, between those two "freeze" points is an open question.

This sounds reasonable if we can come up with not too fuzzy a
definition of "large or risky" (if we leave this to the maintainers'/
committers'/release manager's discretion, there's always going to
be arguments about this).

> == 4) Better upfront planning of what is desired to be in a release. In 
> particular for larger patches. ==

Yeah, as you say multiple times further down - this can be tried, but it's
not clear whether this actually can work well. The main coordination
effort should anyway inside larger contributing entities, since no matter
how close a release is, submitting multiple large series (almost) at once
is always going to cause frustration if dealing with the submissions is
limited to very few people.

Jan

Re: [xen 4.6 retrospective] [bad] review load near freeze point

[Lars Kurth]

> On 28 Aug 2015, at 16:21, Jan Beulich <jbeulich@suse.com> wrote:
> 
>> B) Not enough coordination amongst committers
> 
> Can you be more specific (perhaps with examples) about this one?

A few concrete example were several of Vitaly's series (I will let him point out a couple of examples, as he raised this one).
Anyone else who has such examples?

I have seen a few, but would need to get back and investigate. in particular the fault-line seems to be around patches that affect both hypervisor and tools. Th feedback was that there can be weeks between hypervisor and tools portions of a series being reviewed, leading to lost elapsed times.
 
> 
>> C) Poor communication: the community didn't read Wei's emails on the change 
>> of the release process and was surprised by it
> 
> How is people not reading mails poor communication?

At some point we have to draw a line. It's understandable that someone may forget about an announcement made a few months back. If the same reminder comes every month and people don't read these mails, then they only have themselves to blame. 

>>>  = Possible Solution / Improvement =
>>> [...]
>> A few other possible solutions, which may help alleviate the freeze point 
>> crunch may be:
>> 
>> == 1) Highlight any changes to the process in the release manager's 
>> boilerplate in every release mail ==
>> 
>> This would fix C)
> 
> Would it? People not reading mails means people not reading mails.

See above. At least, we would have done our best. It would also cover us for the case, where someone started following the list *after* we made a change such as the hard freeze.

>> == 2) Get rid of freeze extensions altogether ==
>> 
>> This should shorten the stressful period a little; albeit the effect may be 
>> that the stress may start somewhat earlier and help with A)
>> Even though in the short term, this may not immediately lead to contributors 
>> planning to get patches in at the beginning of the release cycle, in the long 
>> run, this should do it. 
> 
> I think there's always going to be reasons for making exceptions.
> We may be able to further limit their amount, but I don't think it's
> realistic to exclude them altogether.

I just put this out there. Maybe the case, is for making them more exceptional.

>> == 3) Earlier hard freeze for large and risky patches ==
>> 
>> Another thing we could do, is have an earlier freeze point for large and 
>> risky patches. Aka, anything of a certain size or high risk has to go in 
>> earlier than the hard freeze, allowing for some re-work or tidy up between 
>> that early freeze point and the final one. It could would act as a filter and 
>> create some focus. As such it should help with A) and B). Whether we allow 
>> exceptions, between those two "freeze" points is an open question.
> 
> This sounds reasonable if we can come up with not too fuzzy a
> definition of "large or risky" (if we leave this to the maintainers'/
> committers'/release manager's discretion, there's always going to
> be arguments about this).

Agreed. Without thinking this through much, here are some ideas
1) Large could be defined in terms of characters changed, number of files touched, number of patches, ...
2) Risky is a lot harder. One way of looking at it may be: if some patch is of a certain size, but only at revision X (say 3), then it is likely to be risky

The question is whether we need to consider risky at all: if the main problem is review capacity, then probably starting with size may suffice

>> == 4) Better upfront planning of what is desired to be in a release. In 
>> particular for larger patches. ==
> 
> Yeah, as you say multiple times further down - this can be tried, but it's
> not clear whether this actually can work well. The main coordination
> effort should anyway inside larger contributing entities, since no matter
> how close a release is, submitting multiple large series (almost) at once
> is always going to cause frustration if dealing with the submissions is
> limited to very few people.

Agreed. I am not sure this would always work. 
But we could prompt larger contributors to highlight their priorities in the earlier development update mails, and track that information.
At the very best, it sets expectations earlier that there is a limit to what we can do close to a freeze

Lars

Re: [xen 4.6 retrospective] [bad] review load near freeze point

[Jan Beulich]

>>> On 28.08.15 at 18:04, <lars.kurth.xen@gmail.com> wrote:
>> On 28 Aug 2015, at 16:21, Jan Beulich <jbeulich@suse.com> wrote:
>> 
>>> B) Not enough coordination amongst committers
>> 
>> Can you be more specific (perhaps with examples) about this one?
> 
> A few concrete example were several of Vitaly's series (I will let him point 
> out a couple of examples, as he raised this one).
> Anyone else who has such examples?
> 
> I have seen a few, but would need to get back and investigate. in particular 
> the fault-line seems to be around patches that affect both hypervisor and 
> tools. Th feedback was that there can be weeks between hypervisor and tools 
> portions of a series being reviewed, leading to lost elapsed times.

But I'm pretty convinced this isn't because of bad coordination between
maintainers (not committers btw), but because of a lack of bandwidth.
Just because reviewers for one side have the cycles doesn't means the
ones on the other side have too.

>>> == 3) Earlier hard freeze for large and risky patches ==
>>> 
>>> Another thing we could do, is have an earlier freeze point for large and 
>>> risky patches. Aka, anything of a certain size or high risk has to go in 
>>> earlier than the hard freeze, allowing for some re-work or tidy up between 
>>> that early freeze point and the final one. It could would act as a filter and 
>>> create some focus. As such it should help with A) and B). Whether we allow 
>>> exceptions, between those two "freeze" points is an open question.
>> 
>> This sounds reasonable if we can come up with not too fuzzy a
>> definition of "large or risky" (if we leave this to the maintainers'/
>> committers'/release manager's discretion, there's always going to
>> be arguments about this).
> 
> Agreed. Without thinking this through much, here are some ideas
> 1) Large could be defined in terms of characters changed, number of files 
> touched, number of patches, ...
> 2) Risky is a lot harder. One way of looking at it may be: if some patch is 
> of a certain size, but only at revision X (say 3), then it is likely to be 
> risky

Specifically no - imo normally patches shouldn't require more than a
couple of iterations. I said that in my original mail: The quality of
submissions often isn't good enough, or else we wouldn't have to go
through so many cycles.

> The question is whether we need to consider risky at all: if the main 
> problem is review capacity, then probably starting with size may suffice

Agreed. Albeit you certainly realize there are small patches hard to
review (and possibly quite risky) and large patches that are pretty
trivial to approve.

Jan

Re: [xen 4.6 retrospective] [bad] review load near freeze point

[Lars Kurth]

> On 28 Aug 2015, at 17:22, Jan Beulich <JBeulich@suse.com> wrote:
> 
>>>> On 28.08.15 at 18:04, <lars.kurth.xen@gmail.com> wrote:
>>> On 28 Aug 2015, at 16:21, Jan Beulich <jbeulich@suse.com> wrote:
>>> 
>>>> B) Not enough coordination amongst committers
>>> 
>>> Can you be more specific (perhaps with examples) about this one?
>> 
>> A few concrete example were several of Vitaly's series (I will let him point 
>> out a couple of examples, as he raised this one).
>> Anyone else who has such examples?
>> 
>> I have seen a few, but would need to get back and investigate. in particular 
>> the fault-line seems to be around patches that affect both hypervisor and 
>> tools. The feedback was that there can be weeks between hypervisor and tools 
>> portions of a series being reviewed, leading to lost elapsed times.
> 
> But I'm pretty convinced this isn't because of bad coordination between
> maintainers (not committers btw),

Should have been s/committers/reviewers/ - apologies

> but because of a lack of bandwidth.
> Just because reviewers for one side have the cycles doesn't means the
> ones on the other side have too.

Not disagreeing. It would be good though if we can eliminate a "need for better coordination" as a primary cause for these delays.
I am somewhat stuck in a hard place, because everyone I talk to has a different opinion on what is going wrong. 

Re: [xen 4.6 retrospective] Kicking off a retrospective for Xen 4.6 (deadline August 28th)

[Wu, Feng]

[-- Attachment #1.1: Type: text/plain, Size: 6086 bytes --]

It is very helpful if the contributors can post a design document before sending the patches to community,
It has the following advantages:

1.       A design document can give people a whole picture of the feature before going to code details, which
Makes it easier to follow the code when doing review.

2.       The related maintainers and other contributors can discuss the design, and make an agreement on it.
So that the contributor can do the right thing in the right direction. This can save lots of efforts for both the
maintainers and the contributors.

3.       This is the common software development process, design first, then coding.

The whole point of having such design discussions is to get maintainer's feedbacks as early as possible
and have everyone agree on the solution architecture up front. This approach has worked great when all
relevant maintainers actively participate in the discussions like the case of PML patchset. Our assumption is
that all relevant maintainers agree with the design if they remain silent during design discussion. However,
when we come up with an implementation based on the agreed design and maintainers who did not participate
in design discussion raise design problems later, it really made engineers frustrated and defeated the whole
motivation of having design discussion up front.

Thanks,
Feng


From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Lars Kurth
Sent: Tuesday, August 04, 2015 8:53 PM
To: xen devel
Subject: [Xen-devel] [xen 4.6 retrospective] Kicking off a retrospective for Xen 4.6 (deadline August 28th)


Hi all,

I have been asked by a number of people to kick off a retrospective for
* what worked well
* didn't work well
in the Xen 4.6 release cycle.

As most of the stress of the last few weeks is now out of the way, I
thought it would be appropriate to start this process now, and let it run
for a few weeks. We have not done this type of retrospective over e-mail
before (we have done so at developer meetings as well as Hackathons).
As we have a global and distributed community, I think it is worthwhile
trying this by email. To make this work, you need to follow a few ground
rules.

= Ground Rules =
We would like to hear
a) what worked well and
b) what didn't work well
in this release cycle.

Please provide feedback before August 28th.

== What to discuss / what not to discuss ==

* Do NOT send issues which are personal to the mailing list. This means,
do NOT send anything related to individuals or companies that may have
occurred in this release cycle. We do not want to have divisive flame
wars in our community: this does not fit our values and will help
no-one. If you believe that we do have an issue, that requires referring
to individuals/companies, please send me a private email following the
same formatting conventions as outlined below. I can then work with you
to figure out how to best approach this issue or piece of feedback.

* If you are not sure, contact me beforehand and we can discuss the best
way forward.

* It is entirely acceptable to discuss process issues in public. Examples
of topics that are suitable are: "the hard freeze did not work", "the hard
freeze did work well", "we should have longer freeze exception windows",
"we should have no freeze exceptions at all", "we should open master to
contributions after RC2", "we should rename freeze to something else as
it encourages misunderstandings", "we should have a shorter release cycle",
etc. - these are just examples.

* If you agree with an issue/solution you can reply to it with a comment
or a "+1" in the normal way

* If you disagree or have some concerns about what has been proposed,
feel free to reply. You can use the "-1 comment" format.

Please be MINDFUL when you reply to one of the ideas and suggestions that
were raised. The ground rule about NOT becoming personal also applies to
replies.

== Formatting of Mails ==

* Please only cover only *one* topic per feedback e-mail. In other
words please do not mix topics. This makes it easier to collate feedback
and for everyone to follow the threads.

* Use the [xen 4.6 retrospective] tag in the subject line to xen-devel@

* You can use the [good] or [bad] tag in the subject line to xen-devel@

* Make sure you CC community.manager@xenproject.org<mailto:community.manager@xenproject.org>

* Use [private] and the tags above, and do NOT send the mail to xen-devel@
if this is a question/issue ONLY directed at me. This will help me manage
my inbox

* Use [urgent], if your issue should be discussed at the upcoming
Developer Summit Aug 18-19, OR if it is something which affects the current
release cycle (e.g. "we should open master to contributions after RC2")

* Use a descriptive subject line describing the issue/feedback, e.g.
"release cadence was too long", ...

* If you can, reply to this e-mail thread and change the title as
described above. That way, most of the feedback will be nicely threaded

== Content ==
Ideally, you would follow the following template

---
  Subject: [xen 4.6 retrospective] ...

  = Issue / Observation =
  Describe what you have noticed
  Described whether this was positive or negative and how
  ...

  = Possible Solution / Improvement =
  Describe how you think we could improve the situation
  Explain why your proposal would make a positive impact
  ...
---

== Closing the Discussion ==

I will monitor the post-mortem and I will act as facilitator: for example
I may call out improper behaviour in public or private to keep the
retrospective focussed.

At some point after August 28th, we will look at the issues and
suggestions raised and see which ones we can address and how. For the
ones we can address, we will condense feedback into concrete proposals
which we may put forward for voting (unless everyone agrees that something
is a great idea, or strictly speaking no vote is necessary).

Best Regards
Lars

[-- Attachment #1.2: Type: text/html, Size: 27270 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel