Yocto weekly bug trend charts -- WW11

Yocto weekly bug trend charts -- WW11

[Xu, Jiajun]

Hi all,
	The overall open bug trend increased a lot in last week. The new submitted vs. fixed bug number is 64 vs. 52. Some fixed bugs are enhancement bug, which are not calculated into WDD data. WDD number and Open Bug number are 938 and 188. Bug status of WW11 could be found on https://wiki.yoctoproject.org/wiki/Yocto_Bug_Trend.

Best Regards,
Jiajun

Re: Yocto weekly bug trend charts -- WW11

[Richard Purdie]

Hi Jiajun,

On Mon, 2012-03-19 at 08:49 +0000, Xu, Jiajun wrote:
> 	The overall open bug trend increased a lot in last week. The new
> submitted vs. fixed bug number is 64 vs. 52. Some fixed bugs are
> enhancement bug, which are not calculated into WDD data. WDD number
> and Open Bug number are 938 and 188. Bug status of WW11 could be found
> on https://wiki.yoctoproject.org/wiki/Yocto_Bug_Trend.

I agree that there were 64 vs 52 bugs last week which is a change of 12
bugs. The top two charts show increases of 50 and 30 bugs though so the
numbers don't seem to add up for me :/. I appreciate the bugzilla update
upset the accounting but it looks like some of the historical data is
inaccurate somewhere :(

Cheers,

Richard

Re: Yocto weekly bug trend charts -- WW11

[Xu, Jiajun]

> Hi Jiajun,
> 
> On Mon, 2012-03-19 at 08:49 +0000, Xu, Jiajun wrote:
>> 	The overall open bug trend increased a lot in last week. The new
>> submitted vs. fixed bug number is 64 vs. 52. Some fixed bugs are
>> enhancement bug, which are not calculated into WDD data. WDD number
>> and Open Bug number are 938 and 188. Bug status of WW11 could be
>> found on https://wiki.yoctoproject.org/wiki/Yocto_Bug_Trend.
> 
> I agree that there were 64 vs 52 bugs last week which is a change of 12 bugs.
> The top two charts show increases of 50 and 30 bugs though so the
> numbers don't seem to add up for me :/. I appreciate the bugzilla
> update upset the accounting but it looks like some of the historical
> data is inaccurate somewhere :(

Good catch. :)
For the first picture, it is the trend of open bugs by importance(High/Medium/Low/Undecided) with all bugs included. For the second picture, it is the trend of open bugs by severity(minor/normal/major/critical) without enhancement bugs. The first picture is to give us a whole picture of all open bugs. The second one is to give us a true picture of the real bugs - without bugs created for feature enabling.
BTW, when I check the open bug data, I find that a new variable is introduced with new bugzilla for field "severity" - janitors. We do not include the bugs marked as janitors in "Yocto Weekly Open bug Trend(Severity)". My thinking is that janitors is similar with enhancement. We could treat it as enhancement(with weight value "0"), or if we think it should be included into WDD, we could set a weight value for it. How do you think of it?
Fortunately, we only have 4 bugs marked as janitors, and they are all new reported last week. We could simply update the bug trend once we make the decision.

> 
> Cheers,
> 
> Richard

Best Regards,
Jiajun

Re: Yocto weekly bug trend charts -- WW11

[Stewart, David C]

>From: yocto-bounces@yoctoproject.org [mailto:yocto-
>bounces@yoctoproject.org] On Behalf Of Xu, Jiajun
>Sent: Monday, March 19, 2012 8:33 AM
>
>> Hi Jiajun,
>>
>> On Mon, 2012-03-19 at 08:49 +0000, Xu, Jiajun wrote:
>>> 	The overall open bug trend increased a lot in last week. The new
>>> submitted vs. fixed bug number is 64 vs. 52. Some fixed bugs are
>>> enhancement bug, which are not calculated into WDD data. WDD number
>>> and Open Bug number are 938 and 188. Bug status of WW11 could be
>>> found on https://wiki.yoctoproject.org/wiki/Yocto_Bug_Trend.
>>
>> I agree that there were 64 vs 52 bugs last week which is a change of 12 bugs.
>> The top two charts show increases of 50 and 30 bugs though so the
>> numbers don't seem to add up for me :/. I appreciate the bugzilla
>> update upset the accounting but it looks like some of the historical
>> data is inaccurate somewhere :(
>
>Good catch. :)
>For the first picture, it is the trend of open bugs by
>importance(High/Medium/Low/Undecided) with all bugs included. For the
>second picture, it is the trend of open bugs by
>severity(minor/normal/major/critical) without enhancement bugs. The first
>picture is to give us a whole picture of all open bugs. The second one is to give
>us a true picture of the real bugs - without bugs created for feature enabling.

We need to get this data really clean because we make a lot of decisions based on it. Please work with RP to address this

>BTW, when I check the open bug data, I find that a new variable is introduced
>with new bugzilla for field "severity" - janitors. We do not include the bugs
>marked as janitors in "Yocto Weekly Open bug Trend(Severity)". My thinking
>is that janitors is similar with enhancement. We could treat it as
>enhancement(with weight value "0"), or if we think it should be included into
>WDD, we could set a weight value for it. How do you think of it?
>Fortunately, we only have 4 bugs marked as janitors, and they are all new
>reported last week. We could simply update the bug trend once we make the
>decision.

I suggest we make the janitor severity the same as low in the WDD calculation.

>>
>> Cheers,
>>
>> Richard
>
>Best Regards,
>Jiajun
>
>
>_______________________________________________
>yocto mailing list
>yocto@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/yocto

Re: Yocto weekly bug trend charts -- WW11

[Darren Hart]

On 03/19/2012 09:18 AM, Stewart, David C wrote:
>> From: yocto-bounces@yoctoproject.org [mailto:yocto-
>> bounces@yoctoproject.org] On Behalf Of Xu, Jiajun
>> Sent: Monday, March 19, 2012 8:33 AM

>> BTW, when I check the open bug data, I find that a new variable is introduced
>> with new bugzilla for field "severity" - janitors. We do not include the bugs
>> marked as janitors in "Yocto Weekly Open bug Trend(Severity)". My thinking
>> is that janitors is similar with enhancement. We could treat it as
>> enhancement(with weight value "0"), or if we think it should be included into
>> WDD, we could set a weight value for it. How do you think of it?
>> Fortunately, we only have 4 bugs marked as janitors, and they are all new
>> reported last week. We could simply update the bug trend once we make the
>> decision.
> 
> I suggest we make the janitor severity the same as low in the WDD calculation.
> 

I don't know what "low" would map to. The key says:

"The weight we use for each severity: Critical:10, Major:7, Normal:5,
Minor:3, Enhancement:0 "

In my view, Janitors should be marked the same as Enhancement, 0. If a
bug is important enough to track and impact release, it shouldn't be a
janitors bug, which by definition are intended to sit in a pool for
new-comers and irregular contributors to pick up.

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: Yocto weekly bug trend charts -- WW11

[Richard Purdie]

On Mon, 2012-03-19 at 10:13 -0700, Darren Hart wrote:
> On 03/19/2012 09:18 AM, Stewart, David C wrote:
> >> From: yocto-bounces@yoctoproject.org [mailto:yocto-
> >> bounces@yoctoproject.org] On Behalf Of Xu, Jiajun
> >> Sent: Monday, March 19, 2012 8:33 AM
> 
> >> BTW, when I check the open bug data, I find that a new variable is introduced
> >> with new bugzilla for field "severity" - janitors. We do not include the bugs
> >> marked as janitors in "Yocto Weekly Open bug Trend(Severity)". My thinking
> >> is that janitors is similar with enhancement. We could treat it as
> >> enhancement(with weight value "0"), or if we think it should be included into
> >> WDD, we could set a weight value for it. How do you think of it?
> >> Fortunately, we only have 4 bugs marked as janitors, and they are all new
> >> reported last week. We could simply update the bug trend once we make the
> >> decision.
> > 
> > I suggest we make the janitor severity the same as low in the WDD calculation.
> > 
> 
> I don't know what "low" would map to. The key says:
> 
> "The weight we use for each severity: Critical:10, Major:7, Normal:5,
> Minor:3, Enhancement:0 "
> 
> In my view, Janitors should be marked the same as Enhancement, 0. If a
> bug is important enough to track and impact release, it shouldn't be a
> janitors bug, which by definition are intended to sit in a pool for
> new-comers and irregular contributors to pick up.

I did talk briefly with Dave earlier and it was hard to decide whether
these were minor or enhancement in nature. Enhancements are about adding
new functionality, bugs of whatever severity are about issues in
existing features. Janitor class issues could be in either category so
you can argue this both ways from that perspective.

Did we end up deciding to have a janitor user to assign these too in the
end or are they going to remain with various people until assigned? If
the latter, I'd hate to see the WDD being skewed against people due to
high numbers of janitor bugs so that might sway my argument.

Cheers,

Richard

Re: Yocto weekly bug trend charts -- WW11

[Darren Hart]

On 03/19/2012 01:18 PM, Richard Purdie wrote:
> On Mon, 2012-03-19 at 10:13 -0700, Darren Hart wrote:
>> On 03/19/2012 09:18 AM, Stewart, David C wrote:
>>>> From: yocto-bounces@yoctoproject.org [mailto:yocto-
>>>> bounces@yoctoproject.org] On Behalf Of Xu, Jiajun
>>>> Sent: Monday, March 19, 2012 8:33 AM
>>
>>>> BTW, when I check the open bug data, I find that a new variable is introduced
>>>> with new bugzilla for field "severity" - janitors. We do not include the bugs
>>>> marked as janitors in "Yocto Weekly Open bug Trend(Severity)". My thinking
>>>> is that janitors is similar with enhancement. We could treat it as
>>>> enhancement(with weight value "0"), or if we think it should be included into
>>>> WDD, we could set a weight value for it. How do you think of it?
>>>> Fortunately, we only have 4 bugs marked as janitors, and they are all new
>>>> reported last week. We could simply update the bug trend once we make the
>>>> decision.
>>>
>>> I suggest we make the janitor severity the same as low in the WDD calculation.
>>>
>>
>> I don't know what "low" would map to. The key says:
>>
>> "The weight we use for each severity: Critical:10, Major:7, Normal:5,
>> Minor:3, Enhancement:0 "
>>
>> In my view, Janitors should be marked the same as Enhancement, 0. If a
>> bug is important enough to track and impact release, it shouldn't be a
>> janitors bug, which by definition are intended to sit in a pool for
>> new-comers and irregular contributors to pick up.
> 
> I did talk briefly with Dave earlier and it was hard to decide whether
> these were minor or enhancement in nature. Enhancements are about adding
> new functionality, bugs of whatever severity are about issues in
> existing features. Janitor class issues could be in either category so
> you can argue this both ways from that perspective.
> 
> Did we end up deciding to have a janitor user to assign these too in the
> end or are they going to remain with various people until assigned? If
> the latter, I'd hate to see the WDD being skewed against people due to
> high numbers of janitor bugs so that might sway my argument.

Last I spoke with Michael (added to CC), the plan was to assign these to
a default "Yocto Janitor" id. Unfortunately, we don't have a way to set
this up for automatic assignment like we do for "component owner", as
Janitor bugs are currently identified by severity.

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: Yocto weekly bug trend charts -- WW11

[Saul Wold]

On 03/19/2012 01:22 PM, Darren Hart wrote:
> On 03/19/2012 01:18 PM, Richard Purdie wrote:
>> On Mon, 2012-03-19 at 10:13 -0700, Darren Hart wrote:
>>> On 03/19/2012 09:18 AM, Stewart, David C wrote:
>>>>> From: yocto-bounces@yoctoproject.org [mailto:yocto-
>>>>> bounces@yoctoproject.org] On Behalf Of Xu, Jiajun
>>>>> Sent: Monday, March 19, 2012 8:33 AM
>>>
>>>>> BTW, when I check the open bug data, I find that a new variable is introduced
>>>>> with new bugzilla for field "severity" - janitors. We do not include the bugs
>>>>> marked as janitors in "Yocto Weekly Open bug Trend(Severity)". My thinking
>>>>> is that janitors is similar with enhancement. We could treat it as
>>>>> enhancement(with weight value "0"), or if we think it should be included into
>>>>> WDD, we could set a weight value for it. How do you think of it?
>>>>> Fortunately, we only have 4 bugs marked as janitors, and they are all new
>>>>> reported last week. We could simply update the bug trend once we make the
>>>>> decision.
>>>>
>>>> I suggest we make the janitor severity the same as low in the WDD calculation.
>>>>
>>>
>>> I don't know what "low" would map to. The key says:
>>>
>>> "The weight we use for each severity: Critical:10, Major:7, Normal:5,
>>> Minor:3, Enhancement:0 "
>>>
>>> In my view, Janitors should be marked the same as Enhancement, 0. If a
>>> bug is important enough to track and impact release, it shouldn't be a
>>> janitors bug, which by definition are intended to sit in a pool for
>>> new-comers and irregular contributors to pick up.
>>
>> I did talk briefly with Dave earlier and it was hard to decide whether
>> these were minor or enhancement in nature. Enhancements are about adding
>> new functionality, bugs of whatever severity are about issues in
>> existing features. Janitor class issues could be in either category so
>> you can argue this both ways from that perspective.
>>
>> Did we end up deciding to have a janitor user to assign these too in the
>> end or are they going to remain with various people until assigned? If
>> the latter, I'd hate to see the WDD being skewed against people due to
>> high numbers of janitor bugs so that might sway my argument.
>
> Last I spoke with Michael (added to CC), the plan was to assign these to
> a default "Yocto Janitor" id. Unfortunately, we don't have a way to set
> this up for automatic assignment like we do for "component owner", as
> Janitor bugs are currently identified by severity.
>
We can just take care of this during bug creation or triage manually.

Sau!

Re: Yocto weekly bug trend charts -- WW11

[Richard Purdie]

On Mon, 2012-03-19 at 15:32 +0000, Xu, Jiajun wrote:
> > Hi Jiajun,
> > 
> > On Mon, 2012-03-19 at 08:49 +0000, Xu, Jiajun wrote:
> >> 	The overall open bug trend increased a lot in last week. The new
> >> submitted vs. fixed bug number is 64 vs. 52. Some fixed bugs are
> >> enhancement bug, which are not calculated into WDD data. WDD number
> >> and Open Bug number are 938 and 188. Bug status of WW11 could be
> >> found on https://wiki.yoctoproject.org/wiki/Yocto_Bug_Trend.
> > 
> > I agree that there were 64 vs 52 bugs last week which is a change of 12 bugs.
> > The top two charts show increases of 50 and 30 bugs though so the
> > numbers don't seem to add up for me :/. I appreciate the bugzilla
> > update upset the accounting but it looks like some of the historical
> > data is inaccurate somewhere :(
> 
> Good catch. :)
> For the first picture, it is the trend of open bugs by
> importance(High/Medium/Low/Undecided) with all bugs included. For the
> second picture, it is the trend of open bugs by
> severity(minor/normal/major/critical) without enhancement bugs. The
> first picture is to give us a whole picture of all open bugs. The
> second one is to give us a true picture of the real bugs - without
> bugs created for feature enabling.

Right, I like the trend charts and the information they generate is
extremely useful but I do worry that if the information is inaccurate,
we can make bad decisions based upon it. What really worries me now is
I'm in a position where I don't trust the data and it doesn't make
sense?

Do we have any way to correct the data? If I understand how you
currently generate these, I suspect it might be hard since you rely on
output from bugzilla whine emails?

Would it be possible to query the bugzilla data differently to be able
to completely build the chart profiles directly from absolute data
rather than indirectly though incremental data?

I'm wondering about the options of:

a) Any other API bugzilla exposes for queries?
b) Run scripts against bugzilla web API to obtain the necessary data?
c) Query the bugzilla database directly

I've cc'd Michael since he might be able to help us get access to the
data. Just for Michael's reference, what we need is to be able to figure
out:

a) What was the open bug count at time X of undecided, low, medium and
high bugs?
b) What was the open bug count at time X of minor, normal, major and
critical bugs?
c) Between times W and X, what number of bugs were submitted of type
undecided, low, medium and high?
d) Between times W and X, what number of bugs were resolved of type
undecided, low, medium and high?

Where W-X is 7 days and X is in general the end of a week so we can
generate the charts shown on:

https://wiki.yoctoproject.org/wiki/Yocto_Bug_Trend

I can think of ways I could script this although it might look like a
DoS attack against the server :) I'm open to ideas on how we could fix
this. Bonus marks for automatically generating the charts (which
thinking about it, I have done before and wasn't that hard)!

> BTW, when I check the open bug data, I find that a new variable is
> introduced with new bugzilla for field "severity" - janitors. We do
> not include the bugs marked as janitors in "Yocto Weekly Open bug
> Trend(Severity)". My thinking is that janitors is similar with
> enhancement. We could treat it as enhancement(with weight value "0"),
> or if we think it should be included into WDD, we could set a weight
> value for it. How do you think of it?
> Fortunately, we only have 4 bugs marked as janitors, and they are all
> new reported last week. We could simply update the bug trend once we
> make the decision.

The janitor changes happened very recently and I think Dave has
commented on this piece.

Cheers,

Richard

Yocto weekly bug trend charts -- WW11

[Xu, Jiajun]

[-- Attachment #1: Type: text/plain, Size: 253 bytes --]

Hi all,
This is latest Yocto bug trend chart for WW11. QA finished RC2 fullpass testing and is doing RC3 testing now. Open bug number increased a lot from 138 to 152. There remains 66 open bugs targeted to be fixed in 1.0 M4.

Best Regards,
Jiajun

[-- Attachment #2: WW11_weighted_defect_density.JPG --]
[-- Type: image/jpeg, Size: 22006 bytes --]

[-- Attachment #3: WW11_submitted_fixed_bug_trend.JPG --]
[-- Type: image/jpeg, Size: 31838 bytes --]

[-- Attachment #4: WW11_open_bug_trend.JPG --]
[-- Type: image/jpeg, Size: 31083 bytes --]