[Buildroot] autobuild statistics graph

[Buildroot] autobuild statistics graph

[Thomas De Schampheleire]

Hi Thomas,

The autobuild statistics shows a graph with the number of
failed/successful/total builds at:
http://autobuild.buildroot.org/stats.php

However, given that the total number of builds executed each day is
varying, it is difficult to visually draw conclusions from the graph.
I would like to propose adding another graph (or replacing the current
one) that plots the percentages of successful/failed builds. Hence,
the reference will always be 100%, and a 'success' line going up means
improvement.
In fact, in this case it makes little sense to plot both the success
and failure lines, so we can choose either one.

What do you think?
Is this difficult to realize?
If you give me some source code of how the graph is generated, I'm
willing to look into this.

Best regards,
Thomas

[Buildroot] autobuild statistics graph

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Wed, 28 Aug 2013 08:46:50 +0200, Thomas De Schampheleire wrote:

> The autobuild statistics shows a graph with the number of
> failed/successful/total builds at:
> http://autobuild.buildroot.org/stats.php
> 
> However, given that the total number of builds executed each day is
> varying, it is difficult to visually draw conclusions from the graph.

Yeah, now you realize that I was sleeping during all those statistics
courses at the university :)

> I would like to propose adding another graph (or replacing the current
> one) that plots the percentages of successful/failed builds. Hence,
> the reference will always be 100%, and a 'success' line going up means
> improvement.
> In fact, in this case it makes little sense to plot both the success
> and failure lines, so we can choose either one.
> 
> What do you think?

Sounds sane. How would you compute what the reference is? Look at the
build results of the last 30 days (i.e the ones that appear on the
plot) ?

> Is this difficult to realize?
> If you give me some source code of how the graph is generated, I'm
> willing to look into this.

The code that generates the graph is available at
http://git.buildroot.net/buildroot-test/tree/web/graph.php. Warning:
ugly PHP code inside!

Thanks!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] autobuild statistics graph

[Thomas De Schampheleire]

Hi Thomas,

On Wed, Aug 28, 2013 at 9:10 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Thomas De Schampheleire,
>
> On Wed, 28 Aug 2013 08:46:50 +0200, Thomas De Schampheleire wrote:
>
>> The autobuild statistics shows a graph with the number of
>> failed/successful/total builds at:
>> http://autobuild.buildroot.org/stats.php
>>
>> However, given that the total number of builds executed each day is
>> varying, it is difficult to visually draw conclusions from the graph.
>
> Yeah, now you realize that I was sleeping during all those statistics
> courses at the university :)
>
>> I would like to propose adding another graph (or replacing the current
>> one) that plots the percentages of successful/failed builds. Hence,
>> the reference will always be 100%, and a 'success' line going up means
>> improvement.
>> In fact, in this case it makes little sense to plot both the success
>> and failure lines, so we can choose either one.
>>
>> What do you think?
>
> Sounds sane. How would you compute what the reference is? Look at the
> build results of the last 30 days (i.e the ones that appear on the
> plot) ?

I was actually thinking of using the current calculated percentages.
So say on day 1 there are 100 builds, with 70 successful, and on day 2
there have been 50 builds with 25 successful, then I would plot 70%
for the first day and 50% for the second.
Given the variation in the configurations that are being built, a
rising success line from one day to the next cannot be interpreted as
an absolute reduction of bugs (could have been 'chance' because the
bad configurations weren't built on the second day) but if you look at
the trend it should tell us something.

There probably are more sophisticated ways to plot this data, but I'm
not sure it's needed for our purposes.

>
>> Is this difficult to realize?
>> If you give me some source code of how the graph is generated, I'm
>> willing to look into this.
>
> The code that generates the graph is available at
> http://git.buildroot.net/buildroot-test/tree/web/graph.php. Warning:
> ugly PHP code inside!

Here is some untested code that I think should do the trick (I don't
have a PHP environment at hand here):

diff --git a/web/graph.php b/web/graph.php
index c37bc88..7ac37d8 100644
--- a/web/graph.php
+++ b/web/graph.php
@@ -26,18 +26,16 @@ $total_data = array();

 while($current = mysql_fetch_object($ret)) {
   array_push($dates_data, $current->day);
-  array_push($success_data, $current->success);
-  array_push($failures_data, $current->failures);
-  array_push($timeouts_data, $current->timeouts);
-  array_push($total_data, $current->total);
+  array_push($success_data, $current->success * 100 / $current->total);
+  array_push($failures_data, $current->failures * 100 / $current->total);
+  array_push($timeouts_data, $current->timeouts * 100 / $current->total);
 }

 /* Add data in your dataset */
-$myData->addPoints($success_data, "success");
-$myData->addPoints($failures_data, "failure");
-$myData->addPoints($timeouts_data, "timeout");
-$myData->addPoints($total_data, "total");
-$myData->setAxisName(0,"Number of builds");
+$myData->addPoints($success_data, "success %");
+$myData->addPoints($failures_data, "failure %");
+$myData->addPoints($timeouts_data, "timeout %");
+$myData->setAxisName(0,"Percentage of builds");

 $myData->addPoints($dates_data, "Labels");
 $myData->setSerieDescription("Labels","Dates");
@@ -59,4 +57,4 @@
$myPicture->drawLegend(20,20,array("Style"=>LEGEND_NOBORDER,"Mode"=>LEGEND_HORIZ
 $myPicture->drawLineChart();

 $myPicture->Stroke();
-?>
\ No newline at end of file
+?>

---

Best regards,
Thomas

[Buildroot] autobuild statistics graph

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Wed, 28 Aug 2013 09:42:45 +0200, Thomas De Schampheleire wrote:

> > Sounds sane. How would you compute what the reference is? Look at the
> > build results of the last 30 days (i.e the ones that appear on the
> > plot) ?
> 
> I was actually thinking of using the current calculated percentages.
> So say on day 1 there are 100 builds, with 70 successful, and on day 2
> there have been 50 builds with 25 successful, then I would plot 70%
> for the first day and 50% for the second.

Ah right, makes sense. The percentages are in fact already shown in the
table above.

> Given the variation in the configurations that are being built, a
> rising success line from one day to the next cannot be interpreted as
> an absolute reduction of bugs (could have been 'chance' because the
> bad configurations weren't built on the second day) but if you look at
> the trend it should tell us something.

Right.

> Here is some untested code that I think should do the trick (I don't
> have a PHP environment at hand here):

Your patch is now live at http://autobuild.buildroot.org/stats.php and
it seems to work. If you confirm, then I'll commit and push your patch,
with you as the author.

Thanks!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] autobuild statistics graph

[Thomas De Schampheleire]

Hi Thomas,

On Wed, Aug 28, 2013 at 9:48 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Thomas De Schampheleire,
>
> On Wed, 28 Aug 2013 09:42:45 +0200, Thomas De Schampheleire wrote:
>
>> > Sounds sane. How would you compute what the reference is? Look at the
>> > build results of the last 30 days (i.e the ones that appear on the
>> > plot) ?
>>
>> I was actually thinking of using the current calculated percentages.
>> So say on day 1 there are 100 builds, with 70 successful, and on day 2
>> there have been 50 builds with 25 successful, then I would plot 70%
>> for the first day and 50% for the second.
>
> Ah right, makes sense. The percentages are in fact already shown in the
> table above.
>
>> Given the variation in the configurations that are being built, a
>> rising success line from one day to the next cannot be interpreted as
>> an absolute reduction of bugs (could have been 'chance' because the
>> bad configurations weren't built on the second day) but if you look at
>> the trend it should tell us something.
>
> Right.
>
>> Here is some untested code that I think should do the trick (I don't
>> have a PHP environment at hand here):
>
> Your patch is now live at http://autobuild.buildroot.org/stats.php and
> it seems to work. If you confirm, then I'll commit and push your patch,
> with you as the author.

That looks good, yes, thanks for testing so quickly.
One consideration: although the table doesn't have to be larger, the
30 days for the graph is somewhat limited. If we want to see a trend,
then I think a few months can be more interesting. So what about
changing the SQL query to, say, 180 days (6 months) ?

Best regards,
Thomas

[Buildroot] autobuild statistics graph

[Thomas De Schampheleire]

Hi Thomas,

On Wed, Aug 28, 2013 at 10:02 AM, Thomas De Schampheleire
<patrickdepinguin+buildroot@gmail.com> wrote:
> Hi Thomas,
>
> On Wed, Aug 28, 2013 at 9:48 AM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
>> Dear Thomas De Schampheleire,
>>
>> On Wed, 28 Aug 2013 09:42:45 +0200, Thomas De Schampheleire wrote:
>>
>>> > Sounds sane. How would you compute what the reference is? Look at the
>>> > build results of the last 30 days (i.e the ones that appear on the
>>> > plot) ?
>>>
>>> I was actually thinking of using the current calculated percentages.
>>> So say on day 1 there are 100 builds, with 70 successful, and on day 2
>>> there have been 50 builds with 25 successful, then I would plot 70%
>>> for the first day and 50% for the second.
>>
>> Ah right, makes sense. The percentages are in fact already shown in the
>> table above.
>>
>>> Given the variation in the configurations that are being built, a
>>> rising success line from one day to the next cannot be interpreted as
>>> an absolute reduction of bugs (could have been 'chance' because the
>>> bad configurations weren't built on the second day) but if you look at
>>> the trend it should tell us something.
>>
>> Right.
>>
>>> Here is some untested code that I think should do the trick (I don't
>>> have a PHP environment at hand here):
>>
>> Your patch is now live at http://autobuild.buildroot.org/stats.php and
>> it seems to work. If you confirm, then I'll commit and push your patch,
>> with you as the author.
>
> That looks good, yes, thanks for testing so quickly.
> One consideration: although the table doesn't have to be larger, the
> 30 days for the graph is somewhat limited. If we want to see a trend,
> then I think a few months can be more interesting. So what about
> changing the SQL query to, say, 180 days (6 months) ?

It seems you already made this change, thanks.

The x axis is now a bit crowded, below untested patch should make it a
bit more readable, could you check?

diff --git a/web/graph.php b/web/graph.php
index 7ac37d8..5585ee5 100644
--- a/web/graph.php
+++ b/web/graph.php
@@ -50,7 +50,7 @@
$myPicture->setFontProperties(array("FontName"=>"../externals/pchart/fonts/verda
 /* Define the boundaries of the graph area */
 $myPicture->setGraphArea(70,50,650,400);

-$myPicture->drawScale(array("LabelRotation" => 90));
+$myPicture->drawScale(array("LabelRotation" => 90, "LabelSkip" => 10));

 $myPicture->drawLegend(20,20,array("Style"=>LEGEND_NOBORDER,"Mode"=>LEGEND_HORIZONTAL));


Best regards,
Thomas

[Buildroot] autobuild statistics graph

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Thu, 29 Aug 2013 09:40:41 +0200, Thomas De Schampheleire wrote:

> > That looks good, yes, thanks for testing so quickly.
> > One consideration: although the table doesn't have to be larger, the
> > 30 days for the graph is somewhat limited. If we want to see a trend,
> > then I think a few months can be more interesting. So what about
> > changing the SQL query to, say, 180 days (6 months) ?
> 
> It seems you already made this change, thanks.
> 
> The x axis is now a bit crowded, below untested patch should make it a
> bit more readable, could you check?

Yeah, I did some experiment, but as you say the X axis was crowded, and
I didn't had the time to investigate.

> diff --git a/web/graph.php b/web/graph.php
> index 7ac37d8..5585ee5 100644
> --- a/web/graph.php
> +++ b/web/graph.php
> @@ -50,7 +50,7 @@
> $myPicture->setFontProperties(array("FontName"=>"../externals/pchart/fonts/verda
>  /* Define the boundaries of the graph area */
>  $myPicture->setGraphArea(70,50,650,400);
> 
> -$myPicture->drawScale(array("LabelRotation" => 90));
> +$myPicture->drawScale(array("LabelRotation" => 90, "LabelSkip" => 10));
> 
>  $myPicture->drawLegend(20,20,array("Style"=>LEGEND_NOBORDER,"Mode"=>LEGEND_HORIZONTAL));

I'll definitely try this out, thanks!

Another great thing would be to draw some thin vertical lines at the
dates of the various releases (-rc1, -rc2, final and so on), so we can
see if the rate of failures/success correlate with releases or not.
It'd be fun, but I'm not sure it's really worth the effort because
besides the releases themselves, another major thing causes big
variations in the rate of success/failure: when I had a new toolchain
for a new architecture, or a new configuration type (eg. static
builds). I believe those changes in the configuration of the
autobuilders are causing more variations in the results than the
release cycles themselves.

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] autobuild statistics graph

[Thomas De Schampheleire]

Hi Thomas,

On Thu, Aug 29, 2013 at 10:15 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:

>
> Another great thing would be to draw some thin vertical lines at the
> dates of the various releases (-rc1, -rc2, final and so on), so we can
> see if the rate of failures/success correlate with releases or not.
> It'd be fun, but I'm not sure it's really worth the effort because
> besides the releases themselves, another major thing causes big
> variations in the rate of success/failure: when I had a new toolchain
> for a new architecture, or a new configuration type (eg. static
> builds). I believe those changes in the configuration of the
> autobuilders are causing more variations in the results than the
> release cycles themselves.
>

Yes, I also considered that having the releases indicated in the graph
would be interesting. In fact, if we'd have these big moments shown in
the graph, the actual dates wouldn't really matter anymore and could
be hidden completely IMO.

If you still have the info about when some of the big changes (like
new toolchain etc.) was done, we can definitely put that alongside the
release milestones.

How to best store the milestones (in the database or in the graph.php
file itself), I'm not sure. They'll probably need to be manually
updated, but that could be done both in the database as in graph.php.
How to show these milestones in the graph is also not entirely clear.
I can't find a way to pass some custom x values to  pChart. One
solution is to manipulate the date array from php before passing it to
pChart. This way, if we have a map between date (as it appears in the
database) and event, we can replace the dates by the events, and make
all other strings empty. This should give the desired result I think.

[Buildroot] autobuild statistics graph

[Arnout Vandecappelle]

On 08/29/13 10:40, Thomas De Schampheleire wrote:
> Hi Thomas,
>
> On Thu, Aug 29, 2013 at 10:15 AM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
>
>>
>> Another great thing would be to draw some thin vertical lines at the
>> dates of the various releases (-rc1, -rc2, final and so on), so we can
>> see if the rate of failures/success correlate with releases or not.
>> It'd be fun, but I'm not sure it's really worth the effort because
>> besides the releases themselves, another major thing causes big
>> variations in the rate of success/failure: when I had a new toolchain
>> for a new architecture, or a new configuration type (eg. static
>> builds). I believe those changes in the configuration of the
>> autobuilders are causing more variations in the results than the
>> release cycles themselves.
>>
>
> Yes, I also considered that having the releases indicated in the graph
> would be interesting. In fact, if we'd have these big moments shown in
> the graph, the actual dates wouldn't really matter anymore and could
> be hidden completely IMO.
>
> If you still have the info about when some of the big changes (like
> new toolchain etc.) was done, we can definitely put that alongside the
> release milestones.
>
> How to best store the milestones (in the database or in the graph.php
> file itself), I'm not sure. They'll probably need to be manually
> updated, but that could be done both in the database as in graph.php.
> How to show these milestones in the graph is also not entirely clear.
> I can't find a way to pass some custom x values to  pChart. One
> solution is to manipulate the date array from php before passing it to
> pChart. This way, if we have a map between date (as it appears in the
> database) and event, we can replace the dates by the events, and make
> all other strings empty. This should give the desired result I think.

  Ideally the horizontal axis should be based on git commit IDs instead 
of build date, and the labels could be the tags.  And you could manually 
add tags to the clone from which the commit IDs are extracted to indicate 
those interesting points where toolchains are added etc. But that would 
be a bit too much effort I guess :-)

  Regards,
  Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] autobuild statistics graph

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Thu, 29 Aug 2013 17:59:09 +0200, Arnout Vandecappelle wrote:

> > Yes, I also considered that having the releases indicated in the graph
> > would be interesting. In fact, if we'd have these big moments shown in
> > the graph, the actual dates wouldn't really matter anymore and could
> > be hidden completely IMO.
> >
> > If you still have the info about when some of the big changes (like
> > new toolchain etc.) was done, we can definitely put that alongside the
> > release milestones.
> >
> > How to best store the milestones (in the database or in the graph.php
> > file itself), I'm not sure. They'll probably need to be manually
> > updated, but that could be done both in the database as in graph.php.
> > How to show these milestones in the graph is also not entirely clear.
> > I can't find a way to pass some custom x values to  pChart. One
> > solution is to manipulate the date array from php before passing it to
> > pChart. This way, if we have a map between date (as it appears in the
> > database) and event, we can replace the dates by the events, and make
> > all other strings empty. This should give the desired result I think.
> 
>   Ideally the horizontal axis should be based on git commit IDs instead 
> of build date, and the labels could be the tags.  And you could manually 
> add tags to the clone from which the commit IDs are extracted to indicate 
> those interesting points where toolchains are added etc. But that would 
> be a bit too much effort I guess :-)

Hum, yeah, I don't mind spending a bit of time on writing/maintaining a
few tools around QA for Buildroot, but I'd like to also keep some time
for real Buildroot development, and not be completely full with stupid
PHP development tasks :-)

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] autobuild statistics graph

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Thu, 29 Aug 2013 09:40:41 +0200, Thomas De Schampheleire wrote:

> -$myPicture->drawScale(array("LabelRotation" => 90));
> +$myPicture->drawScale(array("LabelRotation" => 90, "LabelSkip" =>
> 10));

Thanks, this works, I've committed this and deployed on autobuild.b.o.

See
http://git.buildroot.net/buildroot-test/commit/?id=51317ea5ce4ea0244744ff284ecfb98a2743520d.

Thanks!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] autobuild statistics graph

[Thomas De Schampheleire]

Hi Thomas,

On Sun, Sep 8, 2013 at 3:57 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Thomas De Schampheleire,
>
> On Thu, 29 Aug 2013 09:40:41 +0200, Thomas De Schampheleire wrote:
>
>> -$myPicture->drawScale(array("LabelRotation" => 90));
>> +$myPicture->drawScale(array("LabelRotation" => 90, "LabelSkip" =>
>> 10));
>
> Thanks, this works, I've committed this and deployed on autobuild.b.o.
>
> See
> http://git.buildroot.net/buildroot-test/commit/?id=51317ea5ce4ea0244744ff284ecfb98a2743520d.

Thanks, that looks much nicer than before!

What about adding a link to this page from the daily e-mail? Something like:

diff --git a/utils/mail.php b/utils/mail.php
index 93cc2e8..c7cfaa0 100644
--- a/utils/mail.php
+++ b/utils/mail.php
@@ -71,6 +71,8 @@ $contents .= sprintf("%15s : %-3d\n", "failures", $failures);
 $contents .= sprintf("%15s : %-3d\n", "timeouts", $timeouts);
 $contents .= sprintf("%15s : %-3d\n", "TOTAL", $total);

+$contents .= sprintf("\nHistorical overview:
http://autobuild.buildroot.org/stats.php\n");
+
 $sql = "select reason,count(id) as reason_count from results " .
   "where date(builddate) = date(now() - interval 1 day) and status != 0 " .
   $condition .



Best regards,
Thomas