[ANN] FSL Test Plan Announcement 1.5

[ANN] FSL Test Plan Announcement 1.5

[Daiane Angolini]

We'd like to invite everyone to help us to test the Freescale Community 
BSP 1.5 - Dora release.

Please, test your favorite board (or boards), using your favorite image. 
We would like to have at least one test for each 35 boards (supported by 
both meta-fsl-arm or meta-fsl-arm-extra).

We've made a form which guides you through a small set of steps and help 
us to track which boards has been tested. Please, make sure you know the 
packages needed for some tests listed at top of form.

If you have any other question you want to be include in the test form, 
please, let us know.
The '1.5' is being developed in the 'dora' branch, so use this branch 
for the tests (please do not send reports for 'master' branch, any 
error/bug on master should be reported on mailing list).

We'll be receiving and reviewing the test results until Oct 20th 2013. 
However please start the tests as soon as you can, so we can have the 
most solid release as possible. The release will be done together with 
the Yocto release (planned for Oct 18th 2013) and any issues found (if 
applicable) will need to be fixed for the 1.5 point releases (1.5.1, 
1.5.2, ...).

The goal test plan for this release is:

MUST HAVE:
* all 35 boards MUST reach the user space prompt
* all 35 boards MUST have some[1] stability
* all IMX6 boards MUST have X11 up and running accelerated

SHOULD HAVE:
* all boards SHOULD play at least one movie
* all boards SHOULD encode at least one kind of movie
* all boards SHOULD have minimal kernel defconfig enabled by default 
(USB/SDcard/aplay)

BONUS POINT:
* all IMX6 boards SHOULD have GPU and VPU support for both SFP and HFP
* all board SHOULD have overnight stability[2]

We need your help to accomplish our test goals and we don't have enough 
manpower resources to do it all by ourselves. We count on the community 
to help on this.

Please, report any bug/error to mailing list, we will discuss 
case-by-case and create needed bug report.

After the test cycle, we will include the summary of the test result on 
the release notes so we can start to build credibility by showing our 
test results for all the boards, giving to users and partners a central 
place for checking for expected behavior, and known issues, for a 
specific board.

[1] some stability now is defined as being alive during the form tests
[2] overnight stability now is defined by the board being alive after 
more than 10 hours testing memtester 50M (last question from form, Bonus 
Point)

Regards,
Daiane

Re: [ANN] FSL Test Plan Announcement 1.5

[Daiane Angolini]

Sorry about that. I forgot the link

https://docs.google.com/forms/d/1y68-QjiDmv4ekc9Aq8vIOXUePZ5wvOu_5_1RpdfK7zY/viewform

Re: [ANN] FSL Test Plan Announcement 1.5

[Sébastien Taylor]

Will the 3.10.9-1.0.0_alpha changes be making it into this Dora branch?

On 2013-10-03, at 1:05 PM, Daiane Angolini <daiane.angolini@freescale.com> wrote:

> Sorry about that. I forgot the link
> 
> https://docs.google.com/forms/d/1y68-QjiDmv4ekc9Aq8vIOXUePZ5wvOu_5_1RpdfK7zY/viewform
> 
> _______________________________________________
> meta-freescale mailing list
> meta-freescale@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/meta-freescale

Re: [ANN] FSL Test Plan Announcement 1.5

[Otavio Salvador]

Hello Sebastien,

On Thu, Oct 3, 2013 at 4:44 PM, Sébastien Taylor <me@staylor.ca> wrote:
> Will the 3.10.9-1.0.0_alpha changes be making it into this Dora branch?

This is still under discussion. You should have seen Lauren's message
about this topic.

However, we must to understand that the 3.10.9-1.0.0-alpha is just for
MX6 based platforms (just 7 boards) but the Dora release covers all
the 35 boards supported by our BSP.

If it is decided to include the 3.10.9 BSP into Dora, this will only happen if:

* there're backwards compatibility with 3.0.35-4.1.0 kernel;
* there're no regressions regarding current Dora BSP;

So after all this will only happen after the release, and the test
results, so we can check the full status of the boards and do a
conscious decision about this.

Regards,

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [ANN] FSL Test Plan Announcement 1.5

[Alexandre Belloni]

Hi Daiane

I just have some time to test now and I'm not able to submit my results.
The form keeps asking me for the board name although I selected it...

On 03/10/2013 20:48, Daiane Angolini wrote:
> We'd like to invite everyone to help us to test the Freescale
> Community BSP 1.5 - Dora release.
>
> Please, test your favorite board (or boards), using your favorite
> image. We would like to have at least one test for each 35 boards
> (supported by both meta-fsl-arm or meta-fsl-arm-extra).
>
> We've made a form which guides you through a small set of steps and
> help us to track which boards has been tested. Please, make sure you
> know the packages needed for some tests listed at top of form.
>
> If you have any other question you want to be include in the test
> form, please, let us know.
> The '1.5' is being developed in the 'dora' branch, so use this branch
> for the tests (please do not send reports for 'master' branch, any
> error/bug on master should be reported on mailing list).
>
> We'll be receiving and reviewing the test results until Oct 20th 2013.
> However please start the tests as soon as you can, so we can have the
> most solid release as possible. The release will be done together with
> the Yocto release (planned for Oct 18th 2013) and any issues found (if
> applicable) will need to be fixed for the 1.5 point releases (1.5.1,
> 1.5.2, ...).
>
> The goal test plan for this release is:
>
> MUST HAVE:
> * all 35 boards MUST reach the user space prompt
> * all 35 boards MUST have some[1] stability
> * all IMX6 boards MUST have X11 up and running accelerated
>
> SHOULD HAVE:
> * all boards SHOULD play at least one movie
> * all boards SHOULD encode at least one kind of movie
> * all boards SHOULD have minimal kernel defconfig enabled by default
> (USB/SDcard/aplay)
>
> BONUS POINT:
> * all IMX6 boards SHOULD have GPU and VPU support for both SFP and HFP
> * all board SHOULD have overnight stability[2]
>
> We need your help to accomplish our test goals and we don't have
> enough manpower resources to do it all by ourselves. We count on the
> community to help on this.
>
> Please, report any bug/error to mailing list, we will discuss
> case-by-case and create needed bug report.
>
> After the test cycle, we will include the summary of the test result
> on the release notes so we can start to build credibility by showing
> our test results for all the boards, giving to users and partners a
> central place for checking for expected behavior, and known issues,
> for a specific board.
>
> [1] some stability now is defined as being alive during the form tests
> [2] overnight stability now is defined by the board being alive after
> more than 10 hours testing memtester 50M (last question from form,
> Bonus Point)
>
> Regards,
> Daiane
>
> _______________________________________________
> meta-freescale mailing list
> meta-freescale@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/meta-freescale


-- 
Alexandre Belloni, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Re: [ANN] FSL Test Plan Announcement 1.5

[Otavio Salvador]

Hello Alexandre,

On Sat, Oct 19, 2013 at 10:00 AM, Alexandre Belloni
<alexandre.belloni@free-electrons.com> wrote:
> I just have some time to test now and I'm not able to submit my results.
> The form keeps asking me for the board name although I selected it...

What is the problem you're facing?

I think I fixed the error for cfa10036 (which were duplicated), please
give it a new try.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [ANN] FSL Test Plan Announcement 1.5

[Daiane Angolini]

On 10/19/2013 10:00 AM, Alexandre Belloni wrote:
> Hi Daiane
>
> I just have some time to test now and I'm not able to submit my results.
> The form keeps asking me for the board name although I selected it...

I saw several entries for your boards on results spreeadsheet.

Do you think this was a temporary error?

(I tried the doc submition with my google accound and *without* it, and 
I have never faced errors... If you were able to colect any log I may 
try to debug it)


Daiane
>
> On 03/10/2013 20:48, Daiane Angolini wrote:
>> We'd like to invite everyone to help us to test the Freescale
>> Community BSP 1.5 - Dora release.
>>
>> Please, test your favorite board (or boards), using your favorite
>> image. We would like to have at least one test for each 35 boards
>> (supported by both meta-fsl-arm or meta-fsl-arm-extra).
>>
>> We've made a form which guides you through a small set of steps and
>> help us to track which boards has been tested. Please, make sure you
>> know the packages needed for some tests listed at top of form.
>>
>> If you have any other question you want to be include in the test
>> form, please, let us know.
>> The '1.5' is being developed in the 'dora' branch, so use this branch
>> for the tests (please do not send reports for 'master' branch, any
>> error/bug on master should be reported on mailing list).
>>
>> We'll be receiving and reviewing the test results until Oct 20th 2013.
>> However please start the tests as soon as you can, so we can have the
>> most solid release as possible. The release will be done together with
>> the Yocto release (planned for Oct 18th 2013) and any issues found (if
>> applicable) will need to be fixed for the 1.5 point releases (1.5.1,
>> 1.5.2, ...).
>>
>> The goal test plan for this release is:
>>
>> MUST HAVE:
>> * all 35 boards MUST reach the user space prompt
>> * all 35 boards MUST have some[1] stability
>> * all IMX6 boards MUST have X11 up and running accelerated
>>
>> SHOULD HAVE:
>> * all boards SHOULD play at least one movie
>> * all boards SHOULD encode at least one kind of movie
>> * all boards SHOULD have minimal kernel defconfig enabled by default
>> (USB/SDcard/aplay)
>>
>> BONUS POINT:
>> * all IMX6 boards SHOULD have GPU and VPU support for both SFP and HFP
>> * all board SHOULD have overnight stability[2]
>>
>> We need your help to accomplish our test goals and we don't have
>> enough manpower resources to do it all by ourselves. We count on the
>> community to help on this.
>>
>> Please, report any bug/error to mailing list, we will discuss
>> case-by-case and create needed bug report.
>>
>> After the test cycle, we will include the summary of the test result
>> on the release notes so we can start to build credibility by showing
>> our test results for all the boards, giving to users and partners a
>> central place for checking for expected behavior, and known issues,
>> for a specific board.
>>
>> [1] some stability now is defined as being alive during the form tests
>> [2] overnight stability now is defined by the board being alive after
>> more than 10 hours testing memtester 50M (last question from form,
>> Bonus Point)
>>
>> Regards,
>> Daiane
>>
>> _______________________________________________
>> meta-freescale mailing list
>> meta-freescale@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/meta-freescale
>
>


-- 
Daiane

Re: [ANN] FSL Test Plan Announcement 1.5

[Otavio Salvador]

Hello Daiane,

On Mon, Oct 21, 2013 at 9:03 AM, Daiane Angolini
<daiane.angolini@freescale.com> wrote:
> On 10/19/2013 10:00 AM, Alexandre Belloni wrote:
>> I just have some time to test now and I'm not able to submit my results.
>> The form keeps asking me for the board name although I selected it...
>
> I saw several entries for your boards on results spreeadsheet.
>
> Do you think this was a temporary error?
>
> (I tried the doc submition with my google accound and *without* it, and I
> have never faced errors... If you were able to colect any log I may try to
> debug it)

There was an error but I was able to fix it and Alexandre succeeded to
send his reports.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [ANN] FSL Test Plan Announcement 1.5

[Daiane Angolini]

On Thu, Oct 3, 2013 at 3:48 PM, Daiane Angolini
<daiane.angolini@freescale.com> wrote:
> We'd like to invite everyone to help us to test the Freescale Community BSP
> 1.5 - Dora release.
>
> Please, test your favorite board (or boards), using your favorite image. We
> would like to have at least one test for each 35 boards (supported by both
> meta-fsl-arm or meta-fsl-arm-extra).


I just closed the Test Form for accepting new test entries.


We appreciate your your help on test cycle. The test result is on Release Notes:

http://freescale.github.io/doc/release-notes/1.5/index.html#document-test-results


I want to send a special thanks to Lucas Nunes and Mario Goulart, from
O.S. Systems
for helping me to create the Test Result Table!


The overall test result is that 12 of 34 boards was considered OK:

cfa10036
cfa10037
cfa10049
cfa10055
cfa10056
cfa10057
cfa10058
imx28evk
imx53qsb
imx53ard
imx6qsabrelite
wandboard-dual

This test cycle had more submitted boards than 1.4 release but we are
still far away from a
full tested release (only 35% of supported boards had the Test Goals
accomplished) and
we must do even better in 1.6, in April.



Best Regards,
Daiane

>
> We've made a form which guides you through a small set of steps and help us
> to track which boards has been tested. Please, make sure you know the
> packages needed for some tests listed at top of form.
>
> If you have any other question you want to be include in the test form,
> please, let us know.
> The '1.5' is being developed in the 'dora' branch, so use this branch for
> the tests (please do not send reports for 'master' branch, any error/bug on
> master should be reported on mailing list).
>
> We'll be receiving and reviewing the test results until Oct 20th 2013.
> However please start the tests as soon as you can, so we can have the most
> solid release as possible. The release will be done together with the Yocto
> release (planned for Oct 18th 2013) and any issues found (if applicable)
> will need to be fixed for the 1.5 point releases (1.5.1, 1.5.2, ...).
>
> The goal test plan for this release is:
>
> MUST HAVE:
> * all 35 boards MUST reach the user space prompt
> * all 35 boards MUST have some[1] stability
> * all IMX6 boards MUST have X11 up and running accelerated
>
> SHOULD HAVE:
> * all boards SHOULD play at least one movie
> * all boards SHOULD encode at least one kind of movie
> * all boards SHOULD have minimal kernel defconfig enabled by default
> (USB/SDcard/aplay)
>
> BONUS POINT:
> * all IMX6 boards SHOULD have GPU and VPU support for both SFP and HFP
> * all board SHOULD have overnight stability[2]
>
> We need your help to accomplish our test goals and we don't have enough
> manpower resources to do it all by ourselves. We count on the community to
> help on this.
>
> Please, report any bug/error to mailing list, we will discuss case-by-case
> and create needed bug report.
>
> After the test cycle, we will include the summary of the test result on the
> release notes so we can start to build credibility by showing our test
> results for all the boards, giving to users and partners a central place for
> checking for expected behavior, and known issues, for a specific board.
>
> [1] some stability now is defined as being alive during the form tests
> [2] overnight stability now is defined by the board being alive after more
> than 10 hours testing memtester 50M (last question from form, Bonus Point)
>
> Regards,
> Daiane
>
> _______________________________________________
> meta-freescale mailing list
> meta-freescale@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/meta-freescale