1.3 M3 Full Pass test results

1.3 M3 Full Pass test results

[Serban, Laurentiu]

[-- Attachment #1.1: Type: text/plain, Size: 6470 bytes --]

Hello,

Here are the results for the full pas tests on 1.3 M3 RC2. The commit used for testing is 8b8748c8f963900b83dc0fdd7757556f917fe4fd.
Some details about the encountered issues below:

BSP - Sudoku-savant project build issue (2878)
ADT - the relocatable sdk issue (2980) causes 13 test cases to be on faile/blocked state , also the Clutter C template issue is unsolved (2577)
Core Build System - x32 is still an issue (2888), cleaning sstate issue is still not solved (2897), incremental RPM image generation (2969), source archiving (2619), the kvm issue was reproduced by another colleague (2790) Yocto BSP creation via JSON (2693) or for qemu (2991) fails, multilib issue (2918 - this requires a little more investigation from QA),
HOB - all seems ok for RC2
Self-hosted-image  - cannot start on Virtual Box (X issue), it is very slow on qemu and it has a m4 package build (3005) issue on VMWare. If the self-hosted-image is used on machine with internet connectivity via proxy there will be an initial sanity check failure, but this is not a blocking issue.

A mention for the performance testing: on a Ubbuntu 12.04  i7 machine using 8 threads the build time was 83 minutes (with prior fetching).

Test Result Summary

Component

Target

Status

Comments

BSP

Beagleboard

GOOD

Sudoku-savant project issue

Routerstationpro

GOOD

Sudoku-savant project issue

Mpc8315e-rdb

GOOD

Sudoku-savant project issue

eMenlow-sato-sdk

GOOD

Sudoku-savant project issue

Blacksand-sato

GOOD

The sdk image was not available

Crownbay-emgd-sato-sdk

GOOD

Sudoku-savant project issue

FRI2-sato-sdk

BUGGY

X Server issue blocks many test cases
Suspend issue
Dmesg issue

HuronRiver-sato-sdk

GOOD

Sudoku-savant project issue

Jasperforest-lsb-sdk

GOOD

Sudoku-savant project issue

QEMU

Qemuarm

GOOD

Everything runs well

Qemuppc

GOOD

Everything runs well

Qemumips

GOOD

Everything runs well

qemux86

GOOD

Everything runs well except for zypper bug 2694 for sato-sdk image

qemux86-64

GOOD

Everything runs well except for zypper bug 2694 for sato-sdk image

Core build system

　

BUGGY

x32 is still an issue (2888), cleaning sstate issue is still not solved (2897), incremental RPM image generation (2969), source archiving (2619), the kvm issue was reproduced by another colleague (2790), Yocto BSP creation via JSON (2693) or for qemu (2991) fails, multilib issue (2918)

HOB

　

GOOD

Everything runs well except for HOB recipe building from command line and and cross toolchain build (2695 and3001).

Compliance

GOOD

Sugarbay: Board unavailable; HuronRiver: Pass; Blacksand: Blocked

Stress

GOOD

Both Crashme and Helltest on Jasperforest could pass 24 hours stress testing

Distribution Support

GOOD

Everything runs well for Ubuntu 12.04, Ubuntu 12.10 - nightly, OpenSuse 12.1, OpenSuse 12.2 RC2, Fedora 17, CentOs 6.3, Fedora 16

Power and Performance

GOOD

one qemux86 sato build on a Core i7 machine costs 83 minutes with the packages previously fetched.

ADT

　

BUGGY

the relocatable sdk issue (2980) causes 13 test cases to be on faile/blocked state , also the Clutter C template issue is unsolved (2577) Also 2768 bug affects native autoconf



Detailed Test Result for each component

Target

Total TCs

Not Run

Passed

Failed

Not testable (Blocked)

Beagleboard Sato-SDK

42

0

41

1 (2878)

0

Routerstationpro Sato-SDK

35

0

34

1 (2878)

0

Mpc8315e-rdb Sato-SDK

37

0

36

1 (2878)

0

eMenlow Sato-SDK

67

0

66

1 (2878)

0

n450 Sato

56

0

42

1(bug 2646)

13

Crownbay-noemgdSato-SDK

67

2

64

1 (2878)

0

FRI2 Sato-SDK

86

1

42

4 (bugs 3004, 2989, 2415)

39  (3004, 2037, 2415)

HuronRiver Sato-SDK

70

0

69

1(bug 2791)

0

Jasperforest lsb-SDK

34

0

33

1 (2878)

0

Qemuarm Sato

32

0

32

0

0

Qemuarm Sato-SDK

37

0

37

0

0

Qemumips Sato

32

0

32

0

0

Qemumips Sato-SDK

37

0

37

0

0

Qemuppc Sato

32

0

32

0

0

Qemuppc Sato-SDK

37

0

37

0

0

Qemux86-64 Sato

32

0

32

0

0

Qemux86-64 Sato-SDK

37

0

37

0

0

Qemux86 Sato

32

0

32

0

0

Qemux86 Sato-SDK

37

0

37

0

0

Core build system

64

0

43

12 (2969, 2619, 2897, 2790, 2888, 2693, 2918, 3005, 2991)

9 (bugs 2888 and 2684, 2996)

HOB

38

0

36

2 (bugs 2695 and 3001)

0

ADT

53

0

23

14 (2980, 2577, 2768)

16(2980)

Compliance

3

0

2

0

1 (n-450 image not available)

Stress

2

0

2

0

0

Distribution Support

8

0

8

0

0

Total

1007

3

886

40

78


Component

Bug Number

Target Milestone

System & Core OS

System Usage

Bug 2878 pkgconfig is missing from sato-sdk image

Bug 2989 [FRI2] dmesg PowerButton error

1.3

Bug 2415 connman doesn't provide a 3g configuration utility

1.3

Bug 2037 [fri2] system cannot enter S3 standby mode

1.2

Bug 3004 [FRI2]X server fails to start

Core build system

Bug 2969 Incremental RPM image generation is not working as expected.

Bug 2619 do_package_write_rpm fails - fakeroot issue

Bug 2897 sstate-cache-management.sh doesn't work with new sstate-cache layout

Bug 2790 eglibc-initial installs to /usr

Bug 2888 x32 build fails at gcc cross-initial do_compile

Bug 2693 Yocto BSP create via JSON file fails

Bug 2918 lib64-core-image-sato-sdk fails due to do_rootfs issue.

Bug 3005 1.3 M3 build-appliance comes with old poky tree and m4-native compile fails

Bug 2991  the image created with yocto-bsp tool fails to start - "unsupported machine type"

ADT

Bug 2980 toolchain issue - segmentation fault while configuring packages

1.3 M4

Bug 2768 iptables 1.4.13 configure fail using "autoreconf -fi"

1.3 M4

Bug 2577 Segmentation fault on qemuarm from /usr/lib/libust.so.0

1.3 M4

HOB

Bug 2695 [HOB]toolchain arch in settings is not saved

1.3 M4

Bug 3001 bblayers-hob.conf does not correspond with bblayers.conf





Laurentiu Serban
QA Engineer
Open Source Technology Center
System Software Division Romania
Desk: +40 31 8604742
iNET: 88451042


[-- Attachment #1.2: Type: text/html, Size: 127736 bytes --]

[-- Attachment #2: 1.3_M3_RC2_FullPass_TR.xlsx --]
[-- Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet, Size: 60435 bytes --]

Re: 1.3 M3 Full Pass test results

[Richard Purdie]

On Thu, 2012-08-23 at 20:13 +0100, Serban, Laurentiu wrote:

> Here are the results for the full pas tests on 1.3 M3 RC2. The commit
> used for testing is 8b8748c8f963900b83dc0fdd7757556f917fe4fd.
>
> Some details about the encountered issues below:
>  
> BSP – Sudoku-savant project build issue (2878)
>
> ADT – the relocatable sdk issue (2980) causes 13 test cases to be on
> faile/blocked state

I thought it worked as long as you didn't have to relocate it so no
tests should have been blocked, we just have the relocation issue?

>  , also the Clutter C template issue is unsolved (2577)
>
> Core Build System – x32 is still an issue (2888), cleaning sstate
> issue is still not solved (2897), incremental RPM image generation
> (2969), source archiving (2619), the kvm issue was reproduced by
> another colleague (2790) Yocto BSP creation via JSON (2693) or for
> qemu (2991) fails, multilib issue (2918 – this requires a little more
> investigation from QA), 
>
> HOB - all seems ok for RC2
>
> Self-hosted-image  - cannot start on Virtual Box (X issue), it is very
> slow on qemu and it has a m4 package build (3005) issue on VMWare. If
> the self-hosted-image is used on machine with internet connectivity
> via proxy there will be an initial sanity check failure, but this is
> not a blocking issue.
>  
> A mention for the performance testing: on a Ubbuntu 12.04  i7 machine
> using 8 threads the build time was 83 minutes (with prior fetching).

How does this compare with our other performance numbers. From what I
remember, we used to hover around the 105-115 minute mark. Did we have
some significant speed gains or is this just an artefact of changing the
test machine?

Cheers,

Richard

Re: 1.3 M3 Full Pass test results

[Serban, Laurentiu]

Hello Richard,

Even if the installer is used in the default mode, issues still occur (see comment 7). I think the root cause for these is the same, so I did not submit a new bug.

Thank you,
Laurentiu 

-----Original Message-----
From: Purdie, Richard 
Sent: Friday, August 24, 2012 1:22 PM
To: Serban, Laurentiu
Cc: yocto@yoctoproject.org; Liu, Song; Stewart, David C; Wold, Saul
Subject: Re: 1.3 M3 Full Pass test results

On Thu, 2012-08-23 at 20:13 +0100, Serban, Laurentiu wrote:

> Here are the results for the full pas tests on 1.3 M3 RC2. The commit 
> used for testing is 8b8748c8f963900b83dc0fdd7757556f917fe4fd.
>
> Some details about the encountered issues below:
>  
> BSP – Sudoku-savant project build issue (2878)
>
> ADT – the relocatable sdk issue (2980) causes 13 test cases to be on 
> faile/blocked state

I thought it worked as long as you didn't have to relocate it so no tests should have been blocked, we just have the relocation issue?

>  , also the Clutter C template issue is unsolved (2577)
>
> Core Build System – x32 is still an issue (2888), cleaning sstate 
> issue is still not solved (2897), incremental RPM image generation 
> (2969), source archiving (2619), the kvm issue was reproduced by 
> another colleague (2790) Yocto BSP creation via JSON (2693) or for 
> qemu (2991) fails, multilib issue (2918 – this requires a little more 
> investigation from QA),
>
> HOB - all seems ok for RC2
>
> Self-hosted-image  - cannot start on Virtual Box (X issue), it is very 
> slow on qemu and it has a m4 package build (3005) issue on VMWare. If 
> the self-hosted-image is used on machine with internet connectivity 
> via proxy there will be an initial sanity check failure, but this is 
> not a blocking issue.
>  
> A mention for the performance testing: on a Ubbuntu 12.04  i7 machine 
> using 8 threads the build time was 83 minutes (with prior fetching).

How does this compare with our other performance numbers. From what I remember, we used to hover around the 105-115 minute mark. Did we have some significant speed gains or is this just an artefact of changing the test machine?

Cheers,

Richard

Re: 1.3 M3 Full Pass test results

[Liu, Song]

Hi Laurentiu, 

Do you have any comment on the performance question Richard asked? I know your team is using a machine with different configuration from the one used by the Shanghai team. The performance figure from the Shanghai team has been hovering around 110 mins. That's the case for 1.2 release and 1.3 M2 milestone report. 1.3 M1 milestone report has the build time as 95 minutes, which I believe is from your team. So my question is whether you used the same machine for M3 performance testing as for M1. Another factor that might have caused the difference (between 95 and 83 minutes) is your testing procedure/environment such as other processes running at the same time, memory usage, sstate cache, etc. If you used the same machine and same testing procedure/environment, then we have some improvement in M3 compared with M1. Please let me know your thoughts.

Thanks,
Song

-----Original Message-----
From: Serban, Laurentiu 
Sent: Friday, August 24, 2012 6:34 AM
To: Purdie, Richard
Cc: yocto@yoctoproject.org; Liu, Song; Stewart, David C; Wold, Saul
Subject: RE: 1.3 M3 Full Pass test results

Hello Richard,

Even if the installer is used in the default mode, issues still occur (see comment 7). I think the root cause for these is the same, so I did not submit a new bug.

Thank you,
Laurentiu 

-----Original Message-----
From: Purdie, Richard
Sent: Friday, August 24, 2012 1:22 PM
To: Serban, Laurentiu
Cc: yocto@yoctoproject.org; Liu, Song; Stewart, David C; Wold, Saul
Subject: Re: 1.3 M3 Full Pass test results

On Thu, 2012-08-23 at 20:13 +0100, Serban, Laurentiu wrote:

> Here are the results for the full pas tests on 1.3 M3 RC2. The commit 
> used for testing is 8b8748c8f963900b83dc0fdd7757556f917fe4fd.
>
> Some details about the encountered issues below:
>  
> BSP – Sudoku-savant project build issue (2878)
>
> ADT – the relocatable sdk issue (2980) causes 13 test cases to be on 
> faile/blocked state

I thought it worked as long as you didn't have to relocate it so no tests should have been blocked, we just have the relocation issue?

>  , also the Clutter C template issue is unsolved (2577)
>
> Core Build System – x32 is still an issue (2888), cleaning sstate 
> issue is still not solved (2897), incremental RPM image generation 
> (2969), source archiving (2619), the kvm issue was reproduced by 
> another colleague (2790) Yocto BSP creation via JSON (2693) or for 
> qemu (2991) fails, multilib issue (2918 – this requires a little more 
> investigation from QA),
>
> HOB - all seems ok for RC2
>
> Self-hosted-image  - cannot start on Virtual Box (X issue), it is very 
> slow on qemu and it has a m4 package build (3005) issue on VMWare. If 
> the self-hosted-image is used on machine with internet connectivity 
> via proxy there will be an initial sanity check failure, but this is 
> not a blocking issue.
>  
> A mention for the performance testing: on a Ubbuntu 12.04  i7 machine 
> using 8 threads the build time was 83 minutes (with prior fetching).

How does this compare with our other performance numbers. From what I remember, we used to hover around the 105-115 minute mark. Did we have some significant speed gains or is this just an artefact of changing the test machine?

Cheers,

Richard

Re: 1.3 M3 Full Pass test results

[Serban, Laurentiu]

Hello guys,

I am sorry I did not notice the last line in the e-mail.

When we ran the performance testing we pre-downloaded all the sources, because last week had a lot of connectivity issues. I mentioned in the e-mail that the build time refers to an image built after all the sources were downloaded.
The machine on which the test was ran is the same as for M1.

I will ask Stefan to re-run the performance tests for the different milestones so far, M1 and M2, in the same conditions as for M3 so we would have a clear view.

I add Jiajun in the loop so he can help us if he can with a test to see if this is related to an artifact as Richard said, but also there were some improvements made by Beth on filesystem generation.

Br,
Laurentiu

-----Original Message-----
From: Liu, Song 
Sent: Monday, August 27, 2012 8:50 PM
To: Serban, Laurentiu; Purdie, Richard
Cc: yocto@yoctoproject.org; Stewart, David C; Wold, Saul
Subject: RE: 1.3 M3 Full Pass test results

Hi Laurentiu, 

Do you have any comment on the performance question Richard asked? I know your team is using a machine with different configuration from the one used by the Shanghai team. The performance figure from the Shanghai team has been hovering around 110 mins. That's the case for 1.2 release and 1.3 M2 milestone report. 1.3 M1 milestone report has the build time as 95 minutes, which I believe is from your team. So my question is whether you used the same machine for M3 performance testing as for M1. Another factor that might have caused the difference (between 95 and 83 minutes) is your testing procedure/environment such as other processes running at the same time, memory usage, sstate cache, etc. If you used the same machine and same testing procedure/environment, then we have some improvement in M3 compared with M1. Please let me know your thoughts.

Thanks,
Song

-----Original Message-----
From: Serban, Laurentiu
Sent: Friday, August 24, 2012 6:34 AM
To: Purdie, Richard
Cc: yocto@yoctoproject.org; Liu, Song; Stewart, David C; Wold, Saul
Subject: RE: 1.3 M3 Full Pass test results

Hello Richard,

Even if the installer is used in the default mode, issues still occur (see comment 7). I think the root cause for these is the same, so I did not submit a new bug.

Thank you,
Laurentiu 

-----Original Message-----
From: Purdie, Richard
Sent: Friday, August 24, 2012 1:22 PM
To: Serban, Laurentiu
Cc: yocto@yoctoproject.org; Liu, Song; Stewart, David C; Wold, Saul
Subject: Re: 1.3 M3 Full Pass test results

On Thu, 2012-08-23 at 20:13 +0100, Serban, Laurentiu wrote:

> Here are the results for the full pas tests on 1.3 M3 RC2. The commit 
> used for testing is 8b8748c8f963900b83dc0fdd7757556f917fe4fd.
>
> Some details about the encountered issues below:
>  
> BSP – Sudoku-savant project build issue (2878)
>
> ADT – the relocatable sdk issue (2980) causes 13 test cases to be on 
> faile/blocked state

I thought it worked as long as you didn't have to relocate it so no tests should have been blocked, we just have the relocation issue?

>  , also the Clutter C template issue is unsolved (2577)
>
> Core Build System – x32 is still an issue (2888), cleaning sstate 
> issue is still not solved (2897), incremental RPM image generation 
> (2969), source archiving (2619), the kvm issue was reproduced by 
> another colleague (2790) Yocto BSP creation via JSON (2693) or for 
> qemu (2991) fails, multilib issue (2918 – this requires a little more 
> investigation from QA),
>
> HOB - all seems ok for RC2
>
> Self-hosted-image  - cannot start on Virtual Box (X issue), it is very 
> slow on qemu and it has a m4 package build (3005) issue on VMWare. If 
> the self-hosted-image is used on machine with internet connectivity 
> via proxy there will be an initial sanity check failure, but this is 
> not a blocking issue.
>  
> A mention for the performance testing: on a Ubbuntu 12.04  i7 machine 
> using 8 threads the build time was 83 minutes (with prior fetching).

How does this compare with our other performance numbers. From what I remember, we used to hover around the 105-115 minute mark. Did we have some significant speed gains or is this just an artefact of changing the test machine?

Cheers,

Richard

Re: 1.3 M3 Full Pass test results

[Xu, Jiajun]

We used to get 115m and 127m for 1.3 M1 and M2 on our test machine.

> Hello guys,
> 
> I am sorry I did not notice the last line in the e-mail.
> 
> When we ran the performance testing we pre-downloaded all the sources,
> because last week had a lot of connectivity issues. I mentioned in the
> e-mail that the build time refers to an image built after all the
> sources were downloaded.
> The machine on which the test was ran is the same as for M1.
> 
> I will ask Stefan to re-run the performance tests for the different
> milestones so far, M1 and M2, in the same conditions as for M3 so we
> would have a clear view.
> 
> I add Jiajun in the loop so he can help us if he can with a test to
> see if this is related to an artifact as Richard said, but also there
> were some improvements made by Beth on filesystem generation.
> 
> Br,
> Laurentiu
> 
> -----Original Message-----
> From: Liu, Song
> Sent: Monday, August 27, 2012 8:50 PM
> To: Serban, Laurentiu; Purdie, Richard
> Cc: yocto@yoctoproject.org; Stewart, David C; Wold, Saul
> Subject: RE: 1.3 M3 Full Pass test results
> 
> Hi Laurentiu,
> 
> Do you have any comment on the performance question Richard asked? I
> know your team is using a machine with different configuration from
> the one used by the Shanghai team. The performance figure from the
> Shanghai team has been hovering around 110 mins. That's the case for
> 1.2 release and 1.3 M2 milestone report. 1.3 M1 milestone report has
> the build time as 95 minutes, which I believe is from your team. So my
> question is whether you used the same machine for M3 performance
> testing as for M1. Another factor that might have caused the
> difference (between 95 and 83 minutes) is your testing
> procedure/environment such as other processes running at the same
> time, memory usage, sstate cache, etc. If you used the same machine
> and same testing procedure/environment, then we have some improvement in M3 compared with M1. Please let me know your thoughts.
> 
> Thanks,
> Song
> 
> -----Original Message-----
> From: Serban, Laurentiu
> Sent: Friday, August 24, 2012 6:34 AM
> To: Purdie, Richard
> Cc: yocto@yoctoproject.org; Liu, Song; Stewart, David C; Wold, Saul
> Subject: RE: 1.3 M3 Full Pass test results
> 
> Hello Richard,
> 
> Even if the installer is used in the default mode, issues still occur
> (see comment 7). I think the root cause for these is the same, so I did not submit a new bug.
> 
> Thank you,
> Laurentiu
> 
> -----Original Message-----
> From: Purdie, Richard
> Sent: Friday, August 24, 2012 1:22 PM
> To: Serban, Laurentiu
> Cc: yocto@yoctoproject.org; Liu, Song; Stewart, David C; Wold, Saul
> Subject: Re: 1.3 M3 Full Pass test results
> 
> On Thu, 2012-08-23 at 20:13 +0100, Serban, Laurentiu wrote:
> 
>> Here are the results for the full pas tests on 1.3 M3 RC2. The
>> commit used for testing is 8b8748c8f963900b83dc0fdd7757556f917fe4fd.
>> 
>> Some details about the encountered issues below:
>> 
>> BSP – Sudoku-savant project build issue (2878)
>> 
>> ADT – the relocatable sdk issue (2980) causes 13 test cases to be on
>> faile/blocked state
> 
> I thought it worked as long as you didn't have to relocate it so no
> tests should have been blocked, we just have the relocation issue?
> 
>>  , also the Clutter C template issue is unsolved (2577)
>> Core Build System – x32 is still an issue (2888), cleaning sstate
>> issue is still not solved (2897), incremental RPM image generation
>> (2969), source archiving (2619), the kvm issue was reproduced by
>> another colleague (2790) Yocto BSP creation via JSON (2693) or for
>> qemu (2991) fails, multilib issue (2918 – this requires a little
>> more investigation from QA),
>> 
>> HOB - all seems ok for RC2
>> 
>> Self-hosted-image  - cannot start on Virtual Box (X issue), it is
>> very slow on qemu and it has a m4 package build (3005) issue on
>> VMWare. If the self-hosted-image is used on machine with internet
>> connectivity via proxy there will be an initial sanity check
>> failure, but this is not a blocking issue.
>> 
>> A mention for the performance testing: on a Ubbuntu 12.04  i7
>> machine using 8 threads the build time was 83 minutes (with prior fetching).
> 
> How does this compare with our other performance numbers. From what I
> remember, we used to hover around the 105-115 minute mark. Did we have
> some significant speed gains or is this just an artefact of changing
> the test machine?
> 
> Cheers,
> 
> Richard
> 
>

Best Regards,
Jiajun

Re: 1.3 M3 Full Pass test results

[Saxena, Rahul]

[-- Attachment #1: Type: text/plain, Size: 6831 bytes --]

Does this 1.3 M3 testing includes testing with the linux-yocto3.0 kernel ?


Thanks
Rahul

From: yocto-bounces@yoctoproject.org [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Serban, Laurentiu
Sent: Thursday, August 23, 2012 12:13 PM
To: yocto@yoctoproject.org
Cc: Purdie, Richard; Wold, Saul
Subject: [yocto] 1.3 M3 Full Pass test results

Hello,

Here are the results for the full pas tests on 1.3 M3 RC2. The commit used for testing is 8b8748c8f963900b83dc0fdd7757556f917fe4fd.
Some details about the encountered issues below:

BSP - Sudoku-savant project build issue (2878)
ADT - the relocatable sdk issue (2980) causes 13 test cases to be on faile/blocked state , also the Clutter C template issue is unsolved (2577)
Core Build System - x32 is still an issue (2888), cleaning sstate issue is still not solved (2897), incremental RPM image generation (2969), source archiving (2619), the kvm issue was reproduced by another colleague (2790) Yocto BSP creation via JSON (2693) or for qemu (2991) fails, multilib issue (2918 - this requires a little more investigation from QA),
HOB - all seems ok for RC2
Self-hosted-image  - cannot start on Virtual Box (X issue), it is very slow on qemu and it has a m4 package build (3005) issue on VMWare. If the self-hosted-image is used on machine with internet connectivity via proxy there will be an initial sanity check failure, but this is not a blocking issue.

A mention for the performance testing: on a Ubbuntu 12.04  i7 machine using 8 threads the build time was 83 minutes (with prior fetching).


Test Result Summary

Component

Target

Status

Comments

BSP

Beagleboard

GOOD

Sudoku-savant project issue

Routerstationpro

GOOD

Sudoku-savant project issue

Mpc8315e-rdb

GOOD

Sudoku-savant project issue

eMenlow-sato-sdk

GOOD

Sudoku-savant project issue

Blacksand-sato

GOOD

The sdk image was not available

Crownbay-emgd-sato-sdk

GOOD

Sudoku-savant project issue

FRI2-sato-sdk

BUGGY

X Server issue blocks many test cases
Suspend issue
Dmesg issue

HuronRiver-sato-sdk

GOOD

Sudoku-savant project issue

Jasperforest-lsb-sdk

GOOD

Sudoku-savant project issue

QEMU

Qemuarm

GOOD

Everything runs well

Qemuppc

GOOD

Everything runs well

Qemumips

GOOD

Everything runs well

qemux86

GOOD

Everything runs well except for zypper bug 2694 for sato-sdk image

qemux86-64

GOOD

Everything runs well except for zypper bug 2694 for sato-sdk image

Core build system

　

BUGGY

x32 is still an issue (2888), cleaning sstate issue is still not solved (2897), incremental RPM image generation (2969), source archiving (2619), the kvm issue was reproduced by another colleague (2790), Yocto BSP creation via JSON (2693) or for qemu (2991) fails, multilib issue (2918)

HOB

　

GOOD

Everything runs well except for HOB recipe building from command line and and cross toolchain build (2695 and3001).

Compliance

GOOD

Sugarbay: Board unavailable; HuronRiver: Pass; Blacksand: Blocked

Stress

GOOD

Both Crashme and Helltest on Jasperforest could pass 24 hours stress testing

Distribution Support

GOOD

Everything runs well for Ubuntu 12.04, Ubuntu 12.10 - nightly, OpenSuse 12.1, OpenSuse 12.2 RC2, Fedora 17, CentOs 6.3, Fedora 16

Power and Performance

GOOD

one qemux86 sato build on a Core i7 machine costs 83 minutes with the packages previously fetched.

ADT

　

BUGGY

the relocatable sdk issue (2980) causes 13 test cases to be on faile/blocked state , also the Clutter C template issue is unsolved (2577) Also 2768 bug affects native autoconf



Detailed Test Result for each component

Target

Total TCs

Not Run

Passed

Failed

Not testable (Blocked)

Beagleboard Sato-SDK

42

0

41

1 (2878)

0

Routerstationpro Sato-SDK

35

0

34

1 (2878)

0

Mpc8315e-rdb Sato-SDK

37

0

36

1 (2878)

0

eMenlow Sato-SDK

67

0

66

1 (2878)

0

n450 Sato

56

0

42

1(bug 2646)

13

Crownbay-noemgdSato-SDK

67

2

64

1 (2878)

0

FRI2 Sato-SDK

86

1

42

4 (bugs 3004, 2989, 2415)

39  (3004, 2037, 2415)

HuronRiver Sato-SDK

70

0

69

1(bug 2791)

0

Jasperforest lsb-SDK

34

0

33

1 (2878)

0

Qemuarm Sato

32

0

32

0

0

Qemuarm Sato-SDK

37

0

37

0

0

Qemumips Sato

32

0

32

0

0

Qemumips Sato-SDK

37

0

37

0

0

Qemuppc Sato

32

0

32

0

0

Qemuppc Sato-SDK

37

0

37

0

0

Qemux86-64 Sato

32

0

32

0

0

Qemux86-64 Sato-SDK

37

0

37

0

0

Qemux86 Sato

32

0

32

0

0

Qemux86 Sato-SDK

37

0

37

0

0

Core build system

64

0

43

12 (2969, 2619, 2897, 2790, 2888, 2693, 2918, 3005, 2991)

9 (bugs 2888 and 2684, 2996)

HOB

38

0

36

2 (bugs 2695 and 3001)

0

ADT

53

0

23

14 (2980, 2577, 2768)

16(2980)

Compliance

3

0

2

0

1 (n-450 image not available)

Stress

2

0

2

0

0

Distribution Support

8

0

8

0

0

Total

1007

3

886

40

78


Component

Bug Number

Target Milestone

System & Core OS

System Usage

Bug 2878 pkgconfig is missing from sato-sdk image

Bug 2989 [FRI2] dmesg PowerButton error

1.3

Bug 2415 connman doesn't provide a 3g configuration utility

1.3

Bug 2037 [fri2] system cannot enter S3 standby mode

1.2

Bug 3004 [FRI2]X server fails to start

Core build system

Bug 2969 Incremental RPM image generation is not working as expected.

Bug 2619 do_package_write_rpm fails - fakeroot issue

Bug 2897 sstate-cache-management.sh doesn't work with new sstate-cache layout

Bug 2790 eglibc-initial installs to /usr

Bug 2888 x32 build fails at gcc cross-initial do_compile

Bug 2693 Yocto BSP create via JSON file fails

Bug 2918 lib64-core-image-sato-sdk fails due to do_rootfs issue.

Bug 3005 1.3 M3 build-appliance comes with old poky tree and m4-native compile fails

Bug 2991  the image created with yocto-bsp tool fails to start - "unsupported machine type"

ADT

Bug 2980 toolchain issue - segmentation fault while configuring packages

1.3 M4

Bug 2768 iptables 1.4.13 configure fail using "autoreconf -fi"

1.3 M4

Bug 2577 Segmentation fault on qemuarm from /usr/lib/libust.so.0

1.3 M4

HOB

Bug 2695 [HOB]toolchain arch in settings is not saved

1.3 M4

Bug 3001 bblayers-hob.conf does not correspond with bblayers.conf





Laurentiu Serban
QA Engineer
Open Source Technology Center
System Software Division Romania
Desk: +40 31 8604742
iNET: 88451042


[-- Attachment #2: Type: text/html, Size: 116770 bytes --]

Re: 1.3 M3 Full Pass test results

[Serban, Laurentiu]

[-- Attachment #1: Type: text/plain, Size: 7178 bytes --]

Hello Rahul,

1.3 M3 does not include testing with linux-yocto 3.0.

Br,
Laurentiu

From: Saxena, Rahul
Sent: Monday, August 27, 2012 7:27 PM
To: Serban, Laurentiu; yocto@yoctoproject.org
Cc: Purdie, Richard; Wold, Saul
Subject: RE: 1.3 M3 Full Pass test results

Does this 1.3 M3 testing includes testing with the linux-yocto3.0 kernel ?


Thanks
Rahul

From: yocto-bounces@yoctoproject.org<mailto:yocto-bounces@yoctoproject.org> [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Serban, Laurentiu
Sent: Thursday, August 23, 2012 12:13 PM
To: yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>
Cc: Purdie, Richard; Wold, Saul
Subject: [yocto] 1.3 M3 Full Pass test results

Hello,

Here are the results for the full pas tests on 1.3 M3 RC2. The commit used for testing is 8b8748c8f963900b83dc0fdd7757556f917fe4fd.
Some details about the encountered issues below:

BSP - Sudoku-savant project build issue (2878)
ADT - the relocatable sdk issue (2980) causes 13 test cases to be on faile/blocked state , also the Clutter C template issue is unsolved (2577)
Core Build System - x32 is still an issue (2888), cleaning sstate issue is still not solved (2897), incremental RPM image generation (2969), source archiving (2619), the kvm issue was reproduced by another colleague (2790) Yocto BSP creation via JSON (2693) or for qemu (2991) fails, multilib issue (2918 - this requires a little more investigation from QA),
HOB - all seems ok for RC2
Self-hosted-image  - cannot start on Virtual Box (X issue), it is very slow on qemu and it has a m4 package build (3005) issue on VMWare. If the self-hosted-image is used on machine with internet connectivity via proxy there will be an initial sanity check failure, but this is not a blocking issue.

A mention for the performance testing: on a Ubbuntu 12.04  i7 machine using 8 threads the build time was 83 minutes (with prior fetching).


Test Result Summary

Component

Target

Status

Comments

BSP

Beagleboard

GOOD

Sudoku-savant project issue

Routerstationpro

GOOD

Sudoku-savant project issue

Mpc8315e-rdb

GOOD

Sudoku-savant project issue

eMenlow-sato-sdk

GOOD

Sudoku-savant project issue

Blacksand-sato

GOOD

The sdk image was not available

Crownbay-emgd-sato-sdk

GOOD

Sudoku-savant project issue

FRI2-sato-sdk

BUGGY

X Server issue blocks many test cases
Suspend issue
Dmesg issue

HuronRiver-sato-sdk

GOOD

Sudoku-savant project issue

Jasperforest-lsb-sdk

GOOD

Sudoku-savant project issue

QEMU

Qemuarm

GOOD

Everything runs well

Qemuppc

GOOD

Everything runs well

Qemumips

GOOD

Everything runs well

qemux86

GOOD

Everything runs well except for zypper bug 2694 for sato-sdk image

qemux86-64

GOOD

Everything runs well except for zypper bug 2694 for sato-sdk image

Core build system

　

BUGGY

x32 is still an issue (2888), cleaning sstate issue is still not solved (2897), incremental RPM image generation (2969), source archiving (2619), the kvm issue was reproduced by another colleague (2790), Yocto BSP creation via JSON (2693) or for qemu (2991) fails, multilib issue (2918)

HOB

　

GOOD

Everything runs well except for HOB recipe building from command line and and cross toolchain build (2695 and3001).

Compliance

GOOD

Sugarbay: Board unavailable; HuronRiver: Pass; Blacksand: Blocked

Stress

GOOD

Both Crashme and Helltest on Jasperforest could pass 24 hours stress testing

Distribution Support

GOOD

Everything runs well for Ubuntu 12.04, Ubuntu 12.10 - nightly, OpenSuse 12.1, OpenSuse 12.2 RC2, Fedora 17, CentOs 6.3, Fedora 16

Power and Performance

GOOD

one qemux86 sato build on a Core i7 machine costs 83 minutes with the packages previously fetched.

ADT

　

BUGGY

the relocatable sdk issue (2980) causes 13 test cases to be on faile/blocked state , also the Clutter C template issue is unsolved (2577) Also 2768 bug affects native autoconf



Detailed Test Result for each component

Target

Total TCs

Not Run

Passed

Failed

Not testable (Blocked)

Beagleboard Sato-SDK

42

0

41

1 (2878)

0

Routerstationpro Sato-SDK

35

0

34

1 (2878)

0

Mpc8315e-rdb Sato-SDK

37

0

36

1 (2878)

0

eMenlow Sato-SDK

67

0

66

1 (2878)

0

n450 Sato

56

0

42

1(bug 2646)

13

Crownbay-noemgdSato-SDK

67

2

64

1 (2878)

0

FRI2 Sato-SDK

86

1

42

4 (bugs 3004, 2989, 2415)

39  (3004, 2037, 2415)

HuronRiver Sato-SDK

70

0

69

1(bug 2791)

0

Jasperforest lsb-SDK

34

0

33

1 (2878)

0

Qemuarm Sato

32

0

32

0

0

Qemuarm Sato-SDK

37

0

37

0

0

Qemumips Sato

32

0

32

0

0

Qemumips Sato-SDK

37

0

37

0

0

Qemuppc Sato

32

0

32

0

0

Qemuppc Sato-SDK

37

0

37

0

0

Qemux86-64 Sato

32

0

32

0

0

Qemux86-64 Sato-SDK

37

0

37

0

0

Qemux86 Sato

32

0

32

0

0

Qemux86 Sato-SDK

37

0

37

0

0

Core build system

64

0

43

12 (2969, 2619, 2897, 2790, 2888, 2693, 2918, 3005, 2991)

9 (bugs 2888 and 2684, 2996)

HOB

38

0

36

2 (bugs 2695 and 3001)

0

ADT

53

0

23

14 (2980, 2577, 2768)

16(2980)

Compliance

3

0

2

0

1 (n-450 image not available)

Stress

2

0

2

0

0

Distribution Support

8

0

8

0

0

Total

1007

3

886

40

78


Component

Bug Number

Target Milestone

System & Core OS

System Usage

Bug 2878 pkgconfig is missing from sato-sdk image

Bug 2989 [FRI2] dmesg PowerButton error

1.3

Bug 2415 connman doesn't provide a 3g configuration utility

1.3

Bug 2037 [fri2] system cannot enter S3 standby mode

1.2

Bug 3004 [FRI2]X server fails to start

Core build system

Bug 2969 Incremental RPM image generation is not working as expected.

Bug 2619 do_package_write_rpm fails - fakeroot issue

Bug 2897 sstate-cache-management.sh doesn't work with new sstate-cache layout

Bug 2790 eglibc-initial installs to /usr

Bug 2888 x32 build fails at gcc cross-initial do_compile

Bug 2693 Yocto BSP create via JSON file fails

Bug 2918 lib64-core-image-sato-sdk fails due to do_rootfs issue.

Bug 3005 1.3 M3 build-appliance comes with old poky tree and m4-native compile fails

Bug 2991  the image created with yocto-bsp tool fails to start - "unsupported machine type"

ADT

Bug 2980 toolchain issue - segmentation fault while configuring packages

1.3 M4

Bug 2768 iptables 1.4.13 configure fail using "autoreconf -fi"

1.3 M4

Bug 2577 Segmentation fault on qemuarm from /usr/lib/libust.so.0

1.3 M4

HOB

Bug 2695 [HOB]toolchain arch in settings is not saved

1.3 M4

Bug 3001 bblayers-hob.conf does not correspond with bblayers.conf





Laurentiu Serban
QA Engineer
Open Source Technology Center
System Software Division Romania
Desk: +40 31 8604742
iNET: 88451042


[-- Attachment #2: Type: text/html, Size: 118894 bytes --]