[LTP] Yocto 1.0 M2 assessment -- DONE

[LTP] Yocto 1.0 M2 assessment -- DONE

[Tian, Kevin]

[-- Attachment #1.1: Type: text/plain, Size: 2020 bytes --]

Hi, all,



I'd like to share the current LTP status, that I've basically done the assessment for

all the failures except routerstation (2.6.34 kernel has some IPC issues which causes

lots of unexpected failures).

https://wiki.yoctoproject.org/wiki/LTP_result#Current_Milestone_.281.0.2FM2.29



Total failures are 271 if setting aside routerstation:



[cid:image002.png@01CBC0A0.CFF918E0]



Out of 271 there are quite some happening on multiple targets. So after removing the

duplicated failures, I managed to audit totally 51 failures, within them:



-          7 failures are fixed (5 are bug fixes, 2 are fixed in 2.6.37 kernel)

http://bugzilla.pokylinux.org/show_bug.cgi?id=672 [LTP]PATH contain ending colon for root user

http://bugzilla.pokylinux.org/show_bug.cgi?id=673 [LTP]normal user can't use cron

http://bugzilla.pokylinux.org/show_bug.cgi?id=674 [LTP]bash is not default /bin/sh in lsb profile

http://bugzilla.pokylinux.org/show_bug.cgi?id=675 [LTP]normal user can't use at

http://bugzilla.pokylinux.org/show_bug.cgi?id=676 [LTP]no 'file' and 'ldd' in sdk profile

-          5 failures are identified as upstream LTP bugs

-          3 failures are pending for further solution

http://bugzilla.pokylinux.org/show_bug.cgi?id=678 [LTP] fallocate always gets -EFBIG on qemuarm/qemumips

-          The majority (36) failures are categorized as Not-A-Bug, which mostly comes from

the fact that SDK profile doesn't include necessary packages for test, and then are

verified PASS by adding them back with necessary tweaking. There're also a few

coming from lacking of memory/disk resources, which will be avoided in the next test.



Qemux86 is the base target which I spent most time. So for most failure analysis you can look at:

https://wiki.yoctoproject.org/wiki/Qemux86-ltp



Qemuarm contains 2nd most fresh analysis, which you may be also interested:

https://wiki.yoctoproject.org/wiki/Qemuarm-ltp



Thanks

Kevin

[-- Attachment #1.2: Type: text/html, Size: 10536 bytes --]

[-- Attachment #2: image002.png --]
[-- Type: image/png, Size: 25998 bytes --]

Re: [LTP] Yocto 1.0 M2 assessment -- DONE

[Saul Wold]

On 01/30/2011 01:12 AM, Tian, Kevin wrote:
> Hi, all,
>
> I'd like to share the current LTP status, that I've basically done the
> assessment for
>
> all the failures except routerstation (2.6.34 kernel has some IPC issues
> which causes
>
> lots of unexpected failures).
>
Kevin,

Thanks for the excellent assesment, as we move forward, I will consult 
with you about the contents of the images.  Is there a reason you chose 
the SDK image over the LSB image?

As you may have heard, we are working on creating sets of images that 
will meet various developers.  Soon, there will be an LSB-dev and 
LSD-sdk set of images that will contain headers/libraries and then 
headers/libraries and developer tools in LSB.  We might want to consider 
testing with the lsb-sdk image in the future since it will have some of 
the missing things that you have tested.

Thoughts?

	Sau!


> https://wiki.yoctoproject.org/wiki/LTP_result#Current_Milestone_.281.0.2FM2.29
>
> Total failures are *271* if setting aside routerstation:
>
> Out of *271* there are quite some happening on multiple targets. So
> after removing the
>
> duplicated failures, I managed to audit totally *51* failures, within them:
>
> - 7 failures are fixed (5 are bug fixes, 2 are fixed in 2.6.37 kernel)
>
> http://bugzilla.pokylinux.org/show_bug.cgi?id=672 [LTP]PATH contain
> ending colon for root user
>
> http://bugzilla.pokylinux.org/show_bug.cgi?id=673 [LTP]normal user can't
> use cron
>
> http://bugzilla.pokylinux.org/show_bug.cgi?id=674 [LTP]bash is not
> default /bin/sh in lsb profile
>
> http://bugzilla.pokylinux.org/show_bug.cgi?id=675 [LTP]normal user can't
> use at
>
> http://bugzilla.pokylinux.org/show_bug.cgi?id=676 [LTP]no 'file' and
> 'ldd' in sdk profile
>
> - 5 failures are identified as upstream LTP bugs
>
> - 3 failures are pending for further solution
>
> http://bugzilla.pokylinux.org/show_bug.cgi?id=678 [LTP] fallocate always
> gets -EFBIG on qemuarm/qemumips
>
> - The majority (36) failures are categorized as Not-A-Bug, which mostly
> comes from
>
> the fact that SDK profile doesn’t include necessary packages for test,
> and then are
>
> verified PASS by adding them back with necessary tweaking. There’re also
> a few
>
> coming from lacking of memory/disk resources, which will be avoided in
> the next test.
>
Do you have the list os packages that you had to add?  I do not see a 
summary of that list.

Again thanks for this great effort, and I look forward to the POSIX 
evaluation at somepoint in the near future.

Sau!

> Qemux86 is the base target which I spent most time. So for most failure
> analysis you can look at:
>
> https://wiki.yoctoproject.org/wiki/Qemux86-ltp
>
> Qemuarm contains 2^nd most fresh analysis, which you may be also interested:
>
> https://wiki.yoctoproject.org/wiki/Qemuarm-ltp
>
> Thanks
>
> Kevin
>
>
>
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: [LTP] Yocto 1.0 M2 assessment -- DONE

[Tian, Kevin]

> From: Saul Wold
> Sent: Monday, January 31, 2011 4:41 AM
> 
> On 01/30/2011 01:12 AM, Tian, Kevin wrote:
> > Hi, all,
> >
> > I'd like to share the current LTP status, that I've basically done the
> > assessment for
> >
> > all the failures except routerstation (2.6.34 kernel has some IPC issues
> > which causes
> >
> > lots of unexpected failures).
> >
> Kevin,
> 
> Thanks for the excellent assesment, as we move forward, I will consult
> with you about the contents of the images.  Is there a reason you chose
> the SDK image over the LSB image?

I talked with Jiajun before. The major reason using SDK is to reduce the QA effort,
because LSB image doesn't run on real hardware, and doesn't contain enough
development tools for openposix test.

But regarding to the fact that I still need manually run LSB images to verify most
failures, it's perhaps a good idea to ask QA to use LSB for Qemu LTP test, and
then use SDK for real-hw LTP and all posix tests. This does add some extra effort
but that's what we have to do in current stage.

> 
> As you may have heard, we are working on creating sets of images that
> will meet various developers.  Soon, there will be an LSB-dev and
> LSD-sdk set of images that will contain headers/libraries and then
> headers/libraries and developer tools in LSB.  We might want to consider
> testing with the lsb-sdk image in the future since it will have some of
> the missing things that you have tested.
> 
> Thoughts?

That's great. If this can be ready in M3, we can move to it to save more time.
Also it's better to have LSB running on real-hw which further unifies the effort.  :-)

> > - The majority (36) failures are categorized as Not-A-Bug, which mostly
> > comes from
> >
> > the fact that SDK profile doesn't include necessary packages for test,
> > and then are
> >
> > verified PASS by adding them back with necessary tweaking. There're also
> > a few
> >
> > coming from lacking of memory/disk resources, which will be avoided in
> > the next test.
> >
> Do you have the list os packages that you had to add?  I do not see a
> summary of that list.

All the packages I have to add are contained in LSB profile already. The only
missing bits, as you said earlier, is the development part and graphics part.

Of course even with LSB profile there're some errors such as cron/at, which
have been fixed in earlier bug lists.

> 
> Again thanks for this great effort, and I look forward to the POSIX
> evaluation at somepoint in the near future.
> 

POSIX is a bit lagging behind, because I want to focus on LTP first. So this work
will extend into M4 a bit. But I hope to have a major break before CNY, if there
may be one reason for the majority of the failures. :-)

Thanks
Kevin