[LSB]About lsb test result and plan

[LSB]About lsb test result and plan

[Xiaofeng Yan]

[-- Attachment #1: Type: text/plain, Size: 6767 bytes --]

Hi everyone,

This is table for lsb test result and we give a preliminary plan for lsb 
test under table.

> Target 	Test Classification 	Test Items 	Original lsb-image
> (The image without adding anything) 	Add lsb package
> Add perl-module-*
> Add eglibc-utils,localedef 	Remark
> Run Test 	Result 	Run Test 	Result
> X86 	Command Tests 	Command Check
> 4.0.0-2 	F 	No 	T 	Test Cases Total: 150
> Test Passed: 139
> Failures: 11
> Test Waived: 8 	The reason for Failures is that some commands are 
> absent in lsb-image,for example, sendmail. We can add these packages 
> including absent commands to lsb-image
> Static Interface Tests 	Library Check
> 4.0.2-1 	F 	No 	T 	Test Cases Total: 2947
> Test Passed: 1557
> Test Failed: 1390 	The reasons for Failures are below:
> 1 Some libraries are absent in lsb-image
> 2 Some functions of libraries are absent
> Runtime Interface Tests 	Core Tests 4.0.5-1 	F 	No 	F 	No 	Add 
> pam-plugin-*, e2fsprogs-mke2fs, eglibc-localedata-posix
> Run test 	Result
> T 	Test Cases Total 10593
> Tests Passed 4100
> Failures 1782
> Warnings 4
> Tests Waived 47
> FIPs 19
> Cpp-T2C Tests
> 4.0.2-1 	F 	No 	T 	Test Cases Total: 1854
> Test Passed: 1589
> Failures: 265
> Test Waived: 144 	The reason for Failures is that some file could be 
> absent in lsb test suite or perhaps some other resons. We need to 
> ensure this probleme later.
> Desktop Test
> 4.0.4-2 	F 	No 	F 	No 	Don't install lsb test suite for it
> Desktop-T2C Test
> 4.0.2-1 	F 	No 	T 	Test Cases Total: 5132
> Test Passed: 4964
> Failures: 168
> Test Waived: 49 	We need ensure the reason later
> OLVER Core Tests
> 4.0.1-1 	F 	No 	T/M
> F/G 	No 	Absent some resources needed by lsb test suite
> Perl Tests 4.0.7-1 	F 	No 	T 	Test Cases Total 853
> Tests Passed 843
> Failures 10 	Add perl-pod, perl-misc, perl-doc
> Modify perl package(add a patch)
> Run test 	Result
> T 	Tests Total 853
> Tests Passed 853
> Printing Tests
> 4.0.4-3 	F 	No 	F 	No 	Absent some resources needed by lsb test suite
> Python Running Tests
> 4.0.2-1 	F 	No 	F 	No 	Absent some resources needed by lsb test suite
> Qt3-Azov Tests
> 4.0.2-1 	F 	No 	F 	No 	Absent some resources needed by lsb test suite
> Qt4-Azov Tests
> 4.0.2-1 	F 	No 	F 	No 	Absent some resources needed by lsb test suite
> Xts5 Tests
> 5.1.5-19 	F 	No 	F 	No 	Absent some resources needed by lsb test suite
> Automated Application Battery Tests 	Apache Tests
> 2.2.8-2 	- 	- 	T 	Passed 	
> Expect Tests
> 5.43.0-7 	- 	- 	T 	Passed 	
> Groff Tests
> 1.19.2-4 	- 	- 	T 	Passed 	
> Python Tests
> 2.4.5-2 	F 	No 	T 	Test Cases Total 291
> Tests Passed 250
> Failures 4
> Warnings 1 	
> Raptor Tests
> 1.4.16-2 	- 	- 	T 	Passed 	
> Rsync Tests
> 3.0.0-2 	- 	- 	T 	Passed 	
> Samba Tests
> 3.0.28a-3 	- 	- 	T 	- 	
> Tcl Tests
> 8.5.1-2 	- 	- 	T 	Passed 	
> Xpdf Tests
> 1.01-7 	- 	- 	T 	Passed 	
> X86-64 	Command Tests 	Command Check
> 4.0.0-2 	F 	No 	T 	Test Cases Total: 150
> Test Passed: 139
> Failures: 11
> Test Waived: 8 	AS X86
> Static Interface Tests 	Library Check
> 4.0.2-1 	F 	No 	T 	Test Cases Total: 2947
> Test Passed: 1557
> Test Failed: 1390 	AS X86
> Runtime Interface Tests 	Core Tests 4.0.5-1 	F 	No 	F 	No 	We will add 
> some packages as x86 to lsb-image and test it.
> Cpp-T2C Tests
> 4.0.2-1 	F 	No 	T 	Test Cases Total: 1854
> Test Passed: 1589
> Failures: 265
> Test Waived: 144 	AS X86
> Desktop Test
> 4.0.4-2 	F 	No 	F 	No 	Absent some resources needed by lsb test suite
> Desktop-T2C Test
> 4.0.2-1 	F 	No 	T 	Test Cases Total: 5132
> Test Passed: 4963
> Failures: 169
> Test Waived: 49 	AS X86
> OLVER Core Tests
> 4.0.1-1 	F 	No 	T/M
> F/G 	No 	Absent some resources needed by lsb test suite
> Perl Tests 4.0.7-1 	F 	No 	T 	Test Cases Total 853
> Tests Passed 18
> Failures 17 	We will add some packages as x86 to lsb-image and test it.
> Printing Tests
> 4.0.4-3 	F 	No 	F 	Test Cases Excepted:97
> Tests Passed 3
> Failures 21
> Test suite error:1 	Absent cups libraries
> Python Running Tests
> 4.0.2-1 	F 	No 	F 	No 	Absent some resources needed by lsb test suite
> Qt3-Azov Tests
> 4.0.2-1 	F 	No 	F 	No 	Absent some resources needed by lsb test suite
> Qt4-Azov Tests
> 4.0.2-1 	F 	No 	F 	No 	Absent some resources needed by lsb test suite
> Xts5 Tests
> 5.1.5-19 	F 	No 	F 	No 	Absent some resources needed by lsb test suite
> Automated Application Battery Tests 	Apache Tests
> 2.2.8-2 	- 	- 	- 	- 	Don't install lsb test suite for it
> Expect Tests
> 5.43.0-7 	- 	- 	- 	- 	Don't install lsb test suite for it
> Groff Tests
> 1.19.2-4 	- 	- 	- 	- 	Don't install lsb test suite for it
> Python Tests
> 2.4.5-2 	F 	No 	T 	No 	Absent some resources needed by lsb test suite
> Raptor Tests
> 1.4.16-2 	- 	- 	- 	- 	Don't install lsb test suite for it
> Rsync Tests
> 3.0.0-2 	- 	- 	- 	- 	Don't install lsb test suite for it
> Samba Tests
> 3.0.28a-3 	- 	- 	- 	- 	Don't install lsb test suite for it
> Tcl Tests
> 8.5.1-2 	- 	- 	- 	- 	Don't install lsb test suite for it
> Xpdf Tests
> 1.01-7 	- 	- 	- 	- 	Don't install lsb test suite for it
>
> 	Note:
> Run test:
> T: LSB test suite can run completely.
> F: There are some issues that block LSB test suite itself running 
> completely.
> T/M: LSB can be tested with manual method.
> T/G: LSB can be tested with graphic method.
> Result:
> No: There is no result because LSB test suite cannot run.
>
>

It is a  preliminary analysis for lsb test. We will give more 
information later(include ppc).
A plan about how to fix bugs:
step 1: We will solve bugs which some of lsb test suite can't still run 
on lsb-image (The work is now in progress)
step 2: Make two auto script for setting up lsb test environment. One is 
for setting on pc , we want to add this script to poky/scripts; the 
other is for setting for lsb-image, we want to add this script to 
lsb-image in the mode of bb file.  (sync up with step 1)
step 3: Write a user manual about how to test lsb-imgae by lsb test 
suite and we want to this user manual to lsb-image (after step 2)
step 4: List bugs tested by lsb test suite(sync up with step 1)
step 5: Solve bugs tested by lsb test suite(everyone can attend to fix bugs)
Method for fixing bugs:
- We will fix bugs for lsb test suite itself on platform x86 firstly and 
then apply the result to  other platform when step 1 is over.
- We will fix bugs tested by lsb test suite on platform x86 and then 
apply the result to other platform.

We will open a bugzilla for failed items tested by lsb test suite and 
make a milestone for fixing bugs with detailed time plan on the next week.
Please give your suggestion.

Thanks,
Yan


[-- Attachment #2: Type: text/html, Size: 109471 bytes --]