[PATCH 00/14] testimage: restore 'auto' behaviour and a few other fixes

[PATCH 00/14] testimage: restore 'auto' behaviour and a few other fixes

[Chen Qi]

The following changes since commit 719d068bde55ef29a3468bc0779d4cb0c11e8c1d:

  bitbake: fetch2: fix import error for Python 3.6.5 (2018-05-29 21:07:46 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib ChenQi/testimage-auto
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/testimage-auto

Chen Qi (14):
  oeqa/core/loader.py: support the 'auto' keyword
  testimage.bbclass: fix behavior of empty TEST_SUITES
  oeqa/core/decorator/__init__.py: set metaclass to ABCMeta
  oeqa/core/decorator/__init__.py: use 'cls' instead of 'obj'
  oeqa/runtime/cases/dnf_runtime.py: skip test if PACKAGE_FEED_URIS is
    not set
  oeqa/runtime/cases/selftest.py: rename to _selftest.py
  testimage.bbclass: also check 'auto' to create rpm index
  oeqa/core/decorator/data.py: fix skipIfNotInDataVar
  oeqa/runtime/cases/rpm.py: skip if rpm not available
  oeqa/runtime/cases/multilib.py: skip if needed packages are not
    available
  oeqa/runtime/cases/multilib.py: fix test_file_connman skipping logic
  bitbake.conf: fix HOSTTOOLS setting related to image testing
  testimage-auto.bbclass: execute the same codes as testimage task
  oeqa/core/target/ssh.py: increase maximum read bytes from 1024 to 4096

 .../oeqa/runtime/cases/{selftest.py => _selftest.py}    |  0
 meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py     |  3 +++
 meta/classes/testimage-auto.bbclass                     |  6 ++++++
 meta/classes/testimage.bbclass                          | 10 ++++++++--
 meta/conf/bitbake.conf                                  |  2 +-
 meta/lib/oeqa/core/decorator/__init__.py                | 10 +++++-----
 meta/lib/oeqa/core/decorator/data.py                    |  4 ++--
 meta/lib/oeqa/core/loader.py                            | 17 +++++++++++++----
 meta/lib/oeqa/core/target/ssh.py                        |  2 +-
 meta/lib/oeqa/runtime/cases/multilib.py                 |  4 +++-
 meta/lib/oeqa/runtime/cases/rpm.py                      |  1 +
 meta/lib/oeqa/selftest/cases/runtime_test.py            |  2 +-
 12 files changed, 44 insertions(+), 17 deletions(-)
 rename meta-selftest/lib/oeqa/runtime/cases/{selftest.py => _selftest.py} (100%)

-- 
1.9.1

[PATCH 01/14] oeqa/core/loader.py: support the 'auto' keyword

[Chen Qi]

In previous OEQA, having 'auto' in TEST_SUITES results in executing
as many test cases as possible.

This behaviour is broken for now. From the codes in core/loader.py,
I can see that it tries to use another keyword 'all'. But in fact,
it does not work.

I've checked the current manual. The manual says using 'auto'.
Below is the current information in manual.

  """
  Alternatively, you can provide the "auto" option to have all applicable
  tests run against the image.

  TEST_SUITES_append = " auto"
  """

So we should restore this behaviour. This patch does so.

Also, output warning message is some module is named as 'auto', as this
is a reserved keyword.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/lib/oeqa/core/loader.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oeqa/core/loader.py b/meta/lib/oeqa/core/loader.py
index a4744de..98fc0f6 100644
--- a/meta/lib/oeqa/core/loader.py
+++ b/meta/lib/oeqa/core/loader.py
@@ -155,7 +155,16 @@ class OETestLoader(unittest.TestLoader):
         class_name = case.__class__.__name__
         test_name = case._testMethodName
 
-        if self.modules:
+        # 'auto' is a reserved key word to run test cases automatically
+        # warn users if their test case belong to a module named 'auto'
+        if module_name_small == "auto":
+            bb.warn("'auto' is a reserved key word for TEST_SUITES. "
+                    "But test case '%s' is detected to belong to auto module. "
+                    "Please condier using a new name for your module." % str(case))
+
+        # check if case belongs to any specified module
+        # if 'auto' is specified, such check is skipped
+        if self.modules and not 'auto' in self.modules:
             module = None
             try:
                 module = self.modules[module_name_small]
@@ -245,7 +254,7 @@ class OETestLoader(unittest.TestLoader):
         for tcName in testCaseNames:
             case = self._getTestCase(testCaseClass, tcName)
             # Filer by case id
-            if not (self.tests and not 'all' in self.tests
+            if not (self.tests and not 'auto' in self.tests
                     and not getCaseID(case) in self.tests):
                 self._handleTestCaseDecorators(case)
 
@@ -309,14 +318,14 @@ class OETestLoader(unittest.TestLoader):
         module_name = module.__name__
 
         # Normal test modules are loaded if no modules were specified,
-        # if module is in the specified module list or if 'all' is in
+        # if module is in the specified module list or if 'auto' is in
         # module list.
         # Underscore modules are loaded only if specified in module list.
         load_module = True if not module_name.startswith('_') \
                               and (not self.modules \
                                    or module_name in self.modules \
                                    or module_name_small in self.modules \
-                                   or 'all' in self.modules) \
+                                   or 'auto' in self.modules) \
                            else False
 
         load_underscore = True if module_name.startswith('_') \
-- 
1.9.1

Re: [PATCH 01/14] oeqa/core/loader.py: support the 'auto' keyword

[Alexander Kanavin]

2018-05-31 11:32 GMT+03:00 Chen Qi <Qi.Chen@windriver.com>:
> In previous OEQA, having 'auto' in TEST_SUITES results in executing
> as many test cases as possible.
>
> This behaviour is broken for now. From the codes in core/loader.py,
> I can see that it tries to use another keyword 'all'. But in fact,
> it does not work.
>
> I've checked the current manual. The manual says using 'auto'.
> Below is the current information in manual.
>
>   """
>   Alternatively, you can provide the "auto" option to have all applicable
>   tests run against the image.
>
>   TEST_SUITES_append = " auto"
>   """
>
> So we should restore this behaviour. This patch does so.

Nice! Can we also try to simplify the setting of DEFAULT_TEST_SUITES
in testimage.bbclass using this keyword? Right now it's basically
manually created lists, which tend to go out of date.

Alex

Re: [PATCH 01/14] oeqa/core/loader.py: support the 'auto' keyword

[ChenQi]

On 05/31/2018 09:57 PM, Alexander Kanavin wrote:
> 2018-05-31 11:32 GMT+03:00 Chen Qi <Qi.Chen@windriver.com>:
>> In previous OEQA, having 'auto' in TEST_SUITES results in executing
>> as many test cases as possible.
>>
>> This behaviour is broken for now. From the codes in core/loader.py,
>> I can see that it tries to use another keyword 'all'. But in fact,
>> it does not work.
>>
>> I've checked the current manual. The manual says using 'auto'.
>> Below is the current information in manual.
>>
>>    """
>>    Alternatively, you can provide the "auto" option to have all applicable
>>    tests run against the image.
>>
>>    TEST_SUITES_append = " auto"
>>    """
>>
>> So we should restore this behaviour. This patch does so.
> Nice! Can we also try to simplify the setting of DEFAULT_TEST_SUITES
> in testimage.bbclass using this keyword? Right now it's basically
> manually created lists, which tend to go out of date.
>
> Alex
>

Hi Alex,

I agree with you that we need to re-evaluate the settings of 
DEFAULT_TEST_SUITES.

I plan to do this work from next week. And I plan to put such changes in 
another patchset.

Looking at the codes, I can see that the DEFAULT_TEST_SUITES are set 
with consideration of qemu machines and libc.
e.g.
# aarch64 has no graphics
DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"
# musl doesn't support systemtap
DEFAULT_TEST_SUITES_remove_libc-musl = "stap"

And I guess this work would take some time.

Back to this patchset, I'll sent out V2 soon.

Best Regards,
Chen Qi

Re: [PATCH 01/14] oeqa/core/loader.py: support the 'auto' keyword

[Alexander Kanavin]

2018-06-01 7:42 GMT+03:00 ChenQi <Qi.Chen@windriver.com>:
> I agree with you that we need to re-evaluate the settings of
> DEFAULT_TEST_SUITES.
>
> I plan to do this work from next week. And I plan to put such changes in
> another patchset.
>
> Looking at the codes, I can see that the DEFAULT_TEST_SUITES are set with
> consideration of qemu machines and libc.
> e.g.
> # aarch64 has no graphics
> DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"
> # musl doesn't support systemtap
> DEFAULT_TEST_SUITES_remove_libc-musl = "stap"

I suppose we can introduce TEST_SUITES_EXCLUDE for such cases, which
would be considered when 'auto' is expanded to a specific list?

Alex

Re: [PATCH 01/14] oeqa/core/loader.py: support the 'auto' keyword

[ChenQi]

On 06/01/2018 01:41 PM, Alexander Kanavin wrote:
> 2018-06-01 7:42 GMT+03:00 ChenQi <Qi.Chen@windriver.com>:
>> I agree with you that we need to re-evaluate the settings of
>> DEFAULT_TEST_SUITES.
>>
>> I plan to do this work from next week. And I plan to put such changes in
>> another patchset.
>>
>> Looking at the codes, I can see that the DEFAULT_TEST_SUITES are set with
>> consideration of qemu machines and libc.
>> e.g.
>> # aarch64 has no graphics
>> DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"
>> # musl doesn't support systemtap
>> DEFAULT_TEST_SUITES_remove_libc-musl = "stap"
> I suppose we can introduce TEST_SUITES_EXCLUDE for such cases, which
> would be considered when 'auto' is expanded to a specific list?
>
> Alex
>
I don't know yet. I guess if test case is written with more 
sophisticated considerations, we don't need such variable. But I'm not 
sure about it. I'll start to investigate it from next week.

Best Regards,
Chen Qi

Re: [PATCH 01/14] oeqa/core/loader.py: support the 'auto' keyword

[Richard Purdie]

On Fri, 2018-06-01 at 13:58 +0800, ChenQi wrote:
> On 06/01/2018 01:41 PM, Alexander Kanavin wrote:
> > 2018-06-01 7:42 GMT+03:00 ChenQi <Qi.Chen@windriver.com>:
> > > I agree with you that we need to re-evaluate the settings of
> > > DEFAULT_TEST_SUITES.
> > > 
> > > I plan to do this work from next week. And I plan to put such
> > > changes in
> > > another patchset.
> > > 
> > > Looking at the codes, I can see that the DEFAULT_TEST_SUITES are
> > > set with
> > > consideration of qemu machines and libc.
> > > e.g.
> > > # aarch64 has no graphics
> > > DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"
> > > # musl doesn't support systemtap
> > > DEFAULT_TEST_SUITES_remove_libc-musl = "stap"
> > 
> > I suppose we can introduce TEST_SUITES_EXCLUDE for such cases,
> > which
> > would be considered when 'auto' is expanded to a specific list?
> 
> I don't know yet. I guess if test case is written with more 
> sophisticated considerations, we don't need such variable. But I'm
> not sure about it. I'll start to investigate it from next week.

This patchset looks good, thanks for working on it.

The original intent of some of these values was to indicate which tests
were expected to pass for which images. I think in 2.6 we have some
people working on collecting up the test results for a given build,
then having a comparison tool which compares that set of results with a
list of tests we expect to pass.

The desire here is to detect when something that should get tested is
no longer getting tested because the test isn't running.

I'm pretty sure the existing mechanism isn't fit for purpose though.

I also want to change some things like getting rid of testimage_auto
and changing that to be configurable with a variable. I'd noticed it
was totally broken so I doubt anyone can have been using it...

Cheers,

Richard

[PATCH 02/14] testimage.bbclass: fix behavior of empty TEST_SUITES

[Chen Qi]

The current behaviour of TEST_SUITES is very confusing.

setting: TEST_SUITES = ""
result:  Execute all test cases.

setting: TEST_SUITES = "some_case_not_exist"
result:  Error out with 'Empty test suite' message.

The expected behaviour of TEST_SUITES should be:
1. when 'auto' is in it, execute as many test cases as possible
2. when no valid test case is specified, error out and give user message

The first one is implemented by a previous patch.
The second one is fixed in this patch.

Also add debug message to show test cases to be executed. This is
for easier debugging.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/testimage.bbclass | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index bb688b0..d97da67 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -260,10 +260,16 @@ def testimage_main(d):
     # Load tests before starting the target
     test_paths = get_runtime_paths(d)
     test_modules = d.getVar('TEST_SUITES').split()
+    if not test_modules:
+        bb.fatal('Empty test suite, please verify TEST_SUITES variable')
+
     tc.loadTests(test_paths, modules=test_modules)
 
-    if not getSuiteCases(tc.suites):
+    suitecases = getSuiteCases(tc.suites)
+    if not suitecases:
         bb.fatal('Empty test suite, please verify TEST_SUITES variable')
+    else:
+        bb.debug(2, 'test suites:\n\t%s' % '\n\t'.join([str(c) for c in suitecases]))
 
     package_extraction(d, tc.suites)
 
-- 
1.9.1

[PATCH 03/14] oeqa/core/decorator/__init__.py: set metaclass to ABCMeta

[Chen Qi]

OETestFilter is a subclass of OETestDecorator. It wants to make
use of @abstractmethod decorator. But such decorator requires
metaclass to be ABCMeta to have effect. So add it now to achieve
the designed behaviour.

Comments from python's manual:
"""
Using this decorator requires that the class's metaclass is ABCMeta
or is derived from it.
"""

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/lib/oeqa/core/decorator/__init__.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/core/decorator/__init__.py b/meta/lib/oeqa/core/decorator/__init__.py
index 855b6b9..6ca0acf 100644
--- a/meta/lib/oeqa/core/decorator/__init__.py
+++ b/meta/lib/oeqa/core/decorator/__init__.py
@@ -2,7 +2,7 @@
 # Released under the MIT license (see COPYING.MIT)
 
 from functools import wraps
-from abc import abstractmethod
+from abc import abstractmethod, ABCMeta
 
 decoratorClasses = set()
 
@@ -10,7 +10,7 @@ def registerDecorator(obj):
     decoratorClasses.add(obj)
     return obj
 
-class OETestDecorator(object):
+class OETestDecorator(object, metaclass=ABCMeta):
     case = None # Reference of OETestCase decorated
     attrs = None # Attributes to be loaded by decorator implementation
 
-- 
1.9.1

[PATCH 04/14] oeqa/core/decorator/__init__.py: use 'cls' instead of 'obj'

[Chen Qi]

Use 'cls' instead of 'obj' to better reflect that registerDecorator
actually serves as a class decorator.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/lib/oeqa/core/decorator/__init__.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/core/decorator/__init__.py b/meta/lib/oeqa/core/decorator/__init__.py
index 6ca0acf..14d7bfc 100644
--- a/meta/lib/oeqa/core/decorator/__init__.py
+++ b/meta/lib/oeqa/core/decorator/__init__.py
@@ -6,9 +6,9 @@ from abc import abstractmethod, ABCMeta
 
 decoratorClasses = set()
 
-def registerDecorator(obj):
-    decoratorClasses.add(obj)
-    return obj
+def registerDecorator(cls):
+    decoratorClasses.add(cls)
+    return cls
 
 class OETestDecorator(object, metaclass=ABCMeta):
     case = None # Reference of OETestCase decorated
-- 
1.9.1

[PATCH 05/14] oeqa/runtime/cases/dnf_runtime.py: skip test if PACKAGE_FEED_URIS is not set

[Chen Qi]

This test is to test the behaviour of PACKAGE_FEED_URIS is correct or not.
If it's not even set, it makes no sense to do such test. So skip this
test if PACKAGE_FEED_URIS is not set.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
index 81c50ed..1aa7274 100644
--- a/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
+++ b/meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py
@@ -1,6 +1,7 @@
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.runtime.cases.dnf import DnfTest
 from oeqa.utils.httpserver import HTTPService
+from oeqa.core.decorator.data import skipIfDataVar
 
 class DnfSelftest(DnfTest):
 
@@ -18,6 +19,8 @@ class DnfSelftest(DnfTest):
         cls.temp_dir.cleanup()
 
     @OETestDepends(['dnf.DnfBasicTest.test_dnf_help'])
+    @skipIfDataVar('PACKAGE_FEED_URIS', None,
+                   'Not suitable as PACKAGE_FEED_URIS is not set')
     def test_verify_package_feeds(self):
         """
         Summary: Check correct setting of PACKAGE_FEED_URIS var
-- 
1.9.1

[PATCH 06/14] oeqa/runtime/cases/selftest.py: rename to _selftest.py

[Chen Qi]

This test modules is designed to be invoked only by selftest. It's
not meant to be tested by normal runtime test. So it should be renamed
with '_' prefix, so that it will not be automatically loaded by normal
runtime tests when 'auto' is in TEST_SUITES.

The failure message is as below.

  RESULTS - selftest.Selftest.test_install_package - Testcase -1: FAILED

Also, change selftest/cases/runtime_test.py to use '_selftest' accordingly.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta-selftest/lib/oeqa/runtime/cases/{selftest.py => _selftest.py} | 0
 meta/lib/oeqa/selftest/cases/runtime_test.py                       | 2 +-
 2 files changed, 1 insertion(+), 1 deletion(-)
 rename meta-selftest/lib/oeqa/runtime/cases/{selftest.py => _selftest.py} (100%)

diff --git a/meta-selftest/lib/oeqa/runtime/cases/selftest.py b/meta-selftest/lib/oeqa/runtime/cases/_selftest.py
similarity index 100%
rename from meta-selftest/lib/oeqa/runtime/cases/selftest.py
rename to meta-selftest/lib/oeqa/runtime/cases/_selftest.py
diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py
index 9c9b4b3..def3819 100644
--- a/meta/lib/oeqa/selftest/cases/runtime_test.py
+++ b/meta/lib/oeqa/selftest/cases/runtime_test.py
@@ -122,7 +122,7 @@ class TestImage(OESelftestTestCase):
             self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
 
         features = 'INHERIT += "testimage"\n'
-        features += 'TEST_SUITES = "ping ssh selftest"\n'
+        features += 'TEST_SUITES = "ping ssh _selftest"\n'
         self.write_config(features)
 
         # Build core-image-sato and testimage
-- 
1.9.1

[PATCH 07/14] testimage.bbclass: also check 'auto' to create rpm index

[Chen Qi]

Having 'auto' in TEST_SUITES will also run the 'dnf' test cases,
so also check it to determine whether to create rpm index or not.

This is to fix the following error when TEST_SUITES = "auto".

  RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/testimage.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index d97da67..14252ff 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -121,7 +121,7 @@ python do_testimage() {
     testimage_sanity(d)
 
     if (d.getVar('IMAGE_PKGTYPE') == 'rpm'
-       and 'dnf' in d.getVar('TEST_SUITES')):
+       and ('dnf' in d.getVar('TEST_SUITES') or 'auto' in d.getVar('TEST_SUITES'))):
         create_rpm_index(d)
 
     testimage_main(d)
-- 
1.9.1

[PATCH 08/14] oeqa/core/decorator/data.py: fix skipIfNotInDataVar

[Chen Qi]

The var might not be set, resulting in unexpected error.

  RESULTS - multilib.MultilibTest.test_check_multilib_libc - Testcase 1593: ERROR

The above error is due to MULTILIBS being not set, which is the default
for OE. This patch fixes this problem.

Also, the debugging message in skipIfNotInDataVar is currently confusing.
Instead of
DEBUG: Checking if 'MULTILIBS' value is in 'multilib:lib32' to run the test
it should be
DEBUG: Checking if 'MULTILIBS' value contains 'multilib:lib32' to run the test

This patch also fixes it.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/lib/oeqa/core/decorator/data.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/core/decorator/data.py b/meta/lib/oeqa/core/decorator/data.py
index ff7bdd9..31c6dd6 100644
--- a/meta/lib/oeqa/core/decorator/data.py
+++ b/meta/lib/oeqa/core/decorator/data.py
@@ -61,10 +61,10 @@ class skipIfNotInDataVar(OETestDecorator):
 
     attrs = ('var', 'value', 'msg')
     def setUpDecorator(self):
-        msg = ('Checking if %r value is in %r to run '
+        msg = ('Checking if %r value contains %r to run '
               'the test' % (self.var, self.value))
         self.logger.debug(msg)
-        if not self.value in self.case.td.get(self.var):
+        if not self.value in (self.case.td.get(self.var) or ""):
             self.case.skipTest(self.msg)
 
 @registerDecorator
-- 
1.9.1

[PATCH 09/14] oeqa/runtime/cases/rpm.py: skip if rpm not available

[Chen Qi]

This test case should only run when rpm package is installed.
So skip it if rpm package is not installed. This fixes:

  RESULTS - rpm.RpmBasicTest.test_rpm_help - Testcase 1059: FAILED

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/lib/oeqa/runtime/cases/rpm.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/lib/oeqa/runtime/cases/rpm.py b/meta/lib/oeqa/runtime/cases/rpm.py
index 05b94c7..84c59a6 100644
--- a/meta/lib/oeqa/runtime/cases/rpm.py
+++ b/meta/lib/oeqa/runtime/cases/rpm.py
@@ -16,6 +16,7 @@ class RpmBasicTest(OERuntimeTestCase):
             cls.skipTest('Tests require image to be build from rpm')
 
     @OETestID(960)
+    @OEHasPackage(['rpm'])
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     def test_rpm_help(self):
         status, output = self.target.run('rpm --help')
-- 
1.9.1

[PATCH 10/14] oeqa/runtime/cases/multilib.py: skip if needed packages are not available

[Chen Qi]

1) The test cases use 'readelf' command to do the check. This command
   is from binutils. So skip the test if the needed binutils package is
   not installed.

   The related error message in log.do_testimage is like below.

     Output:  sh: readelf: not found

2) The test case tests /lib/libc.so.6 from lib32-libc6. So skip the test
   if lib32-libc6 is not installed.

   The related error message in log.do_testimage is like below.

     Output:   readelf: Error: 'lib/libc.so.6': No such file

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/lib/oeqa/runtime/cases/multilib.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/lib/oeqa/runtime/cases/multilib.py b/meta/lib/oeqa/runtime/cases/multilib.py
index 8c167f1..970f676 100644
--- a/meta/lib/oeqa/runtime/cases/multilib.py
+++ b/meta/lib/oeqa/runtime/cases/multilib.py
@@ -27,6 +27,8 @@ class MultilibTest(OERuntimeTestCase):
     @skipIfNotInDataVar('MULTILIBS', 'multilib:lib32',
                         "This isn't a multilib:lib32 image")
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['binutils'])
+    @OEHasPackage(['lib32-libc6'])
     def test_check_multilib_libc(self):
         """
         Check that a multilib image has both 32-bit and 64-bit libc in.
-- 
1.9.1

[PATCH 11/14] oeqa/runtime/cases/multilib.py: fix test_file_connman skipping logic

[Chen Qi]

The test_file_connman should be executed only when 'lib32-connman' is
installed and 'connman' is not installed.

When lib32-connman and connman are both installed, the /usr/sbin/connmand
could be from connman or lib32-connman, depending on the installation
order. What we want to check is the connmand command from lib32-connman,
so we need to make sure that connman is not there to cause chaos.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/lib/oeqa/runtime/cases/multilib.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/runtime/cases/multilib.py b/meta/lib/oeqa/runtime/cases/multilib.py
index 970f676..8902038 100644
--- a/meta/lib/oeqa/runtime/cases/multilib.py
+++ b/meta/lib/oeqa/runtime/cases/multilib.py
@@ -38,6 +38,6 @@ class MultilibTest(OERuntimeTestCase):
 
     @OETestID(279)
     @OETestDepends(['multilib.MultilibTest.test_check_multilib_libc'])
-    @OEHasPackage(['lib32-connman'])
+    @OEHasPackage(['lib32-connman', '!connman'])
     def test_file_connman(self):
         self.archtest("/usr/sbin/connmand", "ELF32")
-- 
1.9.1

[PATCH 12/14] bitbake.conf: fix HOSTTOOLS setting related to image testing

[Chen Qi]

A list of tools are added to HOSTTOOLS depending on if we inherit
testimage or not. Unfortunately, if we use TEST_IMAGE variable to
automate the test, these tools are not added to HOSTTOOLS.

Modify the condition to also check TEST_IMAGE to fix the above problem.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 6ecf78b..7b7472c 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -487,7 +487,7 @@ HOSTTOOLS += " \
 "
 
 # Tools needed to run testimage runtime image testing
-HOSTTOOLS += "${@['', 'ip ping ps scp ssh stty'][bb.data.inherits_class('testimage', d)]}"
+HOSTTOOLS += "${@['', 'ip ping ps scp ssh stty'][bb.data.inherits_class('testimage', d) or d.getVar('TEST_IMAGE') == '1']}"
 
 # Link to these if present
 HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc sftp socat ssh sudo"
-- 
1.9.1

Re: [PATCH 12/14] bitbake.conf: fix HOSTTOOLS setting related to image testing

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1665 bytes --]

While you're at it, I'd suggest fixing this to stop using the horrible
indexing a list with a boolean hack and use a proper inline conditional.
'foo' if bar else 'baz'.

On Thu, May 31, 2018 at 1:43 AM Chen Qi <Qi.Chen@windriver.com> wrote:

> A list of tools are added to HOSTTOOLS depending on if we inherit
> testimage or not. Unfortunately, if we use TEST_IMAGE variable to
> automate the test, these tools are not added to HOSTTOOLS.
>
> Modify the condition to also check TEST_IMAGE to fix the above problem.
>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/conf/bitbake.conf | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 6ecf78b..7b7472c 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -487,7 +487,7 @@ HOSTTOOLS += " \
>  "
>
>  # Tools needed to run testimage runtime image testing
> -HOSTTOOLS += "${@['', 'ip ping ps scp ssh
> stty'][bb.data.inherits_class('testimage', d)]}"
> +HOSTTOOLS += "${@['', 'ip ping ps scp ssh
> stty'][bb.data.inherits_class('testimage', d) or d.getVar('TEST_IMAGE') ==
> '1']}"
>
>  # Link to these if present
>  HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc sftp socat
> ssh sudo"
> --
> 1.9.1
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>


-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2505 bytes --]

Re: [PATCH 12/14] bitbake.conf: fix HOSTTOOLS setting related to image testing

[ChenQi]

[-- Attachment #1: Type: text/plain, Size: 2059 bytes --]

Done.
V2 will be sent out soon.

Thanks,
Chen Qi

On 05/31/2018 10:57 PM, Christopher Larson wrote:
> While you're at it, I'd suggest fixing this to stop using the horrible 
> indexing a list with a boolean hack and use a proper inline 
> conditional. 'foo' if bar else 'baz'.
>
> On Thu, May 31, 2018 at 1:43 AM Chen Qi <Qi.Chen@windriver.com 
> <mailto:Qi.Chen@windriver.com>> wrote:
>
>     A list of tools are added to HOSTTOOLS depending on if we inherit
>     testimage or not. Unfortunately, if we use TEST_IMAGE variable to
>     automate the test, these tools are not added to HOSTTOOLS.
>
>     Modify the condition to also check TEST_IMAGE to fix the above
>     problem.
>
>     Signed-off-by: Chen Qi <Qi.Chen@windriver.com
>     <mailto:Qi.Chen@windriver.com>>
>     ---
>      meta/conf/bitbake.conf | 2 +-
>      1 file changed, 1 insertion(+), 1 deletion(-)
>
>     diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>     index 6ecf78b..7b7472c 100644
>     --- a/meta/conf/bitbake.conf
>     +++ b/meta/conf/bitbake.conf
>     @@ -487,7 +487,7 @@ HOSTTOOLS += " \
>      "
>
>      # Tools needed to run testimage runtime image testing
>     -HOSTTOOLS += "${@['', 'ip ping ps scp ssh
>     stty'][bb.data.inherits_class('testimage', d)]}"
>     +HOSTTOOLS += "${@['', 'ip ping ps scp ssh
>     stty'][bb.data.inherits_class('testimage', d) or
>     d.getVar('TEST_IMAGE') == '1']}"
>
>      # Link to these if present
>      HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc
>     sftp socat ssh sudo"
>     -- 
>     1.9.1
>
>     -- 
>     _______________________________________________
>     Openembedded-core mailing list
>     Openembedded-core@lists.openembedded.org
>     <mailto:Openembedded-core@lists.openembedded.org>
>     http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>
>
> -- 
> Christopher Larson
> kergoth at gmail dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Senior Software Engineer, Mentor Graphics



[-- Attachment #2: Type: text/html, Size: 4088 bytes --]

[PATCH 13/14] testimage-auto.bbclass: execute the same codes as testimage task

[Chen Qi]

testimage-auto is expected to run testimage task's codes automatically.
But in fact, it's currently missing some codes, including testimage_sanity
and create_rpm_index.

This leads to the problem of unexpected runtime failure of test_dnf_makecache.
The error message is as below.

  RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR

This error is caused by the fact that create_rpm_index is not executed
before running the tests.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/testimage-auto.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/testimage-auto.bbclass b/meta/classes/testimage-auto.bbclass
index e0a22b7..7da516a 100644
--- a/meta/classes/testimage-auto.bbclass
+++ b/meta/classes/testimage-auto.bbclass
@@ -16,6 +16,12 @@
 inherit testimage
 
 python do_testimage_auto() {
+    testimage_sanity(d)
+
+    if (d.getVar('IMAGE_PKGTYPE') == 'rpm'
+       and ('dnf' in d.getVar('TEST_SUITES') or 'auto' in d.getVar('TEST_SUITES'))):
+        create_rpm_index(d)
+
     testimage_main(d)
 }
 addtask testimage_auto before do_build after do_image_complete
-- 
1.9.1

Re: [PATCH 13/14] testimage-auto.bbclass: execute the same codes as testimage task

[Alexander Kanavin]

2018-05-31 11:45 GMT+03:00 Chen Qi <Qi.Chen@windriver.com>:
>  python do_testimage_auto() {
> +    testimage_sanity(d)
> +
> +    if (d.getVar('IMAGE_PKGTYPE') == 'rpm'
> +       and ('dnf' in d.getVar('TEST_SUITES') or 'auto' in d.getVar('TEST_SUITES'))):
> +        create_rpm_index(d)
> +
>      testimage_main(d)

Why not simply move these lines to testimage_main()? They might become
out of sync again in the future otherwise.

Alex

Re: [PATCH 13/14] testimage-auto.bbclass: execute the same codes as testimage task

[ChenQi]

On 05/31/2018 09:50 PM, Alexander Kanavin wrote:
> 2018-05-31 11:45 GMT+03:00 Chen Qi <Qi.Chen@windriver.com>:
>>   python do_testimage_auto() {
>> +    testimage_sanity(d)
>> +
>> +    if (d.getVar('IMAGE_PKGTYPE') == 'rpm'
>> +       and ('dnf' in d.getVar('TEST_SUITES') or 'auto' in d.getVar('TEST_SUITES'))):
>> +        create_rpm_index(d)
>> +
>>       testimage_main(d)
> Why not simply move these lines to testimage_main()? They might become
> out of sync again in the future otherwise.
>
> Alex
>
Done.
V2 will be sent out soon.

Thanks,
Chen Qi

[PATCH 14/14] oeqa/core/target/ssh.py: increase maximum read bytes from 1024 to 4096

[Chen Qi]

When running testimage task for core-image-sato-sdk, the following
error appeared.

  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 at position 0: invalid start byte

Checking the codes, I found it's caused by setting a 1024 limit for the
read method of the StreamReader object.

Comments from the manual:
"""
The chars argument indicates the number of decoded code points or bytes to
return. The read() method will never return more data than requested, but
it might return less, if there is not enough available.
"""

When running `systemctl status --full' on target, this error occurs.

This patch increase the bytes limit to 4096 to fix the error.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/lib/oeqa/core/target/ssh.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/core/target/ssh.py b/meta/lib/oeqa/core/target/ssh.py
index 151b99a..8ff1f6c 100644
--- a/meta/lib/oeqa/core/target/ssh.py
+++ b/meta/lib/oeqa/core/target/ssh.py
@@ -208,7 +208,7 @@ def SSHCall(command, logger, timeout=None, **opts):
                 try:
                     if select.select([process.stdout], [], [], 5)[0] != []:
                         reader = codecs.getreader('utf-8')(process.stdout)
-                        data = reader.read(1024, 1024)
+                        data = reader.read(1024, 4096)
                         if not data:
                             process.stdout.close()
                             eof = True
-- 
1.9.1