[PATCH 1/1] selftest/runtime-test.py: Add test for import test from other layers

[PATCH 1/1] selftest/runtime-test.py: Add test for import test from other layers

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

There are some features in testimage/testexport that are not tested;
this might lead to break some of these features without notice.

This adds a new test in order to test two features of testimage:
  - Import test from other layers.
  - Install/Unistall in the DUT without a package manager.

[YOCTO #9764]
[YOCTO #9766]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta-selftest/lib/oeqa/runtime/selftest.json       |  6 +++++
 meta-selftest/lib/oeqa/runtime/selftest.py         | 31 ++++++++++++++++++++++
 .../selftest/{testexport.py => runtime-test.py}    | 19 +++++++++++++
 3 files changed, 56 insertions(+)
 create mode 100644 meta-selftest/lib/oeqa/runtime/selftest.json
 create mode 100644 meta-selftest/lib/oeqa/runtime/selftest.py
 rename meta/lib/oeqa/selftest/{testexport.py => runtime-test.py} (76%)

diff --git a/meta-selftest/lib/oeqa/runtime/selftest.json b/meta-selftest/lib/oeqa/runtime/selftest.json
new file mode 100644
index 0000000..e5ae46e
--- /dev/null
+++ b/meta-selftest/lib/oeqa/runtime/selftest.json
@@ -0,0 +1,6 @@
+{
+    "test_install_package": {
+        "pkg": "socat",
+        "rm": true
+    }
+}
diff --git a/meta-selftest/lib/oeqa/runtime/selftest.py b/meta-selftest/lib/oeqa/runtime/selftest.py
new file mode 100644
index 0000000..b9c9b23
--- /dev/null
+++ b/meta-selftest/lib/oeqa/runtime/selftest.py
@@ -0,0 +1,31 @@
+from oeqa.oetest import oeRuntimeTest, skipModule
+from oeqa.utils.decorators import *
+
+class Selftest(oeRuntimeTest):
+
+    @skipUnlessPassed("test_ssh")
+    def test_install_package(self):
+        """
+        Summary: Check basic package installation functionality.
+        Expected: 1. Before the test socat must be installed using scp.
+                  2. After the test socat must be unistalled using ssh.
+                     This can't be checked in this test.
+        Product: oe-core
+        Author: Mariano Lopez <mariano.lopez@intel.com>
+        """
+
+        (status, output) = self.target.run("socat -V")
+        self.assertEqual(status, 0, msg="socat is not installed")
+
+    @skipUnlessPassed("test_install_package")
+    def test_verify_unistall(self):
+        """
+        Summary: Check basic package installation functionality.
+        Expected: 1. test_install_package must unistall socat.
+                     This test is just to verify that.
+        Product: oe-core
+        Author: Mariano Lopez <mariano.lopez@intel.com>
+        """
+
+        (status, output) = self.target.run("socat -V")
+        self.assertNotEqual(status, 0, msg="socat is still installed")
diff --git a/meta/lib/oeqa/selftest/testexport.py b/meta/lib/oeqa/selftest/runtime-test.py
similarity index 76%
rename from meta/lib/oeqa/selftest/testexport.py
rename to meta/lib/oeqa/selftest/runtime-test.py
index 5823b13..1e1d3b0 100644
--- a/meta/lib/oeqa/selftest/testexport.py
+++ b/meta/lib/oeqa/selftest/runtime-test.py
@@ -42,3 +42,22 @@ class TestExport(oeSelfTest):
             # Verify ping test was succesful
             failure = True if 'FAIL' in result.output else False
             self.assertNotEqual(True, failure, 'ping test failed')
+
+
+class TestImage(oeSelfTest):
+
+    def test_testimage_install(self):
+        """
+        Summary: Check install packages functionality for testimage/testexport.
+        Expected: 1. Import tests from a directory other than meta.
+                  2. Check install/unistall of socat.
+        Product: oe-core
+        Author: Mariano Lopez <mariano.lopez@intel.com>
+        """
+
+        features = 'INHERIT += "testexport"\n'
+        features = 'TEST_SUITES = "ping ssh selftest"\n'
+
+        # Build core-image-sato and testimage
+        bitbake('core-image-sato socat')
+        bitbake('-c testimage core-image-sato')
-- 
2.6.6

Re: [PATCH 1/1] selftest/runtime-test.py: Add test for import test from other layers

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

On 13 June 2016 at 14:32, <mariano.lopez@linux.intel.com> wrote:

> +        features = 'INHERIT += "testexport"\n'
> +        features = 'TEST_SUITES = "ping ssh selftest"\n'
> +
> +        # Build core-image-sato and testimage
> +        bitbake('core-image-sato socat')
> +        bitbake('-c testimage core-image-sato')
>

If we're just running ping/ssh/selftest, can the image be something a lot
smaller such as core-image-base?

Ross

[-- Attachment #2: Type: text/html, Size: 952 bytes --]

Re: [PATCH 1/1] selftest/runtime-test.py: Add test for import test from other layers

[Mariano Lopez]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]



On 06/14/2016 04:26 AM, Burton, Ross wrote:
>
> On 13 June 2016 at 14:32, <mariano.lopez@linux.intel.com 
> <mailto:mariano.lopez@linux.intel.com>> wrote:
>
>     +   features = 'INHERIT += "testexport"\n'
>     +        features = 'TEST_SUITES = "ping ssh selftest"\n'
>     +
>     +        # Build core-image-sato and testimage
>     +        bitbake('core-image-sato socat')
>     +        bitbake('-c testimage core-image-sato')
>
>
> If we're just running ping/ssh/selftest, can the image be something a 
> lot smaller such as core-image-base?
>
> Ross

Unfortunately, base or minimal don't have ssh installed, and ssh it is 
required for the runtime tests.
I could add the ssh to DISTRO_FEATURES but personally I think that is 
not clean. What do you think?

Mariano

[-- Attachment #2: Type: text/html, Size: 2004 bytes --]

Re: [PATCH 1/1] selftest/runtime-test.py: Add test for import test from other layers

[Khem Raj]

On Tue, Jun 14, 2016 at 3:08 PM, Mariano Lopez
<mariano.lopez@linux.intel.com> wrote:
>
>
> On 06/14/2016 04:26 AM, Burton, Ross wrote:
>
>
> On 13 June 2016 at 14:32, <mariano.lopez@linux.intel.com> wrote:
>>
>> +        features = 'INHERIT += "testexport"\n'
>> +        features = 'TEST_SUITES = "ping ssh selftest"\n'
>> +
>> +        # Build core-image-sato and testimage
>> +        bitbake('core-image-sato socat')
>> +        bitbake('-c testimage core-image-sato')
>
>
> If we're just running ping/ssh/selftest, can the image be something a lot
> smaller such as core-image-base?
>
> Ross
>
>
> Unfortunately, base or minimal don't have ssh installed, and ssh it is
> required for the runtime tests.
> I could add the ssh to DISTRO_FEATURES but personally I think that is not
> clean. What do you think?

Can ssh be added to images automatically if testing is enabled ?

>
> Mariano
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

Re: [PATCH 1/1] selftest/runtime-test.py: Add test for import test from other layers

[Mariano Lopez]

On 06/15/2016 03:47 PM, Khem Raj wrote:
> On Tue, Jun 14, 2016 at 3:08 PM, Mariano Lopez
> <mariano.lopez@linux.intel.com> wrote:
>> If we're just running ping/ssh/selftest, can the image be something a lot
>> smaller such as core-image-base?
>>
>> Ross
>>
>>
>> Unfortunately, base or minimal don't have ssh installed, and ssh it is
>> required for the runtime tests.
>> I could add the ssh to DISTRO_FEATURES but personally I think that is not
>> clean. What do you think?
> Can ssh be added to images automatically if testing is enabled ?

IMO that is not a good practice, because the image being tested is 
polluted with packages not meant to be there.

Mariano

Re: [PATCH 1/1] selftest/runtime-test.py: Add test for import test from other layers

[Khem Raj]

On Wed, Jun 15, 2016 at 2:01 PM, Mariano Lopez
<mariano.lopez@linux.intel.com> wrote:
>
>
> On 06/15/2016 03:47 PM, Khem Raj wrote:
>>
>> On Tue, Jun 14, 2016 at 3:08 PM, Mariano Lopez
>> <mariano.lopez@linux.intel.com> wrote:
>>>
>>> If we're just running ping/ssh/selftest, can the image be something a lot
>>> smaller such as core-image-base?
>>>
>>> Ross
>>>
>>>
>>> Unfortunately, base or minimal don't have ssh installed, and ssh it is
>>> required for the runtime tests.
>>> I could add the ssh to DISTRO_FEATURES but personally I think that is not
>>> clean. What do you think?
>>
>> Can ssh be added to images automatically if testing is enabled ?
>
>
> IMO that is not a good practice, because the image being tested is polluted
> with packages not meant to be there.

I thought you mentioned that ssh is required for testing, I dont see
that as polluting.
and its only added to image with have ptest turned on.

>
> Mariano

Re: [PATCH 1/1] selftest/runtime-test.py: Add test for import test from other layers

[Mariano Lopez]

On 06/15/2016 04:05 PM, Khem Raj wrote:
> On Wed, Jun 15, 2016 at 2:01 PM, Mariano Lopez
> <mariano.lopez@linux.intel.com> wrote:
>>
>> On 06/15/2016 03:47 PM, Khem Raj wrote:
>>> On Tue, Jun 14, 2016 at 3:08 PM, Mariano Lopez
>>> <mariano.lopez@linux.intel.com> wrote:
>>>> If we're just running ping/ssh/selftest, can the image be something a lot
>>>> smaller such as core-image-base?
>>>>
>>>> Ross
>>>>
>>>>
>>>> Unfortunately, base or minimal don't have ssh installed, and ssh it is
>>>> required for the runtime tests.
>>>> I could add the ssh to DISTRO_FEATURES but personally I think that is not
>>>> clean. What do you think?
>>> Can ssh be added to images automatically if testing is enabled ?
>>
>> IMO that is not a good practice, because the image being tested is polluted
>> with packages not meant to be there.
> I thought you mentioned that ssh is required for testing, I dont see
> that as polluting.
> and its only added to image with have ptest turned on.

This is for testimage/testexport and there is a TEST_SUITE specific for 
minimal image.

If the resource consumption seems to be the problem; adding ssh to 
minimal or base would trigger the build of a new rootfs/image; in the 
other hand is very likely core-image-sato is already built when running 
this test.

Re: [PATCH 1/1] selftest/runtime-test.py: Add test for import test from other layers

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]

On 13 June 2016 at 14:32, <mariano.lopez@linux.intel.com> wrote:

> +        features = 'INHERIT += "testexport"\n'
>

You mean testimage here as that class isn't inherited by default.


> +        features = 'TEST_SUITES = "ping ssh selftest"\n'
>

This overwrites the first assignment to features, and neither is actually
written using self.write_config().


> +        # Build core-image-sato and testimage
> +        bitbake('core-image-sato socat')
> +        bitbake('-c testimage core-image-sato')
>

I personally believe that core-image-sato is massive overkill for selftest
and core-image-full-cmdline is just as comprehensive (has ssh, for example)
without building all of Sato.

Ross

[-- Attachment #2: Type: text/html, Size: 1612 bytes --]

Re: [PATCH 1/1] selftest/runtime-test.py: Add test for import test from other layers

[Mariano Lopez]

[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]



On 07/25/2016 05:38 PM, Burton, Ross wrote:
>
> On 13 June 2016 at 14:32, <mariano.lopez@linux.intel.com 
> <mailto:mariano.lopez@linux.intel.com>> wrote:
>
>     +   features = 'INHERIT += "testexport"\n'
>
>
> You mean testimage here as that class isn't inherited by default.
>
>     +        features = 'TEST_SUITES = "ping ssh selftest"\n'
>
>
> This overwrites the first assignment to features, and neither is 
> actually written using self.write_config().
>
>     +   # Build core-image-sato and testimage
>     +        bitbake('core-image-sato socat')
>     +        bitbake('-c testimage core-image-sato')
>
>
> I personally believe that core-image-sato is massive overkill for 
> selftest and core-image-full-cmdline is just as comprehensive (has 
> ssh, for example) without building all of Sato.

I don't know what happened when I made this commit and submitted the 
patch, this patch is just wrong. I swear I tested it locally, perhaps I 
messed up the branches or something like that. I'll work on a v2 of this.

Mariano

[-- Attachment #2: Type: text/html, Size: 2855 bytes --]