[PATCH] selftest/manifest.py: Test to verify rootfs manifest

[PATCH] selftest/manifest.py: Test to verify rootfs manifest

[Benjamin Esquivel]

Adding a new test to verify if the packages in the
manifest files actually exists in pkgdata.
 -adding a setUpClass for when more tests get created here
 -check for the paths and fail gracefully if not there
 -skip the test when there are no manifest files to check
 -debug prints for failure analysis

[YOCTO#8028]

Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/lib/oeqa/selftest/manifest.py | 52 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/manifest.py

diff --git a/meta/lib/oeqa/selftest/manifest.py b/meta/lib/oeqa/selftest/manifest.py
new file mode 100644
index 0000000..c2bc945
--- /dev/null
+++ b/meta/lib/oeqa/selftest/manifest.py
@@ -0,0 +1,52 @@
+import unittest
+import os
+from glob import glob
+
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import get_bb_var
+from oeqa.utils.decorators import testcase
+
+class VerifyManifest(oeSelfTest):
+    '''Tests for the manifest files and contents of an image'''
+
+    @classmethod
+    def setUpClass(self):
+
+        # get directory locations from variable values and check them
+        self.deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE');
+        if not self.deploy_dir_image \
+            or not os.path.isdir(self.deploy_dir_image):
+            raise unittest.SkipTest("{}: DEPLOY_DIR_IMAGE does not exist: {}"\
+                    .format("VerifyManifest", self.deploy_dir_image))
+
+        self.pkgdata_dir = get_bb_var('PKGDATA_DIR');
+        if not self.pkgdata_dir \
+            or not os.path.isdir(self.pkgdata_dir):
+            raise unittest.SkipTest("{}: PKGDATA_DIR does not exist: {}"\
+                    .format("VerifyManifest", self.pkgdata_dir))
+
+        # get the manifest files
+        # no need to try: since glob would return an empty list if
+        # the path is non-existant
+        self.manifest_files = glob("%s/*.manifest" % self.deploy_dir_image)
+        self.log.debug("manifest files: {}".format(self.manifest_files))
+        if not self.manifest_files:
+            raise unittest.SkipTest("{}: No manifest files found in: {}"\
+                .format("VerifyManifest", self.deploy_dir_image))
+
+
+    def test_manifest_entries(self):
+        '''Verifying the manifest entries as packages '''
+        testname = self.id().split('.')[-1]
+        rundir = os.path.join(self.pkgdata_dir, "runtime-reverse")
+
+        errmsg = "ERROR: Package %s is in manifest but not in pkgdata"
+        for manifest in self.manifest_files:
+            with open(manifest, "r") as mfile:
+                for manifest_entries in mfile:
+                    pkg = manifest_entries.split()[0]
+                    pkgfile = os.path.join(rundir, pkg)
+                    self.log.debug("{}: looking for {}"\
+                            .format(testname, pkgfile))
+                    self.assertTrue(os.path.isfile(pkgfile),
+                            errmsg % pkg)
-- 
2.3.0

Re: [PATCH] selftest/manifest.py: Test to verify rootfs manifest

[Paul Eggleton]

Hi Benjamin / Mariano,

On Wednesday 26 August 2015 12:26:02 Benjamin Esquivel wrote:
> Adding a new test to verify if the packages in the
> manifest files actually exists in pkgdata.
>  -adding a setUpClass for when more tests get created here
>  -check for the paths and fail gracefully if not there
>  -skip the test when there are no manifest files to check
>  -debug prints for failure analysis
> 
> [YOCTO#8028]
> 
> Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
> ---
>  meta/lib/oeqa/selftest/manifest.py | 52
> ++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+)
>  create mode 100644 meta/lib/oeqa/selftest/manifest.py
> 
> diff --git a/meta/lib/oeqa/selftest/manifest.py
> b/meta/lib/oeqa/selftest/manifest.py new file mode 100644
> index 0000000..c2bc945
> --- /dev/null
> +++ b/meta/lib/oeqa/selftest/manifest.py
> @@ -0,0 +1,52 @@
> +import unittest
> +import os
> +from glob import glob
> +
> +from oeqa.selftest.base import oeSelfTest
> +from oeqa.utils.commands import get_bb_var
> +from oeqa.utils.decorators import testcase
> +
> +class VerifyManifest(oeSelfTest):
> +    '''Tests for the manifest files and contents of an image'''
> +
> +    @classmethod
> +    def setUpClass(self):
> +
> +        # get directory locations from variable values and check them
> +        self.deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE');
> +        if not self.deploy_dir_image \
> +            or not os.path.isdir(self.deploy_dir_image):
> +            raise unittest.SkipTest("{}: DEPLOY_DIR_IMAGE does not exist:
> {}"\ +                    .format("VerifyManifest", self.deploy_dir_image))
> +
> +        self.pkgdata_dir = get_bb_var('PKGDATA_DIR');
> +        if not self.pkgdata_dir \
> +            or not os.path.isdir(self.pkgdata_dir):
> +            raise unittest.SkipTest("{}: PKGDATA_DIR does not exist: {}"\
> +                    .format("VerifyManifest", self.pkgdata_dir))
> +
> +        # get the manifest files
> +        # no need to try: since glob would return an empty list if
> +        # the path is non-existant
> +        self.manifest_files = glob("%s/*.manifest" % self.deploy_dir_image)
> +        self.log.debug("manifest files: {}".format(self.manifest_files)) +
>        if not self.manifest_files:
> +            raise unittest.SkipTest("{}: No manifest files found in: {}"\
> +                .format("VerifyManifest", self.deploy_dir_image))
> +
> +
> +    def test_manifest_entries(self):
> +        '''Verifying the manifest entries as packages '''
> +        testname = self.id().split('.')[-1]
> +        rundir = os.path.join(self.pkgdata_dir, "runtime-reverse")
> +
> +        errmsg = "ERROR: Package %s is in manifest but not in pkgdata"
> +        for manifest in self.manifest_files:
> +            with open(manifest, "r") as mfile:
> +                for manifest_entries in mfile:
> +                    pkg = manifest_entries.split()[0]
> +                    pkgfile = os.path.join(rundir, pkg)
> +                    self.log.debug("{}: looking for {}"\
> +                            .format(testname, pkgfile))
> +                    self.assertTrue(os.path.isfile(pkgfile),
> +                            errmsg % pkg)

Shouldn't this test actually be building an image (e.g. using
bitbake('core-image-minimal') ) in order to ensure there's a manifest file for 
it to look at? Then you can make the test fail rather than skipping if the file 
isn't there or doesn't contain a small set of packages we know should be 
listed.

Also while you're adding this could you add a similar test function for the 
SDK (both host + target manifests).

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] selftest/manifest.py: Test to verify rootfs manifest

[Benjamin Esquivel]

Hey Paul, thanks for looking at this, check the comments below.

On Thu, 2015-08-27 at 09:34 +0100, Paul Eggleton wrote:
> Hi Benjamin / Mariano,
> 
> On Wednesday 26 August 2015 12:26:02 Benjamin Esquivel wrote:
> > Adding a new test to verify if the packages in the
> > manifest files actually exists in pkgdata.
> >  -adding a setUpClass for when more tests get created here
> >  -check for the paths and fail gracefully if not there
> >  -skip the test when there are no manifest files to check
> >  -debug prints for failure analysis
> > 
> > [YOCTO#8028]
> > 
> > Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com
> > >
> > Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
> > ---
> >  meta/lib/oeqa/selftest/manifest.py | 52
> > ++++++++++++++++++++++++++++++++++++++ 1 file changed, 52
> > insertions(+)
> >  create mode 100644 meta/lib/oeqa/selftest/manifest.py
> > 
> > diff --git a/meta/lib/oeqa/selftest/manifest.py
> > b/meta/lib/oeqa/selftest/manifest.py new file mode 100644
> > index 0000000..c2bc945
> > --- /dev/null
> > +++ b/meta/lib/oeqa/selftest/manifest.py
> > @@ -0,0 +1,52 @@
> > +import unittest
> > +import os
> > +from glob import glob
> > +
> > +from oeqa.selftest.base import oeSelfTest
> > +from oeqa.utils.commands import get_bb_var
> > +from oeqa.utils.decorators import testcase
> > +
> > +class VerifyManifest(oeSelfTest):
> > +    '''Tests for the manifest files and contents of an image'''
> > +
> > +    @classmethod
> > +    def setUpClass(self):
> > +
> > +        # get directory locations from variable values and check
> > them
> > +        self.deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE');
> > +        if not self.deploy_dir_image \
> > +            or not os.path.isdir(self.deploy_dir_image):
> > +            raise unittest.SkipTest("{}: DEPLOY_DIR_IMAGE does not
> > exist:
> > {}"\ +                    .format("VerifyManifest",
> > self.deploy_dir_image))
> > +
> > +        self.pkgdata_dir = get_bb_var('PKGDATA_DIR');
> > +        if not self.pkgdata_dir \
> > +            or not os.path.isdir(self.pkgdata_dir):
> > +            raise unittest.SkipTest("{}: PKGDATA_DIR does not
> > exist: {}"\
> > +                    .format("VerifyManifest", self.pkgdata_dir))
> > +
> > +        # get the manifest files
> > +        # no need to try: since glob would return an empty list if
> > +        # the path is non-existant
> > +        self.manifest_files = glob("%s/*.manifest" %
> > self.deploy_dir_image)
> > +        self.log.debug("manifest files:
> > {}".format(self.manifest_files)) +
> >        if not self.manifest_files:
> > +            raise unittest.SkipTest("{}: No manifest files found
> > in: {}"\
> > +                .format("VerifyManifest", self.deploy_dir_image))
> > +
> > +
> > +    def test_manifest_entries(self):
> > +        '''Verifying the manifest entries as packages '''
> > +        testname = self.id().split('.')[-1]
> > +        rundir = os.path.join(self.pkgdata_dir, "runtime-reverse")
> > +
> > +        errmsg = "ERROR: Package %s is in manifest but not in
> > pkgdata"
> > +        for manifest in self.manifest_files:
> > +            with open(manifest, "r") as mfile:
> > +                for manifest_entries in mfile:
> > +                    pkg = manifest_entries.split()[0]
> > +                    pkgfile = os.path.join(rundir, pkg)
> > +                    self.log.debug("{}: looking for {}"\
> > +                            .format(testname, pkgfile))
> > +                    self.assertTrue(os.path.isfile(pkgfile),
> > +                            errmsg % pkg)
> 
> Shouldn't this test actually be building an image (e.g. using
> bitbake('core-image-minimal') ) in order to ensure there's a manifest
> file for 
> it to look at? Then you can make the test fail rather than skipping
> if the file 
> isn't there or doesn't contain a small set of packages we know should
> be 
> listed.
The reason the actual build of an image was not included in the test is
because it is abstracted from it. That's why it contains checks for
when the manifests are not there it skips the test instead of failing.

This would give you the chance to write some level of abstraction that
is able to have test subjects (i.e. subject A: core-image-minimal with
systemd and ipk). Otherwise you're locking the test to a single
scenario which is usually the most common. All the corner cases are
hidden from that scope of testing.

On the other hand, if we had that design I'm talking about we could
easily have a default behavior that does exactly what you're proposing.

What do you think?
 
> 
> Also while you're adding this could you add a similar test function
> for the 
> SDK (both host + target manifests).
> 
I believe we could add many more tests that's why I started splitting
common tasks to the setup function of the test.

I'll add this in the next patch version.

> Paul
> 

Re: [PATCH] selftest/manifest.py: Test to verify rootfs manifest

[Paul Eggleton]

On Thursday 27 August 2015 10:50:51 Benjamin Esquivel wrote:
> Hey Paul, thanks for looking at this, check the comments below.
> 
> On Thu, 2015-08-27 at 09:34 +0100, Paul Eggleton wrote:
> > Hi Benjamin / Mariano,
> > 
> > On Wednesday 26 August 2015 12:26:02 Benjamin Esquivel wrote:
> > > Adding a new test to verify if the packages in the
> > > manifest files actually exists in pkgdata.
> > > 
> > >  -adding a setUpClass for when more tests get created here
> > >  -check for the paths and fail gracefully if not there
> > >  -skip the test when there are no manifest files to check
> > >  -debug prints for failure analysis
> > > 
> > > [YOCTO#8028]
> > > 
> > > Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com
> > > 
> > > Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
> > > ---
> > > 
> > >  meta/lib/oeqa/selftest/manifest.py | 52
> > > 
> > > ++++++++++++++++++++++++++++++++++++++ 1 file changed, 52
> > > insertions(+)
> > > 
> > >  create mode 100644 meta/lib/oeqa/selftest/manifest.py
> > > 
> > > diff --git a/meta/lib/oeqa/selftest/manifest.py
> > > b/meta/lib/oeqa/selftest/manifest.py new file mode 100644
> > > index 0000000..c2bc945
> > > --- /dev/null
> > > +++ b/meta/lib/oeqa/selftest/manifest.py
> > > @@ -0,0 +1,52 @@
> > > +import unittest
> > > +import os
> > > +from glob import glob
> > > +
> > > +from oeqa.selftest.base import oeSelfTest
> > > +from oeqa.utils.commands import get_bb_var
> > > +from oeqa.utils.decorators import testcase
> > > +
> > > +class VerifyManifest(oeSelfTest):
> > > +    '''Tests for the manifest files and contents of an image'''
> > > +
> > > +    @classmethod
> > > +    def setUpClass(self):
> > > +
> > > +        # get directory locations from variable values and check
> > > them
> > > +        self.deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE');
> > > +        if not self.deploy_dir_image \
> > > +            or not os.path.isdir(self.deploy_dir_image):
> > > +            raise unittest.SkipTest("{}: DEPLOY_DIR_IMAGE does not
> > > exist:
> > > {}"\ +                    .format("VerifyManifest",
> > > self.deploy_dir_image))
> > > +
> > > +        self.pkgdata_dir = get_bb_var('PKGDATA_DIR');
> > > +        if not self.pkgdata_dir \
> > > +            or not os.path.isdir(self.pkgdata_dir):
> > > +            raise unittest.SkipTest("{}: PKGDATA_DIR does not
> > > exist: {}"\
> > > +                    .format("VerifyManifest", self.pkgdata_dir))
> > > +
> > > +        # get the manifest files
> > > +        # no need to try: since glob would return an empty list if
> > > +        # the path is non-existant
> > > +        self.manifest_files = glob("%s/*.manifest" %
> > > self.deploy_dir_image)
> > > +        self.log.debug("manifest files:
> > > {}".format(self.manifest_files)) +
> > > 
> > >        if not self.manifest_files:
> > > +            raise unittest.SkipTest("{}: No manifest files found
> > > in: {}"\
> > > +                .format("VerifyManifest", self.deploy_dir_image))
> > > +
> > > +
> > > +    def test_manifest_entries(self):
> > > +        '''Verifying the manifest entries as packages '''
> > > +        testname = self.id().split('.')[-1]
> > > +        rundir = os.path.join(self.pkgdata_dir, "runtime-reverse")
> > > +
> > > +        errmsg = "ERROR: Package %s is in manifest but not in
> > > pkgdata"
> > > +        for manifest in self.manifest_files:
> > > +            with open(manifest, "r") as mfile:
> > > +                for manifest_entries in mfile:
> > > +                    pkg = manifest_entries.split()[0]
> > > +                    pkgfile = os.path.join(rundir, pkg)
> > > +                    self.log.debug("{}: looking for {}"\
> > > +                            .format(testname, pkgfile))
> > > +                    self.assertTrue(os.path.isfile(pkgfile),
> > > +                            errmsg % pkg)
> > 
> > Shouldn't this test actually be building an image (e.g. using
> > bitbake('core-image-minimal') ) in order to ensure there's a manifest
> > file for
> > it to look at? Then you can make the test fail rather than skipping
> > if the file
> > isn't there or doesn't contain a small set of packages we know should
> > be
> > listed.
> 
> The reason the actual build of an image was not included in the test is
> because it is abstracted from it. That's why it contains checks for
> when the manifests are not there it skips the test instead of failing.
> 
> This would give you the chance to write some level of abstraction that
> is able to have test subjects (i.e. subject A: core-image-minimal with
> systemd and ipk). Otherwise you're locking the test to a single
> scenario which is usually the most common. All the corner cases are
> hidden from that scope of testing.
>
> On the other hand, if we had that design I'm talking about we could
> easily have a default behavior that does exactly what you're proposing.
> 
> What do you think?

We want oe-selftest to be able to objectively test that expected functionality 
works when run unattended (as it is on the autobuilder), so if there are 
multiple scenarios to test then we probably need to be explicitly setting 
those up and testing them here rather than expecting the user to set them up 
beforehand. If that adds time to the oe-selftest run, well, so be it.

> > Also while you're adding this could you add a similar test function
> > for the
> > SDK (both host + target manifests).
> 
> I believe we could add many more tests that's why I started splitting
> common tasks to the setup function of the test.
> 
> I'll add this in the next patch version.

Great, thanks.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH v2] selftest/manifest.py: Test support for manifests

[Benjamin Esquivel]

adding support for tests to verify manifest collections
(i.e. all the manifests found in a given dir) checking them
to provide entries that exist in an specified pkgdata dir

tests added:
 -adding an SDK manifest test
 -adding a core-image-minimal manifest test
test support written for future tests:
 -adding a setUpClass that supports other tests
 -a get dir from bb var function that verifies if the dir exists
 -an initial ManifestCollection and ManifestEntry classes defined
 -check for the paths and fail gracefully if not there
 -debug prints for failure analysis

[YOCTO#8028]

Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/lib/oeqa/selftest/manifest.py | 165 +++++++++++++++++++++++++++++++++++++
 1 file changed, 165 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/manifest.py

diff --git a/meta/lib/oeqa/selftest/manifest.py b/meta/lib/oeqa/selftest/manifest.py
new file mode 100644
index 0000000..4843622
--- /dev/null
+++ b/meta/lib/oeqa/selftest/manifest.py
@@ -0,0 +1,165 @@
+import unittest
+import os
+
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import get_bb_var, bitbake
+from oeqa.utils.decorators import testcase
+from glob import glob
+
+class ManifestEntry:
+    '''A manifest item of a collection able to list missing packages'''
+    def __init__(self, entry):
+        self.entry = entry
+        self.missing = []
+
+class ManifestCollection:
+    '''A manifest collection definition with error check and manifest entries'''
+    def __init__(self, mfiles_path = os.getcwd()):
+        self.entries = glob("%s/*.manifest" % mfiles_path)
+        if not self.entries: raise OSError
+        self.entry = {}
+        for e in self.entries:
+            self.entry[e] = ManifestEntry(e)
+        self.errors = False
+
+class VerifyManifest(oeSelfTest):
+    '''Tests for the manifest files and contents of an image'''
+
+    @classmethod
+    def check_manifest_entries(self, manifest, path):
+        manifest_errors = []
+        try:
+            with open(manifest, "r") as mfile:
+                for line in mfile:
+                    manifest_entry = os.path.join(path, line.split()[0])
+                    self.log.debug("{}: looking for {}"\
+                                    .format(self.classname, manifest_entry))
+                    if not os.path.isfile(manifest_entry):
+                        manifest_errors.append(manifest_entry)
+                        self.log.debug("{}: {} not found"\
+                                    .format(self.classname, manifest_entry))
+        except OSError as e:
+            self.log.debug("{}: checking of {} failed"\
+                    .format(self.classname, manifest))
+            raise e
+
+        return manifest_errors
+
+    #this will possibly move from here
+    @classmethod
+    def get_dir_from_bb_var(self, bb_var, target = None):
+        target == self.buildtarget if target == None else target
+        directory = get_bb_var(bb_var, target);
+        if not directory or not os.path.isdir(directory):
+            self.log.debug("{}: {} points to {} when target = {}"\
+                    .format(self.classname, bb_var, directory, target))
+            raise OSError
+        return directory
+
+    @classmethod
+    def setUpClass(self):
+
+        self.buildtarget = 'core-image-minimal'
+        self.classname = 'VerifyManifest'
+
+        self.log.info("{}: doing bitbake {} as a prerequisite of the test"\
+                .format(self.classname, self.buildtarget))
+        if bitbake(self.buildtarget).status:
+            self.log.debug("{} Failed to setup {}"\
+                    .format(self.classname, self.buildtarget))
+            unittest.SkipTest("{}: Cannot setup testing scenario"\
+                    .format(self.classname))
+
+    def test_SDK_manifest_entries(self):
+        '''Verifying the SDK manifest entries exist, this may take a build'''
+        self.testname = self.id().split('.')[-1]
+        # the setup should bitbake core-image-minimal and here it is required
+        # to do an additional setup for the sdk
+        sdktask = '-c populate_sdk'
+        bbargs = sdktask + ' ' + self.buildtarget
+        self.log.debug("{}: doing bitbake {} as a prerequisite of the test"\
+                .format(self.classname, bbargs))
+        if bitbake(bbargs).status:
+            self.log.debug("{} Failed to bitbake {}"\
+                    .format(self.classname, bbargs))
+            unittest.SkipTest("{}: Cannot setup testing scenario"\
+                    .format(self.classname))
+
+
+        pkgdata_dir = reverse_dir = {}
+        # get manifest location based on target to query about
+        try:
+            mdir = self.get_dir_from_bb_var('SDK_DEPLOY',
+                                                self.buildtarget)
+            pkgdata_dir['target'] = self.get_dir_from_bb_var('PKGDATA_DIR',
+                                                        self.buildtarget)
+            reverse_dir['target'] = os.path.join(pkgdata_dir['target'],
+                                            'runtime-reverse')
+            pkgdata_dir['host'] = self.get_dir_from_bb_var('PKGDATA_DIR',
+                                             'nativesdk-packagegroup-sdk-host')
+            reverse_dir['host'] = os.path.join(pkgdata_dir['host'],
+                                            'runtime-reverse')
+        except OSError:
+            raise unittest.SkipTest("{}: Error in obtaining manifest dirs"\
+                .format(self.classname))
+
+        m_collection = ManifestCollection(mdir)
+
+        for mfile in m_collection.entries:
+            self.log.debug("{}: Check manifest {}"\
+                                .format(self.classname, mfile))
+            #assume target
+            revdir = reverse_dir['target']
+            # if this doesn't end with host.manifest then we need another
+            # consistent way of checking what is this manifest for?
+            if mfile.endswith('host.manifest'):
+                revdir = reverse_dir['host']
+
+            if not os.path.exists(revdir):
+                self.fail("unable to find {} to work on it".format(revdir))
+
+            m_collection.entry[mfile].missing = self.check_manifest_entries(\
+                                                        mfile,revdir)
+            if m_collection.entry[mfile].missing:
+                m_collection.errors = True
+                msg = '{}: {} has missing entries'\
+                        .format(self.classname, mfile)
+                logmsg = msg+':\n'+'\n'.join(m_collection.entry[mfile].missing)
+                self.log.debug(logmsg)
+                self.log.info(msg)
+
+        if m_collection.errors: self.fail()
+
+    def test_image_manifest_entries(self):
+        '''Verifying the image manifest entries exist'''
+        self.testname = self.id().split('.')[-1]
+
+        # get manifest location based on target to query about
+        try:
+            mdir = self.get_dir_from_bb_var('DEPLOY_DIR_IMAGE',
+                                                self.buildtarget)
+            pkgdata_dir = {}
+            pkgdata_dir = self.get_dir_from_bb_var('PKGDATA_DIR',
+                                                self.buildtarget)
+            revdir = os.path.join(pkgdata_dir, 'runtime-reverse')
+            if not os.path.exists(revdir): raise OSError
+        except OSError:
+            raise unittest.SkipTest("{}: Error in obtaining manifest dirs"\
+                .format(self.classname))
+
+        m_collection = ManifestCollection(mdir)
+
+        for mfile in m_collection.entries:
+            self.log.debug("{}: Check manifest {}"\
+                                .format(self.classname, mfile))
+            m_collection.entry[mfile].missing = self.check_manifest_entries(\
+                                                        mfile,revdir)
+            if m_collection.entry[mfile].missing:
+                m_collection.errors = True
+                msg = '{}: {} has missing entries'\
+                        .format(self.classname, mfile)
+                logmsg = msg+':\n'+'\n'.join(m_collection.entry[mfile].missing)
+                self.log.debug(logmsg)
+                self.log.info(msg)
+
+        if m_collection.errors: self.fail()
-- 
2.3.0

Re: [PATCH v2] selftest/manifest.py: Test support for manifests

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1207 bytes --]

On 2 September 2015 at 21:18, Benjamin Esquivel <
benjamin.esquivel@linux.intel.com> wrote:

> adding support for tests to verify manifest collections
> (i.e. all the manifests found in a given dir) checking them
> to provide entries that exist in an specified pkgdata dir
>
> tests added:
>  -adding an SDK manifest test
>  -adding a core-image-minimal manifest test
> test support written for future tests:
>  -adding a setUpClass that supports other tests
>  -a get dir from bb var function that verifies if the dir exists
>  -an initial ManifestCollection and ManifestEntry classes defined
>  -check for the paths and fail gracefully if not there
>  -debug prints for failure analysis
>

Just ran this on the autobuilder:

FAIL: test_image_manifest_entries (oeqa.selftest.manifest.VerifyManifest)
Verifying the image manifest entries exist
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/manifest.py",
line 165, in test_image_manifest_entries
    if m_collection.errors: self.fail()
AssertionError: None

Ross

[-- Attachment #2: Type: text/html, Size: 1782 bytes --]

Re: [PATCH v2] selftest/manifest.py: Test support for manifests

[Benjamin Esquivel]

[-- Attachment #1: Type: text/plain, Size: 1383 bytes --]

Thanks for the notice, I'll check it up.
Benjamin
On Thu, 2015-09-10 at 10:52 +0100, Burton, Ross wrote:
> 
> On 2 September 2015 at 21:18, Benjamin Esquivel <
> benjamin.esquivel@linux.intel.com> wrote:
> > adding support for tests to verify manifest collections
> > (i.e. all the manifests found in a given dir) checking them
> > to provide entries that exist in an specified pkgdata dir
> > 
> > tests added:
> >  -adding an SDK manifest test
> >  -adding a core-image-minimal manifest test
> > test support written for future tests:
> >  -adding a setUpClass that supports other tests
> >  -a get dir from bb var function that verifies if the dir exists
> >  -an initial ManifestCollection and ManifestEntry classes defined
> >  -check for the paths and fail gracefully if not there
> >  -debug prints for failure analysis
> > 
> Just ran this on the autobuilder:
> 
> FAIL: test_image_manifest_entries
> (oeqa.selftest.manifest.VerifyManifest)
> Verifying the image manifest entries exist
> ---------------------------------------------------------------------
> -
> Traceback (most recent call last):
>   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe
> -selftest/build/meta/lib/oeqa/selftest/manifest.py", line 165, in
> test_image_manifest_entries
>     if m_collection.errors: self.fail()
> AssertionError: None
> 
> Ross

[-- Attachment #2: Type: text/html, Size: 1920 bytes --]

[PATCH V3] selftest/manifest.py: Test support for manifests

[Benjamin Esquivel]

adding support for tests to verify that manifest contents
contain entries that exist in an specified pkgdata dir

tests added:
 - manifest.VerifyManifest.test_image_manifest_entries
 - manifest.VerifyManifest.test_SDK_manifest_entries
test support written for future tests:
 -adding a setUpClass that supports other manifest tests
 -a get dir from bb var function that verifies if the dir exists
 -a ManifestEntry class defined with missing items list
 -check for the paths and fail gracefully if not there
 -debug prints for failure analysis

[YOCTO#8028]

Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/lib/oeqa/selftest/manifest.py | 163 +++++++++++++++++++++++++++++++++++++
 1 file changed, 163 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/manifest.py

diff --git a/meta/lib/oeqa/selftest/manifest.py b/meta/lib/oeqa/selftest/manifest.py
new file mode 100644
index 0000000..c813c1d
--- /dev/null
+++ b/meta/lib/oeqa/selftest/manifest.py
@@ -0,0 +1,163 @@
+import unittest
+import os
+
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import get_bb_var, bitbake
+from oeqa.utils.decorators import testcase
+
+class ManifestEntry:
+    '''A manifest item of a collection able to list missing packages'''
+    def __init__(self, entry):
+        self.file = entry
+        self.missing = []
+
+class VerifyManifest(oeSelfTest):
+    '''Tests for the manifest files and contents of an image'''
+
+    @classmethod
+    def check_manifest_entries(self, manifest, path):
+        manifest_errors = []
+        try:
+            with open(manifest, "r") as mfile:
+                for line in mfile:
+                    manifest_entry = os.path.join(path, line.split()[0])
+                    self.log.debug("{}: looking for {}"\
+                                    .format(self.classname, manifest_entry))
+                    if not os.path.isfile(manifest_entry):
+                        manifest_errors.append(manifest_entry)
+                        self.log.debug("{}: {} not found"\
+                                    .format(self.classname, manifest_entry))
+        except OSError as e:
+            self.log.debug("{}: checking of {} failed"\
+                    .format(self.classname, manifest))
+            raise e
+
+        return manifest_errors
+
+    #this will possibly move from here
+    @classmethod
+    def get_dir_from_bb_var(self, bb_var, target = None):
+        target == self.buildtarget if target == None else target
+        directory = get_bb_var(bb_var, target);
+        if not directory or not os.path.isdir(directory):
+            self.log.debug("{}: {} points to {} when target = {}"\
+                    .format(self.classname, bb_var, directory, target))
+            raise OSError
+        return directory
+
+    @classmethod
+    def setUpClass(self):
+
+        self.buildtarget = 'core-image-minimal'
+        self.classname = 'VerifyManifest'
+
+        self.log.info("{}: doing bitbake {} as a prerequisite of the test"\
+                .format(self.classname, self.buildtarget))
+        if bitbake(self.buildtarget).status:
+            self.log.debug("{} Failed to setup {}"\
+                    .format(self.classname, self.buildtarget))
+            unittest.SkipTest("{}: Cannot setup testing scenario"\
+                    .format(self.classname))
+
+    def test_SDK_manifest_entries(self):
+        '''Verifying the SDK manifest entries exist, this may take a build'''
+
+        # the setup should bitbake core-image-minimal and here it is required
+        # to do an additional setup for the sdk
+        sdktask = '-c populate_sdk'
+        bbargs = sdktask + ' ' + self.buildtarget
+        self.log.debug("{}: doing bitbake {} as a prerequisite of the test"\
+                .format(self.classname, bbargs))
+        if bitbake(bbargs).status:
+            self.log.debug("{} Failed to bitbake {}"\
+                    .format(self.classname, bbargs))
+            unittest.SkipTest("{}: Cannot setup testing scenario"\
+                    .format(self.classname))
+
+
+        pkgdata_dir = reverse_dir = {}
+        mfilename = mpath = m_entry = {}
+        # get manifest location based on target to query about
+        d_target= dict(target = self.buildtarget,
+                         host = 'nativesdk-packagegroup-sdk-host')
+        try:
+            mdir = self.get_dir_from_bb_var('SDK_DEPLOY', self.buildtarget)
+            for k in d_target.keys():
+                mfilename[k] = "{}-toolchain-{}.{}.manifest".format(
+                        get_bb_var("SDK_NAME", self.buildtarget),
+                        get_bb_var("SDK_VERSION", self.buildtarget),
+                        k)
+                mpath[k] = os.path.join(mdir, mfilename[k])
+                if not os.path.isfile(mpath[k]):
+                    self.log.debug("{}: {} does not exist".format(
+                        self.classname, mpath[k]))
+                    raise IOError
+                m_entry[k] = ManifestEntry(mpath[k])
+
+                pkgdata_dir[k] = self.get_dir_from_bb_var('PKGDATA_DIR',
+                        d_target[k])
+                reverse_dir[k] = os.path.join(pkgdata_dir[k],
+                        'runtime-reverse')
+                if not os.path.exists(reverse_dir[k]):
+                    self.log.debug("{}: {} does not exist".format(
+                        self.classname, reverse_dir[k]))
+                    raise IOError
+        except OSError:
+            raise unittest.SkipTest("{}: Error in obtaining manifest dirs"\
+                .format(self.classname))
+        except IOError:
+            msg = "{}: Error cannot find manifests in the specified dir:\n{}"\
+                    .format(self.classname, mdir)
+            self.fail(msg)
+
+        for k in d_target.keys():
+            self.log.debug("{}: Check manifest {}".format(
+                self.classname, m_entry[k].file))
+
+            m_entry[k].missing = self.check_manifest_entries(\
+                                               m_entry[k].file,reverse_dir[k])
+            if m_entry[k].missing:
+                msg = '{}: {} Error has the following missing entries'\
+                        .format(self.classname, m_entry[k].file)
+                logmsg = msg+':\n'+'\n'.join(m_entry[k].missing)
+                self.log.debug(logmsg)
+                self.log.info(msg)
+                self.fail(logmsg)
+
+    def test_image_manifest_entries(self):
+        '''Verifying the image manifest entries exist'''
+
+        # get manifest location based on target to query about
+        try:
+            mdir = self.get_dir_from_bb_var('DEPLOY_DIR_IMAGE',
+                                                self.buildtarget)
+            mfilename = get_bb_var("IMAGE_LINK_NAME", self.buildtarget)\
+                    + ".manifest"
+            mpath = os.path.join(mdir, mfilename)
+            if not os.path.isfile(mpath): raise IOError
+            m_entry = ManifestEntry(mpath)
+
+            pkgdata_dir = {}
+            pkgdata_dir = self.get_dir_from_bb_var('PKGDATA_DIR',
+                                                self.buildtarget)
+            revdir = os.path.join(pkgdata_dir, 'runtime-reverse')
+            if not os.path.exists(revdir): raise IOError
+        except OSError:
+            raise unittest.SkipTest("{}: Error in obtaining manifest dirs"\
+                .format(self.classname))
+        except IOError:
+            msg = "{}: Error cannot find manifests in dir:\n{}"\
+                    .format(self.classname, mdir)
+            self.fail(msg)
+
+        self.log.debug("{}: Check manifest {}"\
+                            .format(self.classname, m_entry.file))
+        m_entry.missing = self.check_manifest_entries(\
+                                                    m_entry.file, revdir)
+        if m_entry.missing:
+            msg = '{}: {} Error has the following missing entries'\
+                    .format(self.classname, m_entry.file)
+            logmsg = msg+':\n'+'\n'.join(m_entry.missing)
+            self.log.debug(logmsg)
+            self.log.info(msg)
+            self.fail(logmsg)
-- 
2.4.3

Re: [PATCH V3] selftest/manifest.py: Test support for manifests

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 372 bytes --]

On 7 October 2015 at 20:41, Benjamin Esquivel <
benjamin.esquivel@linux.intel.com> wrote:

> tests added:
>  - manifest.VerifyManifest.test_image_manifest_entries
>

This test case isn't exposing a known bug in the manifests, where the
installed package list and the manifest don't match, specifically the
manifest is missing the locale packages (8444).

Ross

[-- Attachment #2: Type: text/html, Size: 819 bytes --]

Re: [PATCH V3] selftest/manifest.py: Test support for manifests

[Benjamin Esquivel]

[-- Attachment #1: Type: text/plain, Size: 843 bytes --]

this test does a one way check from the .manifest to the pkgdata dir.
This test is looking for entries in the manifest that do not exist in
the pkgdata dir. The bug you mention is the other way around, things
that are installed in the rootfs but not listed in the manifest. I
believe a test for this would be necessary too though I am not sure
that the bug is a genuine error yet.
Benjamin 
On Fri, 2015-10-16 at 17:20 +0100, Burton, Ross wrote:
> On 7 October 2015 at 20:41, Benjamin Esquivel <
> benjamin.esquivel@linux.intel.com> wrote:
> > tests added:
> >  - manifest.VerifyManifest.test_image_manifest_entries
> > 
> This test case isn't exposing a known bug in the manifests, where the
> installed package list and the manifest don't match, specifically the
> manifest is missing the locale packages (8444).
> 
> Ross

[-- Attachment #2: Type: text/html, Size: 1330 bytes --]