[PATCH 0/4] Make oe.lsb.distro_identifier() more consistent

[PATCH 0/4] Make oe.lsb.distro_identifier() more consistent

[Joshua Lock]

The oe.lsb.distro_identifier() method call will return different identification
information depending on the source which is found to provide that information.

This series attempts to address this in two ways:
1) preferring os-release(5) as the source of distribution identification. this
  increasingly common standard mechanism is available on each of the build host
  distributions we commonly test on.
2) converting the distribution identifier to lower case before including it in
  the distro_identifier return value. This ensures that, for most of the tested
  distros, the identifier returned via the LSB code paths matches that returned
  by the os-release code paths.

The following changes since commit 98c6ebf1e05158c689e01b785d32757847cdb10c:

  oeqa/selftest/kernel.py: Add new file destined for kernel related tests (2016-11-01 10:05:40 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib joshuagl/liboe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=joshuagl/liboe

Joshua Lock (4):
  lib/oe/path: remove duplicate import
  lib/oe/lsb: make the release dict keys consistent regardless of source
  lib/oe/lsb: prefer /etc/os-release for distribution data
  lib/oe/lsb: attempt to ensure consistent distro id regardless of
    source

 meta/lib/oe/lsb.py  | 70 +++++++++++++++++++++++++++++++++++------------------
 meta/lib/oe/path.py |  1 -
 2 files changed, 46 insertions(+), 25 deletions(-)

-- 
2.7.4

[PATCH 1/4] lib/oe/path: remove duplicate import

[Joshua Lock]

There's no need to import glob inside copyhardlinktree() as it's
already imported for the entire path module.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/lib/oe/path.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 06a5af2..f73fd4a 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -81,7 +81,6 @@ def copyhardlinktree(src, dst):
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
         source = ''
         if os.path.isdir(src):
-            import glob
             if len(glob.glob('%s/.??*' % src)) > 0:
                 source = '%s/.??* ' % src
             source = source + '%s/*' % src
-- 
2.7.4

[PATCH 2/4] lib/oe/lsb: make the release dict keys consistent regardless of source

[Joshua Lock]

Rather than have the distro_identifier method look for different keys in
the dict depending on the source ensure that each function for retrieving
release data uses the same key names in the returned dict.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/lib/oe/lsb.py | 36 +++++++++++++++++++++---------------
 1 file changed, 21 insertions(+), 15 deletions(-)

diff --git a/meta/lib/oe/lsb.py b/meta/lib/oe/lsb.py
index e0bdfba..0bb7686 100644
--- a/meta/lib/oe/lsb.py
+++ b/meta/lib/oe/lsb.py
@@ -1,5 +1,5 @@
-def release_dict():
-    """Return the output of lsb_release -ir as a dictionary"""
+def release_dict_lsb():
+    """ Return the output of lsb_release -ir as a dictionary """
     from subprocess import PIPE
 
     try:
@@ -7,19 +7,28 @@ def release_dict():
     except bb.process.CmdError as exc:
         return None
 
+    lsb_map = { 'Distributor ID': 'DISTRIB_ID',
+                'Release': 'DISTRIB_RELEASE'}
+    lsb_keys = lsb_map.keys()
+
     data = {}
     for line in output.splitlines():
-        if line.startswith("-e"): line = line[3:]
+        if line.startswith("-e"):
+            line = line[3:]
         try:
             key, value = line.split(":\t", 1)
         except ValueError:
             continue
-        else:
-            data[key] = value
+        if key in lsb_keys:
+            data[lsb_map[key]] = value
+
+    if len(data.keys()) != 2:
+        return None
+
     return data
 
 def release_dict_file():
-    """ Try to gather LSB release information manually when lsb_release tool is unavailable """
+    """ Try to gather release information manually when other methods fail """
     data = None
     try:
         if os.path.exists('/etc/lsb-release'):
@@ -64,15 +73,12 @@ def distro_identifier(adjust_hook=None):
 
     import re
 
-    lsb_data = release_dict()
-    if lsb_data:
-        distro_id, release = lsb_data['Distributor ID'], lsb_data['Release']
-    else:
-        lsb_data_file = release_dict_file()
-        if lsb_data_file:
-            distro_id, release = lsb_data_file['DISTRIB_ID'], lsb_data_file.get('DISTRIB_RELEASE', None)
-        else:
-            distro_id, release = None, None
+    distro_data = release_dict_lsb()
+    if not distro_data:
+        distro_data = release_dict_file()
+
+    distro_id = distro_data['DISTRIB_ID']
+    release = distro_data['DISTRIB_RELEASE']
 
     if adjust_hook:
         distro_id, release = adjust_hook(distro_id, release)
-- 
2.7.4

[PATCH 3/4] lib/oe/lsb: prefer /etc/os-release for distribution data

[Joshua Lock]

os-release(5) is an increasingly standard source of operating system
identification and more likely to be present on modern OS deployments, i.e.
many container variants of common distros include os-release and not the
lsb_release tool.

Therefore we should favour parsing /etc/os-release in distro_identifier(),
try lsb_release when that fails and finally fall back on various distro
specific sources of OS identification.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/lib/oe/lsb.py | 34 +++++++++++++++++++++++++---------
 1 file changed, 25 insertions(+), 9 deletions(-)

diff --git a/meta/lib/oe/lsb.py b/meta/lib/oe/lsb.py
index 0bb7686..bdc893f 100644
--- a/meta/lib/oe/lsb.py
+++ b/meta/lib/oe/lsb.py
@@ -1,3 +1,22 @@
+def release_dict_osr():
+    """ Populate a dict with pertinent values from /etc/os-release """
+    if not os.path.exists('/etc/os-release'):
+        return None
+
+    data = {}
+    with open('/etc/os-release') as f:
+        for line in f:
+            key, val = line.rstrip().split('=', 1)
+            if key == 'NAME':
+                data['DISTRIB_ID'] = val
+            if key == 'VERSION_ID':
+                data['DISTRIB_RELEASE'] = val
+
+    if len(data.keys()) != 2:
+        return None
+
+    return data
+
 def release_dict_lsb():
     """ Return the output of lsb_release -ir as a dictionary """
     from subprocess import PIPE
@@ -46,14 +65,6 @@ def release_dict_file():
             if match:
                 data['DISTRIB_ID'] = match.group(1)
                 data['DISTRIB_RELEASE'] = match.group(2)
-        elif os.path.exists('/etc/os-release'):
-            data = {}
-            with open('/etc/os-release') as f:
-                for line in f:
-                    if line.startswith('NAME='):
-                        data['DISTRIB_ID'] = line[5:].rstrip().strip('"')
-                    if line.startswith('VERSION_ID='):
-                        data['DISTRIB_RELEASE'] = line[11:].rstrip().strip('"')
         elif os.path.exists('/etc/SuSE-release'):
             data = {}
             data['DISTRIB_ID'] = 'SUSE LINUX'
@@ -73,7 +84,12 @@ def distro_identifier(adjust_hook=None):
 
     import re
 
-    distro_data = release_dict_lsb()
+    # Try /etc/os-release first, then the output of `lsb_release -ir` and
+    # finally fall back on parsing various release files in order to determine
+    # host distro name and version.
+    distro_data = release_dict_osr()
+    if not distro_data:
+        distro_data = release_dict_lsb()
     if not distro_data:
         distro_data = release_dict_file()
 
-- 
2.7.4

[PATCH 4/4] lib/oe/lsb: attempt to ensure consistent distro id regardless of source

[Joshua Lock]

The LSB Distributor ID and os-release NAME differ for most of the
distributions tested by the Yocto Project (CentOS, Debian, Fedora,
openSUSE and Ubuntu) however for all but openSUSE the os-release ID
matches the LSB Distributor ID when both are lowered before
comparison.

Therefore, in order to improve the consistency of identification of
a distribution, switch to using the os-release ID and converting
the ID value to lowercase.

Table showing comparison of LSB Distributor ID to os-release fields NAME
and ID for current Yocto Project supported host distributions:

Distribution | Version | Distributor ID   | NAME             | ID       |
-------------------------------------------------------------------------
CentOS       | 7       | CentOS           | CentOS Linux     | centos   |
Debian       | 8       | Debian           | Debian GNU/Linux | debian   |
Fedora       | 23      | Fedora           | Fedora           | fedora   |
Fedora       | 24      | Fedora           | Fedora           | fedora   |
openSUSE     | 13.2    | openSUSE project | openSUSE         | opensuse |
openSUSE     | 42.1    | SUSE LINUX       | openSUSE Leap    | opensuse |
Ubuntu       | 14.04   | Ubuntu           | Ubuntu           | ubuntu   |
Ubuntu       | 16.04   | Ubuntu           | Ubuntu           | ubuntu   |

[YOCTO #10591]

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/lib/oe/lsb.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/lsb.py b/meta/lib/oe/lsb.py
index bdc893f..cd43731 100644
--- a/meta/lib/oe/lsb.py
+++ b/meta/lib/oe/lsb.py
@@ -7,7 +7,7 @@ def release_dict_osr():
     with open('/etc/os-release') as f:
         for line in f:
             key, val = line.rstrip().split('=', 1)
-            if key == 'NAME':
+            if key == 'ID':
                 data['DISTRIB_ID'] = val
             if key == 'VERSION_ID':
                 data['DISTRIB_RELEASE'] = val
@@ -104,7 +104,7 @@ def distro_identifier(adjust_hook=None):
     distro_id = re.sub(r'\W', '', distro_id)
 
     if release:
-        id_str = '{0}-{1}'.format(distro_id, release)
+        id_str = '{0}-{1}'.format(distro_id.lower(), release)
     else:
         id_str = distro_id
     return id_str.replace(' ','-').replace('/','-')
-- 
2.7.4