[PATCH v2 0/4] Make oe.lsb.distro_identifier() more consistent

[PATCH v2 0/4] Make oe.lsb.distro_identifier() more consistent

[Joshua Lock]

The oe.lsb.distro_identifier() method call will return different identification
information depending on the source which is found to provide that information.

This series attempts to address this in two ways:
1) preferring os-release(5) as the source of distribution identification. this
  increasingly common standard mechanism is available on each of the build host
  distributions we commonly test on.
2) converting the distribution identifier to lower case before including it in
  the distro_identifier return value. This ensures that, for most of the tested
  distros, the identifier returned via the LSB code paths matches that returned
  by the os-release code paths.

Changes since v1:
* improve release_dict_osr() in patch 3 to handle empty lines and values with surrounding quotation marks in /etc/os-release, as used in CentOS


The following changes since commit 9303d8055c45a0f6af295d70a6f6a8b9d8d8a7c9:

  devtool: add "rename" subcommand (2016-11-07 11:04:17 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib joshuagl/liboe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=joshuagl/liboe

Joshua Lock (4):
  lib/oe/path: remove duplicate import
  lib/oe/lsb: make the release dict keys consistent regardless of source
  lib/oe/lsb: prefer /etc/os-release for distribution data
  lib/oe/lsb: attempt to ensure consistent distro id regardless of
    source

 meta/lib/oe/lsb.py  | 73 +++++++++++++++++++++++++++++++++++------------------
 meta/lib/oe/path.py |  1 -
 2 files changed, 49 insertions(+), 25 deletions(-)

-- 
2.7.4

[PATCH v2 1/4] lib/oe/path: remove duplicate import

[Joshua Lock]

There's no need to import glob inside copyhardlinktree() as it's
already imported for the entire path module.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/lib/oe/path.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 06a5af2..f73fd4a 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -81,7 +81,6 @@ def copyhardlinktree(src, dst):
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
         source = ''
         if os.path.isdir(src):
-            import glob
             if len(glob.glob('%s/.??*' % src)) > 0:
                 source = '%s/.??* ' % src
             source = source + '%s/*' % src
-- 
2.7.4

[PATCH v2 2/4] lib/oe/lsb: make the release dict keys consistent regardless of source

[Joshua Lock]

Rather than have the distro_identifier method look for different keys in
the dict depending on the source ensure that each function for retrieving
release data uses the same key names in the returned dict.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/lib/oe/lsb.py | 36 +++++++++++++++++++++---------------
 1 file changed, 21 insertions(+), 15 deletions(-)

diff --git a/meta/lib/oe/lsb.py b/meta/lib/oe/lsb.py
index e0bdfba..0bb7686 100644
--- a/meta/lib/oe/lsb.py
+++ b/meta/lib/oe/lsb.py
@@ -1,5 +1,5 @@
-def release_dict():
-    """Return the output of lsb_release -ir as a dictionary"""
+def release_dict_lsb():
+    """ Return the output of lsb_release -ir as a dictionary """
     from subprocess import PIPE
 
     try:
@@ -7,19 +7,28 @@ def release_dict():
     except bb.process.CmdError as exc:
         return None
 
+    lsb_map = { 'Distributor ID': 'DISTRIB_ID',
+                'Release': 'DISTRIB_RELEASE'}
+    lsb_keys = lsb_map.keys()
+
     data = {}
     for line in output.splitlines():
-        if line.startswith("-e"): line = line[3:]
+        if line.startswith("-e"):
+            line = line[3:]
         try:
             key, value = line.split(":\t", 1)
         except ValueError:
             continue
-        else:
-            data[key] = value
+        if key in lsb_keys:
+            data[lsb_map[key]] = value
+
+    if len(data.keys()) != 2:
+        return None
+
     return data
 
 def release_dict_file():
-    """ Try to gather LSB release information manually when lsb_release tool is unavailable """
+    """ Try to gather release information manually when other methods fail """
     data = None
     try:
         if os.path.exists('/etc/lsb-release'):
@@ -64,15 +73,12 @@ def distro_identifier(adjust_hook=None):
 
     import re
 
-    lsb_data = release_dict()
-    if lsb_data:
-        distro_id, release = lsb_data['Distributor ID'], lsb_data['Release']
-    else:
-        lsb_data_file = release_dict_file()
-        if lsb_data_file:
-            distro_id, release = lsb_data_file['DISTRIB_ID'], lsb_data_file.get('DISTRIB_RELEASE', None)
-        else:
-            distro_id, release = None, None
+    distro_data = release_dict_lsb()
+    if not distro_data:
+        distro_data = release_dict_file()
+
+    distro_id = distro_data['DISTRIB_ID']
+    release = distro_data['DISTRIB_RELEASE']
 
     if adjust_hook:
         distro_id, release = adjust_hook(distro_id, release)
-- 
2.7.4

[PATCH v2 3/4] lib/oe/lsb: prefer /etc/os-release for distribution data

[Joshua Lock]

os-release(5) is an increasingly standard source of operating system
identification and more likely to be present on modern OS deployments, i.e.
many container variants of common distros include os-release and not the
lsb_release tool.

Therefore we should favour parsing /etc/os-release in distro_identifier(),
try lsb_release when that fails and finally fall back on various distro
specific sources of OS identification.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/lib/oe/lsb.py | 37 ++++++++++++++++++++++++++++---------
 1 file changed, 28 insertions(+), 9 deletions(-)

diff --git a/meta/lib/oe/lsb.py b/meta/lib/oe/lsb.py
index 0bb7686..8018c7b 100644
--- a/meta/lib/oe/lsb.py
+++ b/meta/lib/oe/lsb.py
@@ -1,3 +1,25 @@
+def release_dict_osr():
+    """ Populate a dict with pertinent values from /etc/os-release """
+    if not os.path.exists('/etc/os-release'):
+        return None
+
+    data = {}
+    with open('/etc/os-release') as f:
+        for line in f:
+            try:
+                key, val = line.rstrip().split('=', 1)
+            except ValueError:
+                continue
+            if key == 'NAME':
+                data['DISTRIB_ID'] = val.strip('"')
+            if key == 'VERSION_ID':
+                data['DISTRIB_RELEASE'] = val.strip('"')
+
+    if len(data.keys()) != 2:
+        return None
+
+    return data
+
 def release_dict_lsb():
     """ Return the output of lsb_release -ir as a dictionary """
     from subprocess import PIPE
@@ -46,14 +68,6 @@ def release_dict_file():
             if match:
                 data['DISTRIB_ID'] = match.group(1)
                 data['DISTRIB_RELEASE'] = match.group(2)
-        elif os.path.exists('/etc/os-release'):
-            data = {}
-            with open('/etc/os-release') as f:
-                for line in f:
-                    if line.startswith('NAME='):
-                        data['DISTRIB_ID'] = line[5:].rstrip().strip('"')
-                    if line.startswith('VERSION_ID='):
-                        data['DISTRIB_RELEASE'] = line[11:].rstrip().strip('"')
         elif os.path.exists('/etc/SuSE-release'):
             data = {}
             data['DISTRIB_ID'] = 'SUSE LINUX'
@@ -73,7 +87,12 @@ def distro_identifier(adjust_hook=None):
 
     import re
 
-    distro_data = release_dict_lsb()
+    # Try /etc/os-release first, then the output of `lsb_release -ir` and
+    # finally fall back on parsing various release files in order to determine
+    # host distro name and version.
+    distro_data = release_dict_osr()
+    if not distro_data:
+        distro_data = release_dict_lsb()
     if not distro_data:
         distro_data = release_dict_file()
 
-- 
2.7.4

[PATCH v2 4/4] lib/oe/lsb: attempt to ensure consistent distro id regardless of source

[Joshua Lock]

The LSB Distributor ID and os-release NAME differ for most of the
distributions tested by the Yocto Project (CentOS, Debian, Fedora,
openSUSE and Ubuntu) however for all but openSUSE the os-release ID
matches the LSB Distributor ID when both are lowered before
comparison.

Therefore, in order to improve the consistency of identification of
a distribution, switch to using the os-release ID and converting
the ID value to lowercase.

Table showing comparison of LSB Distributor ID to os-release fields NAME
and ID for current Yocto Project supported host distributions:

Distribution | Version | Distributor ID   | NAME             | ID       |
-------------------------------------------------------------------------
CentOS       | 7       | CentOS           | CentOS Linux     | centos   |
Debian       | 8       | Debian           | Debian GNU/Linux | debian   |
Fedora       | 23      | Fedora           | Fedora           | fedora   |
Fedora       | 24      | Fedora           | Fedora           | fedora   |
openSUSE     | 13.2    | openSUSE project | openSUSE         | opensuse |
openSUSE     | 42.1    | SUSE LINUX       | openSUSE Leap    | opensuse |
Ubuntu       | 14.04   | Ubuntu           | Ubuntu           | ubuntu   |
Ubuntu       | 16.04   | Ubuntu           | Ubuntu           | ubuntu   |

[YOCTO #10591]

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/lib/oe/lsb.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/lsb.py b/meta/lib/oe/lsb.py
index 8018c7b..5a795a1 100644
--- a/meta/lib/oe/lsb.py
+++ b/meta/lib/oe/lsb.py
@@ -10,7 +10,7 @@ def release_dict_osr():
                 key, val = line.rstrip().split('=', 1)
             except ValueError:
                 continue
-            if key == 'NAME':
+            if key == 'ID':
                 data['DISTRIB_ID'] = val.strip('"')
             if key == 'VERSION_ID':
                 data['DISTRIB_RELEASE'] = val.strip('"')
@@ -107,7 +107,7 @@ def distro_identifier(adjust_hook=None):
     distro_id = re.sub(r'\W', '', distro_id)
 
     if release:
-        id_str = '{0}-{1}'.format(distro_id, release)
+        id_str = '{0}-{1}'.format(distro_id.lower(), release)
     else:
         id_str = distro_id
     return id_str.replace(' ','-').replace('/','-')
-- 
2.7.4