[PATCH 0/3] sanity: improve mirrors check

[PATCH 0/3] sanity: improve mirrors check

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

The following changes since commit 217aa4d9802609d2c3628c8751a27a5d25900898:

  autogen-native: inherit pkgconfig to fix a build failure (2014-08-25 10:25:54 +0100)

are available in the git repository at:

  git@github.com:kergoth/openembedded-core improve-sanity-mirrors-check

for you to fetch changes up to 47b5636f4c87979aecfbcb743235a9e010bedd23:

  sanity: refactor mirrors checks to be more pythonic (2014-08-25 15:45:19 -0700)

----------------------------------------------------------------
Christopher Larson (3):
      sanity: handle both \n and \n in mirror vars
      sanity: fix support for regex schemes in mirrors check
      sanity: refactor mirrors checks to be more pythonic

 meta/classes/sanity.bbclass | 67 ++++++++++++++++++++++++---------------------
 1 file changed, 36 insertions(+), 31 deletions(-)

-- 
1.8.3.4

[PATCH 1/3] sanity: handle both \n and \n in mirror vars

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/sanity.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index dbcc26b..7cfc4be 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d):
         'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
         'bzr://', 'cvs://']
     for mir_type in mir_types:
-        mirros = (d.getVar(mir_type, True) or '').split('\\n')
+        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')
         for mir in mirros:
             mir_list = mir.split()
             # Should be two members.
-- 
1.8.3.4

[PATCH 2/3] sanity: fix support for regex schemes in mirrors check

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/sanity.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 7cfc4be..f655d8b 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -754,8 +754,9 @@ def check_sanity_everybuild(status, d):
         status.addresult("Error, you have a space in your COREBASE directory path. Please move the installation to a directory which doesn't include a space since autotools doesn't support this.")
 
     # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS
+    import re
     mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
-    protocols = ['http://', 'ftp://', 'file://', 'https://', 'https?$://', \
+    protocols = ['http://', 'ftp://', 'file://', 'https://', \
         'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
         'bzr://', 'cvs://']
     for mir_type in mir_types:
@@ -767,12 +768,19 @@ def check_sanity_everybuild(status, d):
                 bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \
                     % (mir_type, mir.strip(), len(mir_list)))
             elif len(mir_list) == 2:
+                decoded = bb.fetch2.decodeurl(mir_list[0])
+                try:
+                    pattern_scheme = re.compile(decoded[0])
+                except re.error as exc:
+                    bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip()))
+                    continue
+
                 # Each member should start with protocols
                 valid_protocol_0 = False
                 valid_protocol_1 = False
                 file_absolute = True
                 for protocol in protocols:
-                    if not valid_protocol_0 and mir_list[0].startswith(protocol):
+                    if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]):
                         valid_protocol_0 = True
                     if not valid_protocol_1 and mir_list[1].startswith(protocol):
                         valid_protocol_1 = True
-- 
1.8.3.4

[PATCH 3/3] sanity: refactor mirrors checks to be more pythonic

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

- Use clearer variable names
- Use variable unpacking to reference elements by name rather than index
- Sacrifice a small amount of time (iterate over protocols twice per entry
  rather than once) for clarity: use readable generator expressions with any()
  rather than maintaining state.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/sanity.bbclass | 73 ++++++++++++++++++++++-----------------------
 1 file changed, 35 insertions(+), 38 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index f655d8b..5be5efb 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -755,44 +755,41 @@ def check_sanity_everybuild(status, d):
 
     # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS
     import re
-    mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
-    protocols = ['http://', 'ftp://', 'file://', 'https://', \
-        'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
-        'bzr://', 'cvs://']
-    for mir_type in mir_types:
-        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')
-        for mir in mirros:
-            mir_list = mir.split()
-            # Should be two members.
-            if len(mir_list) not in [0, 2]:
-                bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \
-                    % (mir_type, mir.strip(), len(mir_list)))
-            elif len(mir_list) == 2:
-                decoded = bb.fetch2.decodeurl(mir_list[0])
-                try:
-                    pattern_scheme = re.compile(decoded[0])
-                except re.error as exc:
-                    bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip()))
-                    continue
-
-                # Each member should start with protocols
-                valid_protocol_0 = False
-                valid_protocol_1 = False
-                file_absolute = True
-                for protocol in protocols:
-                    if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]):
-                        valid_protocol_0 = True
-                    if not valid_protocol_1 and mir_list[1].startswith(protocol):
-                        valid_protocol_1 = True
-                        # The file:// must be an absolute path.
-                        if protocol == 'file://' and not mir_list[1].startswith('file:///'):
-                            file_absolute = False
-                    if valid_protocol_0 and valid_protocol_1:
-                        break
-                if not (valid_protocol_0 and valid_protocol_1):
-                    bb.warn('Invalid protocol in %s: %s' % (mir_type, mir.strip()))
-                if not file_absolute:
-                    bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mir_type, mir.strip()))
+    mirror_vars = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
+    protocols = ['http', 'ftp', 'file', 'https', \
+                 'git', 'gitsm', 'hg', 'osc', 'p4', 'svk', 'svn', \
+                 'bzr', 'cvs']
+    for mirror_var in mirror_vars:
+        mirrors = (d.getVar(mirror_var, True) or '').replace('\\n', '\n').split('\n')
+        for mirror_entry in mirrors:
+            mirror_entry = mirror_entry.strip()
+            if not mirror_entry:
+                # ignore blank lines
+                continue
+
+            try:
+                pattern, mirror = mirror_entry.split()
+            except ValueError:
+                bb.warn('Invalid %s: %s, should be 2 members.' % (mirror_var, mirror_entry.strip()))
+                continue
+
+            decoded = bb.fetch2.decodeurl(pattern)
+            try:
+                pattern_scheme = re.compile(decoded[0])
+            except re.error as exc:
+                bb.warn('Invalid scheme regex (%s) in %s; %s' % (pattern, mirror_var, mirror_entry))
+                continue
+
+            if not any(pattern_scheme.match(protocol) for protocol in protocols):
+                bb.warn('Invalid protocol (%s) in %s: %s' % (decoded[0], mirror_var, mirror_entry))
+                continue
+
+            if not any(mirror.startswith(protocol + '://') for protocol in protocols):
+                bb.warn('Invalid protocol in %s: %s' % (mirror_var, mirror_entry))
+                continue
+
+            if mirror.startswith('file://') and not mirror.startswith('file:///'):
+                bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mirror_var, mirror_entry))
 
     # Check that TMPDIR hasn't changed location since the last time we were run
     tmpdir = d.getVar('TMPDIR', True)
-- 
1.8.3.4

Re: [PATCH 1/3] sanity: handle both \n and \n in mirror vars

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 258 bytes --]

On Mon, Aug 25, 2014 at 3:46 PM, Christopher Larson <kergoth@gmail.com>
wrote:

> From: Christopher Larson <chris_larson@mentor.com>
>
> Signed-off-by: Christopher Larson <chris_larson@mentor.com>
>


Bah, typo in the subject, fixing. :)

-Chris

[-- Attachment #2: Type: text/html, Size: 786 bytes --]

[PATCHv2 0/3] sanity: improve mirrors check

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

v2: fixed subject line of first commit to mention \\n

The following changes since commit 217aa4d9802609d2c3628c8751a27a5d25900898:

  autogen-native: inherit pkgconfig to fix a build failure (2014-08-25 10:25:54 +0100)

are available in the git repository at:

  git@github.com:kergoth/openembedded-core improve-sanity-mirrors-check

for you to fetch changes up to 8fa6adaaa363a88c888f2afab2f0117b199a54c0:

  sanity: refactor mirrors checks to be more pythonic (2014-08-25 15:50:53 -0700)

----------------------------------------------------------------
Christopher Larson (3):
      sanity: handle both \n and \\n in mirror vars
      sanity: fix support for regex schemes in mirrors check
      sanity: refactor mirrors checks to be more pythonic

 meta/classes/sanity.bbclass | 67 ++++++++++++++++++++++++---------------------
 1 file changed, 36 insertions(+), 31 deletions(-)

-- 
1.8.3.4

[PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/sanity.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index dbcc26b..7cfc4be 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d):
         'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
         'bzr://', 'cvs://']
     for mir_type in mir_types:
-        mirros = (d.getVar(mir_type, True) or '').split('\\n')
+        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')
         for mir in mirros:
             mir_list = mir.split()
             # Should be two members.
-- 
1.8.3.4

[PATCHv2 2/3] sanity: fix support for regex schemes in mirrors check

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/sanity.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 7cfc4be..f655d8b 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -754,8 +754,9 @@ def check_sanity_everybuild(status, d):
         status.addresult("Error, you have a space in your COREBASE directory path. Please move the installation to a directory which doesn't include a space since autotools doesn't support this.")
 
     # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS
+    import re
     mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
-    protocols = ['http://', 'ftp://', 'file://', 'https://', 'https?$://', \
+    protocols = ['http://', 'ftp://', 'file://', 'https://', \
         'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
         'bzr://', 'cvs://']
     for mir_type in mir_types:
@@ -767,12 +768,19 @@ def check_sanity_everybuild(status, d):
                 bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \
                     % (mir_type, mir.strip(), len(mir_list)))
             elif len(mir_list) == 2:
+                decoded = bb.fetch2.decodeurl(mir_list[0])
+                try:
+                    pattern_scheme = re.compile(decoded[0])
+                except re.error as exc:
+                    bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip()))
+                    continue
+
                 # Each member should start with protocols
                 valid_protocol_0 = False
                 valid_protocol_1 = False
                 file_absolute = True
                 for protocol in protocols:
-                    if not valid_protocol_0 and mir_list[0].startswith(protocol):
+                    if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]):
                         valid_protocol_0 = True
                     if not valid_protocol_1 and mir_list[1].startswith(protocol):
                         valid_protocol_1 = True
-- 
1.8.3.4

[PATCHv2 3/3] sanity: refactor mirrors checks to be more pythonic

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

- Use clearer variable names
- Use variable unpacking to reference elements by name rather than index
- Sacrifice a small amount of time (iterate over protocols twice per entry
  rather than once) for clarity: use readable generator expressions with any()
  rather than maintaining state.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/sanity.bbclass | 73 ++++++++++++++++++++++-----------------------
 1 file changed, 35 insertions(+), 38 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index f655d8b..5be5efb 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -755,44 +755,41 @@ def check_sanity_everybuild(status, d):
 
     # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS
     import re
-    mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
-    protocols = ['http://', 'ftp://', 'file://', 'https://', \
-        'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
-        'bzr://', 'cvs://']
-    for mir_type in mir_types:
-        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')
-        for mir in mirros:
-            mir_list = mir.split()
-            # Should be two members.
-            if len(mir_list) not in [0, 2]:
-                bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \
-                    % (mir_type, mir.strip(), len(mir_list)))
-            elif len(mir_list) == 2:
-                decoded = bb.fetch2.decodeurl(mir_list[0])
-                try:
-                    pattern_scheme = re.compile(decoded[0])
-                except re.error as exc:
-                    bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip()))
-                    continue
-
-                # Each member should start with protocols
-                valid_protocol_0 = False
-                valid_protocol_1 = False
-                file_absolute = True
-                for protocol in protocols:
-                    if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]):
-                        valid_protocol_0 = True
-                    if not valid_protocol_1 and mir_list[1].startswith(protocol):
-                        valid_protocol_1 = True
-                        # The file:// must be an absolute path.
-                        if protocol == 'file://' and not mir_list[1].startswith('file:///'):
-                            file_absolute = False
-                    if valid_protocol_0 and valid_protocol_1:
-                        break
-                if not (valid_protocol_0 and valid_protocol_1):
-                    bb.warn('Invalid protocol in %s: %s' % (mir_type, mir.strip()))
-                if not file_absolute:
-                    bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mir_type, mir.strip()))
+    mirror_vars = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS']
+    protocols = ['http', 'ftp', 'file', 'https', \
+                 'git', 'gitsm', 'hg', 'osc', 'p4', 'svk', 'svn', \
+                 'bzr', 'cvs']
+    for mirror_var in mirror_vars:
+        mirrors = (d.getVar(mirror_var, True) or '').replace('\\n', '\n').split('\n')
+        for mirror_entry in mirrors:
+            mirror_entry = mirror_entry.strip()
+            if not mirror_entry:
+                # ignore blank lines
+                continue
+
+            try:
+                pattern, mirror = mirror_entry.split()
+            except ValueError:
+                bb.warn('Invalid %s: %s, should be 2 members.' % (mirror_var, mirror_entry.strip()))
+                continue
+
+            decoded = bb.fetch2.decodeurl(pattern)
+            try:
+                pattern_scheme = re.compile(decoded[0])
+            except re.error as exc:
+                bb.warn('Invalid scheme regex (%s) in %s; %s' % (pattern, mirror_var, mirror_entry))
+                continue
+
+            if not any(pattern_scheme.match(protocol) for protocol in protocols):
+                bb.warn('Invalid protocol (%s) in %s: %s' % (decoded[0], mirror_var, mirror_entry))
+                continue
+
+            if not any(mirror.startswith(protocol + '://') for protocol in protocols):
+                bb.warn('Invalid protocol in %s: %s' % (mirror_var, mirror_entry))
+                continue
+
+            if mirror.startswith('file://') and not mirror.startswith('file:///'):
+                bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mirror_var, mirror_entry))
 
     # Check that TMPDIR hasn't changed location since the last time we were run
     tmpdir = d.getVar('TMPDIR', True)
-- 
1.8.3.4

Re: [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1301 bytes --]

On Mon, Aug 25, 2014 at 03:57:42PM -0700, Christopher Larson wrote:
> From: Christopher Larson <chris_larson@mentor.com>
> 
> Signed-off-by: Christopher Larson <chris_larson@mentor.com>
> ---
>  meta/classes/sanity.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index dbcc26b..7cfc4be 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d):
>          'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
>          'bzr://', 'cvs://']
>      for mir_type in mir_types:
> -        mirros = (d.getVar(mir_type, True) or '').split('\\n')
> +        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')

Would you mind fixing the typo in variable name as well?

>          for mir in mirros:
>              mir_list = mir.split()
>              # Should be two members.
> -- 
> 1.8.3.4
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1554 bytes --]

On Tue, Aug 26, 2014 at 12:51:49PM +0200, Martin Jansa wrote:
> On Mon, Aug 25, 2014 at 03:57:42PM -0700, Christopher Larson wrote:
> > From: Christopher Larson <chris_larson@mentor.com>
> > 
> > Signed-off-by: Christopher Larson <chris_larson@mentor.com>
> > ---
> >  meta/classes/sanity.bbclass | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> > index dbcc26b..7cfc4be 100644
> > --- a/meta/classes/sanity.bbclass
> > +++ b/meta/classes/sanity.bbclass
> > @@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d):
> >          'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
> >          'bzr://', 'cvs://']
> >      for mir_type in mir_types:
> > -        mirros = (d.getVar(mir_type, True) or '').split('\\n')
> > +        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')
> 
> Would you mind fixing the typo in variable name as well?

Nevermind, it's refactored in 3/3, sorry for noise.

> >          for mir in mirros:
> >              mir_list = mir.split()
> >              # Should be two members.
> > -- 
> > 1.8.3.4
> > 
> > -- 
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
> 
> -- 
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com



-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars

[Koen Kooi]

Op 26 aug. 2014, om 00:57 heeft Christopher Larson <kergoth@gmail.com> het volgende geschreven:

> From: Christopher Larson <chris_larson@mentor.com>
> 
> Signed-off-by: Christopher Larson <chris_larson@mentor.com>

Tested-by: Koen Kooi <koen@dominion.thruhere.net>

> ---
> meta/classes/sanity.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index dbcc26b..7cfc4be 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d):
>         'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \
>         'bzr://', 'cvs://']
>     for mir_type in mir_types:
> -        mirros = (d.getVar(mir_type, True) or '').split('\\n')
> +        mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n')
>         for mir in mirros:
>             mir_list = mir.split()
>             # Should be two members.
> -- 
> 1.8.3.4
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>