* [PATCH 0/3] sanity: improve mirrors check
@ 2014-08-25 22:46 Christopher Larson
2014-08-25 22:46 ` [PATCH 1/3] sanity: handle both \n and \n in mirror vars Christopher Larson
` (3 more replies)
0 siblings, 4 replies; 12+ messages in thread
From: Christopher Larson @ 2014-08-25 22:46 UTC (permalink / raw)
To: openembedded-core; +Cc: Christopher Larson
From: Christopher Larson <chris_larson@mentor.com>
The following changes since commit 217aa4d9802609d2c3628c8751a27a5d25900898:
autogen-native: inherit pkgconfig to fix a build failure (2014-08-25 10:25:54 +0100)
are available in the git repository at:
git@github.com:kergoth/openembedded-core improve-sanity-mirrors-check
for you to fetch changes up to 47b5636f4c87979aecfbcb743235a9e010bedd23:
sanity: refactor mirrors checks to be more pythonic (2014-08-25 15:45:19 -0700)
----------------------------------------------------------------
Christopher Larson (3):
sanity: handle both \n and \n in mirror vars
sanity: fix support for regex schemes in mirrors check
sanity: refactor mirrors checks to be more pythonic
meta/classes/sanity.bbclass | 67 ++++++++++++++++++++++++---------------------
1 file changed, 36 insertions(+), 31 deletions(-)
--
1.8.3.4
^ permalink raw reply [flat|nested] 12+ messages in thread* [PATCH 1/3] sanity: handle both \n and \n in mirror vars 2014-08-25 22:46 [PATCH 0/3] sanity: improve mirrors check Christopher Larson @ 2014-08-25 22:46 ` Christopher Larson 2014-08-25 22:50 ` Christopher Larson 2014-08-25 22:46 ` [PATCH 2/3] sanity: fix support for regex schemes in mirrors check Christopher Larson ` (2 subsequent siblings) 3 siblings, 1 reply; 12+ messages in thread From: Christopher Larson @ 2014-08-25 22:46 UTC (permalink / raw) To: openembedded-core; +Cc: Christopher Larson From: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Christopher Larson <chris_larson@mentor.com> --- meta/classes/sanity.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass index dbcc26b..7cfc4be 100644 --- a/meta/classes/sanity.bbclass +++ b/meta/classes/sanity.bbclass @@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d): 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ 'bzr://', 'cvs://'] for mir_type in mir_types: - mirros = (d.getVar(mir_type, True) or '').split('\\n') + mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n') for mir in mirros: mir_list = mir.split() # Should be two members. -- 1.8.3.4 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH 1/3] sanity: handle both \n and \n in mirror vars 2014-08-25 22:46 ` [PATCH 1/3] sanity: handle both \n and \n in mirror vars Christopher Larson @ 2014-08-25 22:50 ` Christopher Larson 0 siblings, 0 replies; 12+ messages in thread From: Christopher Larson @ 2014-08-25 22:50 UTC (permalink / raw) To: Patches and discussions about the oe-core layer; +Cc: Christopher Larson [-- Attachment #1: Type: text/plain, Size: 258 bytes --] On Mon, Aug 25, 2014 at 3:46 PM, Christopher Larson <kergoth@gmail.com> wrote: > From: Christopher Larson <chris_larson@mentor.com> > > Signed-off-by: Christopher Larson <chris_larson@mentor.com> > Bah, typo in the subject, fixing. :) -Chris [-- Attachment #2: Type: text/html, Size: 786 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 2/3] sanity: fix support for regex schemes in mirrors check 2014-08-25 22:46 [PATCH 0/3] sanity: improve mirrors check Christopher Larson 2014-08-25 22:46 ` [PATCH 1/3] sanity: handle both \n and \n in mirror vars Christopher Larson @ 2014-08-25 22:46 ` Christopher Larson 2014-08-25 22:46 ` [PATCH 3/3] sanity: refactor mirrors checks to be more pythonic Christopher Larson 2014-08-25 22:57 ` [PATCHv2 0/3] sanity: improve mirrors check Christopher Larson 3 siblings, 0 replies; 12+ messages in thread From: Christopher Larson @ 2014-08-25 22:46 UTC (permalink / raw) To: openembedded-core; +Cc: Christopher Larson From: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Christopher Larson <chris_larson@mentor.com> --- meta/classes/sanity.bbclass | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass index 7cfc4be..f655d8b 100644 --- a/meta/classes/sanity.bbclass +++ b/meta/classes/sanity.bbclass @@ -754,8 +754,9 @@ def check_sanity_everybuild(status, d): status.addresult("Error, you have a space in your COREBASE directory path. Please move the installation to a directory which doesn't include a space since autotools doesn't support this.") # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS + import re mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS'] - protocols = ['http://', 'ftp://', 'file://', 'https://', 'https?$://', \ + protocols = ['http://', 'ftp://', 'file://', 'https://', \ 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ 'bzr://', 'cvs://'] for mir_type in mir_types: @@ -767,12 +768,19 @@ def check_sanity_everybuild(status, d): bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \ % (mir_type, mir.strip(), len(mir_list))) elif len(mir_list) == 2: + decoded = bb.fetch2.decodeurl(mir_list[0]) + try: + pattern_scheme = re.compile(decoded[0]) + except re.error as exc: + bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip())) + continue + # Each member should start with protocols valid_protocol_0 = False valid_protocol_1 = False file_absolute = True for protocol in protocols: - if not valid_protocol_0 and mir_list[0].startswith(protocol): + if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]): valid_protocol_0 = True if not valid_protocol_1 and mir_list[1].startswith(protocol): valid_protocol_1 = True -- 1.8.3.4 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 3/3] sanity: refactor mirrors checks to be more pythonic 2014-08-25 22:46 [PATCH 0/3] sanity: improve mirrors check Christopher Larson 2014-08-25 22:46 ` [PATCH 1/3] sanity: handle both \n and \n in mirror vars Christopher Larson 2014-08-25 22:46 ` [PATCH 2/3] sanity: fix support for regex schemes in mirrors check Christopher Larson @ 2014-08-25 22:46 ` Christopher Larson 2014-08-25 22:57 ` [PATCHv2 0/3] sanity: improve mirrors check Christopher Larson 3 siblings, 0 replies; 12+ messages in thread From: Christopher Larson @ 2014-08-25 22:46 UTC (permalink / raw) To: openembedded-core; +Cc: Christopher Larson From: Christopher Larson <chris_larson@mentor.com> - Use clearer variable names - Use variable unpacking to reference elements by name rather than index - Sacrifice a small amount of time (iterate over protocols twice per entry rather than once) for clarity: use readable generator expressions with any() rather than maintaining state. Signed-off-by: Christopher Larson <chris_larson@mentor.com> --- meta/classes/sanity.bbclass | 73 ++++++++++++++++++++++----------------------- 1 file changed, 35 insertions(+), 38 deletions(-) diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass index f655d8b..5be5efb 100644 --- a/meta/classes/sanity.bbclass +++ b/meta/classes/sanity.bbclass @@ -755,44 +755,41 @@ def check_sanity_everybuild(status, d): # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS import re - mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS'] - protocols = ['http://', 'ftp://', 'file://', 'https://', \ - 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ - 'bzr://', 'cvs://'] - for mir_type in mir_types: - mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n') - for mir in mirros: - mir_list = mir.split() - # Should be two members. - if len(mir_list) not in [0, 2]: - bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \ - % (mir_type, mir.strip(), len(mir_list))) - elif len(mir_list) == 2: - decoded = bb.fetch2.decodeurl(mir_list[0]) - try: - pattern_scheme = re.compile(decoded[0]) - except re.error as exc: - bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip())) - continue - - # Each member should start with protocols - valid_protocol_0 = False - valid_protocol_1 = False - file_absolute = True - for protocol in protocols: - if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]): - valid_protocol_0 = True - if not valid_protocol_1 and mir_list[1].startswith(protocol): - valid_protocol_1 = True - # The file:// must be an absolute path. - if protocol == 'file://' and not mir_list[1].startswith('file:///'): - file_absolute = False - if valid_protocol_0 and valid_protocol_1: - break - if not (valid_protocol_0 and valid_protocol_1): - bb.warn('Invalid protocol in %s: %s' % (mir_type, mir.strip())) - if not file_absolute: - bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mir_type, mir.strip())) + mirror_vars = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS'] + protocols = ['http', 'ftp', 'file', 'https', \ + 'git', 'gitsm', 'hg', 'osc', 'p4', 'svk', 'svn', \ + 'bzr', 'cvs'] + for mirror_var in mirror_vars: + mirrors = (d.getVar(mirror_var, True) or '').replace('\\n', '\n').split('\n') + for mirror_entry in mirrors: + mirror_entry = mirror_entry.strip() + if not mirror_entry: + # ignore blank lines + continue + + try: + pattern, mirror = mirror_entry.split() + except ValueError: + bb.warn('Invalid %s: %s, should be 2 members.' % (mirror_var, mirror_entry.strip())) + continue + + decoded = bb.fetch2.decodeurl(pattern) + try: + pattern_scheme = re.compile(decoded[0]) + except re.error as exc: + bb.warn('Invalid scheme regex (%s) in %s; %s' % (pattern, mirror_var, mirror_entry)) + continue + + if not any(pattern_scheme.match(protocol) for protocol in protocols): + bb.warn('Invalid protocol (%s) in %s: %s' % (decoded[0], mirror_var, mirror_entry)) + continue + + if not any(mirror.startswith(protocol + '://') for protocol in protocols): + bb.warn('Invalid protocol in %s: %s' % (mirror_var, mirror_entry)) + continue + + if mirror.startswith('file://') and not mirror.startswith('file:///'): + bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mirror_var, mirror_entry)) # Check that TMPDIR hasn't changed location since the last time we were run tmpdir = d.getVar('TMPDIR', True) -- 1.8.3.4 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCHv2 0/3] sanity: improve mirrors check 2014-08-25 22:46 [PATCH 0/3] sanity: improve mirrors check Christopher Larson ` (2 preceding siblings ...) 2014-08-25 22:46 ` [PATCH 3/3] sanity: refactor mirrors checks to be more pythonic Christopher Larson @ 2014-08-25 22:57 ` Christopher Larson 2014-08-25 22:57 ` [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars Christopher Larson ` (2 more replies) 3 siblings, 3 replies; 12+ messages in thread From: Christopher Larson @ 2014-08-25 22:57 UTC (permalink / raw) To: openembedded-core; +Cc: Christopher Larson From: Christopher Larson <chris_larson@mentor.com> v2: fixed subject line of first commit to mention \\n The following changes since commit 217aa4d9802609d2c3628c8751a27a5d25900898: autogen-native: inherit pkgconfig to fix a build failure (2014-08-25 10:25:54 +0100) are available in the git repository at: git@github.com:kergoth/openembedded-core improve-sanity-mirrors-check for you to fetch changes up to 8fa6adaaa363a88c888f2afab2f0117b199a54c0: sanity: refactor mirrors checks to be more pythonic (2014-08-25 15:50:53 -0700) ---------------------------------------------------------------- Christopher Larson (3): sanity: handle both \n and \\n in mirror vars sanity: fix support for regex schemes in mirrors check sanity: refactor mirrors checks to be more pythonic meta/classes/sanity.bbclass | 67 ++++++++++++++++++++++++--------------------- 1 file changed, 36 insertions(+), 31 deletions(-) -- 1.8.3.4 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars 2014-08-25 22:57 ` [PATCHv2 0/3] sanity: improve mirrors check Christopher Larson @ 2014-08-25 22:57 ` Christopher Larson 2014-08-26 10:51 ` Martin Jansa 2014-08-27 11:50 ` Koen Kooi 2014-08-25 22:57 ` [PATCHv2 2/3] sanity: fix support for regex schemes in mirrors check Christopher Larson 2014-08-25 22:57 ` [PATCHv2 3/3] sanity: refactor mirrors checks to be more pythonic Christopher Larson 2 siblings, 2 replies; 12+ messages in thread From: Christopher Larson @ 2014-08-25 22:57 UTC (permalink / raw) To: openembedded-core; +Cc: Christopher Larson From: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Christopher Larson <chris_larson@mentor.com> --- meta/classes/sanity.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass index dbcc26b..7cfc4be 100644 --- a/meta/classes/sanity.bbclass +++ b/meta/classes/sanity.bbclass @@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d): 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ 'bzr://', 'cvs://'] for mir_type in mir_types: - mirros = (d.getVar(mir_type, True) or '').split('\\n') + mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n') for mir in mirros: mir_list = mir.split() # Should be two members. -- 1.8.3.4 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars 2014-08-25 22:57 ` [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars Christopher Larson @ 2014-08-26 10:51 ` Martin Jansa 2014-08-26 10:59 ` Martin Jansa 2014-08-27 11:50 ` Koen Kooi 1 sibling, 1 reply; 12+ messages in thread From: Martin Jansa @ 2014-08-26 10:51 UTC (permalink / raw) To: Christopher Larson; +Cc: Christopher Larson, openembedded-core [-- Attachment #1: Type: text/plain, Size: 1301 bytes --] On Mon, Aug 25, 2014 at 03:57:42PM -0700, Christopher Larson wrote: > From: Christopher Larson <chris_larson@mentor.com> > > Signed-off-by: Christopher Larson <chris_larson@mentor.com> > --- > meta/classes/sanity.bbclass | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass > index dbcc26b..7cfc4be 100644 > --- a/meta/classes/sanity.bbclass > +++ b/meta/classes/sanity.bbclass > @@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d): > 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ > 'bzr://', 'cvs://'] > for mir_type in mir_types: > - mirros = (d.getVar(mir_type, True) or '').split('\\n') > + mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n') Would you mind fixing the typo in variable name as well? > for mir in mirros: > mir_list = mir.split() > # Should be two members. > -- > 1.8.3.4 > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 188 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars 2014-08-26 10:51 ` Martin Jansa @ 2014-08-26 10:59 ` Martin Jansa 0 siblings, 0 replies; 12+ messages in thread From: Martin Jansa @ 2014-08-26 10:59 UTC (permalink / raw) To: Christopher Larson; +Cc: Christopher Larson, openembedded-core [-- Attachment #1: Type: text/plain, Size: 1554 bytes --] On Tue, Aug 26, 2014 at 12:51:49PM +0200, Martin Jansa wrote: > On Mon, Aug 25, 2014 at 03:57:42PM -0700, Christopher Larson wrote: > > From: Christopher Larson <chris_larson@mentor.com> > > > > Signed-off-by: Christopher Larson <chris_larson@mentor.com> > > --- > > meta/classes/sanity.bbclass | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass > > index dbcc26b..7cfc4be 100644 > > --- a/meta/classes/sanity.bbclass > > +++ b/meta/classes/sanity.bbclass > > @@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d): > > 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ > > 'bzr://', 'cvs://'] > > for mir_type in mir_types: > > - mirros = (d.getVar(mir_type, True) or '').split('\\n') > > + mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n') > > Would you mind fixing the typo in variable name as well? Nevermind, it's refactored in 3/3, sorry for noise. > > for mir in mirros: > > mir_list = mir.split() > > # Should be two members. > > -- > > 1.8.3.4 > > > > -- > > _______________________________________________ > > Openembedded-core mailing list > > Openembedded-core@lists.openembedded.org > > http://lists.openembedded.org/mailman/listinfo/openembedded-core > > -- > Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 188 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars 2014-08-25 22:57 ` [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars Christopher Larson 2014-08-26 10:51 ` Martin Jansa @ 2014-08-27 11:50 ` Koen Kooi 1 sibling, 0 replies; 12+ messages in thread From: Koen Kooi @ 2014-08-27 11:50 UTC (permalink / raw) To: Christopher Larson; +Cc: Christopher Larson, openembedded-core Op 26 aug. 2014, om 00:57 heeft Christopher Larson <kergoth@gmail.com> het volgende geschreven: > From: Christopher Larson <chris_larson@mentor.com> > > Signed-off-by: Christopher Larson <chris_larson@mentor.com> Tested-by: Koen Kooi <koen@dominion.thruhere.net> > --- > meta/classes/sanity.bbclass | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass > index dbcc26b..7cfc4be 100644 > --- a/meta/classes/sanity.bbclass > +++ b/meta/classes/sanity.bbclass > @@ -759,7 +759,7 @@ def check_sanity_everybuild(status, d): > 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ > 'bzr://', 'cvs://'] > for mir_type in mir_types: > - mirros = (d.getVar(mir_type, True) or '').split('\\n') > + mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n') > for mir in mirros: > mir_list = mir.split() > # Should be two members. > -- > 1.8.3.4 > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core > ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCHv2 2/3] sanity: fix support for regex schemes in mirrors check 2014-08-25 22:57 ` [PATCHv2 0/3] sanity: improve mirrors check Christopher Larson 2014-08-25 22:57 ` [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars Christopher Larson @ 2014-08-25 22:57 ` Christopher Larson 2014-08-25 22:57 ` [PATCHv2 3/3] sanity: refactor mirrors checks to be more pythonic Christopher Larson 2 siblings, 0 replies; 12+ messages in thread From: Christopher Larson @ 2014-08-25 22:57 UTC (permalink / raw) To: openembedded-core; +Cc: Christopher Larson From: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Christopher Larson <chris_larson@mentor.com> --- meta/classes/sanity.bbclass | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass index 7cfc4be..f655d8b 100644 --- a/meta/classes/sanity.bbclass +++ b/meta/classes/sanity.bbclass @@ -754,8 +754,9 @@ def check_sanity_everybuild(status, d): status.addresult("Error, you have a space in your COREBASE directory path. Please move the installation to a directory which doesn't include a space since autotools doesn't support this.") # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS + import re mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS'] - protocols = ['http://', 'ftp://', 'file://', 'https://', 'https?$://', \ + protocols = ['http://', 'ftp://', 'file://', 'https://', \ 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ 'bzr://', 'cvs://'] for mir_type in mir_types: @@ -767,12 +768,19 @@ def check_sanity_everybuild(status, d): bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \ % (mir_type, mir.strip(), len(mir_list))) elif len(mir_list) == 2: + decoded = bb.fetch2.decodeurl(mir_list[0]) + try: + pattern_scheme = re.compile(decoded[0]) + except re.error as exc: + bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip())) + continue + # Each member should start with protocols valid_protocol_0 = False valid_protocol_1 = False file_absolute = True for protocol in protocols: - if not valid_protocol_0 and mir_list[0].startswith(protocol): + if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]): valid_protocol_0 = True if not valid_protocol_1 and mir_list[1].startswith(protocol): valid_protocol_1 = True -- 1.8.3.4 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCHv2 3/3] sanity: refactor mirrors checks to be more pythonic 2014-08-25 22:57 ` [PATCHv2 0/3] sanity: improve mirrors check Christopher Larson 2014-08-25 22:57 ` [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars Christopher Larson 2014-08-25 22:57 ` [PATCHv2 2/3] sanity: fix support for regex schemes in mirrors check Christopher Larson @ 2014-08-25 22:57 ` Christopher Larson 2 siblings, 0 replies; 12+ messages in thread From: Christopher Larson @ 2014-08-25 22:57 UTC (permalink / raw) To: openembedded-core; +Cc: Christopher Larson From: Christopher Larson <chris_larson@mentor.com> - Use clearer variable names - Use variable unpacking to reference elements by name rather than index - Sacrifice a small amount of time (iterate over protocols twice per entry rather than once) for clarity: use readable generator expressions with any() rather than maintaining state. Signed-off-by: Christopher Larson <chris_larson@mentor.com> --- meta/classes/sanity.bbclass | 73 ++++++++++++++++++++++----------------------- 1 file changed, 35 insertions(+), 38 deletions(-) diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass index f655d8b..5be5efb 100644 --- a/meta/classes/sanity.bbclass +++ b/meta/classes/sanity.bbclass @@ -755,44 +755,41 @@ def check_sanity_everybuild(status, d): # Check the format of MIRRORS, PREMIRRORS and SSTATE_MIRRORS import re - mir_types = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS'] - protocols = ['http://', 'ftp://', 'file://', 'https://', \ - 'git://', 'gitsm://', 'hg://', 'osc://', 'p4://', 'svk://', 'svn://', \ - 'bzr://', 'cvs://'] - for mir_type in mir_types: - mirros = (d.getVar(mir_type, True) or '').replace('\\n', '\n').split('\n') - for mir in mirros: - mir_list = mir.split() - # Should be two members. - if len(mir_list) not in [0, 2]: - bb.warn('Invalid %s: %s, should be 2 members, but found %s.' \ - % (mir_type, mir.strip(), len(mir_list))) - elif len(mir_list) == 2: - decoded = bb.fetch2.decodeurl(mir_list[0]) - try: - pattern_scheme = re.compile(decoded[0]) - except re.error as exc: - bb.warn('Invalid scheme regex (%s) in %s: %s' % (decoded[0], mir_type, mir.strip())) - continue - - # Each member should start with protocols - valid_protocol_0 = False - valid_protocol_1 = False - file_absolute = True - for protocol in protocols: - if not valid_protocol_0 and pattern_scheme.match(protocol[:-3]): - valid_protocol_0 = True - if not valid_protocol_1 and mir_list[1].startswith(protocol): - valid_protocol_1 = True - # The file:// must be an absolute path. - if protocol == 'file://' and not mir_list[1].startswith('file:///'): - file_absolute = False - if valid_protocol_0 and valid_protocol_1: - break - if not (valid_protocol_0 and valid_protocol_1): - bb.warn('Invalid protocol in %s: %s' % (mir_type, mir.strip())) - if not file_absolute: - bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mir_type, mir.strip())) + mirror_vars = ['MIRRORS', 'PREMIRRORS', 'SSTATE_MIRRORS'] + protocols = ['http', 'ftp', 'file', 'https', \ + 'git', 'gitsm', 'hg', 'osc', 'p4', 'svk', 'svn', \ + 'bzr', 'cvs'] + for mirror_var in mirror_vars: + mirrors = (d.getVar(mirror_var, True) or '').replace('\\n', '\n').split('\n') + for mirror_entry in mirrors: + mirror_entry = mirror_entry.strip() + if not mirror_entry: + # ignore blank lines + continue + + try: + pattern, mirror = mirror_entry.split() + except ValueError: + bb.warn('Invalid %s: %s, should be 2 members.' % (mirror_var, mirror_entry.strip())) + continue + + decoded = bb.fetch2.decodeurl(pattern) + try: + pattern_scheme = re.compile(decoded[0]) + except re.error as exc: + bb.warn('Invalid scheme regex (%s) in %s; %s' % (pattern, mirror_var, mirror_entry)) + continue + + if not any(pattern_scheme.match(protocol) for protocol in protocols): + bb.warn('Invalid protocol (%s) in %s: %s' % (decoded[0], mirror_var, mirror_entry)) + continue + + if not any(mirror.startswith(protocol + '://') for protocol in protocols): + bb.warn('Invalid protocol in %s: %s' % (mirror_var, mirror_entry)) + continue + + if mirror.startswith('file://') and not mirror.startswith('file:///'): + bb.warn('Invalid file url in %s: %s, must be absolute path (file:///)' % (mirror_var, mirror_entry)) # Check that TMPDIR hasn't changed location since the last time we were run tmpdir = d.getVar('TMPDIR', True) -- 1.8.3.4 ^ permalink raw reply related [flat|nested] 12+ messages in thread
end of thread, other threads:[~2014-08-27 11:50 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-08-25 22:46 [PATCH 0/3] sanity: improve mirrors check Christopher Larson 2014-08-25 22:46 ` [PATCH 1/3] sanity: handle both \n and \n in mirror vars Christopher Larson 2014-08-25 22:50 ` Christopher Larson 2014-08-25 22:46 ` [PATCH 2/3] sanity: fix support for regex schemes in mirrors check Christopher Larson 2014-08-25 22:46 ` [PATCH 3/3] sanity: refactor mirrors checks to be more pythonic Christopher Larson 2014-08-25 22:57 ` [PATCHv2 0/3] sanity: improve mirrors check Christopher Larson 2014-08-25 22:57 ` [PATCHv2 1/3] sanity: handle both \n and \\n in mirror vars Christopher Larson 2014-08-26 10:51 ` Martin Jansa 2014-08-26 10:59 ` Martin Jansa 2014-08-27 11:50 ` Koen Kooi 2014-08-25 22:57 ` [PATCHv2 2/3] sanity: fix support for regex schemes in mirrors check Christopher Larson 2014-08-25 22:57 ` [PATCHv2 3/3] sanity: refactor mirrors checks to be more pythonic Christopher Larson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox