[PATCH] (PRE)MIRRORS: add pattern for npm:// without slash

[PATCH] (PRE)MIRRORS: add pattern for npm:// without slash

[Olaf Mandel]

For URLs with the npm:// transport but with no other slash in it, the
common MIRRORS and PREMIRRORS pattern of npm://.*/.* fails to match.
Add a new pattern in the mirros.bbclass and own-mirrors.bbclass classes
for these URLs.

Many URLs with the npm:// transport have no slash after the host part:

  npm://registry.npmjs.org;name=foo;version=0.1.2

This means that MIRRORS and PREMIRRORS containing entries like the
first one will not match these URLs:

  npm://.*/.*   # fails to match
  npm://.*      # matches, but not those URLs with a slash

Because the npm:// URLs without a slash seem to be more ubiquitous (they
are generated by recipetool), place the new pattern first.

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
---
 meta/classes/mirrors.bbclass     | 2 ++
 meta/classes/own-mirrors.bbclass | 1 +
 2 files changed, 3 insertions(+)

diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
index b98684f5c6..710df3c7bf 100644
--- a/meta/classes/mirrors.bbclass
+++ b/meta/classes/mirrors.bbclass
@@ -48,6 +48,7 @@ p4://.*/.*      http://downloads.yoctoproject.org/mirror/sources/ \n \
 osc://.*/.*     http://downloads.yoctoproject.org/mirror/sources/ \n \
 https?$://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
 ftp://.*/.*     http://downloads.yoctoproject.org/mirror/sources/ \n \
+npm://.*        http://downloads.yoctoproject.org/mirror/sources/ \n \
 npm://.*/.*     http://downloads.yoctoproject.org/mirror/sources/ \n \
 cvs://.*/.*     http://sources.openembedded.org/ \n \
 svn://.*/.*     http://sources.openembedded.org/ \n \
@@ -58,6 +59,7 @@ p4://.*/.*      http://sources.openembedded.org/ \n \
 osc://.*/.*     http://sources.openembedded.org/ \n \
 https?$://.*/.* http://sources.openembedded.org/ \n \
 ftp://.*/.*     http://sources.openembedded.org/ \n \
+npm://.*        http://sources.openembedded.org/ \n \
 npm://.*/.*     http://sources.openembedded.org/ \n \
 ${CPAN_MIRROR}  http://cpan.metacpan.org/ \n \
 ${CPAN_MIRROR}  http://search.cpan.org/CPAN/ \n \
diff --git a/meta/classes/own-mirrors.bbclass b/meta/classes/own-mirrors.bbclass
index 12b42675bc..253ed9cb7e 100644
--- a/meta/classes/own-mirrors.bbclass
+++ b/meta/classes/own-mirrors.bbclass
@@ -9,5 +9,6 @@ p4://.*/.*      ${SOURCE_MIRROR_URL}
 osc://.*/.*     ${SOURCE_MIRROR_URL}
 https?$://.*/.* ${SOURCE_MIRROR_URL}
 ftp://.*/.*     ${SOURCE_MIRROR_URL}
+npm://.*        ${SOURCE_MIRROR_URL}
 npm://.*/.*     ${SOURCE_MIRROR_URL}
 }
-- 
2.11.0

Re: [PATCH] (PRE)MIRRORS: add pattern for npm:// without slash

[Paul Eggleton]

Hi Olaf,

On Friday, 25 August 2017 6:19:46 AM NZST Olaf Mandel wrote:
> For URLs with the npm:// transport but with no other slash in it, the
> common MIRRORS and PREMIRRORS pattern of npm://.*/.* fails to match.
> Add a new pattern in the mirros.bbclass and own-mirrors.bbclass classes
> for these URLs.
> 
> Many URLs with the npm:// transport have no slash after the host part:
> 
>   npm://registry.npmjs.org;name=foo;version=0.1.2
> 
> This means that MIRRORS and PREMIRRORS containing entries like the
> first one will not match these URLs:
> 
>   npm://.*/.*   # fails to match
>   npm://.*      # matches, but not those URLs with a slash
> 
> Because the npm:// URLs without a slash seem to be more ubiquitous (they
> are generated by recipetool), place the new pattern first.

Couldn't we just do away with npm://.*/.* ?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] (PRE)MIRRORS: add pattern for npm:// without slash

[Olaf Mandel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hello Paul,

Am 24.08.2017 um 22:53 schrieb Paul Eggleton:
> On Friday, 25 August 2017 6:19:46 AM NZST Olaf Mandel wrote:
>> For URLs with the npm:// transport but with no other slash in it,
>> the common MIRRORS and PREMIRRORS pattern of npm://.*/.* fails to
>> match. Add a new pattern in the mirros.bbclass and
>> own-mirrors.bbclass classes for these URLs.
- -Snipp-
> 
> Couldn't we just do away with npm://.*/.* ?
> 
While that would fix my specific problem, it would introduce a
backwards-breaking change: if someone _did_ have a SRC_URI with a
slash in it, this would change where on the mirror the package is
searched for:

SRC_URI = npm://foo.tld/bar;name=baz;version=0.1.2
npm://.*/.* =>     baz-0.1.2.tar.xz
npm://.*    => bar/baz-0.1.2.tar.xz

So keeping both lines is the better solution. But while testing this I
found that the order I chose (first npm://.*, then npm://.*/.*) is
wrong: this first tries the changed (wrong) location on the mirror and
only then the original (correct) one. It is much more efficient to
sort the lines the other way around: this prevents extra network
traffic, if the mirror is not local. I will send an updated batch soon.

Best regards,
Olaf
- -- 
Olaf Mandel

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJZn+/7AAoJEFS9nbRiUh3/0EMIAJTUoXVsvSLWDUm8phbAEoXb
jQZW7hfP+CHzn1V7X6kTu4CZlQWhxOzYbv07sd2cz/4+Hr2U/7JJCul2QTQGizc/
NxVKaixR8V0tM0irn8NeAHexLXLj+dKdmbvaG/H/BeHhFZc4eQOt6gmbGZ1F7V2g
njcP7nui13HbYpHViGFKMT5hnU9B2gKWQ0FAtY6kGZFS1UDvyGFXyHXn1kU0hzjl
6nRuG0q18uueUEEFxqjo4Cb5BTNEjC/QIGIKbSTsqBIU1r3t7uaPSHIYihnRGZ9M
bCpoFKB2PFgi8qFl+cUaFBmtGMeh+loEIDIi1eZ1HmRO7ST0glDmxB0SKM1N06I=
=QS0a
-----END PGP SIGNATURE-----

[PATCH v2] (PRE)MIRRORS: add pattern for npm:// without slash

[Olaf Mandel]

For URIs with the npm:// transport but with no other slash in it, the
common MIRRORS and PREMIRRORS pattern of npm://.*/.* fails to match.
Add a new pattern in the mirros.bbclass and own-mirrors.bbclass classes
for these URIs.

Many URIs with the npm:// transport have no slash after the host part:

  npm://registry.npmjs.org;name=foo;version=0.1.2

This means that MIRRORS and PREMIRRORS containing entries like the
first one will not match these URIs:

  npm://.*/.*   # fails to match
  npm://.*      # matches

The npm:// URIs without a slash also matches URIs with a slash, but
then the file is searched for on the mirror with the same path that was
in the URI. Because of that, the pattern with the slash should be first
in the list, to prevent extra requests to the mirrors containing the
wrong path.

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
---
 meta/classes/mirrors.bbclass     | 2 ++
 meta/classes/own-mirrors.bbclass | 1 +
 2 files changed, 3 insertions(+)

diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
index b98684f5c6..7ac654494b 100644
--- a/meta/classes/mirrors.bbclass
+++ b/meta/classes/mirrors.bbclass
@@ -49,6 +49,7 @@ osc://.*/.*     http://downloads.yoctoproject.org/mirror/sources/ \n \
 https?$://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
 ftp://.*/.*     http://downloads.yoctoproject.org/mirror/sources/ \n \
 npm://.*/.*     http://downloads.yoctoproject.org/mirror/sources/ \n \
+npm://.*        http://downloads.yoctoproject.org/mirror/sources/ \n \
 cvs://.*/.*     http://sources.openembedded.org/ \n \
 svn://.*/.*     http://sources.openembedded.org/ \n \
 git://.*/.*     http://sources.openembedded.org/ \n \
@@ -59,6 +60,7 @@ osc://.*/.*     http://sources.openembedded.org/ \n \
 https?$://.*/.* http://sources.openembedded.org/ \n \
 ftp://.*/.*     http://sources.openembedded.org/ \n \
 npm://.*/.*     http://sources.openembedded.org/ \n \
+npm://.*        http://sources.openembedded.org/ \n \
 ${CPAN_MIRROR}  http://cpan.metacpan.org/ \n \
 ${CPAN_MIRROR}  http://search.cpan.org/CPAN/ \n \
 "
diff --git a/meta/classes/own-mirrors.bbclass b/meta/classes/own-mirrors.bbclass
index 12b42675bc..c70b94e7dc 100644
--- a/meta/classes/own-mirrors.bbclass
+++ b/meta/classes/own-mirrors.bbclass
@@ -10,4 +10,5 @@ osc://.*/.*     ${SOURCE_MIRROR_URL}
 https?$://.*/.* ${SOURCE_MIRROR_URL}
 ftp://.*/.*     ${SOURCE_MIRROR_URL}
 npm://.*/.*     ${SOURCE_MIRROR_URL}
+npm://.*        ${SOURCE_MIRROR_URL}
 }
-- 
2.11.0

Re: [PATCH v2] (PRE)MIRRORS: add pattern for npm:// without slash

[Leonardo Sandoval]

>  npm://.*/.*     http://sources.openembedded.org/ \n \
> +npm://.*        http://sources.openembedded.org/ \n \

instead of adding a new npm regex, not sure if below one would the same
job

npm://.*(/.*)?

Re: [PATCH v2] (PRE)MIRRORS: add pattern for npm:// without slash

[Olaf Mandel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hello Leonardo,

Am 25.08.2017 um 16:32 schrieb Leonardo Sandoval:
> instead of adding a new npm regex, not sure if below one would the
> same job
> 
> npm://.*(/.*)?
> 
No: the string is split into different parts before the RE is
considered. So the fetch step fails with this error:

Exception: sre_constants.error: missing ), unterminated subpattern at
position 2

Probably because the pattern was split to:

['npm', '.*(', '/.*)?', '', '', OrderedDict()]

Best regards,
Olaf
- -- 
Olaf Mandel

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJZoEzpAAoJEFS9nbRiUh3/KgEIAKjWmkwrlR3CXNk1zYgS9XKb
FQsE47IYogdtLMSgjgbjo+tUVj34a4NpPY4kNXdwadXNSOQZJPDcu2hzkvxAyAVd
9pX0YtxASx4A25/kgBKiTzOL62gGf3vYAnCg3hCC4nPnZE5d49Xg/oNvhQ2E34xv
vZMaohiPwK5B2QJpPDYtVXEC72ExoMzeoEy1x46hIplQQTh3CE481TKHbFj1he/a
UVwRalNhePQbpScDu49oYdrg8bIl+GfmuDwFE5zLAWBdkkgFaDlLdbwUoz+om3xa
GUHA9eL0lu8CWmrZe1Y4RZuGgj42yJd1CDd7NA3SkeHcB6gsAPzwyn6/UF2rivo=
=mQeF
-----END PGP SIGNATURE-----

Re: [PATCH v2] (PRE)MIRRORS: add pattern for npm:// without slash

[Olaf Mandel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hello again,

Am 25.08.2017 um 18:14 schrieb Olaf Mandel:
> Am 25.08.2017 um 16:32 schrieb Leonardo Sandoval:
>> instead of adding a new npm regex, not sure if below one would
>> the same job
> 
>> npm://.*(/.*)?
> 
> No: the string is split into different parts before the RE is 
> considered. [...]
> 
- -Snipp-
> ['npm', '.*(', '/.*)?', '', '', OrderedDict()]
> 
So npm://.*(/.*)? does not work. But npm://.*/?.* works and _is_ more
elegant than the two pattern lines from my last patches.

Thank you for the suggestion, I will send a new patch.

Best regards,
Olaf
- -- 
Olaf Mandel
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJZoFehAAoJEFS9nbRiUh3/FzsH/06OUDbQPrf3yrH5OBMJe+wF
5Ky8SUworYqA9uGmzR+3nGqmlxlLlhZFMCsM4+a00nr7kSNLu0KF+BISG9fJT5Up
6PJKDZZBChWYHqhkO8CgB3caiQiJKTlFtKmVznNBzJuIPf+ffTiUb0OX7RgFoHHP
jSuEXqKxQTUJO1k1tGhD53F6+W0rzfk2NXqtI2xfURCwZFcTHvrIYHflwoGZzF3F
gc5J37x5oFSxb3Lfy711ZWDTCOljFzoecdpKbhIj2qPr8sO5WVOTE07/FzW4Mb3P
KUlOpbR3D5eSma8f+lacqNQVx0toWRo15yHOVWssLbcoV6V8rzUu6sF1T1Od7Tw=
=RJXU
-----END PGP SIGNATURE-----

[PATCH v3] (PRE)MIRRORS: fix pattern for npm:// without slash

[Olaf Mandel]

For URIs with the npm:// transport but with no other slash in it, the
common MIRRORS and PREMIRRORS pattern of npm://.*/.* fails to match.
Make the last slash in the pattern optional in the mirros.bbclass and
own-mirrors.bbclass classes.

Many URIs with the npm:// transport have no slash after the host part:

  npm://registry.npmjs.org;name=foo;version=0.1.2

This means that MIRRORS and PREMIRRORS containing entries like the
first one will not match these URIs:

  npm://.*/.*   # fails to match
  npm://.*/?.*  # matches this and URIs with path components

For normal regular expressions, a pattern like 'npm://.*(/.*)?' would
probably be preferred, but that won't work here: the pattern gets split
into the substrings 'npm', '.*(' and '/.*)?', which are not valid
regular expressions individually.

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
---
 meta/classes/mirrors.bbclass     | 4 ++--
 meta/classes/own-mirrors.bbclass | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/mirrors.bbclass b/meta/classes/mirrors.bbclass
index b98684f5c6..766f1cb6fa 100644
--- a/meta/classes/mirrors.bbclass
+++ b/meta/classes/mirrors.bbclass
@@ -48,7 +48,7 @@ p4://.*/.*      http://downloads.yoctoproject.org/mirror/sources/ \n \
 osc://.*/.*     http://downloads.yoctoproject.org/mirror/sources/ \n \
 https?$://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
 ftp://.*/.*     http://downloads.yoctoproject.org/mirror/sources/ \n \
-npm://.*/.*     http://downloads.yoctoproject.org/mirror/sources/ \n \
+npm://.*/?.*    http://downloads.yoctoproject.org/mirror/sources/ \n \
 cvs://.*/.*     http://sources.openembedded.org/ \n \
 svn://.*/.*     http://sources.openembedded.org/ \n \
 git://.*/.*     http://sources.openembedded.org/ \n \
@@ -58,7 +58,7 @@ p4://.*/.*      http://sources.openembedded.org/ \n \
 osc://.*/.*     http://sources.openembedded.org/ \n \
 https?$://.*/.* http://sources.openembedded.org/ \n \
 ftp://.*/.*     http://sources.openembedded.org/ \n \
-npm://.*/.*     http://sources.openembedded.org/ \n \
+npm://.*/?.*    http://sources.openembedded.org/ \n \
 ${CPAN_MIRROR}  http://cpan.metacpan.org/ \n \
 ${CPAN_MIRROR}  http://search.cpan.org/CPAN/ \n \
 "
diff --git a/meta/classes/own-mirrors.bbclass b/meta/classes/own-mirrors.bbclass
index 12b42675bc..0296d545be 100644
--- a/meta/classes/own-mirrors.bbclass
+++ b/meta/classes/own-mirrors.bbclass
@@ -9,5 +9,5 @@ p4://.*/.*      ${SOURCE_MIRROR_URL}
 osc://.*/.*     ${SOURCE_MIRROR_URL}
 https?$://.*/.* ${SOURCE_MIRROR_URL}
 ftp://.*/.*     ${SOURCE_MIRROR_URL}
-npm://.*/.*     ${SOURCE_MIRROR_URL}
+npm://.*/?.*    ${SOURCE_MIRROR_URL}
 }
-- 
2.11.0