[PATCH 1/3] elfutils: use HTTP instead of FTP to fetch

[PATCH 1/3] elfutils: use HTTP instead of FTP to fetch

[Ross Burton]

FTP is inferiour to HTTP is all respects, so use the HTTP URL for the tarball.

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/recipes-devtools/elfutils/elfutils_0.168.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/elfutils/elfutils_0.168.bb b/meta/recipes-devtools/elfutils/elfutils_0.168.bb
index c0a4f234020..0717616ec0a 100644
--- a/meta/recipes-devtools/elfutils/elfutils_0.168.bb
+++ b/meta/recipes-devtools/elfutils/elfutils_0.168.bb
@@ -5,7 +5,7 @@ LICENSE = "(GPLv3 & Elfutils-Exception)"
 LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
 DEPENDS = "libtool bzip2 zlib virtual/libintl"
 DEPENDS_append_libc-musl = " argp-standalone fts "
-SRC_URI = "ftp://sourceware.org/pub/elfutils/${PV}/${BP}.tar.bz2"
+SRC_URI = "https://sourceware.org/elfutils/ftp/${PV}/${BP}.tar.bz2"
 SRC_URI[md5sum] = "52adfa40758d0d39e5d5c57689bf38d6"
 SRC_URI[sha256sum] = "b88d07893ba1373c7dd69a7855974706d05377766568a7d9002706d5de72c276"
 
-- 
2.11.0

[PATCH 2/3] libpng: use SourceForge mirror

[Ross Burton]

The Gentoo mirror also deletes old versions when they're not used, so revert
back to the canonical SourceForge site, adding /older-releases/ to MIRRORS to
handle new releases moving the version we want.

Original idea by Maxin B. John <maxin.john@intel.com>.

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/recipes-multimedia/libpng/libpng_1.6.29.bb | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-multimedia/libpng/libpng_1.6.29.bb b/meta/recipes-multimedia/libpng/libpng_1.6.29.bb
index 8a3056e1a39..be327caaffc 100644
--- a/meta/recipes-multimedia/libpng/libpng_1.6.29.bb
+++ b/meta/recipes-multimedia/libpng/libpng_1.6.29.bb
@@ -6,11 +6,14 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=f9acafc1325f8f8da80aeee94dbc1bda \
                     file://png.h;endline=144;md5=0b2aa20c7ff55d9411108317df8ff0c9"
 DEPENDS = "zlib"
 
-SRC_URI = "${GENTOO_MIRROR}/libpng-${PV}.tar.xz \
-          "
+LIBV = "16"
+
+SRC_URI = "${SOURCEFORGE_MIRROR}/project/${BPN}/${BPN}${LIBV}/${PV}/${BP}.tar.xz"
 SRC_URI[md5sum] = "3245dbd76ea91e1437507357b858ec97"
 SRC_URI[sha256sum] = "4245b684e8fe829ebb76186327bb37ce5a639938b219882b53d64bd3cfc5f239"
 
+MIRRORS += "${SOURCEFORGE_MIRROR}/project/${BPN}/${BPN}${LIBV}/older-releases/${PV}"
+
 BINCONFIG = "${bindir}/libpng-config ${bindir}/libpng16-config"
 
 inherit autotools binconfig-disabled pkgconfig
-- 
2.11.0

[PATCH 3/3] fetch/wget: mitigate a wget race condition when listing FTP directories

[Ross Burton]

When wget is fetching a listing for a directory over FTP it writes to a
temporary file called .listing in the current directory.  If there are many such
operations happening in parallel - for example during 'bitbake world -c
checkpkg' - then up to BB_NUMBER_THREADS instances of wget will be racing to
write to, read, and delete the same file.

This results in various failures such as the file disappearing before wget has
processed it or the file changing contents, which causes checkpkg to randomly
fail.

Mitigate the race condition by creating a temporary directory to run wget in
when doing directory listings.

[ YOCTO #11828 ]

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 bitbake/lib/bb/fetch2/wget.py | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 208ee9bdd69..8ee9769d39b 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -90,13 +90,13 @@ class Wget(FetchMethod):
 
         self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate"
 
-    def _runwget(self, ud, d, command, quiet):
+    def _runwget(self, ud, d, command, quiet, workdir=None):
 
         progresshandler = WgetProgressHandler(d)
 
         logger.debug(2, "Fetching %s using command '%s'" % (ud.url, command))
         bb.fetch2.check_network_access(d, command, ud.url)
-        runfetchcmd(command + ' --progress=dot -v', d, quiet, log=progresshandler)
+        runfetchcmd(command + ' --progress=dot -v', d, quiet, log=progresshandler, workdir=workdir)
 
     def download(self, ud, d):
         """Fetch urls"""
@@ -422,17 +422,16 @@ class Wget(FetchMethod):
         Run fetch checkstatus to get directory information
         """
         f = tempfile.NamedTemporaryFile()
+        with tempfile.TemporaryDirectory(prefix="wget-index-") as workdir, tempfile.NamedTemporaryFile(dir=workdir, prefix="wget-listing-") as f:
+            agent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/9.10 (karmic) Firefox/3.6.12"
+            fetchcmd = self.basecmd
+            fetchcmd += " -O " + f.name + " --user-agent='" + agent + "' '" + uri + "'"
+            try:
+                self._runwget(ud, d, fetchcmd, True, workdir=workdir)
+                fetchresult = f.read()
+            except bb.fetch2.BBFetchException:
+                fetchresult = ""
 
-        agent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/9.10 (karmic) Firefox/3.6.12"
-        fetchcmd = self.basecmd
-        fetchcmd += " -O " + f.name + " --user-agent='" + agent + "' '" + uri + "'"
-        try:
-            self._runwget(ud, d, fetchcmd, True)
-            fetchresult = f.read()
-        except bb.fetch2.BBFetchException:
-            fetchresult = ""
-
-        f.close()
         return fetchresult
 
     def _check_latest_version(self, url, package, package_regex, current_version, ud, d):
-- 
2.11.0

Re: [PATCH 3/3] fetch/wget: mitigate a wget race condition when listing FTP directories

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 3393 bytes --]

Wrong list, sorry.

Ross

On 25 July 2017 at 11:17, Ross Burton <ross.burton@intel.com> wrote:

> When wget is fetching a listing for a directory over FTP it writes to a
> temporary file called .listing in the current directory.  If there are
> many such
> operations happening in parallel - for example during 'bitbake world -c
> checkpkg' - then up to BB_NUMBER_THREADS instances of wget will be racing
> to
> write to, read, and delete the same file.
>
> This results in various failures such as the file disappearing before wget
> has
> processed it or the file changing contents, which causes checkpkg to
> randomly
> fail.
>
> Mitigate the race condition by creating a temporary directory to run wget
> in
> when doing directory listings.
>
> [ YOCTO #11828 ]
>
> Signed-off-by: Ross Burton <ross.burton@intel.com>
> ---
>  bitbake/lib/bb/fetch2/wget.py | 23 +++++++++++------------
>  1 file changed, 11 insertions(+), 12 deletions(-)
>
> diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
> index 208ee9bdd69..8ee9769d39b 100644
> --- a/bitbake/lib/bb/fetch2/wget.py
> +++ b/bitbake/lib/bb/fetch2/wget.py
> @@ -90,13 +90,13 @@ class Wget(FetchMethod):
>
>          self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget -t
> 2 -T 30 --passive-ftp --no-check-certificate"
>
> -    def _runwget(self, ud, d, command, quiet):
> +    def _runwget(self, ud, d, command, quiet, workdir=None):
>
>          progresshandler = WgetProgressHandler(d)
>
>          logger.debug(2, "Fetching %s using command '%s'" % (ud.url,
> command))
>          bb.fetch2.check_network_access(d, command, ud.url)
> -        runfetchcmd(command + ' --progress=dot -v', d, quiet,
> log=progresshandler)
> +        runfetchcmd(command + ' --progress=dot -v', d, quiet,
> log=progresshandler, workdir=workdir)
>
>      def download(self, ud, d):
>          """Fetch urls"""
> @@ -422,17 +422,16 @@ class Wget(FetchMethod):
>          Run fetch checkstatus to get directory information
>          """
>          f = tempfile.NamedTemporaryFile()
> +        with tempfile.TemporaryDirectory(prefix="wget-index-") as
> workdir, tempfile.NamedTemporaryFile(dir=workdir, prefix="wget-listing-")
> as f:
> +            agent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12)
> Gecko/20101027 Ubuntu/9.10 (karmic) Firefox/3.6.12"
> +            fetchcmd = self.basecmd
> +            fetchcmd += " -O " + f.name + " --user-agent='" + agent + "'
> '" + uri + "'"
> +            try:
> +                self._runwget(ud, d, fetchcmd, True, workdir=workdir)
> +                fetchresult = f.read()
> +            except bb.fetch2.BBFetchException:
> +                fetchresult = ""
>
> -        agent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12)
> Gecko/20101027 Ubuntu/9.10 (karmic) Firefox/3.6.12"
> -        fetchcmd = self.basecmd
> -        fetchcmd += " -O " + f.name + " --user-agent='" + agent + "' '"
> + uri + "'"
> -        try:
> -            self._runwget(ud, d, fetchcmd, True)
> -            fetchresult = f.read()
> -        except bb.fetch2.BBFetchException:
> -            fetchresult = ""
> -
> -        f.close()
>          return fetchresult
>
>      def _check_latest_version(self, url, package, package_regex,
> current_version, ud, d):
> --
> 2.11.0
>
>

[-- Attachment #2: Type: text/html, Size: 4517 bytes --]

✗ patchtest: failure for "elfutils: use HTTP instead of ..." and 2 more

[Patchwork]

== Series Details ==

Series: "elfutils: use HTTP instead of ..." and 2 more
Revision: 1
URL   : https://patchwork.openembedded.org/series/7924/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at d3a41fbd94)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe