From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2191FC4332F for ; Fri, 9 Dec 2022 14:42:14 +0000 (UTC) Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) by mx.groups.io with SMTP id smtpd.web10.2919.1670596929604161147 for ; Fri, 09 Dec 2022 06:42:10 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=kLnCBhKw; spf=pass (domain: bootlin.com, ip: 217.70.178.230, mailfrom: alexandre.belloni@bootlin.com) Received: (Authenticated sender: alexandre.belloni@bootlin.com) by mail.gandi.net (Postfix) with ESMTPSA id 6130524000F; Fri, 9 Dec 2022 14:42:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1670596926; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Uu8icwP92e+dG4IoS0tRyJUJyC8sJMrOm/JVzQ/fy+o=; b=kLnCBhKwhZrVvsZDzepRD7SVx5JovuDHiXI6VfsYe58usGRjdWmWsxS6QaK3ntXDI2qMzL jAaZWtCGYf9WWsDk8LQZI5w/zp1hfNtVNK+UvTbETS4wR5U+XyjOAkVpFXrtCwnyqnVsO+ WbFAMZ+vmJx4g9fhj9t0iKecVX1H4xUCR44sUMTNHjC7zdBKPh5u6TBzT/wncBrrcXFimo Mx+Qpr6nZM5QXraolvLaIKOwF4ONBYrjD8Lpg2GHO3ptImwJrrBP539b+7TkkpCxJxWQoE fOKho0MizJgD++r7SELX34biFpUxqTqwzioF5HI0edk158AlswK0RU07tgdYxg== Date: Fri, 9 Dec 2022 15:42:06 +0100 From: Alexandre Belloni To: Alexander Kanavin Cc: bitbake-devel@lists.openembedded.org, Alexander Kanavin Subject: Re: [bitbake-devel] [PATCH v3] fetch2/wget.py: correctly match versioned directories Message-ID: References: <20221209085840.3733809-1-alex@linutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20221209085840.3733809-1-alex@linutronix.de> List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 09 Dec 2022 14:42:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14162 This patch applied correctly but failed: 2022-12-09 13:21:18 (456 MB/s) - ‘/tmp/wget-index-ypaf69js/wget-listing-m180p7ds’ saved [34419/34419] --2022-12-09 13:21:18-- http://localhost:46501/files/v3.16/ Resolving localhost (localhost)... ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:46501... failed: Connection refused. Connecting to localhost (localhost)|127.0.0.1|:46501... connected. HTTP request sent, awaiting response... 200 OK Length: 17922 (18K) [text/html] Saving to: ‘/tmp/wget-index-3i6kh677/wget-listing-w93f0eg3’ I suspect we can't connect to random ports on the AB On 09/12/2022 09:58:40+0100, Alexander Kanavin wrote: > When obtaining latest upstream versions, the code needs > to check if the existing tarball is in a versioned directory > (e.g. component-name/x.y/component-name-x.y.z.tar.gz) and > if it is, it needs to first obtain the list of all > such versioned directories and then check all of them by going > one step up in the directory hierarchy. > > Existing code was returning a correct match when the component > name did not have numbers, e.g. a check on 'source/epiphany/43/' > would return 43, but was stopping too soon when the component > name itself had numbers ('source/libxml2/2.10/' would return libxml2). > > This change ensures the last match is taken instead of the first. > > Also, adjust the fetcher tests to check that versioned directories > are correctly traversed in this case (e.g. the step to go one level > up is taken and a new tarball is discovered in a different versioned > directory). > > Signed-off-by: Alexander Kanavin > --- > bitbake/lib/bb/fetch2/wget.py | 4 +- > .../software/libxml2/2.10/index.html | 20 ++++++++++ > .../software/libxml2/2.9/index.html | 40 +++++++++++++++++++ > .../software/libxml2/index.html | 19 +++++++++ > bitbake/lib/bb/tests/fetch.py | 3 ++ > 5 files changed, 84 insertions(+), 2 deletions(-) > create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html > create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html > create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html > > diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py > index 821afa5b58..7ccea3ad89 100644 > --- a/bitbake/lib/bb/fetch2/wget.py > +++ b/bitbake/lib/bb/fetch2/wget.py > @@ -644,10 +644,10 @@ class Wget(FetchMethod): > # search for version matches on folders inside the path, like: > # "5.7" in http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz > dirver_regex = re.compile(r"(?P[^/]*(\d+\.)*\d+([-_]r\d+)*)/") > - m = dirver_regex.search(path) > + m = dirver_regex.findall(path) > if m: > pn = d.getVar('PN') > - dirver = m.group('dirver') > + dirver = m[-1][0] > > dirver_pn_regex = re.compile(r"%s\d?" % (re.escape(pn))) > if not dirver_pn_regex.search(dirver): > diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html > new file mode 100644 > index 0000000000..4e41af6d6a > --- /dev/null > +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html > @@ -0,0 +1,20 @@ > + > + > +Index of /sources/libxml2/2.10/ > +

Index of /sources/libxml2/2.10/

> + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
LATEST-IS-2.10.32.5 MiB2022-Oct-14 12:55
libxml2-2.10.0.news7.1 KiB2022-Aug-17 11:55
libxml2-2.10.0.sha256sum174 B2022-Aug-17 11:55
libxml2-2.10.0.tar.xz2.6 MiB2022-Aug-17 11:55
libxml2-2.10.1.news455 B2022-Aug-25 11:33
libxml2-2.10.1.sha256sum174 B2022-Aug-25 11:33
libxml2-2.10.1.tar.xz2.6 MiB2022-Aug-25 11:33
libxml2-2.10.2.news309 B2022-Aug-29 14:56
libxml2-2.10.2.sha256sum174 B2022-Aug-29 14:56
libxml2-2.10.2.tar.xz2.5 MiB2022-Aug-29 14:56
libxml2-2.10.3.news294 B2022-Oct-14 12:55
libxml2-2.10.3.sha256sum174 B2022-Oct-14 12:55
libxml2-2.10.3.tar.xz2.5 MiB2022-Oct-14 12:55
> diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html > new file mode 100644 > index 0000000000..abdfdd0fa2 > --- /dev/null > +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html > @@ -0,0 +1,40 @@ > + > + > +Index of /sources/libxml2/2.9/ > +

Index of /sources/libxml2/2.9/

> + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
LATEST-IS-2.9.143.0 MiB2022-May-02 12:03
libxml2-2.9.0.sha256sum87 B2022-Feb-14 18:27
libxml2-2.9.0.tar.xz3.0 MiB2022-Feb-14 18:27
libxml2-2.9.1.sha256sum87 B2022-Feb-14 18:28
libxml2-2.9.1.tar.xz3.0 MiB2022-Feb-14 18:28
libxml2-2.9.10.sha256sum88 B2022-Feb-14 18:42
libxml2-2.9.10.tar.xz3.2 MiB2022-Feb-14 18:42
libxml2-2.9.11.sha256sum88 B2022-Feb-14 18:43
libxml2-2.9.11.tar.xz3.2 MiB2022-Feb-14 18:43
libxml2-2.9.12.sha256sum88 B2022-Feb-14 18:45
libxml2-2.9.12.tar.xz3.2 MiB2022-Feb-14 18:45
libxml2-2.9.13.news26.6 KiB2022-Feb-20 12:42
libxml2-2.9.13.sha256sum174 B2022-Feb-20 12:42
libxml2-2.9.13.tar.xz3.1 MiB2022-Feb-20 12:42
libxml2-2.9.14.news1.0 KiB2022-May-02 12:03
libxml2-2.9.14.sha256sum174 B2022-May-02 12:03
libxml2-2.9.14.tar.xz3.0 MiB2022-May-02 12:03
libxml2-2.9.2.sha256sum87 B2022-Feb-14 18:30
libxml2-2.9.2.tar.xz3.2 MiB2022-Feb-14 18:30
libxml2-2.9.3.sha256sum87 B2022-Feb-14 18:31
libxml2-2.9.3.tar.xz3.2 MiB2022-Feb-14 18:31
libxml2-2.9.4.sha256sum87 B2022-Feb-14 18:33
libxml2-2.9.4.tar.xz2.9 MiB2022-Feb-14 18:33
libxml2-2.9.5.sha256sum87 B2022-Feb-14 18:35
libxml2-2.9.5.tar.xz3.0 MiB2022-Feb-14 18:35
libxml2-2.9.6.sha256sum87 B2022-Feb-14 18:36
libxml2-2.9.6.tar.xz3.0 MiB2022-Feb-14 18:36
libxml2-2.9.7.sha256sum87 B2022-Feb-14 18:37
libxml2-2.9.7.tar.xz3.0 MiB2022-Feb-14 18:37
libxml2-2.9.8.sha256sum87 B2022-Feb-14 18:39
libxml2-2.9.8.tar.xz3.0 MiB2022-Feb-14 18:39
libxml2-2.9.9.sha256sum87 B2022-Feb-14 18:40
libxml2-2.9.9.tar.xz3.0 MiB2022-Feb-14 18:40
> diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html > new file mode 100644 > index 0000000000..c183e06a55 > --- /dev/null > +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html > @@ -0,0 +1,19 @@ > + > + > +Index of /sources/libxml2/ > +

Index of /sources/libxml2/

> + > + > + > + > + > + > + > + > + > + > + > + > + > + > +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
2.0/-2009-Jul-14 13:04
2.1/-2009-Jul-14 13:04
2.10/-2022-Oct-14 12:55
2.2/-2009-Jul-14 13:04
2.3/-2009-Jul-14 13:05
2.4/-2009-Jul-14 13:05
2.5/-2009-Jul-14 13:05
2.6/-2009-Jul-14 13:05
2.7/-2022-Feb-14 18:24
2.8/-2022-Feb-14 18:26
2.9/-2022-May-02 12:04
cache.json22.8 KiB2022-Oct-14 12:55
> diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py > index 0af06e46e5..ad3d4dea7d 100644 > --- a/bitbake/lib/bb/tests/fetch.py > +++ b/bitbake/lib/bb/tests/fetch.py > @@ -1401,6 +1401,9 @@ class FetchLatestVersionTest(FetcherTest): > # http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz > ("cmake", "/files/v2.8/cmake-2.8.12.1.tar.gz", "", "") > : "2.8.12.1", > + # https://download.gnome.org/sources/libxml2/2.9/libxml2-2.9.14.tar.xz > + ("libxml2", "/software/libxml2/2.9/libxml2-2.9.14.tar.xz", "", "") > + : "2.10.3", > # > # packages with versions only in current directory > # > -- > 2.30.2 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#14158): https://lists.openembedded.org/g/bitbake-devel/message/14158 > Mute This Topic: https://lists.openembedded.org/mt/95556940/3617179 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com] > -=-=-=-=-=-=-=-=-=-=-=- > -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com