From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 291ECC4332F for ; Fri, 9 Dec 2022 14:51:44 +0000 (UTC) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by mx.groups.io with SMTP id smtpd.web10.3170.1670597495204943220 for ; Fri, 09 Dec 2022 06:51:35 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=Pjs/56Pn; spf=pass (domain: bootlin.com, ip: 217.70.183.198, mailfrom: alexandre.belloni@bootlin.com) Received: (Authenticated sender: alexandre.belloni@bootlin.com) by mail.gandi.net (Postfix) with ESMTPSA id F059FC0016; Fri, 9 Dec 2022 14:51:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1670597493; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2cNydvC1noyOLXk9F/nMycWjNLRddYw5L+ziB6dMN1s=; b=Pjs/56PnKl8Gtg7XqAOXrWi3SYhoA9bnRkQI/Vs5CPPvVKsnsF+1RAutGzWPWoS6WnoZmr K34cGw+xjyHgmJFfHkuSbjdGZP4w86WDzmuAdkuhbVLbt6VVfa6IDph3Ne75VSE3tXx9Xh FscZ8kEjiLHtKuDIydLsdixlEHrGl0IVatYZRygOB0Mji2DSFMTePbQQhL2Qs5FlyLFY6Q Z9P1f+m2H4Mrut3FB81NwWaYoRmFOhrNGxr9EzsQWMiLZVVcD7gwMSsd9G/7faB4Jb7/ZO 2eq1yEwwlRUMQL2ylQdVZvUZRz/4hHJ6jaRpUlIZcPfHseQ9GyauImUaWpaimQ== Date: Fri, 9 Dec 2022 15:51:32 +0100 From: Alexandre Belloni To: Alexander Kanavin Cc: bitbake-devel@lists.openembedded.org, Alexander Kanavin Subject: Re: [bitbake-devel] [PATCH v3] fetch2/wget.py: correctly match versioned directories Message-ID: References: <20221209085840.3733809-1-alex@linutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 09 Dec 2022 14:51:44 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14164 On 09/12/2022 15:46:39+0100, Alexander Kanavin wrote: > That is not a failure. It tries ipv6 first, then ipv4 - the former > fails, the latter succeeds. Are you getting an actual red result, and > can I see the link? https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4493/steps/12/logs/stdio I guess the correct one is: Resolving localhost (localhost)... ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:46501... failed: Connection refused. Connecting to localhost (localhost)|127.0.0.1|:46501... connected. HTTP request sent, awaiting response... 404 File not found 2022-12-09 13:21:18 ERROR 404: File not found. > > Alex > > On Fri, 9 Dec 2022 at 15:42, Alexandre Belloni > wrote: > > > > This patch applied correctly but failed: > > > > 2022-12-09 13:21:18 (456 MB/s) - ‘/tmp/wget-index-ypaf69js/wget-listing-m180p7ds’ saved [34419/34419] > > --2022-12-09 13:21:18-- http://localhost:46501/files/v3.16/ > > Resolving localhost (localhost)... ::1, 127.0.0.1 > > Connecting to localhost (localhost)|::1|:46501... failed: Connection refused. > > Connecting to localhost (localhost)|127.0.0.1|:46501... connected. > > HTTP request sent, awaiting response... 200 OK > > Length: 17922 (18K) [text/html] > > Saving to: ‘/tmp/wget-index-3i6kh677/wget-listing-w93f0eg3’ > > > > I suspect we can't connect to random ports on the AB > > > > > > On 09/12/2022 09:58:40+0100, Alexander Kanavin wrote: > > > When obtaining latest upstream versions, the code needs > > > to check if the existing tarball is in a versioned directory > > > (e.g. component-name/x.y/component-name-x.y.z.tar.gz) and > > > if it is, it needs to first obtain the list of all > > > such versioned directories and then check all of them by going > > > one step up in the directory hierarchy. > > > > > > Existing code was returning a correct match when the component > > > name did not have numbers, e.g. a check on 'source/epiphany/43/' > > > would return 43, but was stopping too soon when the component > > > name itself had numbers ('source/libxml2/2.10/' would return libxml2). > > > > > > This change ensures the last match is taken instead of the first. > > > > > > Also, adjust the fetcher tests to check that versioned directories > > > are correctly traversed in this case (e.g. the step to go one level > > > up is taken and a new tarball is discovered in a different versioned > > > directory). > > > > > > Signed-off-by: Alexander Kanavin > > > --- > > > bitbake/lib/bb/fetch2/wget.py | 4 +- > > > .../software/libxml2/2.10/index.html | 20 ++++++++++ > > > .../software/libxml2/2.9/index.html | 40 +++++++++++++++++++ > > > .../software/libxml2/index.html | 19 +++++++++ > > > bitbake/lib/bb/tests/fetch.py | 3 ++ > > > 5 files changed, 84 insertions(+), 2 deletions(-) > > > create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html > > > create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html > > > create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html > > > > > > diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py > > > index 821afa5b58..7ccea3ad89 100644 > > > --- a/bitbake/lib/bb/fetch2/wget.py > > > +++ b/bitbake/lib/bb/fetch2/wget.py > > > @@ -644,10 +644,10 @@ class Wget(FetchMethod): > > > # search for version matches on folders inside the path, like: > > > # "5.7" in http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz > > > dirver_regex = re.compile(r"(?P[^/]*(\d+\.)*\d+([-_]r\d+)*)/") > > > - m = dirver_regex.search(path) > > > + m = dirver_regex.findall(path) > > > if m: > > > pn = d.getVar('PN') > > > - dirver = m.group('dirver') > > > + dirver = m[-1][0] > > > > > > dirver_pn_regex = re.compile(r"%s\d?" % (re.escape(pn))) > > > if not dirver_pn_regex.search(dirver): > > > diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html > > > new file mode 100644 > > > index 0000000000..4e41af6d6a > > > --- /dev/null > > > +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.10/index.html > > > @@ -0,0 +1,20 @@ > > > + > > > + > > > +Index of /sources/libxml2/2.10/ > > > +

Index of /sources/libxml2/2.10/

> > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
LATEST-IS-2.10.32.5 MiB2022-Oct-14 12:55
libxml2-2.10.0.news7.1 KiB2022-Aug-17 11:55
libxml2-2.10.0.sha256sum174 B2022-Aug-17 11:55
libxml2-2.10.0.tar.xz2.6 MiB2022-Aug-17 11:55
libxml2-2.10.1.news455 B2022-Aug-25 11:33
libxml2-2.10.1.sha256sum174 B2022-Aug-25 11:33
libxml2-2.10.1.tar.xz2.6 MiB2022-Aug-25 11:33
libxml2-2.10.2.news309 B2022-Aug-29 14:56
libxml2-2.10.2.sha256sum174 B2022-Aug-29 14:56
libxml2-2.10.2.tar.xz2.5 MiB2022-Aug-29 14:56
libxml2-2.10.3.news294 B2022-Oct-14 12:55
libxml2-2.10.3.sha256sum174 B2022-Oct-14 12:55
libxml2-2.10.3.tar.xz2.5 MiB2022-Oct-14 12:55
> > > diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html > > > new file mode 100644 > > > index 0000000000..abdfdd0fa2 > > > --- /dev/null > > > +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/2.9/index.html > > > @@ -0,0 +1,40 @@ > > > + > > > + > > > +Index of /sources/libxml2/2.9/ > > > +

Index of /sources/libxml2/2.9/

> > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
LATEST-IS-2.9.143.0 MiB2022-May-02 12:03
libxml2-2.9.0.sha256sum87 B2022-Feb-14 18:27
libxml2-2.9.0.tar.xz3.0 MiB2022-Feb-14 18:27
libxml2-2.9.1.sha256sum87 B2022-Feb-14 18:28
libxml2-2.9.1.tar.xz3.0 MiB2022-Feb-14 18:28
libxml2-2.9.10.sha256sum88 B2022-Feb-14 18:42
libxml2-2.9.10.tar.xz3.2 MiB2022-Feb-14 18:42
libxml2-2.9.11.sha256sum88 B2022-Feb-14 18:43
libxml2-2.9.11.tar.xz3.2 MiB2022-Feb-14 18:43
libxml2-2.9.12.sha256sum88 B2022-Feb-14 18:45
libxml2-2.9.12.tar.xz3.2 MiB2022-Feb-14 18:45
libxml2-2.9.13.news26.6 KiB2022-Feb-20 12:42
libxml2-2.9.13.sha256sum174 B2022-Feb-20 12:42
libxml2-2.9.13.tar.xz3.1 MiB2022-Feb-20 12:42
libxml2-2.9.14.news1.0 KiB2022-May-02 12:03
libxml2-2.9.14.sha256sum174 B2022-May-02 12:03
libxml2-2.9.14.tar.xz3.0 MiB2022-May-02 12:03
libxml2-2.9.2.sha256sum87 B2022-Feb-14 18:30
libxml2-2.9.2.tar.xz3.2 MiB2022-Feb-14 18:30
libxml2-2.9.3.sha256sum87 B2022-Feb-14 18:31
libxml2-2.9.3.tar.xz3.2 MiB2022-Feb-14 18:31
libxml2-2.9.4.sha256sum87 B2022-Feb-14 18:33
libxml2-2.9.4.tar.xz2.9 MiB2022-Feb-14 18:33
libxml2-2.9.5.sha256sum87 B2022-Feb-14 18:35
libxml2-2.9.5.tar.xz3.0 MiB2022-Feb-14 18:35
libxml2-2.9.6.sha256sum87 B2022-Feb-14 18:36
libxml2-2.9.6.tar.xz3.0 MiB2022-Feb-14 18:36
libxml2-2.9.7.sha256sum87 B2022-Feb-14 18:37
libxml2-2.9.7.tar.xz3.0 MiB2022-Feb-14 18:37
libxml2-2.9.8.sha256sum87 B2022-Feb-14 18:39
libxml2-2.9.8.tar.xz3.0 MiB2022-Feb-14 18:39
libxml2-2.9.9.sha256sum87 B2022-Feb-14 18:40
libxml2-2.9.9.tar.xz3.0 MiB2022-Feb-14 18:40
> > > diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html > > > new file mode 100644 > > > index 0000000000..c183e06a55 > > > --- /dev/null > > > +++ b/bitbake/lib/bb/tests/fetch-testdata/software/libxml2/index.html > > > @@ -0,0 +1,19 @@ > > > + > > > + > > > +Index of /sources/libxml2/ > > > +

Index of /sources/libxml2/

> > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > + > > > +
File Name  ↓ File Size  ↓ Date  ↓ 
Parent directory/--
2.0/-2009-Jul-14 13:04
2.1/-2009-Jul-14 13:04
2.10/-2022-Oct-14 12:55
2.2/-2009-Jul-14 13:04
2.3/-2009-Jul-14 13:05
2.4/-2009-Jul-14 13:05
2.5/-2009-Jul-14 13:05
2.6/-2009-Jul-14 13:05
2.7/-2022-Feb-14 18:24
2.8/-2022-Feb-14 18:26
2.9/-2022-May-02 12:04
cache.json22.8 KiB2022-Oct-14 12:55
> > > diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py > > > index 0af06e46e5..ad3d4dea7d 100644 > > > --- a/bitbake/lib/bb/tests/fetch.py > > > +++ b/bitbake/lib/bb/tests/fetch.py > > > @@ -1401,6 +1401,9 @@ class FetchLatestVersionTest(FetcherTest): > > > # http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz > > > ("cmake", "/files/v2.8/cmake-2.8.12.1.tar.gz", "", "") > > > : "2.8.12.1", > > > + # https://download.gnome.org/sources/libxml2/2.9/libxml2-2.9.14.tar.xz > > > + ("libxml2", "/software/libxml2/2.9/libxml2-2.9.14.tar.xz", "", "") > > > + : "2.10.3", > > > # > > > # packages with versions only in current directory > > > # > > > -- > > > 2.30.2 > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > Links: You receive all messages sent to this group. > > > View/Reply Online (#14158): https://lists.openembedded.org/g/bitbake-devel/message/14158 > > > Mute This Topic: https://lists.openembedded.org/mt/95556940/3617179 > > > Group Owner: bitbake-devel+owner@lists.openembedded.org > > > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com] > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > > > > -- > > Alexandre Belloni, co-owner and COO, Bootlin > > Embedded Linux and Kernel engineering > > https://bootlin.com -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com