All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2 0/3] distro_check: fix the Ubuntu and openSUSE package lists, drop Clear
@ 2026-11-16 16:50 Tim Orling
  2026-11-16 16:50 ` [PATCH v2 1/3] lib/oe/distro_check: find the latest Ubuntu release from meta-release Tim Orling
                   ` (2 more replies)
  0 siblings, 3 replies; 5+ messages in thread
From: Tim Orling @ 2026-11-16 16:50 UTC (permalink / raw)
  To: openembedded-core; +Cc: Tim Orling

With the fixes for bug 16503, create_distro_packages_list() still got the
wrong Ubuntu release and no openSUSE or Clear lists:

  NOTE: Distro: Ubuntu, Latest Release: xenial, # src packages: 3456
  WARNING: Cannot fetch packages for openSUSE: HTTP Error 404: Not Found
  WARNING: Cannot fetch packages for Clear: <urlopen error [Errno -3] Temporary failure in name resolution>

1: Bug 16509: the Ubuntu archive ignores the "newest first" sort that
   find_latest_ubuntu_release() relied on, so it always took xenial.
   Use the release list that do-release-upgrade reads instead.
2: Bug 16510: download.opensuse.org's listings now link to "./15.6/",
   still include Leap 42.x, and keep the source RPMs under repo/oss/src/;
   Leap 16 has no update repository.
3: Bug 16511: Clear Linux has been discontinued.

With these, all four lists are fetched:

  NOTE: Distro: Debian, Latest Release: 13.7, # src packages: 37593
  NOTE: Distro: Ubuntu, Latest Release: resolute, # src packages: 3053
  NOTE: Distro: Fedora, Latest Release: 45, # src packages: 23237
  NOTE: Distro: openSUSE, Latest Release: 16.1, # src packages: 10638

The new cases are covered by oe-selftest's oelib.distro_check, run with
plain unittest (not oe-selftest).

This applies on top of "Fix distro_check when a distro list cannot be
fetched" (bug 16503):
https://lore.kernel.org/all/20261114200810.31078-1-tim.orling@konsulko.com/

Changes in v2:
- 2/3: use str.removeprefix() for the "./" links (Richard Purdie)

Tim Orling (3):
  lib/oe/distro_check: find the latest Ubuntu release from meta-release
  lib/oe/distro_check: fix the openSUSE Leap package lists
  lib/oe/distro_check: drop Clear Linux

 meta/lib/oe/distro_check.py                   | 59 ++++++++++++-------
 .../oeqa/selftest/cases/oelib/distro_check.py | 49 ++++++++++++++-
 2 files changed, 84 insertions(+), 24 deletions(-)

-- 
2.56.1


^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH v2 1/3] lib/oe/distro_check: find the latest Ubuntu release from meta-release
  2026-11-16 16:50 [PATCH v2 0/3] distro_check: fix the Ubuntu and openSUSE package lists, drop Clear Tim Orling
@ 2026-11-16 16:50 ` Tim Orling
  2026-11-16 16:50 ` [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists Tim Orling
  2026-11-16 16:50 ` [PATCH v2 3/3] lib/oe/distro_check: drop Clear Linux Tim Orling
  2 siblings, 0 replies; 5+ messages in thread
From: Tim Orling @ 2026-11-16 16:50 UTC (permalink / raw)
  To: openembedded-core; +Cc: Tim Orling

find_latest_ubuntu_release() asked the archive for its dists/ listing
newest first and took the first *-updates entry, but the archive ignores
the sort and lists the oldest first, so Ubuntu was always xenial. Read
the release list that do-release-upgrade uses instead, and take the last
supported release.

Tested against changelogs.ubuntu.com and archive.ubuntu.com: resolute,
with main and updates packages.

[YOCTO #16509]

AI-Generated: Claude Opus 5.5
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 meta/lib/oe/distro_check.py                   | 26 ++++++++++---------
 .../oeqa/selftest/cases/oelib/distro_check.py | 26 +++++++++++++++++++
 2 files changed, 40 insertions(+), 12 deletions(-)

diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
index 223bed2ec2..814b35b8d8 100644
--- a/meta/lib/oe/distro_check.py
+++ b/meta/lib/oe/distro_check.py
@@ -131,23 +131,25 @@ def get_latest_released_debian_source_package_list(d):
     package_names |= get_debian_style_source_package_list(url, "updates", d)
     return latest, package_names
 
-def find_latest_ubuntu_release(url, d):
+def find_latest_ubuntu_release(d):
     """
-    Find the latest listed Ubuntu release on the given ubuntu/dists/ URL.
-
-    To avoid matching development releases look for distributions that have
-    updates, so the resulting distro could be any supported release.
+    Find the latest supported Ubuntu release, from the list of releases
+    that do-release-upgrade reads. The releases are listed oldest first.
     """
-    url += "?C=M;O=D" # Descending Sort by Last Modified
-    for link in get_links_from_url(url, d):
-        if "-updates" in link:
-            distro = link.replace("-updates", "")
-            return distro
-    return "_NotFound_"
+    latest = "_NotFound_"
+    dist = None
+    listing = create_socket("https://changelogs.ubuntu.com/meta-release", d).read().decode("utf-8")
+    for line in listing.splitlines():
+        key, _, value = line.partition(":")
+        if key == "Dist":
+            dist = value.strip()
+        elif key == "Supported" and value.strip() == "1" and dist:
+            latest = dist
+    return latest
 
 def get_latest_released_ubuntu_source_package_list(d):
     "Returns list of all the name os packages in the latest ubuntu distro"
-    latest = find_latest_ubuntu_release("http://archive.ubuntu.com/ubuntu/dists/", d)
+    latest = find_latest_ubuntu_release(d)
     url = "http://archive.ubuntu.com/ubuntu/dists/%s/main/source/Sources.gz" % latest
     package_names = get_debian_style_source_package_list(url, "main", d)
     url = "http://archive.ubuntu.com/ubuntu/dists/%s-updates/main/source/Sources.gz" % latest
diff --git a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
index f47d1dd4a4..7545fa0c42 100644
--- a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
+++ b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
@@ -31,6 +31,32 @@ class TestDebianStyleSourcePackageList(TestCase):
     def test_xz(self):
         self.check("http://example.com/Sources.xz", lzma.compress(SOURCES))
 
+META_RELEASE = b"""Dist: noble
+Name: Noble Numbat
+Version: 24.04.5 LTS
+Supported: 1
+
+Dist: questing
+Name: Questing Quokka
+Version: 25.10
+Supported: 0
+
+Dist: resolute
+Name: Resolute Raccoon
+Version: 26.04.1 LTS
+Supported: 1
+
+Dist: zzz
+Name: Not supported any more
+Version: 99.10
+Supported: 0
+"""
+
+class TestFindLatestUbuntuRelease(TestCase):
+    def test_latest_supported(self):
+        with mock.patch.object(oe.distro_check, "create_socket", return_value=io.BytesIO(META_RELEASE)):
+            self.assertEqual(oe.distro_check.find_latest_ubuntu_release(None), "resolute")
+
 class TestCreateDistroPackagesList(TestCase):
     def test_skip_failed_distro(self):
         """
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists
  2026-11-16 16:50 [PATCH v2 0/3] distro_check: fix the Ubuntu and openSUSE package lists, drop Clear Tim Orling
  2026-11-16 16:50 ` [PATCH v2 1/3] lib/oe/distro_check: find the latest Ubuntu release from meta-release Tim Orling
@ 2026-11-16 16:50 ` Tim Orling
  2026-11-20 13:48   ` Paul Barker
  2026-11-16 16:50 ` [PATCH v2 3/3] lib/oe/distro_check: drop Clear Linux Tim Orling
  2 siblings, 1 reply; 5+ messages in thread
From: Tim Orling @ 2026-11-16 16:50 UTC (permalink / raw)
  To: openembedded-core; +Cc: Tim Orling

download.opensuse.org's listings now link to "./15.6/", which
find_latest_numeric_release() couldn't parse, so the Leap URLs had an
empty release and failed with HTTP 404. Accept "./" links, ignore Leap
42.x (which came before 15.0), read the source RPMs from repo/oss/src/,
where they now are, and allow for there being no update repository, as
for Leap 16.

Tested against download.opensuse.org: Leap 16.1 is fetched again.

[YOCTO #16510]

AI-Generated: Claude Opus 5.5
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
Changes in v2: use str.removeprefix() for the "./" links

 meta/lib/oe/distro_check.py                   | 27 ++++++++++++++++---
 .../oeqa/selftest/cases/oelib/distro_check.py | 19 +++++++++++++
 2 files changed, 42 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
index 814b35b8d8..b5c05a6765 100644
--- a/meta/lib/oe/distro_check.py
+++ b/meta/lib/oe/distro_check.py
@@ -22,7 +22,8 @@ def get_links_from_url(url, d):
     soup = BeautifulSoup(create_socket(url,d), "html.parser", parse_only=SoupStrainer("a"))
     hyperlinks = []
     for line in soup.find_all('a', href=True):
-        hyperlinks.append(line['href'].strip('/'))
+        # Some listings (e.g. MirrorCache) link to "./name/"
+        hyperlinks.append(line['href'].removeprefix('./').strip('/'))
     return hyperlinks
 
 def find_latest_numeric_release(url, d):
@@ -84,12 +85,30 @@ def get_latest_released_fedora_source_package_list(d):
     package_names |= get_source_package_list_from_url_by_letter("http://archive.fedoraproject.org/pub/fedora/linux/updates/%s/SRPMS/" % latest, "updates", d)
     return latest, package_names
 
+def find_latest_opensuse_leap_release(url, d):
+    "Find the latest Leap release on the given url, ignoring Leap 42.x, which came before 15.0"
+    releases = []
+    for link in get_links_from_url(url, d):
+        try:
+            version = tuple(int(n) for n in link.split("."))
+        except ValueError:
+            continue
+        if version[0] != 42:
+            releases.append((version, link))
+    return max(releases)[1] if releases else "_NotFound_"
+
 def get_latest_released_opensuse_source_package_list(d):
     "Returns list of all the name os packages in the latest opensuse distro"
-    latest = find_latest_numeric_release("http://download.opensuse.org/source/distribution/leap", d)
+    from urllib.error import HTTPError
+    latest = find_latest_opensuse_leap_release("http://download.opensuse.org/source/distribution/leap/", d)
 
-    package_names = get_source_package_list_from_url("http://download.opensuse.org/source/distribution/leap/%s/repo/oss/suse/src/" % latest, "main", d)
-    package_names |= get_source_package_list_from_url("http://download.opensuse.org/update/leap/%s/oss/src/" % latest, "updates", d)
+    package_names = get_source_package_list_from_url("http://download.opensuse.org/source/distribution/leap/%s/repo/oss/src/" % latest, "main", d)
+    # Leap 16 has no separate update repository
+    try:
+        package_names |= get_source_package_list_from_url("http://download.opensuse.org/update/leap/%s/oss/src/" % latest, "updates", d)
+    except HTTPError as e:
+        if e.code != 404:
+            raise
     return latest, package_names
 
 def get_latest_released_clear_source_package_list(d):
diff --git a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
index 7545fa0c42..7add8d70f9 100644
--- a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
+++ b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
@@ -57,6 +57,25 @@ class TestFindLatestUbuntuRelease(TestCase):
         with mock.patch.object(oe.distro_check, "create_socket", return_value=io.BytesIO(META_RELEASE)):
             self.assertEqual(oe.distro_check.find_latest_ubuntu_release(None), "resolute")
 
+class TestOpenSUSE(TestCase):
+    LISTING = b"""<a href="../">..</a> <a href="./15.0-Current/">15.0-Current/</a>
+<a href="./15.6/">15.6/</a> <a href="./16.0/">16.0/</a> <a href="./42.3/">42.3/</a>"""
+
+    def test_links_relative_to_current_directory(self):
+        try:
+            from bs4 import BeautifulSoup, SoupStrainer
+        except ImportError:
+            self.skipTest("bs4 is not available")
+        with mock.patch.object(oe.distro_check, "create_socket", return_value=io.BytesIO(self.LISTING)):
+            links = oe.distro_check.get_links_from_url("http://example.com/leap/", None)
+        self.assertEqual(links, ["..", "15.0-Current", "15.6", "16.0", "42.3"])
+
+    def test_latest_leap_release(self):
+        # Leap 42.x came before 15.0
+        links = ["..", "15.0-Current", "15.6", "16.0", "16.1", "42.2", "42.3", "42.3-Current"]
+        with mock.patch.object(oe.distro_check, "get_links_from_url", return_value=links):
+            self.assertEqual(oe.distro_check.find_latest_opensuse_leap_release("http://example.com/leap/", None), "16.1")
+
 class TestCreateDistroPackagesList(TestCase):
     def test_skip_failed_distro(self):
         """
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH v2 3/3] lib/oe/distro_check: drop Clear Linux
  2026-11-16 16:50 [PATCH v2 0/3] distro_check: fix the Ubuntu and openSUSE package lists, drop Clear Tim Orling
  2026-11-16 16:50 ` [PATCH v2 1/3] lib/oe/distro_check: find the latest Ubuntu release from meta-release Tim Orling
  2026-11-16 16:50 ` [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists Tim Orling
@ 2026-11-16 16:50 ` Tim Orling
  2 siblings, 0 replies; 5+ messages in thread
From: Tim Orling @ 2026-11-16 16:50 UTC (permalink / raw)
  To: openembedded-core; +Cc: Tim Orling

Clear Linux OS was discontinued in 2025 and download.clearlinux.org no
longer resolves, so its package list could only ever fail.

[YOCTO #16511]

AI-Generated: Claude Opus 5.5
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 meta/lib/oe/distro_check.py                        | 6 ------
 meta/lib/oeqa/selftest/cases/oelib/distro_check.py | 4 ++--
 2 files changed, 2 insertions(+), 8 deletions(-)

diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
index b5c05a6765..d8e487318c 100644
--- a/meta/lib/oe/distro_check.py
+++ b/meta/lib/oe/distro_check.py
@@ -111,11 +111,6 @@ def get_latest_released_opensuse_source_package_list(d):
             raise
     return latest, package_names
 
-def get_latest_released_clear_source_package_list(d):
-    latest = find_latest_numeric_release("https://download.clearlinux.org/releases/", d)
-    package_names = get_source_package_list_from_url("https://download.clearlinux.org/releases/%s/clear/source/SRPMS/" % latest, "main", d)
-    return latest, package_names
-
 def find_latest_debian_release(url, d):
     "Find the latest listed debian release on the given url"
 
@@ -185,7 +180,6 @@ def create_distro_packages_list(distro_check_dir, d):
                             ("Ubuntu", get_latest_released_ubuntu_source_package_list),
                             ("Fedora", get_latest_released_fedora_source_package_list),
                             ("openSUSE", get_latest_released_opensuse_source_package_list),
-                            ("Clear", get_latest_released_clear_source_package_list),
                            )
 
     for name, fetcher_func in per_distro_functions:
diff --git a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
index 7add8d70f9..d783384251 100644
--- a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
+++ b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
@@ -89,7 +89,7 @@ class TestCreateDistroPackagesList(TestCase):
             return "1.0", {"foo:main"}
 
         fetchers = {"get_latest_released_%s_source_package_list" % name: ok
-                    for name in ("ubuntu", "fedora", "opensuse", "clear")}
+                    for name in ("ubuntu", "fedora", "opensuse")}
         fetchers["get_latest_released_debian_source_package_list"] = fail
 
         with tempfile.TemporaryDirectory() as tmpdir, \
@@ -99,6 +99,6 @@ class TestCreateDistroPackagesList(TestCase):
             oe.distro_check.create_distro_packages_list(tmpdir, None)
             lists = sorted(os.listdir(os.path.join(tmpdir, "package_lists")))
 
-        self.assertEqual(lists, ["Clear-1.0", "Fedora-1.0", "Ubuntu-1.0", "openSUSE-1.0"])
+        self.assertEqual(lists, ["Fedora-1.0", "Ubuntu-1.0", "openSUSE-1.0"])
         warn.assert_called_once()
         self.assertIn("Debian", warn.call_args[0][0])
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists
  2026-11-16 16:50 ` [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists Tim Orling
@ 2026-11-20 13:48   ` Paul Barker
  0 siblings, 0 replies; 5+ messages in thread
From: Paul Barker @ 2026-11-20 13:48 UTC (permalink / raw)
  To: Tim Orling, openembedded-core; +Cc: Tim Orling

On Mon Nov 16, 2026 at 4:50 PM GMT, Tim Orling wrote:
> download.opensuse.org's listings now link to "./15.6/", which
> find_latest_numeric_release() couldn't parse, so the Leap URLs had an
> empty release and failed with HTTP 404. Accept "./" links, ignore Leap
> 42.x (which came before 15.0), read the source RPMs from repo/oss/src/,
> where they now are, and allow for there being no update repository, as
> for Leap 16.
>
> Tested against download.opensuse.org: Leap 16.1 is fetched again.
>
> [YOCTO #16510]
>
> AI-Generated: Claude Opus 5.5
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---
> Changes in v2: use str.removeprefix() for the "./" links
>
>  meta/lib/oe/distro_check.py                   | 27 ++++++++++++++++---
>  .../oeqa/selftest/cases/oelib/distro_check.py | 19 +++++++++++++
>  2 files changed, 42 insertions(+), 4 deletions(-)
>
> diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
> index 814b35b8d8..b5c05a6765 100644
> --- a/meta/lib/oe/distro_check.py
> +++ b/meta/lib/oe/distro_check.py
> @@ -22,7 +22,8 @@ def get_links_from_url(url, d):
>      soup = BeautifulSoup(create_socket(url,d), "html.parser", parse_only=SoupStrainer("a"))
>      hyperlinks = []
>      for line in soup.find_all('a', href=True):
> -        hyperlinks.append(line['href'].strip('/'))
> +        # Some listings (e.g. MirrorCache) link to "./name/"
> +        hyperlinks.append(line['href'].removeprefix('./').strip('/'))
>      return hyperlinks
>  
>  def find_latest_numeric_release(url, d):
> @@ -84,12 +85,30 @@ def get_latest_released_fedora_source_package_list(d):
>      package_names |= get_source_package_list_from_url_by_letter("http://archive.fedoraproject.org/pub/fedora/linux/updates/%s/SRPMS/" % latest, "updates", d)
>      return latest, package_names
>  
> +def find_latest_opensuse_leap_release(url, d):
> +    "Find the latest Leap release on the given url, ignoring Leap 42.x, which came before 15.0"
> +    releases = []
> +    for link in get_links_from_url(url, d):
> +        try:
> +            version = tuple(int(n) for n in link.split("."))
> +        except ValueError:
> +            continue
> +        if version[0] != 42:
> +            releases.append((version, link))
> +    return max(releases)[1] if releases else "_NotFound_"
> +
>  def get_latest_released_opensuse_source_package_list(d):
>      "Returns list of all the name os packages in the latest opensuse distro"
> -    latest = find_latest_numeric_release("http://download.opensuse.org/source/distribution/leap", d)
> +    from urllib.error import HTTPError
> +    latest = find_latest_opensuse_leap_release("http://download.opensuse.org/source/distribution/leap/", d)
>  
> -    package_names = get_source_package_list_from_url("http://download.opensuse.org/source/distribution/leap/%s/repo/oss/suse/src/" % latest, "main", d)
> -    package_names |= get_source_package_list_from_url("http://download.opensuse.org/update/leap/%s/oss/src/" % latest, "updates", d)
> +    package_names = get_source_package_list_from_url("http://download.opensuse.org/source/distribution/leap/%s/repo/oss/src/" % latest, "main", d)
> +    # Leap 16 has no separate update repository
> +    try:
> +        package_names |= get_source_package_list_from_url("http://download.opensuse.org/update/leap/%s/oss/src/" % latest, "updates", d)
> +    except HTTPError as e:
> +        if e.code != 404:
> +            raise
>      return latest, package_names
>  
>  def get_latest_released_clear_source_package_list(d):
> diff --git a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
> index 7545fa0c42..7add8d70f9 100644
> --- a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
> +++ b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
> @@ -57,6 +57,25 @@ class TestFindLatestUbuntuRelease(TestCase):
>          with mock.patch.object(oe.distro_check, "create_socket", return_value=io.BytesIO(META_RELEASE)):
>              self.assertEqual(oe.distro_check.find_latest_ubuntu_release(None), "resolute")
>  
> +class TestOpenSUSE(TestCase):
> +    LISTING = b"""<a href="../">..</a> <a href="./15.0-Current/">15.0-Current/</a>
> +<a href="./15.6/">15.6/</a> <a href="./16.0/">16.0/</a> <a href="./42.3/">42.3/</a>"""
> +
> +    def test_links_relative_to_current_directory(self):
> +        try:
> +            from bs4 import BeautifulSoup, SoupStrainer

Hi Tim,

This has been merged to master, but I still wonder... Should this use
our vendored copy of bs4 from bitbake (bb._vendor.bs4)?

Best regards,

-- 
Paul Barker


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-20 13:48 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 16:50 [PATCH v2 0/3] distro_check: fix the Ubuntu and openSUSE package lists, drop Clear Tim Orling
2026-11-16 16:50 ` [PATCH v2 1/3] lib/oe/distro_check: find the latest Ubuntu release from meta-release Tim Orling
2026-11-16 16:50 ` [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists Tim Orling
2026-11-20 13:48   ` Paul Barker
2026-11-16 16:50 ` [PATCH v2 3/3] lib/oe/distro_check: drop Clear Linux Tim Orling

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.