Openembedded Core Discussions
 help / color / mirror / Atom feed
* [PATCH v2 0/3] distro_check: fix the Ubuntu and openSUSE package lists, drop Clear
@ 2026-11-16 16:50 Tim Orling
  2026-11-16 16:50 ` [PATCH v2 1/3] lib/oe/distro_check: find the latest Ubuntu release from meta-release Tim Orling
                   ` (2 more replies)
  0 siblings, 3 replies; 5+ messages in thread
From: Tim Orling @ 2026-11-16 16:50 UTC (permalink / raw)
  To: openembedded-core; +Cc: Tim Orling

With the fixes for bug 16503, create_distro_packages_list() still got the
wrong Ubuntu release and no openSUSE or Clear lists:

  NOTE: Distro: Ubuntu, Latest Release: xenial, # src packages: 3456
  WARNING: Cannot fetch packages for openSUSE: HTTP Error 404: Not Found
  WARNING: Cannot fetch packages for Clear: <urlopen error [Errno -3] Temporary failure in name resolution>

1: Bug 16509: the Ubuntu archive ignores the "newest first" sort that
   find_latest_ubuntu_release() relied on, so it always took xenial.
   Use the release list that do-release-upgrade reads instead.
2: Bug 16510: download.opensuse.org's listings now link to "./15.6/",
   still include Leap 42.x, and keep the source RPMs under repo/oss/src/;
   Leap 16 has no update repository.
3: Bug 16511: Clear Linux has been discontinued.

With these, all four lists are fetched:

  NOTE: Distro: Debian, Latest Release: 13.7, # src packages: 37593
  NOTE: Distro: Ubuntu, Latest Release: resolute, # src packages: 3053
  NOTE: Distro: Fedora, Latest Release: 45, # src packages: 23237
  NOTE: Distro: openSUSE, Latest Release: 16.1, # src packages: 10638

The new cases are covered by oe-selftest's oelib.distro_check, run with
plain unittest (not oe-selftest).

This applies on top of "Fix distro_check when a distro list cannot be
fetched" (bug 16503):
https://lore.kernel.org/all/20261114200810.31078-1-tim.orling@konsulko.com/

Changes in v2:
- 2/3: use str.removeprefix() for the "./" links (Richard Purdie)

Tim Orling (3):
  lib/oe/distro_check: find the latest Ubuntu release from meta-release
  lib/oe/distro_check: fix the openSUSE Leap package lists
  lib/oe/distro_check: drop Clear Linux

 meta/lib/oe/distro_check.py                   | 59 ++++++++++++-------
 .../oeqa/selftest/cases/oelib/distro_check.py | 49 ++++++++++++++-
 2 files changed, 84 insertions(+), 24 deletions(-)

-- 
2.56.1


^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH v2 1/3] lib/oe/distro_check: find the latest Ubuntu release from meta-release
  2026-11-16 16:50 [PATCH v2 0/3] distro_check: fix the Ubuntu and openSUSE package lists, drop Clear Tim Orling
@ 2026-11-16 16:50 ` Tim Orling
  2026-11-16 16:50 ` [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists Tim Orling
  2026-11-16 16:50 ` [PATCH v2 3/3] lib/oe/distro_check: drop Clear Linux Tim Orling
  2 siblings, 0 replies; 5+ messages in thread
From: Tim Orling @ 2026-11-16 16:50 UTC (permalink / raw)
  To: openembedded-core; +Cc: Tim Orling

find_latest_ubuntu_release() asked the archive for its dists/ listing
newest first and took the first *-updates entry, but the archive ignores
the sort and lists the oldest first, so Ubuntu was always xenial. Read
the release list that do-release-upgrade uses instead, and take the last
supported release.

Tested against changelogs.ubuntu.com and archive.ubuntu.com: resolute,
with main and updates packages.

[YOCTO #16509]

AI-Generated: Claude Opus 5.5
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 meta/lib/oe/distro_check.py                   | 26 ++++++++++---------
 .../oeqa/selftest/cases/oelib/distro_check.py | 26 +++++++++++++++++++
 2 files changed, 40 insertions(+), 12 deletions(-)

diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
index 223bed2ec2..814b35b8d8 100644
--- a/meta/lib/oe/distro_check.py
+++ b/meta/lib/oe/distro_check.py
@@ -131,23 +131,25 @@ def get_latest_released_debian_source_package_list(d):
     package_names |= get_debian_style_source_package_list(url, "updates", d)
     return latest, package_names
 
-def find_latest_ubuntu_release(url, d):
+def find_latest_ubuntu_release(d):
     """
-    Find the latest listed Ubuntu release on the given ubuntu/dists/ URL.
-
-    To avoid matching development releases look for distributions that have
-    updates, so the resulting distro could be any supported release.
+    Find the latest supported Ubuntu release, from the list of releases
+    that do-release-upgrade reads. The releases are listed oldest first.
     """
-    url += "?C=M;O=D" # Descending Sort by Last Modified
-    for link in get_links_from_url(url, d):
-        if "-updates" in link:
-            distro = link.replace("-updates", "")
-            return distro
-    return "_NotFound_"
+    latest = "_NotFound_"
+    dist = None
+    listing = create_socket("https://changelogs.ubuntu.com/meta-release", d).read().decode("utf-8")
+    for line in listing.splitlines():
+        key, _, value = line.partition(":")
+        if key == "Dist":
+            dist = value.strip()
+        elif key == "Supported" and value.strip() == "1" and dist:
+            latest = dist
+    return latest
 
 def get_latest_released_ubuntu_source_package_list(d):
     "Returns list of all the name os packages in the latest ubuntu distro"
-    latest = find_latest_ubuntu_release("http://archive.ubuntu.com/ubuntu/dists/", d)
+    latest = find_latest_ubuntu_release(d)
     url = "http://archive.ubuntu.com/ubuntu/dists/%s/main/source/Sources.gz" % latest
     package_names = get_debian_style_source_package_list(url, "main", d)
     url = "http://archive.ubuntu.com/ubuntu/dists/%s-updates/main/source/Sources.gz" % latest
diff --git a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
index f47d1dd4a4..7545fa0c42 100644
--- a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
+++ b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
@@ -31,6 +31,32 @@ class TestDebianStyleSourcePackageList(TestCase):
     def test_xz(self):
         self.check("http://example.com/Sources.xz", lzma.compress(SOURCES))
 
+META_RELEASE = b"""Dist: noble
+Name: Noble Numbat
+Version: 24.04.5 LTS
+Supported: 1
+
+Dist: questing
+Name: Questing Quokka
+Version: 25.10
+Supported: 0
+
+Dist: resolute
+Name: Resolute Raccoon
+Version: 26.04.1 LTS
+Supported: 1
+
+Dist: zzz
+Name: Not supported any more
+Version: 99.10
+Supported: 0
+"""
+
+class TestFindLatestUbuntuRelease(TestCase):
+    def test_latest_supported(self):
+        with mock.patch.object(oe.distro_check, "create_socket", return_value=io.BytesIO(META_RELEASE)):
+            self.assertEqual(oe.distro_check.find_latest_ubuntu_release(None), "resolute")
+
 class TestCreateDistroPackagesList(TestCase):
     def test_skip_failed_distro(self):
         """
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists
  2026-11-16 16:50 [PATCH v2 0/3] distro_check: fix the Ubuntu and openSUSE package lists, drop Clear Tim Orling
  2026-11-16 16:50 ` [PATCH v2 1/3] lib/oe/distro_check: find the latest Ubuntu release from meta-release Tim Orling
@ 2026-11-16 16:50 ` Tim Orling
  2026-11-20 13:48   ` Paul Barker
  2026-11-16 16:50 ` [PATCH v2 3/3] lib/oe/distro_check: drop Clear Linux Tim Orling
  2 siblings, 1 reply; 5+ messages in thread
From: Tim Orling @ 2026-11-16 16:50 UTC (permalink / raw)
  To: openembedded-core; +Cc: Tim Orling

download.opensuse.org's listings now link to "./15.6/", which
find_latest_numeric_release() couldn't parse, so the Leap URLs had an
empty release and failed with HTTP 404. Accept "./" links, ignore Leap
42.x (which came before 15.0), read the source RPMs from repo/oss/src/,
where they now are, and allow for there being no update repository, as
for Leap 16.

Tested against download.opensuse.org: Leap 16.1 is fetched again.

[YOCTO #16510]

AI-Generated: Claude Opus 5.5
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
Changes in v2: use str.removeprefix() for the "./" links

 meta/lib/oe/distro_check.py                   | 27 ++++++++++++++++---
 .../oeqa/selftest/cases/oelib/distro_check.py | 19 +++++++++++++
 2 files changed, 42 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
index 814b35b8d8..b5c05a6765 100644
--- a/meta/lib/oe/distro_check.py
+++ b/meta/lib/oe/distro_check.py
@@ -22,7 +22,8 @@ def get_links_from_url(url, d):
     soup = BeautifulSoup(create_socket(url,d), "html.parser", parse_only=SoupStrainer("a"))
     hyperlinks = []
     for line in soup.find_all('a', href=True):
-        hyperlinks.append(line['href'].strip('/'))
+        # Some listings (e.g. MirrorCache) link to "./name/"
+        hyperlinks.append(line['href'].removeprefix('./').strip('/'))
     return hyperlinks
 
 def find_latest_numeric_release(url, d):
@@ -84,12 +85,30 @@ def get_latest_released_fedora_source_package_list(d):
     package_names |= get_source_package_list_from_url_by_letter("http://archive.fedoraproject.org/pub/fedora/linux/updates/%s/SRPMS/" % latest, "updates", d)
     return latest, package_names
 
+def find_latest_opensuse_leap_release(url, d):
+    "Find the latest Leap release on the given url, ignoring Leap 42.x, which came before 15.0"
+    releases = []
+    for link in get_links_from_url(url, d):
+        try:
+            version = tuple(int(n) for n in link.split("."))
+        except ValueError:
+            continue
+        if version[0] != 42:
+            releases.append((version, link))
+    return max(releases)[1] if releases else "_NotFound_"
+
 def get_latest_released_opensuse_source_package_list(d):
     "Returns list of all the name os packages in the latest opensuse distro"
-    latest = find_latest_numeric_release("http://download.opensuse.org/source/distribution/leap", d)
+    from urllib.error import HTTPError
+    latest = find_latest_opensuse_leap_release("http://download.opensuse.org/source/distribution/leap/", d)
 
-    package_names = get_source_package_list_from_url("http://download.opensuse.org/source/distribution/leap/%s/repo/oss/suse/src/" % latest, "main", d)
-    package_names |= get_source_package_list_from_url("http://download.opensuse.org/update/leap/%s/oss/src/" % latest, "updates", d)
+    package_names = get_source_package_list_from_url("http://download.opensuse.org/source/distribution/leap/%s/repo/oss/src/" % latest, "main", d)
+    # Leap 16 has no separate update repository
+    try:
+        package_names |= get_source_package_list_from_url("http://download.opensuse.org/update/leap/%s/oss/src/" % latest, "updates", d)
+    except HTTPError as e:
+        if e.code != 404:
+            raise
     return latest, package_names
 
 def get_latest_released_clear_source_package_list(d):
diff --git a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
index 7545fa0c42..7add8d70f9 100644
--- a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
+++ b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
@@ -57,6 +57,25 @@ class TestFindLatestUbuntuRelease(TestCase):
         with mock.patch.object(oe.distro_check, "create_socket", return_value=io.BytesIO(META_RELEASE)):
             self.assertEqual(oe.distro_check.find_latest_ubuntu_release(None), "resolute")
 
+class TestOpenSUSE(TestCase):
+    LISTING = b"""<a href="../">..</a> <a href="./15.0-Current/">15.0-Current/</a>
+<a href="./15.6/">15.6/</a> <a href="./16.0/">16.0/</a> <a href="./42.3/">42.3/</a>"""
+
+    def test_links_relative_to_current_directory(self):
+        try:
+            from bs4 import BeautifulSoup, SoupStrainer
+        except ImportError:
+            self.skipTest("bs4 is not available")
+        with mock.patch.object(oe.distro_check, "create_socket", return_value=io.BytesIO(self.LISTING)):
+            links = oe.distro_check.get_links_from_url("http://example.com/leap/", None)
+        self.assertEqual(links, ["..", "15.0-Current", "15.6", "16.0", "42.3"])
+
+    def test_latest_leap_release(self):
+        # Leap 42.x came before 15.0
+        links = ["..", "15.0-Current", "15.6", "16.0", "16.1", "42.2", "42.3", "42.3-Current"]
+        with mock.patch.object(oe.distro_check, "get_links_from_url", return_value=links):
+            self.assertEqual(oe.distro_check.find_latest_opensuse_leap_release("http://example.com/leap/", None), "16.1")
+
 class TestCreateDistroPackagesList(TestCase):
     def test_skip_failed_distro(self):
         """
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH v2 3/3] lib/oe/distro_check: drop Clear Linux
  2026-11-16 16:50 [PATCH v2 0/3] distro_check: fix the Ubuntu and openSUSE package lists, drop Clear Tim Orling
  2026-11-16 16:50 ` [PATCH v2 1/3] lib/oe/distro_check: find the latest Ubuntu release from meta-release Tim Orling
  2026-11-16 16:50 ` [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists Tim Orling
@ 2026-11-16 16:50 ` Tim Orling
  2 siblings, 0 replies; 5+ messages in thread
From: Tim Orling @ 2026-11-16 16:50 UTC (permalink / raw)
  To: openembedded-core; +Cc: Tim Orling

Clear Linux OS was discontinued in 2025 and download.clearlinux.org no
longer resolves, so its package list could only ever fail.

[YOCTO #16511]

AI-Generated: Claude Opus 5.5
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 meta/lib/oe/distro_check.py                        | 6 ------
 meta/lib/oeqa/selftest/cases/oelib/distro_check.py | 4 ++--
 2 files changed, 2 insertions(+), 8 deletions(-)

diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
index b5c05a6765..d8e487318c 100644
--- a/meta/lib/oe/distro_check.py
+++ b/meta/lib/oe/distro_check.py
@@ -111,11 +111,6 @@ def get_latest_released_opensuse_source_package_list(d):
             raise
     return latest, package_names
 
-def get_latest_released_clear_source_package_list(d):
-    latest = find_latest_numeric_release("https://download.clearlinux.org/releases/", d)
-    package_names = get_source_package_list_from_url("https://download.clearlinux.org/releases/%s/clear/source/SRPMS/" % latest, "main", d)
-    return latest, package_names
-
 def find_latest_debian_release(url, d):
     "Find the latest listed debian release on the given url"
 
@@ -185,7 +180,6 @@ def create_distro_packages_list(distro_check_dir, d):
                             ("Ubuntu", get_latest_released_ubuntu_source_package_list),
                             ("Fedora", get_latest_released_fedora_source_package_list),
                             ("openSUSE", get_latest_released_opensuse_source_package_list),
-                            ("Clear", get_latest_released_clear_source_package_list),
                            )
 
     for name, fetcher_func in per_distro_functions:
diff --git a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
index 7add8d70f9..d783384251 100644
--- a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
+++ b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
@@ -89,7 +89,7 @@ class TestCreateDistroPackagesList(TestCase):
             return "1.0", {"foo:main"}
 
         fetchers = {"get_latest_released_%s_source_package_list" % name: ok
-                    for name in ("ubuntu", "fedora", "opensuse", "clear")}
+                    for name in ("ubuntu", "fedora", "opensuse")}
         fetchers["get_latest_released_debian_source_package_list"] = fail
 
         with tempfile.TemporaryDirectory() as tmpdir, \
@@ -99,6 +99,6 @@ class TestCreateDistroPackagesList(TestCase):
             oe.distro_check.create_distro_packages_list(tmpdir, None)
             lists = sorted(os.listdir(os.path.join(tmpdir, "package_lists")))
 
-        self.assertEqual(lists, ["Clear-1.0", "Fedora-1.0", "Ubuntu-1.0", "openSUSE-1.0"])
+        self.assertEqual(lists, ["Fedora-1.0", "Ubuntu-1.0", "openSUSE-1.0"])
         warn.assert_called_once()
         self.assertIn("Debian", warn.call_args[0][0])
-- 
2.56.1


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists
  2026-11-16 16:50 ` [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists Tim Orling
@ 2026-11-20 13:48   ` Paul Barker
  0 siblings, 0 replies; 5+ messages in thread
From: Paul Barker @ 2026-11-20 13:48 UTC (permalink / raw)
  To: Tim Orling, openembedded-core; +Cc: Tim Orling

On Mon Nov 16, 2026 at 4:50 PM GMT, Tim Orling wrote:
> download.opensuse.org's listings now link to "./15.6/", which
> find_latest_numeric_release() couldn't parse, so the Leap URLs had an
> empty release and failed with HTTP 404. Accept "./" links, ignore Leap
> 42.x (which came before 15.0), read the source RPMs from repo/oss/src/,
> where they now are, and allow for there being no update repository, as
> for Leap 16.
>
> Tested against download.opensuse.org: Leap 16.1 is fetched again.
>
> [YOCTO #16510]
>
> AI-Generated: Claude Opus 5.5
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---
> Changes in v2: use str.removeprefix() for the "./" links
>
>  meta/lib/oe/distro_check.py                   | 27 ++++++++++++++++---
>  .../oeqa/selftest/cases/oelib/distro_check.py | 19 +++++++++++++
>  2 files changed, 42 insertions(+), 4 deletions(-)
>
> diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
> index 814b35b8d8..b5c05a6765 100644
> --- a/meta/lib/oe/distro_check.py
> +++ b/meta/lib/oe/distro_check.py
> @@ -22,7 +22,8 @@ def get_links_from_url(url, d):
>      soup = BeautifulSoup(create_socket(url,d), "html.parser", parse_only=SoupStrainer("a"))
>      hyperlinks = []
>      for line in soup.find_all('a', href=True):
> -        hyperlinks.append(line['href'].strip('/'))
> +        # Some listings (e.g. MirrorCache) link to "./name/"
> +        hyperlinks.append(line['href'].removeprefix('./').strip('/'))
>      return hyperlinks
>  
>  def find_latest_numeric_release(url, d):
> @@ -84,12 +85,30 @@ def get_latest_released_fedora_source_package_list(d):
>      package_names |= get_source_package_list_from_url_by_letter("http://archive.fedoraproject.org/pub/fedora/linux/updates/%s/SRPMS/" % latest, "updates", d)
>      return latest, package_names
>  
> +def find_latest_opensuse_leap_release(url, d):
> +    "Find the latest Leap release on the given url, ignoring Leap 42.x, which came before 15.0"
> +    releases = []
> +    for link in get_links_from_url(url, d):
> +        try:
> +            version = tuple(int(n) for n in link.split("."))
> +        except ValueError:
> +            continue
> +        if version[0] != 42:
> +            releases.append((version, link))
> +    return max(releases)[1] if releases else "_NotFound_"
> +
>  def get_latest_released_opensuse_source_package_list(d):
>      "Returns list of all the name os packages in the latest opensuse distro"
> -    latest = find_latest_numeric_release("http://download.opensuse.org/source/distribution/leap", d)
> +    from urllib.error import HTTPError
> +    latest = find_latest_opensuse_leap_release("http://download.opensuse.org/source/distribution/leap/", d)
>  
> -    package_names = get_source_package_list_from_url("http://download.opensuse.org/source/distribution/leap/%s/repo/oss/suse/src/" % latest, "main", d)
> -    package_names |= get_source_package_list_from_url("http://download.opensuse.org/update/leap/%s/oss/src/" % latest, "updates", d)
> +    package_names = get_source_package_list_from_url("http://download.opensuse.org/source/distribution/leap/%s/repo/oss/src/" % latest, "main", d)
> +    # Leap 16 has no separate update repository
> +    try:
> +        package_names |= get_source_package_list_from_url("http://download.opensuse.org/update/leap/%s/oss/src/" % latest, "updates", d)
> +    except HTTPError as e:
> +        if e.code != 404:
> +            raise
>      return latest, package_names
>  
>  def get_latest_released_clear_source_package_list(d):
> diff --git a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
> index 7545fa0c42..7add8d70f9 100644
> --- a/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
> +++ b/meta/lib/oeqa/selftest/cases/oelib/distro_check.py
> @@ -57,6 +57,25 @@ class TestFindLatestUbuntuRelease(TestCase):
>          with mock.patch.object(oe.distro_check, "create_socket", return_value=io.BytesIO(META_RELEASE)):
>              self.assertEqual(oe.distro_check.find_latest_ubuntu_release(None), "resolute")
>  
> +class TestOpenSUSE(TestCase):
> +    LISTING = b"""<a href="../">..</a> <a href="./15.0-Current/">15.0-Current/</a>
> +<a href="./15.6/">15.6/</a> <a href="./16.0/">16.0/</a> <a href="./42.3/">42.3/</a>"""
> +
> +    def test_links_relative_to_current_directory(self):
> +        try:
> +            from bs4 import BeautifulSoup, SoupStrainer

Hi Tim,

This has been merged to master, but I still wonder... Should this use
our vendored copy of bs4 from bitbake (bb._vendor.bs4)?

Best regards,

-- 
Paul Barker


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-20 13:48 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 16:50 [PATCH v2 0/3] distro_check: fix the Ubuntu and openSUSE package lists, drop Clear Tim Orling
2026-11-16 16:50 ` [PATCH v2 1/3] lib/oe/distro_check: find the latest Ubuntu release from meta-release Tim Orling
2026-11-16 16:50 ` [PATCH v2 2/3] lib/oe/distro_check: fix the openSUSE Leap package lists Tim Orling
2026-11-20 13:48   ` Paul Barker
2026-11-16 16:50 ` [PATCH v2 3/3] lib/oe/distro_check: drop Clear Linux Tim Orling

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox