[PATCH 0/9] Let autoreconf run gtkdocize

[PATCH 0/9] Let autoreconf run gtkdocize

[ross.burton]

From: Ross Burton <ross.burton@arm.com>

Hi,

This series is the result of a distraction whilst yak-shaving on autoconf...

Essentially: let autoreconf run gtkdocize, instead of running it by hand. This
needed some small fixes but mostly removes special-cases.

There are some followup patches needed for meta-oe, which I will send when this
is merged.

Ross

Ross Burton (9):
  nativesdk: ensure features don't get backfilled
  glib-2.0: no need to depend on target gtk-doc
  autotools: append to EXTRA_AUTORECONF
  autotools: don't exclude gtkdocize
  gtk-doc: remove obsolete logic
  gtk-doc: don't use docdir set in environment in gtkdocize
  gtk-doc: don't manually call gtkdocize
  kmod: fix configure with autopoint calling gtkdocize
  util-linux: ensure gtkdocize isn't called

 meta/classes-recipe/autotools.bbclass         |  2 +-
 meta/classes-recipe/gtk-doc.bbclass           | 21 ++----------
 meta/classes-recipe/nativesdk.bbclass         |  3 ++
 .../glib-2.0/glib-2.0/native-gtkdoc.patch     | 21 ++++++++++++
 meta/recipes-core/glib-2.0/glib-2.0_2.78.1.bb |  1 +
 meta/recipes-core/glib-2.0/glib.inc           |  2 --
 meta/recipes-core/util-linux/util-linux.inc   |  3 ++
 ...01-Don-t-use-docdir-from-environment.patch | 24 ++++++++++++++
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb  |  1 +
 meta/recipes-kernel/kmod/kmod/gtkdocdir.patch | 33 +++++++++++++++++++
 meta/recipes-kernel/kmod/kmod_31.bb           |  3 +-
 11 files changed, 90 insertions(+), 24 deletions(-)
 create mode 100644 meta/recipes-core/glib-2.0/glib-2.0/native-gtkdoc.patch
 create mode 100644 meta/recipes-gnome/gtk-doc/files/0001-Don-t-use-docdir-from-environment.patch
 create mode 100644 meta/recipes-kernel/kmod/kmod/gtkdocdir.patch

-- 
2.34.1

[PATCH 1/9] nativesdk: ensure features don't get backfilled

[ross.burton]

From: Ross Burton <ross.burton@arm.com>

nativesdk aims to run in a slightly different environment than the
target build, so it unsets MACHINE_FEATURES and lets DISTRO_FEATURES be
filtered by DISTRO_FEATURES_NATIVESDK (in bitbake.conf).

However, feature backfill happens _after_ these operations:

 $ bitbake-getvar -r nativesdk-glib-2.0 MACHINE_FEATURES
 #
 # $MACHINE_FEATURES [5 operations]
 #   set /home/ross/Yocto/poky/meta/conf/machine/include/qemu.inc:14
 #     "alsa bluetooth usbgadget screen vfat"
 #   set /home/ross/Yocto/poky/meta/conf/documentation.conf:284
 #     [doc] "Specifies the list of hardware features the MACHINE supports."
 #   set? /home/ross/Yocto/poky/meta/conf/bitbake.conf:893
 #     ""
 #   set /home/ross/Yocto/poky/meta/classes-recipe/nativesdk.bbclass:18
 #     ""
 #   append utils.py:132 [features_backfill]
 #     " rtc qemu-usermode"
 # pre-expansion value:
 #   " rtc qemu-usermode"
 MACHINE_FEATURES=" rtc qemu-usermode"

I don't believe this is intentional or desired as the machine features
are unrelated to the nativesdk environment.  Specifically, this means
that recipes that look for the qemu-usermode machine feature will fail
in nativesdk as there may not actually be qemu-user support for the
nativesdk environment.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes-recipe/nativesdk.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes-recipe/nativesdk.bbclass b/meta/classes-recipe/nativesdk.bbclass
index 08288fdb733..ccaf411f0d7 100644
--- a/meta/classes-recipe/nativesdk.bbclass
+++ b/meta/classes-recipe/nativesdk.bbclass
@@ -15,7 +15,10 @@ NATIVESDKLIBC ?= "libc-glibc"
 LIBCOVERRIDE = ":${NATIVESDKLIBC}"
 CLASSOVERRIDE = "class-nativesdk"
 MACHINEOVERRIDES = ""
+
 MACHINE_FEATURES = ""
+DISTRO_FEATURES_BACKFILL = ""
+MACHINE_FEATURES_BACKFILL = ""
 
 MULTILIBS = ""
 
-- 
2.34.1

[PATCH 2/9] glib-2.0: no need to depend on target gtk-doc

[ross.burton]

From: Ross Burton <ross.burton@arm.com>

The meson.build looks for a target gtk-doc, but it should be looking for
the native package.

Fixing this means we can drop the need for a target gtk-doc package.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 .../glib-2.0/glib-2.0/native-gtkdoc.patch     | 21 +++++++++++++++++++
 meta/recipes-core/glib-2.0/glib-2.0_2.78.1.bb |  1 +
 meta/recipes-core/glib-2.0/glib.inc           |  2 --
 3 files changed, 22 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-core/glib-2.0/glib-2.0/native-gtkdoc.patch

diff --git a/meta/recipes-core/glib-2.0/glib-2.0/native-gtkdoc.patch b/meta/recipes-core/glib-2.0/glib-2.0/native-gtkdoc.patch
new file mode 100644
index 00000000000..f56c2b85640
--- /dev/null
+++ b/meta/recipes-core/glib-2.0/glib-2.0/native-gtkdoc.patch
@@ -0,0 +1,21 @@
+docs/reference: depend on a native gtk-doc
+
+In a cross-compiled environment we don't need a target gtk-doc, but a
+native gtk-doc.
+
+Upstream-Status: Backport [f3ff370d1b24f8afb51cd4e865593d5b678188a9]
+Signed-off-by: Ross Burton <ross.burton@arm.com>
+
+diff --git a/docs/reference/meson.build b/docs/reference/meson.build
+index c417ffecb..67c1dda56 100644
+--- a/docs/reference/meson.build
++++ b/docs/reference/meson.build
+@@ -42,7 +42,7 @@ ignore_decorators = '|'.join(ignore_decorators)
+ if get_option('gtk_doc')
+   # Check we have the minimum gtk-doc version required. Older versions won't
+   # generate correct documentation.
+-  dependency('gtk-doc', version : '>=1.32.1',
++  dependency('gtk-doc', version : '>=1.32.1', native: true,
+     fallback : ['gtk-doc', 'dummy_dep'],
+     default_options : ['tests=false'])
+ 
diff --git a/meta/recipes-core/glib-2.0/glib-2.0_2.78.1.bb b/meta/recipes-core/glib-2.0/glib-2.0_2.78.1.bb
index a4902621122..3a1dc5ff0c9 100644
--- a/meta/recipes-core/glib-2.0/glib-2.0_2.78.1.bb
+++ b/meta/recipes-core/glib-2.0/glib-2.0_2.78.1.bb
@@ -14,6 +14,7 @@ SRC_URI = "${GNOME_MIRROR}/glib/${SHRT_VER}/glib-${PV}.tar.xz \
            file://0001-Do-not-write-bindir-into-pkg-config-files.patch \
            file://0001-meson-Run-atomics-test-on-clang-as-well.patch \
            file://0001-gio-tests-resources.c-comment-out-a-build-host-only-.patch \
+           file://native-gtkdoc.patch \
            "
 SRC_URI:append:class-native = " file://relocate-modules.patch \ 
                                 file://0001-meson.build-do-not-enable-pidfd-features-on-native-g.patch \
diff --git a/meta/recipes-core/glib-2.0/glib.inc b/meta/recipes-core/glib-2.0/glib.inc
index fbf3a39b15f..a2ce7de20d4 100644
--- a/meta/recipes-core/glib-2.0/glib.inc
+++ b/meta/recipes-core/glib-2.0/glib.inc
@@ -30,8 +30,6 @@ LEAD_SONAME = "libglib-2.0.*"
 
 inherit meson gettext gtk-doc pkgconfig ptest-gnome upstream-version-is-even bash-completion gio-module-cache manpages
 
-DEPENDS:append:class-target = "${@' gtk-doc' if d.getVar('GTKDOC_ENABLED') == 'True' else ''}"
-
 GTKDOC_MESON_OPTION = "gtk_doc"
 
 S = "${WORKDIR}/glib-${PV}"
-- 
2.34.1

[PATCH 3/9] autotools: append to EXTRA_AUTORECONF

[ross.burton]

From: Ross Burton <ross.burton@arm.com>

Inherit order may mean that this class is inherited after assignments,
so extend instead of assign EXTRA_AUTORECONF.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes-recipe/autotools.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes-recipe/autotools.bbclass b/meta/classes-recipe/autotools.bbclass
index 1663307b061..12881a31ec9 100644
--- a/meta/classes-recipe/autotools.bbclass
+++ b/meta/classes-recipe/autotools.bbclass
@@ -37,7 +37,7 @@ inherit siteinfo
 export CONFIG_SITE
 
 acpaths ?= "default"
-EXTRA_AUTORECONF = "--exclude=autopoint --exclude=gtkdocize"
+EXTRA_AUTORECONF += "--exclude=autopoint --exclude=gtkdocize"
 
 export lt_cv_sys_lib_dlsearch_path_spec = "${libdir} ${base_libdir}"
 
-- 
2.34.1

[PATCH 4/9] autotools: don't exclude gtkdocize

[ross.burton]

From: Ross Burton <ross.burton@arm.com>

gtkdocize can now be ran successfully from autoreconf, so there's no
need to exclude it and run it manually.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes-recipe/autotools.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes-recipe/autotools.bbclass b/meta/classes-recipe/autotools.bbclass
index 12881a31ec9..ca76cde0e74 100644
--- a/meta/classes-recipe/autotools.bbclass
+++ b/meta/classes-recipe/autotools.bbclass
@@ -37,7 +37,7 @@ inherit siteinfo
 export CONFIG_SITE
 
 acpaths ?= "default"
-EXTRA_AUTORECONF += "--exclude=autopoint --exclude=gtkdocize"
+EXTRA_AUTORECONF += "--exclude=autopoint"
 
 export lt_cv_sys_lib_dlsearch_path_spec = "${libdir} ${base_libdir}"
 
-- 
2.34.1

[PATCH 5/9] gtk-doc: remove obsolete logic

[ross.burton]

From: Ross Burton <ross.burton@arm.com>

This manual disabling of gtk-doc for autotools and meson in native and
nativesdk builds is replicating logic above for target builds.

Instead, use one assignment for all builds, as we explicitly disable
gtk-doc in native builds and it can be useful in nativesdk.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes-recipe/gtk-doc.bbclass | 11 ++---------
 1 file changed, 2 insertions(+), 9 deletions(-)

diff --git a/meta/classes-recipe/gtk-doc.bbclass b/meta/classes-recipe/gtk-doc.bbclass
index 68fa2cc7451..c980c6bb52d 100644
--- a/meta/classes-recipe/gtk-doc.bbclass
+++ b/meta/classes-recipe/gtk-doc.bbclass
@@ -25,16 +25,9 @@ GTKDOC_MESON_ENABLE_FLAG ?= 'true'
 GTKDOC_MESON_DISABLE_FLAG ?= 'false'
 
 # Auto enable/disable based on GTKDOC_ENABLED
-EXTRA_OECONF:prepend:class-target = "${@bb.utils.contains('GTKDOC_ENABLED', 'True', '--enable-gtk-doc --enable-gtk-doc-html --disable-gtk-doc-pdf', \
+EXTRA_OECONF:prepend = "${@bb.utils.contains('GTKDOC_ENABLED', 'True', '--enable-gtk-doc --enable-gtk-doc-html --disable-gtk-doc-pdf', \
                                                                                     '--disable-gtk-doc', d)} "
-EXTRA_OEMESON:prepend:class-target = "-D${GTKDOC_MESON_OPTION}=${@bb.utils.contains('GTKDOC_ENABLED', 'True', '${GTKDOC_MESON_ENABLE_FLAG}', '${GTKDOC_MESON_DISABLE_FLAG}', d)} "
-
-# When building native recipes, disable gtkdoc, as it is not necessary,
-# pulls in additional dependencies, and makes build times longer
-EXTRA_OECONF:prepend:class-native = "--disable-gtk-doc "
-EXTRA_OECONF:prepend:class-nativesdk = "--disable-gtk-doc "
-EXTRA_OEMESON:prepend:class-native = "-D${GTKDOC_MESON_OPTION}=${GTKDOC_MESON_DISABLE_FLAG} "
-EXTRA_OEMESON:prepend:class-nativesdk = "-D${GTKDOC_MESON_OPTION}=${GTKDOC_MESON_DISABLE_FLAG} "
+EXTRA_OEMESON:prepend = "-D${GTKDOC_MESON_OPTION}=${@bb.utils.contains('GTKDOC_ENABLED', 'True', '${GTKDOC_MESON_ENABLE_FLAG}', '${GTKDOC_MESON_DISABLE_FLAG}', d)} "
 
 # Even though gtkdoc is disabled on -native, gtk-doc package is still
 # needed for m4 macros.
-- 
2.34.1

[PATCH 6/9] gtk-doc: don't use docdir set in environment in gtkdocize

[ross.burton]

From: Ross Burton <ross.burton@arm.com>

The gtkdocize script was accidentally respecting $docdir from the
environment as the default documentation directory. This is a problem as
bitbake.conf exports $docdir, resulting in configure failures.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 ...01-Don-t-use-docdir-from-environment.patch | 24 +++++++++++++++++++
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb  |  1 +
 2 files changed, 25 insertions(+)
 create mode 100644 meta/recipes-gnome/gtk-doc/files/0001-Don-t-use-docdir-from-environment.patch

diff --git a/meta/recipes-gnome/gtk-doc/files/0001-Don-t-use-docdir-from-environment.patch b/meta/recipes-gnome/gtk-doc/files/0001-Don-t-use-docdir-from-environment.patch
new file mode 100644
index 00000000000..f40124877c3
--- /dev/null
+++ b/meta/recipes-gnome/gtk-doc/files/0001-Don-t-use-docdir-from-environment.patch
@@ -0,0 +1,24 @@
+From 72dfeec0e49478b0bfb471c4155044391bad8e6c Mon Sep 17 00:00:00 2001
+From: Ross Burton <ross.burton@arm.com>
+Date: Fri, 8 Dec 2023 10:35:25 +0000
+Subject: [PATCH] Don't use docdir from environment
+
+Upstream-Status: Submitted [https://gitlab.gnome.org/GNOME/gtk-doc/-/merge_requests/73]
+Signed-off-by: Ross Burton <ross.burton@arm.com>
+---
+ buildsystems/autotools/gtkdocize.in | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/buildsystems/autotools/gtkdocize.in b/buildsystems/autotools/gtkdocize.in
+index 83127bf..76dcbfd 100755
+--- a/buildsystems/autotools/gtkdocize.in
++++ b/buildsystems/autotools/gtkdocize.in
+@@ -39,7 +39,7 @@ set - $args
+ 
+ # assume working directory if srcdir is not set
+ test "$srcdir" || srcdir=.
+-test "$docdir" || docdir="$srcdir"
++docdir="$srcdir"
+ 
+ # detect configure script
+ no_configure_found=0
diff --git a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
index 2e8ab1c771a..7c366737da5 100644
--- a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
+++ b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
@@ -26,6 +26,7 @@ SRC_URI += "file://0001-Do-not-hardocode-paths-to-perl-python-in-scripts.patch \
            file://0001-Do-not-error-out-if-xsltproc-is-not-found.patch \
            file://conditionaltests.patch \
            file://no-clobber.patch \
+           file://0001-Don-t-use-docdir-from-environment.patch \
            "
 SRC_URI:append:class-native = " file://pkg-config-native.patch"
 
-- 
2.34.1

[PATCH 7/9] gtk-doc: don't manually call gtkdocize

[ross.burton]

From: Ross Burton <ross.burton@arm.com>

The autoreconf call will now call gtkdocize if needed, so we don't need
to run it manually here.

This obsoletes GTKDOC_DOCDIR.  If this is needed then the replacement is
to ensure that the configure.ac has a call to GTK_DOC_CHECK which passes
--docdir.

For example, this is the change required for kmod:

-GTK_DOC_CHECK([1.14],[--flavour no-tmpl-flat])
+GTK_DOC_CHECK([1.14],[--flavour no-tmpl-flat --docdir libkmod/docs])

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes-recipe/gtk-doc.bbclass | 10 ----------
 1 file changed, 10 deletions(-)

diff --git a/meta/classes-recipe/gtk-doc.bbclass b/meta/classes-recipe/gtk-doc.bbclass
index c980c6bb52d..9d3911966bb 100644
--- a/meta/classes-recipe/gtk-doc.bbclass
+++ b/meta/classes-recipe/gtk-doc.bbclass
@@ -33,21 +33,11 @@ EXTRA_OEMESON:prepend = "-D${GTKDOC_MESON_OPTION}=${@bb.utils.contains('GTKDOC_E
 # needed for m4 macros.
 DEPENDS:append = " gtk-doc-native"
 
-# The documentation directory, where the infrastructure will be copied.
-# gtkdocize has a default of "." so to handle out-of-tree builds set this to $S.
-GTKDOC_DOCDIR ?= "${S}"
-
 export STAGING_DIR_HOST
 
 inherit python3native pkgconfig qemu
 DEPENDS:append = "${@' qemu-native' if d.getVar('GTKDOC_ENABLED') == 'True' else ''}"
 
-do_configure:prepend () {
-	# Need to use ||true as this is only needed if configure.ac both exists
-	# and uses GTK_DOC_CHECK.
-	gtkdocize --srcdir ${S} --docdir ${GTKDOC_DOCDIR} || true
-}
-
 do_compile:prepend:class-target () {
     if [ ${GTKDOC_ENABLED} = True ]; then
         # Write out a qemu wrapper that will be given to gtkdoc-scangobj so that it
-- 
2.34.1

[PATCH 8/9] kmod: fix configure with autopoint calling gtkdocize

[ross.burton]

From: Ross Burton <ross.burton@arm.com>

GTKDOC_DOCDIR is obsolete now, so fix the build by passing the
documentation directory in the GTK_DOC_CHECK call.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/recipes-kernel/kmod/kmod/gtkdocdir.patch | 33 +++++++++++++++++++
 meta/recipes-kernel/kmod/kmod_31.bb           |  3 +-
 2 files changed, 34 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-kernel/kmod/kmod/gtkdocdir.patch

diff --git a/meta/recipes-kernel/kmod/kmod/gtkdocdir.patch b/meta/recipes-kernel/kmod/kmod/gtkdocdir.patch
new file mode 100644
index 00000000000..a34ea466e8e
--- /dev/null
+++ b/meta/recipes-kernel/kmod/kmod/gtkdocdir.patch
@@ -0,0 +1,33 @@
+From dd59095f70f774f6d1e767010e25b35ef6db4c4b Mon Sep 17 00:00:00 2001
+From: Ross Burton <ross.burton@arm.com>
+Date: Fri, 8 Dec 2023 22:35:45 +0000
+Subject: [PATCH] configure: set docdir in GTK_DOC_CHECK
+
+By passing --docdir in the GTK_DOC_CHECK arguments (to match
+autogen.sh) autoreconf will work out of the box.
+
+Without this autoreconf fails due to the documentation not being in
+./docs, the default location.
+
+Upstream-Status: Submitted [https://lore.kernel.org/linux-modules/20231208224511.1363066-1-ross.burton@arm.com/T/#u]
+Signed-off-by: Ross Burton <ross.burton@arm.com>
+---
+ configure.ac | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/configure.ac b/configure.ac
+index de01e08..67696c4 100644
+--- a/configure.ac
++++ b/configure.ac
+@@ -255,7 +255,7 @@ AS_IF([test "x$enable_coverage" = "xyes"], [
+ AM_CONDITIONAL([ENABLE_COVERAGE], [test "x$enable_coverage" = "xyes"])
+ 
+ m4_ifdef([GTK_DOC_CHECK], [
+-GTK_DOC_CHECK([1.14],[--flavour no-tmpl-flat])
++GTK_DOC_CHECK([1.14],[--flavour no-tmpl-flat --docdir libkmod/docs])
+ ], [
+ AM_CONDITIONAL([ENABLE_GTK_DOC], false)])
+ 
+-- 
+2.34.1
+
diff --git a/meta/recipes-kernel/kmod/kmod_31.bb b/meta/recipes-kernel/kmod/kmod_31.bb
index 934a678a062..f8e83d5db63 100644
--- a/meta/recipes-kernel/kmod/kmod_31.bb
+++ b/meta/recipes-kernel/kmod/kmod_31.bb
@@ -20,6 +20,7 @@ SRCREV = "aff617ea871d0568cc491bd116c0be1e857463bb"
 SRC_URI = "git://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git;branch=master;protocol=https \
            file://depmod-search.conf \
            file://avoid_parallel_tests.patch \
+           file://gtkdocdir.patch \
            "
 
 S = "${WORKDIR}/git"
@@ -35,8 +36,6 @@ PACKAGECONFIG[xz] = "--with-xz,--without-xz,xz"
 PACKAGECONFIG[zlib] = "--with-zlib,--without-zlib,zlib"
 PACKAGECONFIG[zstd] = "--with-zstd,--without-zstd,zstd"
 
-GTKDOC_DOCDIR = "${S}/libkmod/docs"
-
 PROVIDES += "module-init-tools-insmod-static module-init-tools-depmod module-init-tools"
 RPROVIDES:${PN} += "module-init-tools-insmod-static module-init-tools-depmod module-init-tools"
 RCONFLICTS:${PN} += "module-init-tools-insmod-static module-init-tools-depmod module-init-tools"
-- 
2.34.1

[PATCH 9/9] util-linux: ensure gtkdocize isn't called

[ross.burton]

From: Ross Burton <ross.burton@arm.com>

autoreconf will see that the configure.ac uses gtk-doc and will call
gtkdocize, but we can't actually build the documentation due to
dependency loops so forcibly tell autoreconf not to do this.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/recipes-core/util-linux/util-linux.inc | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-core/util-linux/util-linux.inc b/meta/recipes-core/util-linux/util-linux.inc
index 952a680a849..5592c14b732 100644
--- a/meta/recipes-core/util-linux/util-linux.inc
+++ b/meta/recipes-core/util-linux/util-linux.inc
@@ -39,3 +39,6 @@ SRC_URI = "${KERNELORG_MIRROR}/linux/utils/util-linux/v${MAJOR_VERSION}/util-lin
            "
 
 SRC_URI[sha256sum] = "87abdfaa8e490f8be6dde976f7c80b9b5ff9f301e1b67e3899e1f05a59a1531f"
+
+# gtk-doc is not enabled as it requires xmlto which requires util-linux
+EXTRA_AUTORECONF += "--exclude=gtkdocize"
-- 
2.34.1

Re: [OE-core] [PATCH 1/9] nativesdk: ensure features don't get backfilled

[Alexandre Belloni]

Hello Ross,

Not obvious but this is the cause of:

https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/8697

On 12/12/2023 14:08:12+0000, Ross Burton wrote:
> From: Ross Burton <ross.burton@arm.com>
> 
> nativesdk aims to run in a slightly different environment than the
> target build, so it unsets MACHINE_FEATURES and lets DISTRO_FEATURES be
> filtered by DISTRO_FEATURES_NATIVESDK (in bitbake.conf).
> 
> However, feature backfill happens _after_ these operations:
> 
>  $ bitbake-getvar -r nativesdk-glib-2.0 MACHINE_FEATURES
>  #
>  # $MACHINE_FEATURES [5 operations]
>  #   set /home/ross/Yocto/poky/meta/conf/machine/include/qemu.inc:14
>  #     "alsa bluetooth usbgadget screen vfat"
>  #   set /home/ross/Yocto/poky/meta/conf/documentation.conf:284
>  #     [doc] "Specifies the list of hardware features the MACHINE supports."
>  #   set? /home/ross/Yocto/poky/meta/conf/bitbake.conf:893
>  #     ""
>  #   set /home/ross/Yocto/poky/meta/classes-recipe/nativesdk.bbclass:18
>  #     ""
>  #   append utils.py:132 [features_backfill]
>  #     " rtc qemu-usermode"
>  # pre-expansion value:
>  #   " rtc qemu-usermode"
>  MACHINE_FEATURES=" rtc qemu-usermode"
> 
> I don't believe this is intentional or desired as the machine features
> are unrelated to the nativesdk environment.  Specifically, this means
> that recipes that look for the qemu-usermode machine feature will fail
> in nativesdk as there may not actually be qemu-user support for the
> nativesdk environment.
> 
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>  meta/classes-recipe/nativesdk.bbclass | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/meta/classes-recipe/nativesdk.bbclass b/meta/classes-recipe/nativesdk.bbclass
> index 08288fdb733..ccaf411f0d7 100644
> --- a/meta/classes-recipe/nativesdk.bbclass
> +++ b/meta/classes-recipe/nativesdk.bbclass
> @@ -15,7 +15,10 @@ NATIVESDKLIBC ?= "libc-glibc"
>  LIBCOVERRIDE = ":${NATIVESDKLIBC}"
>  CLASSOVERRIDE = "class-nativesdk"
>  MACHINEOVERRIDES = ""
> +
>  MACHINE_FEATURES = ""
> +DISTRO_FEATURES_BACKFILL = ""
> +MACHINE_FEATURES_BACKFILL = ""
>  
>  MULTILIBS = ""
>  
> -- 
> 2.34.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#192193): https://lists.openembedded.org/g/openembedded-core/message/192193
> Mute This Topic: https://lists.openembedded.org/mt/103129922/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH 1/9] nativesdk: ensure features don't get backfilled

[Ross Burton]

On 14 Dec 2023, at 07:49, Alexandre Belloni <alexandre.belloni@bootlin.com> wrote:
> Not obvious but this is the cause of:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/8697

That was a fun rabbithole to dig.

The error from the log is:

ERROR: /home/pokybuild/yocto-worker/buildtools/build/meta/recipes-devtools/python/python3_3.11.5.bb: pgo cannot be enabled as there is no qemu-usermode support for this architecture/machine

Which was intentional fallout: MACHINE_FEATURES don’t describe the SDK so that this ever worked was more luck than judgement.  Also the python3 recipe will write the qemu-calling wrapper script using the architecture only for target builds, so we’ve not had PGO nativesdk python since the 3.7.2 upgrade in 2019[1] which changed this from all builds to target only.

Maybe we should allow machine-sdk configurations to extend SDK_MACHINE_FEATURES and then nativesdk can use that?  This might allow the MACHINE_FEATURES to “work” without being based on the target machine.

Alternatively, I can remove the pgo from the buildtools build, as it doesn’t actually do anything right now anyway.

Ross

[1] oe-core 02714c105426b0d687620913c1a7401b386428b6

Re: [OE-core] [PATCH 1/9] nativesdk: ensure features don't get backfilled

[Richard Purdie]

On Thu, 2023-12-14 at 13:31 +0000, Ross Burton wrote:
> On 14 Dec 2023, at 07:49, Alexandre Belloni
> <alexandre.belloni@bootlin.com> wrote:
> > Not obvious but this is the cause of:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/8697
> 
> That was a fun rabbithole to dig.
> 
> The error from the log is:
> 
> ERROR: /home/pokybuild/yocto-worker/buildtools/build/meta/recipes-
> devtools/python/python3_3.11.5.bb: pgo cannot be enabled as there is
> no qemu-usermode support for this architecture/machine
> 
> Which was intentional fallout: MACHINE_FEATURES don’t describe the
> SDK so that this ever worked was more luck than judgement.  Also the
> python3 recipe will write the qemu-calling wrapper script using the
> architecture only for target builds, so we’ve not had PGO nativesdk
> python since the 3.7.2 upgrade in 2019[1] which changed this from all
> builds to target only.
> 
> Maybe we should allow machine-sdk configurations to extend
> SDK_MACHINE_FEATURES and then nativesdk can use that?  This might
> allow the MACHINE_FEATURES to “work” without being based on the
> target machine.

Sadly I suspect we're going to have to go down this route eventually in
some form.

> Alternatively, I can remove the pgo from the buildtools build, as it
> doesn’t actually do anything right now anyway.

It was a decent win at one point so that would seem sad.

Cheers,

Richard

Re: [OE-core] [PATCH 1/9] nativesdk: ensure features don't get backfilled

[Alexander Kanavin]

FWIW in the upcoming python 3.12 update I have removed the pgo patch
(0001-Makefile.pre-use-qemu-wrapper-when-gathering-profile.patch) with
the rationale that "pgo support has been rewritten upstream and needs
to be reenabled and tested separately". I simply don't have bandwidth
to make it work, with a ton of recipes in meta-oe that still need
fixing with 3.12.

Alex

On Thu, 14 Dec 2023 at 14:35, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Thu, 2023-12-14 at 13:31 +0000, Ross Burton wrote:
> > On 14 Dec 2023, at 07:49, Alexandre Belloni
> > <alexandre.belloni@bootlin.com> wrote:
> > > Not obvious but this is the cause of:
> > >
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/8697
> >
> > That was a fun rabbithole to dig.
> >
> > The error from the log is:
> >
> > ERROR: /home/pokybuild/yocto-worker/buildtools/build/meta/recipes-
> > devtools/python/python3_3.11.5.bb: pgo cannot be enabled as there is
> > no qemu-usermode support for this architecture/machine
> >
> > Which was intentional fallout: MACHINE_FEATURES don’t describe the
> > SDK so that this ever worked was more luck than judgement.  Also the
> > python3 recipe will write the qemu-calling wrapper script using the
> > architecture only for target builds, so we’ve not had PGO nativesdk
> > python since the 3.7.2 upgrade in 2019[1] which changed this from all
> > builds to target only.
> >
> > Maybe we should allow machine-sdk configurations to extend
> > SDK_MACHINE_FEATURES and then nativesdk can use that?  This might
> > allow the MACHINE_FEATURES to “work” without being based on the
> > target machine.
>
> Sadly I suspect we're going to have to go down this route eventually in
> some form.
>
> > Alternatively, I can remove the pgo from the buildtools build, as it
> > doesn’t actually do anything right now anyway.
>
> It was a decent win at one point so that would seem sad.
>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#192370): https://lists.openembedded.org/g/openembedded-core/message/192370
> Mute This Topic: https://lists.openembedded.org/mt/103129922/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH 1/9] nativesdk: ensure features don't get backfilled

[Alexander Kanavin]

Correction: they didn't rewrite it, they just moved around the lines:
https://github.com/python/cpython/commit/9084e1b04f8d35721b535a3acd550b920d50f11a
and I didn't look thoroughly enough for the new location.

Neverthless, given that a-full still passed with the patch removed
means pgo can be completely broken or completely working, and we would
have no idea. I lean towards 'completely broken' but I'll reinstate
the patch.

Alex


On Thu, 14 Dec 2023 at 14:49, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
>
> FWIW in the upcoming python 3.12 update I have removed the pgo patch
> (0001-Makefile.pre-use-qemu-wrapper-when-gathering-profile.patch) with
> the rationale that "pgo support has been rewritten upstream and needs
> to be reenabled and tested separately". I simply don't have bandwidth
> to make it work, with a ton of recipes in meta-oe that still need
> fixing with 3.12.
>
> Alex
>
> On Thu, 14 Dec 2023 at 14:35, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Thu, 2023-12-14 at 13:31 +0000, Ross Burton wrote:
> > > On 14 Dec 2023, at 07:49, Alexandre Belloni
> > > <alexandre.belloni@bootlin.com> wrote:
> > > > Not obvious but this is the cause of:
> > > >
> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/8697
> > >
> > > That was a fun rabbithole to dig.
> > >
> > > The error from the log is:
> > >
> > > ERROR: /home/pokybuild/yocto-worker/buildtools/build/meta/recipes-
> > > devtools/python/python3_3.11.5.bb: pgo cannot be enabled as there is
> > > no qemu-usermode support for this architecture/machine
> > >
> > > Which was intentional fallout: MACHINE_FEATURES don’t describe the
> > > SDK so that this ever worked was more luck than judgement.  Also the
> > > python3 recipe will write the qemu-calling wrapper script using the
> > > architecture only for target builds, so we’ve not had PGO nativesdk
> > > python since the 3.7.2 upgrade in 2019[1] which changed this from all
> > > builds to target only.
> > >
> > > Maybe we should allow machine-sdk configurations to extend
> > > SDK_MACHINE_FEATURES and then nativesdk can use that?  This might
> > > allow the MACHINE_FEATURES to “work” without being based on the
> > > target machine.
> >
> > Sadly I suspect we're going to have to go down this route eventually in
> > some form.
> >
> > > Alternatively, I can remove the pgo from the buildtools build, as it
> > > doesn’t actually do anything right now anyway.
> >
> > It was a decent win at one point so that would seem sad.
> >
> > Cheers,
> >
> > Richard
> >
> >
> >
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#192380): https://lists.openembedded.org/g/openembedded-core/message/192380
> Mute This Topic: https://lists.openembedded.org/mt/103129922/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>