[PATCH 1/4] populate_sdk_ext.bbclass: do not symlink unfsd from sdk image sysroot into eSDK tools path

[PATCH 1/4] populate_sdk_ext.bbclass: do not symlink unfsd from sdk image sysroot into eSDK tools path

[Alexander Kanavin]

This was done in 2016 to support Eclipse plugin (long dead),
it's currently broken as image sysroot is not in the SDK
until the image is built in that context, and current tools
all rely on runqemu-export-rootfs which does not rely on PATH
and runs unfsd with full path to recipe-specific sysroots.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/classes-recipe/populate_sdk_ext.bbclass | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass
index 028d0c1ff54..173b3065b00 100644
--- a/meta/classes-recipe/populate_sdk_ext.bbclass
+++ b/meta/classes-recipe/populate_sdk_ext.bbclass
@@ -635,13 +635,6 @@ install_tools() {
 			test -e ${targetscriptfn} || ln -rs ${scriptfn} ${targetscriptfn}
 		done
 	done
-	# We can't use the same method as above because files in the sysroot won't exist at this point
-	# (they get populated from sstate on installation)
-	unfsd_path="${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/unfsd"
-	if [ "${SDK_INCLUDE_TOOLCHAIN}" = "1" -a ! -e $unfsd_path ] ; then
-		binrelpath=${@os.path.relpath(d.getVar('STAGING_BINDIR_NATIVE'), d.getVar('TMPDIR'))}
-		ln -rs ${SDK_OUTPUT}/${SDKPATH}/tmp/$binrelpath/unfsd $unfsd_path
-	fi
 	touch ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase
 
 	# find latest buildtools-tarball and install it
-- 
2.39.2

[PATCH 2/4] meta/lib/oe/copy_buildsystem.py: do not derefence symlinks

[Alexander Kanavin]

This was added (I think) for the purpose of supporting layers that refer
to items outside of the layer via relative symlinks:
https://git.yoctoproject.org/poky-contrib/commit/?id=d31d1ad4e566e42d0bbcf1f41ac25e33181fb517

I do not think copying the link target into the layer that references it is the
correct solution: rather the original target should be included
into the SDK with the same relative path.

This change is done for the sake of preserving symlinks
that are referencing things inside the layer as they are;
particularly the content of scripts/esdk-tools/.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/lib/oe/copy_buildsystem.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index a0d829054e1..81abfbf9e23 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -22,7 +22,7 @@ def _smart_copy(src, dest):
     mode = os.stat(src).st_mode
     if stat.S_ISDIR(mode):
         bb.utils.mkdirhier(dest)
-        cmd = "tar --exclude='.git' --exclude='__pycache__' --xattrs --xattrs-include='*' -chf - -C %s -p . \
+        cmd = "tar --exclude='.git' --exclude='__pycache__' --xattrs --xattrs-include='*' -cf - -C %s -p . \
         | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dest)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
     else:
-- 
2.39.2

[PATCH 3/4] scripts/esdk-tools: use a dedicated, static directory for esdk tools

[Alexander Kanavin]

This allows easier replication of esdk environment (which provides
a curated, limited set of tools that for example does not include bitbake)
in a standard yocto build. Switchover between various sets can be achieved
via PATH manipulation.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/classes-recipe/populate_sdk_ext.bbclass | 16 ++++------------
 scripts/esdk-tools/devtool                   |  1 +
 scripts/esdk-tools/oe-find-native-sysroot    |  1 +
 scripts/esdk-tools/recipetool                |  1 +
 scripts/esdk-tools/runqemu                   |  1 +
 scripts/esdk-tools/runqemu-addptable2image   |  1 +
 scripts/esdk-tools/runqemu-export-rootfs     |  1 +
 scripts/esdk-tools/runqemu-extract-sdk       |  1 +
 scripts/esdk-tools/runqemu-gen-tapdevs       |  1 +
 scripts/esdk-tools/runqemu-ifdown            |  1 +
 scripts/esdk-tools/runqemu-ifup              |  1 +
 scripts/esdk-tools/wic                       |  1 +
 12 files changed, 15 insertions(+), 12 deletions(-)
 create mode 120000 scripts/esdk-tools/devtool
 create mode 120000 scripts/esdk-tools/oe-find-native-sysroot
 create mode 120000 scripts/esdk-tools/recipetool
 create mode 120000 scripts/esdk-tools/runqemu
 create mode 120000 scripts/esdk-tools/runqemu-addptable2image
 create mode 120000 scripts/esdk-tools/runqemu-export-rootfs
 create mode 120000 scripts/esdk-tools/runqemu-extract-sdk
 create mode 120000 scripts/esdk-tools/runqemu-gen-tapdevs
 create mode 120000 scripts/esdk-tools/runqemu-ifdown
 create mode 120000 scripts/esdk-tools/runqemu-ifup
 create mode 120000 scripts/esdk-tools/wic

diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass
index 173b3065b00..53adc868d47 100644
--- a/meta/classes-recipe/populate_sdk_ext.bbclass
+++ b/meta/classes-recipe/populate_sdk_ext.bbclass
@@ -220,16 +220,16 @@ python copy_buildsystem () {
         if os.path.exists(os.path.join(baseoutpath, relpath)):
             conf_initpath = relpath
 
-        relpath = os.path.join('layers', path, 'scripts', 'devtool')
+        relpath = os.path.join('layers', path, 'scripts', 'esdk-tools', 'devtool')
         if os.path.exists(os.path.join(baseoutpath, relpath)):
-            scriptrelpath = os.path.dirname(relpath)
+            esdk_tools_path = os.path.dirname(relpath)
 
         relpath = os.path.join('layers', path, 'meta')
         if os.path.exists(os.path.join(baseoutpath, relpath, 'lib', 'oe')):
             core_meta_subdir = relpath
 
     d.setVar('oe_init_build_env_path', conf_initpath)
-    d.setVar('scriptrelpath', scriptrelpath)
+    d.setVar('esdk_tools_path', esdk_tools_path)
 
     # Write out config file for devtool
     import configparser
@@ -627,14 +627,6 @@ def get_sdk_required_utilities(buildtools_fn, d):
     return ' '.join(sanity_required_utilities)
 
 install_tools() {
-	install -d ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}
-	scripts="devtool recipetool oe-find-native-sysroot runqemu* wic"
-	for script in $scripts; do
-		for scriptfn in `find ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath} -maxdepth 1 -executable -name "$script"`; do
-			targetscriptfn="${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/$(basename $scriptfn)"
-			test -e ${targetscriptfn} || ln -rs ${scriptfn} ${targetscriptfn}
-		done
-	done
 	touch ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase
 
 	# find latest buildtools-tarball and install it
@@ -713,7 +705,7 @@ sdk_ext_postinst() {
 
 	# A bit of another hack, but we need this in the path only for devtool
 	# so put it at the end of $PATH.
-	echo "export PATH=\"$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH\"" >> $env_setup_script
+	echo "export PATH=\"$target_sdk_dir/${esdk_tools_path}:\$PATH\"" >> $env_setup_script
 
 	echo "printf 'SDK environment now set up; additionally you may now run devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $env_setup_script
 
diff --git a/scripts/esdk-tools/devtool b/scripts/esdk-tools/devtool
new file mode 120000
index 00000000000..176a01ca682
--- /dev/null
+++ b/scripts/esdk-tools/devtool
@@ -0,0 +1 @@
+../devtool
\ No newline at end of file
diff --git a/scripts/esdk-tools/oe-find-native-sysroot b/scripts/esdk-tools/oe-find-native-sysroot
new file mode 120000
index 00000000000..d3493f3310a
--- /dev/null
+++ b/scripts/esdk-tools/oe-find-native-sysroot
@@ -0,0 +1 @@
+../oe-find-native-sysroot
\ No newline at end of file
diff --git a/scripts/esdk-tools/recipetool b/scripts/esdk-tools/recipetool
new file mode 120000
index 00000000000..60a95dd9360
--- /dev/null
+++ b/scripts/esdk-tools/recipetool
@@ -0,0 +1 @@
+../recipetool
\ No newline at end of file
diff --git a/scripts/esdk-tools/runqemu b/scripts/esdk-tools/runqemu
new file mode 120000
index 00000000000..ae7e7ad7c22
--- /dev/null
+++ b/scripts/esdk-tools/runqemu
@@ -0,0 +1 @@
+../runqemu
\ No newline at end of file
diff --git a/scripts/esdk-tools/runqemu-addptable2image b/scripts/esdk-tools/runqemu-addptable2image
new file mode 120000
index 00000000000..afcd00e79d9
--- /dev/null
+++ b/scripts/esdk-tools/runqemu-addptable2image
@@ -0,0 +1 @@
+../runqemu-addptable2image
\ No newline at end of file
diff --git a/scripts/esdk-tools/runqemu-export-rootfs b/scripts/esdk-tools/runqemu-export-rootfs
new file mode 120000
index 00000000000..a26fcf6110f
--- /dev/null
+++ b/scripts/esdk-tools/runqemu-export-rootfs
@@ -0,0 +1 @@
+../runqemu-export-rootfs
\ No newline at end of file
diff --git a/scripts/esdk-tools/runqemu-extract-sdk b/scripts/esdk-tools/runqemu-extract-sdk
new file mode 120000
index 00000000000..cc858aaad55
--- /dev/null
+++ b/scripts/esdk-tools/runqemu-extract-sdk
@@ -0,0 +1 @@
+../runqemu-extract-sdk
\ No newline at end of file
diff --git a/scripts/esdk-tools/runqemu-gen-tapdevs b/scripts/esdk-tools/runqemu-gen-tapdevs
new file mode 120000
index 00000000000..dbdf79134cf
--- /dev/null
+++ b/scripts/esdk-tools/runqemu-gen-tapdevs
@@ -0,0 +1 @@
+../runqemu-gen-tapdevs
\ No newline at end of file
diff --git a/scripts/esdk-tools/runqemu-ifdown b/scripts/esdk-tools/runqemu-ifdown
new file mode 120000
index 00000000000..0097693ca3d
--- /dev/null
+++ b/scripts/esdk-tools/runqemu-ifdown
@@ -0,0 +1 @@
+../runqemu-ifdown
\ No newline at end of file
diff --git a/scripts/esdk-tools/runqemu-ifup b/scripts/esdk-tools/runqemu-ifup
new file mode 120000
index 00000000000..41026d2c0a4
--- /dev/null
+++ b/scripts/esdk-tools/runqemu-ifup
@@ -0,0 +1 @@
+../runqemu-ifup
\ No newline at end of file
diff --git a/scripts/esdk-tools/wic b/scripts/esdk-tools/wic
new file mode 120000
index 00000000000..a9d908aa254
--- /dev/null
+++ b/scripts/esdk-tools/wic
@@ -0,0 +1 @@
+../wic
\ No newline at end of file
-- 
2.39.2

[PATCH 4/4] populate_sdk_ext: split copy_buildsystem() into logical steps defined as functions

[Alexander Kanavin]

copy_buildsystem() has become far too large and needs to be split
into shorter and more understandable pieces; some of those
pieces will be reused for the purpose of providing esdk mode
directly from a plain yocto build.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/classes-recipe/populate_sdk_ext.bbclass | 59 ++++++++++++++++----
 1 file changed, 47 insertions(+), 12 deletions(-)

diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass
index 53adc868d47..a20f38fb210 100644
--- a/meta/classes-recipe/populate_sdk_ext.bbclass
+++ b/meta/classes-recipe/populate_sdk_ext.bbclass
@@ -186,12 +186,7 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
         if os.path.exists(localconf + '.bak'):
             os.replace(localconf + '.bak', localconf)
 
-python copy_buildsystem () {
-    import re
-    import shutil
-    import glob
-    import oe.copy_buildsystem
-
+def copy_bitbake_and_layers(d, baseoutpath, derivative):
     oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT')
 
     conf_bbpath = ''
@@ -200,13 +195,7 @@ python copy_buildsystem () {
 
     # Copy in all metadata layers + bitbake (as repositories)
     buildsystem = oe.copy_buildsystem.BuildSystem('extensible SDK', d)
-    baseoutpath = d.getVar('SDK_OUTPUT') + '/' + d.getVar('SDKPATH')
 
-    #check if custome templateconf path is set
-    use_custom_templateconf = d.getVar('SDK_CUSTOM_TEMPLATECONF')
-
-    # Determine if we're building a derivative extensible SDK (from devtool build-sdk)
-    derivative = (d.getVar('SDK_DERIVATIVE') or '') == '1'
     if derivative:
         workspace_name = 'orig-workspace'
     else:
@@ -231,6 +220,9 @@ python copy_buildsystem () {
     d.setVar('oe_init_build_env_path', conf_initpath)
     d.setVar('esdk_tools_path', esdk_tools_path)
 
+    return (conf_initpath, conf_bbpath, core_meta_subdir, sdkbblayers)
+
+def write_devtool_config(d, baseoutpath, conf_bbpath, conf_initpath, core_meta_subdir):
     # Write out config file for devtool
     import configparser
     config = configparser.ConfigParser()
@@ -247,10 +239,12 @@ python copy_buildsystem () {
     with open(os.path.join(baseoutpath, 'conf', 'devtool.conf'), 'w') as f:
         config.write(f)
 
+def write_unlocked_sigs(d, baseoutpath):
     unlockedsigs =  os.path.join(baseoutpath, 'conf', 'unlocked-sigs.inc')
     with open(unlockedsigs, 'w') as f:
         pass
 
+def write_bblayers_conf(d, baseoutpath, sdkbblayers):
     # Create a layer for new recipes / appends
     bbpath = d.getVar('BBPATH')
     env = os.environ.copy()
@@ -279,6 +273,9 @@ python copy_buildsystem () {
         f.write('    $' + '{SDKBASEMETAPATH}/workspace \\\n')
         f.write('    "\n')
 
+def copy_uninative(d, baseoutpath):
+    import shutil
+
     # Copy uninative tarball
     # For now this is where uninative.bbclass expects the tarball
     if bb.data.inherits_class('uninative', d):
@@ -288,6 +285,12 @@ python copy_buildsystem () {
         bb.utils.mkdirhier(uninative_outdir)
         shutil.copy(uninative_file, uninative_outdir)
 
+    return uninative_checksum
+
+def write_local_conf(d, baseoutpath, derivative, core_meta_subdir, uninative_checksum):
+    #check if custome templateconf path is set
+    use_custom_templateconf = d.getVar('SDK_CUSTOM_TEMPLATECONF')
+
     env_passthrough = (d.getVar('BB_ENV_PASSTHROUGH_ADDITIONS') or '').split()
     env_passthrough_values = {}
 
@@ -457,6 +460,9 @@ python copy_buildsystem () {
                 f.write(line)
             f.write('\n')
 
+def prepare_locked_cache(d, baseoutpath, conf_initpath):
+    import shutil
+
     # Filter the locked signatures file to just the sstate tasks we are interested in
     excluded_targets = get_sdk_install_targets(d, images_only=True)
     sigfile = d.getVar('WORKDIR') + '/locked-sigs.inc'
@@ -560,6 +566,9 @@ python copy_buildsystem () {
                 f = os.path.join(root, name)
                 os.remove(f)
 
+def write_manifest(d, baseoutpath):
+    import glob
+
     # Write manifest file
     # Note: at the moment we cannot include the env setup script here to keep
     # it updated, since it gets modified during SDK installation (see
@@ -583,6 +592,32 @@ python copy_buildsystem () {
                     continue
                 chksum = bb.utils.sha256_file(fn)
                 f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
+
+
+python copy_buildsystem () {
+    import oe.copy_buildsystem
+
+    baseoutpath = d.getVar('SDK_OUTPUT') + '/' + d.getVar('SDKPATH')
+
+    # Determine if we're building a derivative extensible SDK (from devtool build-sdk)
+    derivative = (d.getVar('SDK_DERIVATIVE') or '') == '1'
+
+    conf_initpath, conf_bbpath, core_meta_subdir, sdkbblayers = copy_bitbake_and_layers(d, baseoutpath, derivative)
+
+    write_devtool_config(d, baseoutpath, conf_bbpath, conf_initpath, core_meta_subdir)
+
+    write_unlocked_sigs(d, baseoutpath)
+
+    write_bblayers_conf(d, baseoutpath, sdkbblayers)
+
+    uninative_checksum = copy_uninative(d, baseoutpath)
+
+    write_local_conf(d, baseoutpath, derivative, core_meta_subdir, uninative_checksum)
+
+    prepare_locked_cache(d, baseoutpath, conf_initpath)
+
+    write_manifest(d, baseoutpath)
+
 }
 
 def get_current_buildtools(d):
-- 
2.39.2

Re: [OE-core] [PATCH 4/4] populate_sdk_ext: split copy_buildsystem() into logical steps defined as functions

[Alexandre Belloni]

https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/7419/steps/16/logs/stdio

On 03/11/2023 11:28:06+0100, Alexander Kanavin wrote:
> copy_buildsystem() has become far too large and needs to be split
> into shorter and more understandable pieces; some of those
> pieces will be reused for the purpose of providing esdk mode
> directly from a plain yocto build.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  meta/classes-recipe/populate_sdk_ext.bbclass | 59 ++++++++++++++++----
>  1 file changed, 47 insertions(+), 12 deletions(-)
> 
> diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass
> index 53adc868d47..a20f38fb210 100644
> --- a/meta/classes-recipe/populate_sdk_ext.bbclass
> +++ b/meta/classes-recipe/populate_sdk_ext.bbclass
> @@ -186,12 +186,7 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
>          if os.path.exists(localconf + '.bak'):
>              os.replace(localconf + '.bak', localconf)
>  
> -python copy_buildsystem () {
> -    import re
> -    import shutil
> -    import glob
> -    import oe.copy_buildsystem
> -
> +def copy_bitbake_and_layers(d, baseoutpath, derivative):
>      oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT')
>  
>      conf_bbpath = ''
> @@ -200,13 +195,7 @@ python copy_buildsystem () {
>  
>      # Copy in all metadata layers + bitbake (as repositories)
>      buildsystem = oe.copy_buildsystem.BuildSystem('extensible SDK', d)
> -    baseoutpath = d.getVar('SDK_OUTPUT') + '/' + d.getVar('SDKPATH')
>  
> -    #check if custome templateconf path is set
> -    use_custom_templateconf = d.getVar('SDK_CUSTOM_TEMPLATECONF')
> -
> -    # Determine if we're building a derivative extensible SDK (from devtool build-sdk)
> -    derivative = (d.getVar('SDK_DERIVATIVE') or '') == '1'
>      if derivative:
>          workspace_name = 'orig-workspace'
>      else:
> @@ -231,6 +220,9 @@ python copy_buildsystem () {
>      d.setVar('oe_init_build_env_path', conf_initpath)
>      d.setVar('esdk_tools_path', esdk_tools_path)
>  
> +    return (conf_initpath, conf_bbpath, core_meta_subdir, sdkbblayers)
> +
> +def write_devtool_config(d, baseoutpath, conf_bbpath, conf_initpath, core_meta_subdir):
>      # Write out config file for devtool
>      import configparser
>      config = configparser.ConfigParser()
> @@ -247,10 +239,12 @@ python copy_buildsystem () {
>      with open(os.path.join(baseoutpath, 'conf', 'devtool.conf'), 'w') as f:
>          config.write(f)
>  
> +def write_unlocked_sigs(d, baseoutpath):
>      unlockedsigs =  os.path.join(baseoutpath, 'conf', 'unlocked-sigs.inc')
>      with open(unlockedsigs, 'w') as f:
>          pass
>  
> +def write_bblayers_conf(d, baseoutpath, sdkbblayers):
>      # Create a layer for new recipes / appends
>      bbpath = d.getVar('BBPATH')
>      env = os.environ.copy()
> @@ -279,6 +273,9 @@ python copy_buildsystem () {
>          f.write('    $' + '{SDKBASEMETAPATH}/workspace \\\n')
>          f.write('    "\n')
>  
> +def copy_uninative(d, baseoutpath):
> +    import shutil
> +
>      # Copy uninative tarball
>      # For now this is where uninative.bbclass expects the tarball
>      if bb.data.inherits_class('uninative', d):
> @@ -288,6 +285,12 @@ python copy_buildsystem () {
>          bb.utils.mkdirhier(uninative_outdir)
>          shutil.copy(uninative_file, uninative_outdir)
>  
> +    return uninative_checksum
> +
> +def write_local_conf(d, baseoutpath, derivative, core_meta_subdir, uninative_checksum):
> +    #check if custome templateconf path is set
> +    use_custom_templateconf = d.getVar('SDK_CUSTOM_TEMPLATECONF')
> +
>      env_passthrough = (d.getVar('BB_ENV_PASSTHROUGH_ADDITIONS') or '').split()
>      env_passthrough_values = {}
>  
> @@ -457,6 +460,9 @@ python copy_buildsystem () {
>                  f.write(line)
>              f.write('\n')
>  
> +def prepare_locked_cache(d, baseoutpath, conf_initpath):
> +    import shutil
> +
>      # Filter the locked signatures file to just the sstate tasks we are interested in
>      excluded_targets = get_sdk_install_targets(d, images_only=True)
>      sigfile = d.getVar('WORKDIR') + '/locked-sigs.inc'
> @@ -560,6 +566,9 @@ python copy_buildsystem () {
>                  f = os.path.join(root, name)
>                  os.remove(f)
>  
> +def write_manifest(d, baseoutpath):
> +    import glob
> +
>      # Write manifest file
>      # Note: at the moment we cannot include the env setup script here to keep
>      # it updated, since it gets modified during SDK installation (see
> @@ -583,6 +592,32 @@ python copy_buildsystem () {
>                      continue
>                  chksum = bb.utils.sha256_file(fn)
>                  f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
> +
> +
> +python copy_buildsystem () {
> +    import oe.copy_buildsystem
> +
> +    baseoutpath = d.getVar('SDK_OUTPUT') + '/' + d.getVar('SDKPATH')
> +
> +    # Determine if we're building a derivative extensible SDK (from devtool build-sdk)
> +    derivative = (d.getVar('SDK_DERIVATIVE') or '') == '1'
> +
> +    conf_initpath, conf_bbpath, core_meta_subdir, sdkbblayers = copy_bitbake_and_layers(d, baseoutpath, derivative)
> +
> +    write_devtool_config(d, baseoutpath, conf_bbpath, conf_initpath, core_meta_subdir)
> +
> +    write_unlocked_sigs(d, baseoutpath)
> +
> +    write_bblayers_conf(d, baseoutpath, sdkbblayers)
> +
> +    uninative_checksum = copy_uninative(d, baseoutpath)
> +
> +    write_local_conf(d, baseoutpath, derivative, core_meta_subdir, uninative_checksum)
> +
> +    prepare_locked_cache(d, baseoutpath, conf_initpath)
> +
> +    write_manifest(d, baseoutpath)
> +
>  }
>  
>  def get_current_buildtools(d):
> -- 
> 2.39.2
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#190151): https://lists.openembedded.org/g/openembedded-core/message/190151
> Mute This Topic: https://lists.openembedded.org/mt/102361115/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH 4/4] populate_sdk_ext: split copy_buildsystem() into logical steps defined as functions

[Alexander Kanavin]

Thanks, I just sent a corrected patch.

Alex

On Sat, 4 Nov 2023 at 00:50, Alexandre Belloni
<alexandre.belloni@bootlin.com> wrote:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/7419/steps/16/logs/stdio
>
> On 03/11/2023 11:28:06+0100, Alexander Kanavin wrote:
> > copy_buildsystem() has become far too large and needs to be split
> > into shorter and more understandable pieces; some of those
> > pieces will be reused for the purpose of providing esdk mode
> > directly from a plain yocto build.
> >
> > Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> > ---
> >  meta/classes-recipe/populate_sdk_ext.bbclass | 59 ++++++++++++++++----
> >  1 file changed, 47 insertions(+), 12 deletions(-)
> >
> > diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass
> > index 53adc868d47..a20f38fb210 100644
> > --- a/meta/classes-recipe/populate_sdk_ext.bbclass
> > +++ b/meta/classes-recipe/populate_sdk_ext.bbclass
> > @@ -186,12 +186,7 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
> >          if os.path.exists(localconf + '.bak'):
> >              os.replace(localconf + '.bak', localconf)
> >
> > -python copy_buildsystem () {
> > -    import re
> > -    import shutil
> > -    import glob
> > -    import oe.copy_buildsystem
> > -
> > +def copy_bitbake_and_layers(d, baseoutpath, derivative):
> >      oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT')
> >
> >      conf_bbpath = ''
> > @@ -200,13 +195,7 @@ python copy_buildsystem () {
> >
> >      # Copy in all metadata layers + bitbake (as repositories)
> >      buildsystem = oe.copy_buildsystem.BuildSystem('extensible SDK', d)
> > -    baseoutpath = d.getVar('SDK_OUTPUT') + '/' + d.getVar('SDKPATH')
> >
> > -    #check if custome templateconf path is set
> > -    use_custom_templateconf = d.getVar('SDK_CUSTOM_TEMPLATECONF')
> > -
> > -    # Determine if we're building a derivative extensible SDK (from devtool build-sdk)
> > -    derivative = (d.getVar('SDK_DERIVATIVE') or '') == '1'
> >      if derivative:
> >          workspace_name = 'orig-workspace'
> >      else:
> > @@ -231,6 +220,9 @@ python copy_buildsystem () {
> >      d.setVar('oe_init_build_env_path', conf_initpath)
> >      d.setVar('esdk_tools_path', esdk_tools_path)
> >
> > +    return (conf_initpath, conf_bbpath, core_meta_subdir, sdkbblayers)
> > +
> > +def write_devtool_config(d, baseoutpath, conf_bbpath, conf_initpath, core_meta_subdir):
> >      # Write out config file for devtool
> >      import configparser
> >      config = configparser.ConfigParser()
> > @@ -247,10 +239,12 @@ python copy_buildsystem () {
> >      with open(os.path.join(baseoutpath, 'conf', 'devtool.conf'), 'w') as f:
> >          config.write(f)
> >
> > +def write_unlocked_sigs(d, baseoutpath):
> >      unlockedsigs =  os.path.join(baseoutpath, 'conf', 'unlocked-sigs.inc')
> >      with open(unlockedsigs, 'w') as f:
> >          pass
> >
> > +def write_bblayers_conf(d, baseoutpath, sdkbblayers):
> >      # Create a layer for new recipes / appends
> >      bbpath = d.getVar('BBPATH')
> >      env = os.environ.copy()
> > @@ -279,6 +273,9 @@ python copy_buildsystem () {
> >          f.write('    $' + '{SDKBASEMETAPATH}/workspace \\\n')
> >          f.write('    "\n')
> >
> > +def copy_uninative(d, baseoutpath):
> > +    import shutil
> > +
> >      # Copy uninative tarball
> >      # For now this is where uninative.bbclass expects the tarball
> >      if bb.data.inherits_class('uninative', d):
> > @@ -288,6 +285,12 @@ python copy_buildsystem () {
> >          bb.utils.mkdirhier(uninative_outdir)
> >          shutil.copy(uninative_file, uninative_outdir)
> >
> > +    return uninative_checksum
> > +
> > +def write_local_conf(d, baseoutpath, derivative, core_meta_subdir, uninative_checksum):
> > +    #check if custome templateconf path is set
> > +    use_custom_templateconf = d.getVar('SDK_CUSTOM_TEMPLATECONF')
> > +
> >      env_passthrough = (d.getVar('BB_ENV_PASSTHROUGH_ADDITIONS') or '').split()
> >      env_passthrough_values = {}
> >
> > @@ -457,6 +460,9 @@ python copy_buildsystem () {
> >                  f.write(line)
> >              f.write('\n')
> >
> > +def prepare_locked_cache(d, baseoutpath, conf_initpath):
> > +    import shutil
> > +
> >      # Filter the locked signatures file to just the sstate tasks we are interested in
> >      excluded_targets = get_sdk_install_targets(d, images_only=True)
> >      sigfile = d.getVar('WORKDIR') + '/locked-sigs.inc'
> > @@ -560,6 +566,9 @@ python copy_buildsystem () {
> >                  f = os.path.join(root, name)
> >                  os.remove(f)
> >
> > +def write_manifest(d, baseoutpath):
> > +    import glob
> > +
> >      # Write manifest file
> >      # Note: at the moment we cannot include the env setup script here to keep
> >      # it updated, since it gets modified during SDK installation (see
> > @@ -583,6 +592,32 @@ python copy_buildsystem () {
> >                      continue
> >                  chksum = bb.utils.sha256_file(fn)
> >                  f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
> > +
> > +
> > +python copy_buildsystem () {
> > +    import oe.copy_buildsystem
> > +
> > +    baseoutpath = d.getVar('SDK_OUTPUT') + '/' + d.getVar('SDKPATH')
> > +
> > +    # Determine if we're building a derivative extensible SDK (from devtool build-sdk)
> > +    derivative = (d.getVar('SDK_DERIVATIVE') or '') == '1'
> > +
> > +    conf_initpath, conf_bbpath, core_meta_subdir, sdkbblayers = copy_bitbake_and_layers(d, baseoutpath, derivative)
> > +
> > +    write_devtool_config(d, baseoutpath, conf_bbpath, conf_initpath, core_meta_subdir)
> > +
> > +    write_unlocked_sigs(d, baseoutpath)
> > +
> > +    write_bblayers_conf(d, baseoutpath, sdkbblayers)
> > +
> > +    uninative_checksum = copy_uninative(d, baseoutpath)
> > +
> > +    write_local_conf(d, baseoutpath, derivative, core_meta_subdir, uninative_checksum)
> > +
> > +    prepare_locked_cache(d, baseoutpath, conf_initpath)
> > +
> > +    write_manifest(d, baseoutpath)
> > +
> >  }
> >
> >  def get_current_buildtools(d):
> > --
> > 2.39.2
> >
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#190151): https://lists.openembedded.org/g/openembedded-core/message/190151
> > Mute This Topic: https://lists.openembedded.org/mt/102361115/3617179
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Re: [OE-core] [PATCH 4/4] populate_sdk_ext: split copy_buildsystem() into logical steps defined as functions

[Alexandre Belloni]

On 06/11/2023 13:54:16+0100, Alexander Kanavin wrote:
> Thanks, I just sent a corrected patch.
> 

Thanks!

> Alex
> 
> On Sat, 4 Nov 2023 at 00:50, Alexandre Belloni
> <alexandre.belloni@bootlin.com> wrote:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/7419/steps/16/logs/stdio
> >
> > On 03/11/2023 11:28:06+0100, Alexander Kanavin wrote:
> > > copy_buildsystem() has become far too large and needs to be split
> > > into shorter and more understandable pieces; some of those
> > > pieces will be reused for the purpose of providing esdk mode
> > > directly from a plain yocto build.
> > >
> > > Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> > > ---
> > >  meta/classes-recipe/populate_sdk_ext.bbclass | 59 ++++++++++++++++----
> > >  1 file changed, 47 insertions(+), 12 deletions(-)
> > >
> > > diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass
> > > index 53adc868d47..a20f38fb210 100644
> > > --- a/meta/classes-recipe/populate_sdk_ext.bbclass
> > > +++ b/meta/classes-recipe/populate_sdk_ext.bbclass
> > > @@ -186,12 +186,7 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
> > >          if os.path.exists(localconf + '.bak'):
> > >              os.replace(localconf + '.bak', localconf)
> > >
> > > -python copy_buildsystem () {
> > > -    import re
> > > -    import shutil
> > > -    import glob
> > > -    import oe.copy_buildsystem
> > > -
> > > +def copy_bitbake_and_layers(d, baseoutpath, derivative):
> > >      oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT')
> > >
> > >      conf_bbpath = ''
> > > @@ -200,13 +195,7 @@ python copy_buildsystem () {
> > >
> > >      # Copy in all metadata layers + bitbake (as repositories)
> > >      buildsystem = oe.copy_buildsystem.BuildSystem('extensible SDK', d)
> > > -    baseoutpath = d.getVar('SDK_OUTPUT') + '/' + d.getVar('SDKPATH')
> > >
> > > -    #check if custome templateconf path is set
> > > -    use_custom_templateconf = d.getVar('SDK_CUSTOM_TEMPLATECONF')
> > > -
> > > -    # Determine if we're building a derivative extensible SDK (from devtool build-sdk)
> > > -    derivative = (d.getVar('SDK_DERIVATIVE') or '') == '1'
> > >      if derivative:
> > >          workspace_name = 'orig-workspace'
> > >      else:
> > > @@ -231,6 +220,9 @@ python copy_buildsystem () {
> > >      d.setVar('oe_init_build_env_path', conf_initpath)
> > >      d.setVar('esdk_tools_path', esdk_tools_path)
> > >
> > > +    return (conf_initpath, conf_bbpath, core_meta_subdir, sdkbblayers)
> > > +
> > > +def write_devtool_config(d, baseoutpath, conf_bbpath, conf_initpath, core_meta_subdir):
> > >      # Write out config file for devtool
> > >      import configparser
> > >      config = configparser.ConfigParser()
> > > @@ -247,10 +239,12 @@ python copy_buildsystem () {
> > >      with open(os.path.join(baseoutpath, 'conf', 'devtool.conf'), 'w') as f:
> > >          config.write(f)
> > >
> > > +def write_unlocked_sigs(d, baseoutpath):
> > >      unlockedsigs =  os.path.join(baseoutpath, 'conf', 'unlocked-sigs.inc')
> > >      with open(unlockedsigs, 'w') as f:
> > >          pass
> > >
> > > +def write_bblayers_conf(d, baseoutpath, sdkbblayers):
> > >      # Create a layer for new recipes / appends
> > >      bbpath = d.getVar('BBPATH')
> > >      env = os.environ.copy()
> > > @@ -279,6 +273,9 @@ python copy_buildsystem () {
> > >          f.write('    $' + '{SDKBASEMETAPATH}/workspace \\\n')
> > >          f.write('    "\n')
> > >
> > > +def copy_uninative(d, baseoutpath):
> > > +    import shutil
> > > +
> > >      # Copy uninative tarball
> > >      # For now this is where uninative.bbclass expects the tarball
> > >      if bb.data.inherits_class('uninative', d):
> > > @@ -288,6 +285,12 @@ python copy_buildsystem () {
> > >          bb.utils.mkdirhier(uninative_outdir)
> > >          shutil.copy(uninative_file, uninative_outdir)
> > >
> > > +    return uninative_checksum
> > > +
> > > +def write_local_conf(d, baseoutpath, derivative, core_meta_subdir, uninative_checksum):
> > > +    #check if custome templateconf path is set
> > > +    use_custom_templateconf = d.getVar('SDK_CUSTOM_TEMPLATECONF')
> > > +
> > >      env_passthrough = (d.getVar('BB_ENV_PASSTHROUGH_ADDITIONS') or '').split()
> > >      env_passthrough_values = {}
> > >
> > > @@ -457,6 +460,9 @@ python copy_buildsystem () {
> > >                  f.write(line)
> > >              f.write('\n')
> > >
> > > +def prepare_locked_cache(d, baseoutpath, conf_initpath):
> > > +    import shutil
> > > +
> > >      # Filter the locked signatures file to just the sstate tasks we are interested in
> > >      excluded_targets = get_sdk_install_targets(d, images_only=True)
> > >      sigfile = d.getVar('WORKDIR') + '/locked-sigs.inc'
> > > @@ -560,6 +566,9 @@ python copy_buildsystem () {
> > >                  f = os.path.join(root, name)
> > >                  os.remove(f)
> > >
> > > +def write_manifest(d, baseoutpath):
> > > +    import glob
> > > +
> > >      # Write manifest file
> > >      # Note: at the moment we cannot include the env setup script here to keep
> > >      # it updated, since it gets modified during SDK installation (see
> > > @@ -583,6 +592,32 @@ python copy_buildsystem () {
> > >                      continue
> > >                  chksum = bb.utils.sha256_file(fn)
> > >                  f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
> > > +
> > > +
> > > +python copy_buildsystem () {
> > > +    import oe.copy_buildsystem
> > > +
> > > +    baseoutpath = d.getVar('SDK_OUTPUT') + '/' + d.getVar('SDKPATH')
> > > +
> > > +    # Determine if we're building a derivative extensible SDK (from devtool build-sdk)
> > > +    derivative = (d.getVar('SDK_DERIVATIVE') or '') == '1'
> > > +
> > > +    conf_initpath, conf_bbpath, core_meta_subdir, sdkbblayers = copy_bitbake_and_layers(d, baseoutpath, derivative)
> > > +
> > > +    write_devtool_config(d, baseoutpath, conf_bbpath, conf_initpath, core_meta_subdir)
> > > +
> > > +    write_unlocked_sigs(d, baseoutpath)
> > > +
> > > +    write_bblayers_conf(d, baseoutpath, sdkbblayers)
> > > +
> > > +    uninative_checksum = copy_uninative(d, baseoutpath)
> > > +
> > > +    write_local_conf(d, baseoutpath, derivative, core_meta_subdir, uninative_checksum)
> > > +
> > > +    prepare_locked_cache(d, baseoutpath, conf_initpath)
> > > +
> > > +    write_manifest(d, baseoutpath)
> > > +
> > >  }
> > >
> > >  def get_current_buildtools(d):
> > > --
> > > 2.39.2
> > >
> >
> > >
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > > Links: You receive all messages sent to this group.
> > > View/Reply Online (#190151): https://lists.openembedded.org/g/openembedded-core/message/190151
> > > Mute This Topic: https://lists.openembedded.org/mt/102361115/3617179
> > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > >
> >
> >
> > --
> > Alexandre Belloni, co-owner and COO, Bootlin
> > Embedded Linux and Kernel engineering
> > https://bootlin.com

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com