[PATCH v8 0/8] devtool ide plugin

[PATCH v8 0/8] devtool ide plugin

[Adrian Freihofer]

Changes in comparison to v6:

- Fix findigs: https://lists.openembedded.org/g/openembedded-core/message/188995
  - Major refactoring to split IDE specific parts into separate files
  - Use bindir for gdbserver
  - Remove pontless warning for debuginfod
  - Simplified searching for debug symbols
  - Added a new patch that makes the behavior of image-combined-dbg.bbclass the
    default behavior of rootfs-dbg. I think this solves a whole bunch of problems
    not only for GDB but also for all the other tools that need the debug symbols.
- Dropped the workaround for the pseudo breakage with sources in tree
  After a cleanup and improving the script to better replicate what bitbake does
  I was no longer able to reproduce the issue.
- Improve the default behavior for not yet supported build tools. devtool build
  and devtool deploy-target are called.
- Improve the tests:
  - Start Qemu only once (it's expensive)
  - Test also the ide=none
- Improve documentation about the no recipe mode (to docs mailing list)
- Details and bug fixes

According to https://www.yoctoproject.org/community/yocto-project-engineering-request-for-quotation/
one of the proposed areas for development of the Yocto project is "VSCode IDE Integration - New developer tooling".
One aspect of this larger topic is helping application developers configure the IDE to work on a recipe's source code using Yocto's eSDK. This patchset provides a new devtool plugin (devtool ide) that does this fully automatically for recipes inheriting the cmake or the meson bbclass. Support for more programming languages and build tools may be added in the future.

Let's start with a brief introduction of how it looks like from a user's perspective.

Reference setup
---------------

$ . oe-init-build-env

Add the following layers to bblayers.conf file:
- meta
- meta-poky
- meta-yocto-bsp
- meta-selftest

Add the following to the local.conf

IMAGE_GEN_DEBUGFS = "1"
IMAGE_FEATURES += "ssh-server-dropbear debug-tweaks"
IMAGE_INSTALL:append = "\
    cmake-example-ptest \
    meson-example-ptest \
    gdbserver \
"

Working on a recipe
-------------------

Note for those who "hate cmake with passion": Replace cmake by meson in the following commands.

1. Add the recipe which should be modified to the workspace

   $ devtool modify cmake-example

2. Build the eSDK. This provides all the host tools as well as an image which can be used on the target device.
   Since everything is built with one bitbake command also all the debug symbols are expected to be consistent.

   $ devtool ide cmake-example core-image-minimal

3. Install the image on the target device or simply use Qemu

   $ runqemu

4. Start VSCode

   $ code "$BUILDDIR/workspace/sources/cmake-example"

5. Work in VSCode, after installing the proposed plugins

   Ctrl + Shift + p, cmake Select Configure Preset
   Ctrl + Shift + p, cmake Build

6. Execute the unit tests on the host works with Qemu even for cross compiled test executables.

   Ctrl + Shift + p, cmake Test

7. Remote debugging with GDB

   Ctrl + Shift + p, Run Task --> install && deploy-target cmake-example
   F5 (Debug configuration might be selected before)

8. Work on the recipes and call bitbake again to get the SDK updated.
   Building the application by calling cmake (from the IDE) or by calling bitbake or by calling devtool build is expected to produce the exact same results. To make that possible, the devtool ide plugin is designed to configure the IDE to call cmake with the exact same configuration as used by bitbake when building the recipe. Unlike the eSDK, the goal here is clearly that there is no longer a separation between the SDK and the application development process. Regardless of which tool is called, the same source files are edited and the same o-files are generated and re-used.

Working with a SDK without a recipe
-----------------------------------

If devtool ide is called for a recipe named none or meta-ide-support the eSDK with its generic environment file gets generated.
In case of using VSCode and cmake in addition to the well known environment file a cmake-kit
https://vector-of-bool.github.io/docs/vscode-cmake-tools/kits.html is added to the User-Local Kits.
This allows to work with cmake calling the cross-toolchain out of VSCode or a shell with the environment file sourced.

Design
------

The goal of this implementation is to create a configuration for VSCode (or other IDEs) that allows to work on the code of a recipe completely independent from bitbake. bitbake is only called if the configuration or the whole SDK has to be regenerated. But bitbake should not need to be called while working in the IDE. This has two major advantages over calling devtool build from the IDE:
- The IDE provides plugins for integration with cmake, for example. These features are usable, which would not be the case if bitbake or devtool are called from within the IDE.
- It is much faster.

Supporting other IDEs
---------------------

Focus is currently VSCode. But a paramter "--ide=none" is already supported.
With this paramter passed, no VSCode specific config files are generated.
Instead, simple scripts are generated, which should be suitable for integration with other IDEs.

Testing
-------

Reasonable but not yet complete test coverage is provided by:

$ oe-selftest -r devtool.DevtoolIdeTests

What's next?
------------

- Support for more build-tools and programming languages
- Support for more IDEs

Known bugs
----------

- IntelliSense with meson does not work. meson and ninja generate a compile_commands.json which requires to have the compiler in the PATH. (cmake generates the same file with absolute path.)

Adrian Freihofer (8):
  vscode: add minimal configuration
  cmake.bbclass: support qemu
  image-combined-dbg: make this the default
  tests: add a C++ example recipe
  devtool: refactor deploy-target
  devtool: new ide plugin
  oe-selftest devtool: ide tests
  docs: cover devtool ide

 .gitignore                                    |    2 +
 .vscode/settings.json                         |   32 +
 documentation/sdk-manual/extensible.rst       |  153 ++-
 meta-selftest/recipes-test/cpp/.gitignore     |    1 +
 .../recipes-test/cpp/autotools-example.bb     |   18 +
 .../cpp/autotools-example/run-ptest           |   10 +
 .../recipes-test/cpp/cmake-example.bb         |   17 +
 .../recipes-test/cpp/cmake-example/run-ptest  |   10 +
 .../recipes-test/cpp/cpp-example.inc          |   24 +
 .../recipes-test/cpp/files/CMakeLists.txt     |   60 +
 .../recipes-test/cpp/files/Makefile.am        |   13 +
 .../recipes-test/cpp/files/configure.ac       |   11 +
 .../cpp/files/cpp-example-lib.cpp             |   17 +
 .../cpp/files/cpp-example-lib.hpp             |   16 +
 .../recipes-test/cpp/files/cpp-example.cpp    |   16 +
 .../recipes-test/cpp/files/meson.build        |   34 +
 .../cpp/files/test-cpp-example.cpp            |   20 +
 .../recipes-test/cpp/meson-example.bb         |   17 +
 .../recipes-test/cpp/meson-example/run-ptest  |   10 +
 meta/classes-recipe/cmake.bbclass             |   20 +
 .../classes-recipe/image-combined-dbg.bbclass |   15 -
 meta/lib/oe/rootfs.py                         |   35 +-
 meta/lib/oeqa/selftest/cases/devtool.py       |  274 +++++
 scripts/crosstap                              |   28 +-
 scripts/lib/devtool/__init__.py               |    5 +-
 scripts/lib/devtool/deploy.py                 |  230 ++--
 scripts/lib/devtool/ide.py                    | 1095 +++++++++++++++++
 scripts/lib/devtool/ide_handlers/__init__.py  |   23 +
 scripts/lib/devtool/ide_handlers/ide_base.py  |   46 +
 scripts/lib/devtool/ide_handlers/ide_code.py  |  420 +++++++
 scripts/lib/devtool/ide_handlers/ide_none.py  |   91 ++
 31 files changed, 2580 insertions(+), 183 deletions(-)
 create mode 100644 .vscode/settings.json
 create mode 100644 meta-selftest/recipes-test/cpp/.gitignore
 create mode 100644 meta-selftest/recipes-test/cpp/autotools-example.bb
 create mode 100644 meta-selftest/recipes-test/cpp/autotools-example/run-ptest
 create mode 100644 meta-selftest/recipes-test/cpp/cmake-example.bb
 create mode 100644 meta-selftest/recipes-test/cpp/cmake-example/run-ptest
 create mode 100644 meta-selftest/recipes-test/cpp/cpp-example.inc
 create mode 100644 meta-selftest/recipes-test/cpp/files/CMakeLists.txt
 create mode 100644 meta-selftest/recipes-test/cpp/files/Makefile.am
 create mode 100644 meta-selftest/recipes-test/cpp/files/configure.ac
 create mode 100644 meta-selftest/recipes-test/cpp/files/cpp-example-lib.cpp
 create mode 100644 meta-selftest/recipes-test/cpp/files/cpp-example-lib.hpp
 create mode 100644 meta-selftest/recipes-test/cpp/files/cpp-example.cpp
 create mode 100644 meta-selftest/recipes-test/cpp/files/meson.build
 create mode 100644 meta-selftest/recipes-test/cpp/files/test-cpp-example.cpp
 create mode 100644 meta-selftest/recipes-test/cpp/meson-example.bb
 create mode 100644 meta-selftest/recipes-test/cpp/meson-example/run-ptest
 delete mode 100644 meta/classes-recipe/image-combined-dbg.bbclass
 create mode 100755 scripts/lib/devtool/ide.py
 create mode 100644 scripts/lib/devtool/ide_handlers/__init__.py
 create mode 100644 scripts/lib/devtool/ide_handlers/ide_base.py
 create mode 100644 scripts/lib/devtool/ide_handlers/ide_code.py
 create mode 100644 scripts/lib/devtool/ide_handlers/ide_none.py

-- 
2.41.0

[PATCH v8 1/8] vscode: add minimal configuration

[Adrian Freihofer]

It is essential to configure VSCode indexer plugins to ignore the build
folder of bitbake. Otherwise, the indexer plugins run with 100% CPU load
until an OOM exception occurs. In practice, this makes VSCode more or
less unusable for working with Yocto until a file like the one added by
this commit is deployed before VSCode starts. From the user's point of
view, it is not obvious why the system runs at 100% CPU load and
eventually crashes.

It is even more misleading that VSCode starts the indexers immediately,
but does not stop or reconfigure them when the ignore list is updated.
In practice, this means that every time the ignore list is changed,
VSCode immediately starts indexing the build folder until the OOM
exception stops it. Depending on the system's OOM handler, the entire
build machine may crash.
Particularly annoying is the Python plugin that ignores the general
ignore list and requires an extra ignore section.

The settings are suitable for workflows like bitbake, devtool modify,
devtool reset. The settings are not intended to work on the source code
of a recipe. It is assumed that a separate instance of VSCode is used
per workspace folder. These per workspace instances can have different
settings depending on the details of the sources that come with the
recipe. The new devtool ide plugin will generate settings to match this.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 .gitignore            |  2 ++
 .vscode/settings.json | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 34 insertions(+)
 create mode 100644 .vscode/settings.json

diff --git a/.gitignore b/.gitignore
index 8f48d452dab..f6ce090b5fc 100644
--- a/.gitignore
+++ b/.gitignore
@@ -36,3 +36,5 @@ _toaster_clones/
 downloads/
 sstate-cache/
 toaster.sqlite
+.vscode/
+vscode-bitbake-build/
diff --git a/.vscode/settings.json b/.vscode/settings.json
new file mode 100644
index 00000000000..517a86d1bfa
--- /dev/null
+++ b/.vscode/settings.json
@@ -0,0 +1,32 @@
+{
+    "files.watcherExclude": {
+        "**/.git/**": true,
+        "**/cache/**": true,
+        "**/tmp*/**": true,
+        "**/downloads/**": true,
+        "**/sstate-cache/**": true,
+        "**/vscode-bitbake-build/**": true,
+        "**/workspace/sources/**": true,
+        "**/workspace/attic/**": true
+    },
+    "files.exclude": {
+        "**/.git/**": true,
+        "**/cache/**": true,
+        "**/tmp*/**": true,
+        "**/downloads/**": true,
+        "**/sstate-cache/**": true,
+        "**/vscode-bitbake-build/**": true,
+        "**/workspace/sources/**": true,
+        "**/workspace/attic/**": true
+    },
+    "python.analysis.exclude": [
+        "**/.git/**",
+        "**/cache/**",
+        "**/tmp*/**",
+        "**/downloads/**",
+        "**/sstate-cache/**",
+        "**/vscode-bitbake-build/**",
+        "**/workspace/sources/**",
+        "**/workspace/attic/**"
+    ]
+}
-- 
2.41.0

[PATCH v8 2/8] cmake.bbclass: support qemu

[Adrian Freihofer]

Define the CMAKE_CROSSCOMPILING_EMULATOR variable similar to what the
meson bbclass does. This allows for example to execute cross compilied
unit tests on the build machine.

CMAKE_CROSSCOMPILING_EMULATOR is a semi colon separated list of
paramters which could directly handle the -L and the -E parameters.
Creating a wrapper script is not absolutely mandatory. But anyway lets
do it similar to what the meson.bbclass does and also disable pseudo.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 meta/classes-recipe/cmake.bbclass | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/meta/classes-recipe/cmake.bbclass b/meta/classes-recipe/cmake.bbclass
index d978b889440..911c237a3fd 100644
--- a/meta/classes-recipe/cmake.bbclass
+++ b/meta/classes-recipe/cmake.bbclass
@@ -4,6 +4,13 @@
 # SPDX-License-Identifier: MIT
 #
 
+inherit qemu
+
+EXEWRAPPER_ENABLED:class-native = "False"
+EXEWRAPPER_ENABLED:class-nativesdk = "False"
+EXEWRAPPER_ENABLED ?= "${@bb.utils.contains('MACHINE_FEATURES', 'qemu-usermode', 'True', 'False', d)}"
+DEPENDS:append = "${@' qemu-native' if d.getVar('EXEWRAPPER_ENABLED') == 'True' else ''}"
+
 # Path to the CMake file to process.
 OECMAKE_SOURCEPATH ??= "${S}"
 
@@ -156,6 +163,19 @@ EOF
 
 addtask generate_toolchain_file after do_patch before do_configure
 
+cmake_do_generate_toolchain_file:append:class-target() {
+    if [ "${EXEWRAPPER_ENABLED}" = "True" ]; then
+        # Write out a qemu wrapper that will be used as exe_wrapper so that camake
+        # can run target helper binaries through that. This also allows to execute ctest.
+        qemu_binary="${@qemu_wrapper_cmdline(d, '${STAGING_DIR_HOST}', ['${STAGING_DIR_HOST}/${libdir}','${STAGING_DIR_HOST}/${base_libdir}'])}"
+        echo "#!/bin/sh" > "${WORKDIR}/cmake-qemuwrapper"
+        echo "$qemu_binary \"\$@\"" >> "${WORKDIR}/cmake-qemuwrapper"
+        chmod +x "${WORKDIR}/cmake-qemuwrapper"
+        echo "set( CMAKE_CROSSCOMPILING_EMULATOR ${WORKDIR}/cmake-qemuwrapper)" \
+          >> ${WORKDIR}/toolchain.cmake
+    fi
+}
+
 CONFIGURE_FILES = "CMakeLists.txt *.cmake"
 
 do_configure[cleandirs] = "${@d.getVar('B') if d.getVar('S') != d.getVar('B') else ''}"
-- 
2.41.0

Re: [OE-core] [PATCH v8 2/8] cmake.bbclass: support qemu

[Richard Purdie]

On Wed, 2023-11-01 at 12:01 +0100, Adrian Freihofer wrote:
> Define the CMAKE_CROSSCOMPILING_EMULATOR variable similar to what the
> meson bbclass does. This allows for example to execute cross compilied
> unit tests on the build machine.
> 
> CMAKE_CROSSCOMPILING_EMULATOR is a semi colon separated list of
> paramters which could directly handle the -L and the -E parameters.
> Creating a wrapper script is not absolutely mandatory. But anyway lets
> do it similar to what the meson.bbclass does and also disable pseudo.
> 
> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> ---
>  meta/classes-recipe/cmake.bbclass | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/meta/classes-recipe/cmake.bbclass b/meta/classes-recipe/cmake.bbclass
> index d978b889440..911c237a3fd 100644
> --- a/meta/classes-recipe/cmake.bbclass
> +++ b/meta/classes-recipe/cmake.bbclass
> @@ -4,6 +4,13 @@
>  # SPDX-License-Identifier: MIT
>  #
>  
> +inherit qemu
> +
> +EXEWRAPPER_ENABLED:class-native = "False"
> +EXEWRAPPER_ENABLED:class-nativesdk = "False"
> +EXEWRAPPER_ENABLED ?= "${@bb.utils.contains('MACHINE_FEATURES', 'qemu-usermode', 'True', 'False', d)}"
> +DEPENDS:append = "${@' qemu-native' if d.getVar('EXEWRAPPER_ENABLED') == 'True' else ''}"
> +
>  # Path to the CMake file to process.
>  OECMAKE_SOURCEPATH ??= "${S}"
>  
> @@ -156,6 +163,19 @@ EOF
>  
>  addtask generate_toolchain_file after do_patch before do_configure
>  
> +cmake_do_generate_toolchain_file:append:class-target() {
> +    if [ "${EXEWRAPPER_ENABLED}" = "True" ]; then
> +        # Write out a qemu wrapper that will be used as exe_wrapper so that camake
> +        # can run target helper binaries through that. This also allows to execute ctest.
> +        qemu_binary="${@qemu_wrapper_cmdline(d, '${STAGING_DIR_HOST}', ['${STAGING_DIR_HOST}/${libdir}','${STAGING_DIR_HOST}/${base_libdir}'])}"
> +        echo "#!/bin/sh" > "${WORKDIR}/cmake-qemuwrapper"
> +        echo "$qemu_binary \"\$@\"" >> "${WORKDIR}/cmake-qemuwrapper"
> +        chmod +x "${WORKDIR}/cmake-qemuwrapper"
> +        echo "set( CMAKE_CROSSCOMPILING_EMULATOR ${WORKDIR}/cmake-qemuwrapper)" \
> +          >> ${WORKDIR}/toolchain.cmake
> +    fi
> +}
> +
>  CONFIGURE_FILES = "CMakeLists.txt *.cmake"
>  
>  do_configure[cleandirs] = "${@d.getVar('B') if d.getVar('S') != d.getVar('B') else ''}"

We've tried very hard historically to avoid needing qemu usermode. It
has crept in, first because gobject introspection required it and more
recently, meson but that isn't a reason to encourage it.

I'm a bit reluctant to put this in the cmake class itself as it makes
it seem like it is part of the normal best practise and it really
isn't. I'd suggest a cmake-qemu class which people can use if they
really need this.

The reason is that some architectures/platorms don't work under qemu.
Even x86-64 didn't with certain more recent CPU optimisations until
surprisingly recently.

Cheers,

Richard

[PATCH v8 3/8] image-combined-dbg: make this the default

[Adrian Freihofer]

Remove the image-combined-dbg.bbclass and make this the default
behavior for the rootfs-dbg. A rootfs-dbg with only debug symbols but
no executable binaries also causes problems with gdb, which is
probably the most common use case for the roofs-dbg. This change
simplifies and improves the user experience for a slightly larger
rootfs-dbg.

If the rootfs-dbg contains a complete copy of the rootfs, it is also
usable for booting the target device over the network. This in turn
simplifies other use cases with e.g. the use of perf on a device
booted over the network.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 .../classes-recipe/image-combined-dbg.bbclass | 15 --------
 meta/lib/oe/rootfs.py                         | 35 ++++---------------
 scripts/crosstap                              | 28 +--------------
 3 files changed, 7 insertions(+), 71 deletions(-)
 delete mode 100644 meta/classes-recipe/image-combined-dbg.bbclass

diff --git a/meta/classes-recipe/image-combined-dbg.bbclass b/meta/classes-recipe/image-combined-dbg.bbclass
deleted file mode 100644
index 729313739c1..00000000000
--- a/meta/classes-recipe/image-combined-dbg.bbclass
+++ /dev/null
@@ -1,15 +0,0 @@
-#
-# Copyright OpenEmbedded Contributors
-#
-# SPDX-License-Identifier: MIT
-#
-
-IMAGE_PREPROCESS_COMMAND:append = " combine_dbg_image"
-
-combine_dbg_image () {
-        if [ "${IMAGE_GEN_DEBUGFS}" = "1" -a -e ${IMAGE_ROOTFS}-dbg ]; then
-                # copy target files into -dbg rootfs, so it can be used for
-                # debug purposes directly
-                tar -C ${IMAGE_ROOTFS} -cf - . | tar -C ${IMAGE_ROOTFS}-dbg -xf -
-        fi
-}
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 1a48ed10b3f..a1cc28a6dd0 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -111,40 +111,17 @@ class Rootfs(object, metaclass=ABCMeta):
         if gen_debugfs != '1':
            return
 
+        rootfs_orig = self.image_rootfs + '-orig'
+
         bb.note("  Renaming the original rootfs...")
         try:
-            shutil.rmtree(self.image_rootfs + '-orig')
+            shutil.rmtree(rootfs_orig)
         except:
             pass
-        bb.utils.rename(self.image_rootfs, self.image_rootfs + '-orig')
+        bb.utils.rename(self.image_rootfs, rootfs_orig)
 
         bb.note("  Creating debug rootfs...")
-        bb.utils.mkdirhier(self.image_rootfs)
-
-        bb.note("  Copying back package database...")
-        for path in package_paths:
-            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path))
-            if os.path.isdir(self.image_rootfs + '-orig' + path):
-                shutil.copytree(self.image_rootfs + '-orig' + path, self.image_rootfs + path, symlinks=True)
-            elif os.path.isfile(self.image_rootfs + '-orig' + path):
-                shutil.copyfile(self.image_rootfs + '-orig' + path, self.image_rootfs + path)
-
-        # Copy files located in /usr/lib/debug or /usr/src/debug
-        for dir in ["/usr/lib/debug", "/usr/src/debug"]:
-            src = self.image_rootfs + '-orig' + dir
-            if os.path.exists(src):
-                dst = self.image_rootfs + dir
-                bb.utils.mkdirhier(os.path.dirname(dst))
-                shutil.copytree(src, dst)
-
-        # Copy files with suffix '.debug' or located in '.debug' dir.
-        for root, dirs, files in os.walk(self.image_rootfs + '-orig'):
-            relative_dir = root[len(self.image_rootfs + '-orig'):]
-            for f in files:
-                if f.endswith('.debug') or '/.debug' in relative_dir:
-                    bb.utils.mkdirhier(self.image_rootfs + relative_dir)
-                    shutil.copy(os.path.join(root, f),
-                                self.image_rootfs + relative_dir)
+        shutil.copytree(rootfs_orig, self.image_rootfs, symlinks=True)
 
         bb.note("  Install complementary '*-dbg' packages...")
         self.pm.install_complementary('*-dbg')
@@ -178,7 +155,7 @@ class Rootfs(object, metaclass=ABCMeta):
         bb.utils.rename(self.image_rootfs, self.image_rootfs + '-dbg')
 
         bb.note("  Restoring original rootfs...")
-        bb.utils.rename(self.image_rootfs + '-orig', self.image_rootfs)
+        bb.utils.rename(rootfs_orig, self.image_rootfs)
 
     def _exec_shell_cmd(self, cmd):
         try:
diff --git a/scripts/crosstap b/scripts/crosstap
index 5aa72f14d44..87dac33e064 100755
--- a/scripts/crosstap
+++ b/scripts/crosstap
@@ -170,18 +170,6 @@ class BitbakeEnv(object):
         return ret
 
 class ParamDiscovery(object):
-    SYMBOLS_CHECK_MESSAGE = """
-WARNING: image '%s' does not have dbg-pkgs IMAGE_FEATURES enabled and no
-"image-combined-dbg" in inherited classes is specified. As result the image
-does not have symbols for user-land processes DWARF based probes. Consider
-adding 'dbg-pkgs' to EXTRA_IMAGE_FEATURES or adding "image-combined-dbg" to
-USER_CLASSES. I.e add this line 'USER_CLASSES += "image-combined-dbg"' to
-local.conf file.
-
-Or you may use IMAGE_GEN_DEBUGFS="1" option, and then after build you need
-recombine/unpack image and image-dbg tarballs and pass resulting dir location
-with --sysroot option.
-"""
 
     def __init__(self, image):
         self.image = image
@@ -204,8 +192,6 @@ with --sysroot option.
 
         self.staging_dir_native = None
 
-        self.image_combined_dbg = False
-
     def discover(self):
         if self.image:
             benv_image = BitbakeEnv(self.image)
@@ -248,10 +234,6 @@ with --sysroot option.
         (self.staging_dir_native
         ) = benv_systemtap.get_vars(["STAGING_DIR_NATIVE"])
 
-        if self.inherit:
-            if "image-combined-dbg" in self.inherit.split():
-                self.image_combined_dbg = True
-
     def check(self, sysroot_option):
         ret = True
         if self.image_rootfs:
@@ -280,10 +262,6 @@ with --sysroot option.
                 if "dbg-pkgs" in image_features:
                     dbg_pkgs_found = True
 
-            if not dbg_pkgs_found \
-               and not self.image_combined_dbg:
-                print(ParamDiscovery.SYMBOLS_CHECK_MESSAGE % (self.image))
-
         if not ret:
             print("")
 
@@ -310,10 +288,7 @@ with --sysroot option.
         stap.stap = self.staging_dir_native + "/usr/bin/stap"
         if not stap.sysroot:
             if self.image_rootfs:
-                if self.image_combined_dbg:
-                    stap.sysroot = self.image_rootfs + "-dbg"
-                else:
-                    stap.sysroot = self.image_rootfs
+                stap.sysroot = self.image_rootfs + "-dbg"
         stap.runtime = self.staging_dir_native + "/usr/share/systemtap/runtime"
         stap.tapset = self.staging_dir_native + "/usr/share/systemtap/tapset"
         stap.arch = self.__map_systemtap_arch()
@@ -362,7 +337,6 @@ configuration is recommended:
 # enables symbol + target binaries rootfs-dbg in workspace
 IMAGE_GEN_DEBUGFS = "1"
 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
-USER_CLASSES += "image-combined-dbg"
 
 # enables kernel debug symbols
 KERNEL_EXTRA_FEATURES:append = " features/debug/debug-kernel.scc"
-- 
2.41.0

Re: [OE-core] [PATCH v8 3/8] image-combined-dbg: make this the default

[Richard Purdie]

On Wed, 2023-11-01 at 12:01 +0100, Adrian Freihofer wrote:
> Remove the image-combined-dbg.bbclass and make this the default
> behavior for the rootfs-dbg. A rootfs-dbg with only debug symbols but
> no executable binaries also causes problems with gdb, which is
> probably the most common use case for the roofs-dbg. This change
> simplifies and improves the user experience for a slightly larger
> rootfs-dbg.
> 
> If the rootfs-dbg contains a complete copy of the rootfs, it is also
> usable for booting the target device over the network. This in turn
> simplifies other use cases with e.g. the use of perf on a device
> booted over the network.
> 
> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>

I'm being pressured for review on this so I'll just say what the
problem is. I struggle to review this as off the top of my head, I
can't remember the difference between "rootfs-dbg" or "image-combined-
dbg". I understand the patch gets rid of one and the argument appears
to be that gdb doesn't work well with the case that is removed.

What isn't here is any reminder of what the differences are, or a
pointer to the history which lead us to have two different modes in the
first place. We presumably had a reason for adding it.

That means in order to review it, I'd have to dig into the history and
work out the differences, then work out why we added the two modes and
then determine if they're still needed.

What would help speed up review would be a pointer to the original
commits and/or a summary of why the were added. A summary of the
differences between the two modes would also help/

Cheers,

Richard

Re: [OE-core] [PATCH v8 3/8] image-combined-dbg: make this the default

[adrian.freihofer]

On Mon, 2023-11-06 at 13:50 +0000, Richard Purdie wrote:
> On Wed, 2023-11-01 at 12:01 +0100, Adrian Freihofer wrote:
> > Remove the image-combined-dbg.bbclass and make this the default
> > behavior for the rootfs-dbg. A rootfs-dbg with only debug symbols
> > but
> > no executable binaries also causes problems with gdb, which is
> > probably the most common use case for the roofs-dbg. This change
> > simplifies and improves the user experience for a slightly larger
> > rootfs-dbg.
> > 
> > If the rootfs-dbg contains a complete copy of the rootfs, it is
> > also
> > usable for booting the target device over the network. This in turn
> > simplifies other use cases with e.g. the use of perf on a device
> > booted over the network.
> > 
> > Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> 
> I'm being pressured for review on this so I'll just say what the
> problem is. I struggle to review this as off the top of my head, I
> can't remember the difference between "rootfs-dbg" or "image-
> combined-
> dbg". I understand the patch gets rid of one and the argument appears
> to be that gdb doesn't work well with the case that is removed.
> 
> What isn't here is any reminder of what the differences are, or a
> pointer to the history which lead us to have two different modes in
> the
> first place. We presumably had a reason for adding it.
> 
> That means in order to review it, I'd have to dig into the history
> and
> work out the differences, then work out why we added the two modes
> and
> then determine if they're still needed.
> 
> What would help speed up review would be a pointer to the original
> commits and/or a summary of why the were added. A summary of the
> differences between the two modes would also help/
> 

We have tried to work through the git history and to search for use
cases that rely on the old implementation. The git history does not
explain why it is like it is.

Enguerrand came up with theoretical use cases where the rootfs-dbg
could be mounted somehow as overlay at run-time. I can't rule out the
possibility of someone doing something like that.

Maybe we should keep that as it is.

@Ross: do you have a different opinion regarding the discussion we had:
https://lists.openembedded.org/g/openembedded-core/message/188995


Here are my notes. Just in case we would like to come back to this some
when.

- 7a7c6b021f114c6bedfbdc9afd2bf2925b238d19:
  The first implementation of rootfs-dbg only added the *-dbg packages
  and the package database.
- 69d3df9169133d5e05eff25019569fb8974d48c2
- c1ce0d9a9e200e35a9b6f9d537232875683ab9f1
- e73a85be3e020561db92a197c593afe7fd952919
- 1800b1ba7ae07cd010e529653eaa1d7a5841e6e5
  Added some workarounds related to opkg and openssl which got finally
  reverted again.
- 364c4c7d3fc049f2b734a6a235758a75af364ce0
  Support for PACKAGE_DEBUG_SPLIT_STYLE= 'debug-with-srcpkg'
- 5f30534c47571d199d8655b3da51f7b55fe20c04
  Introduce IMAGE_INSTALL_DEBUGFS
- 18f080fbe4cf51824e5f1d73a10e06e3a5724423
  Removes the package database later, as it caused problems with RPMs.

Comparison of
  $ tar xvfj core-image-minimal-qemux86-64.rootfs.tar.bz2
  $ tar xvfj core-image-minimal-qemux86-64.rootfs-dbg.tar.bz2
without this patch verus
  $ tar xvfj core-image-minimal-qemux86-64.rootfs-dbg.tar.bz2
with this patch applied shows some differences. This is expected
because
some of the post rootfs steps run after rootfs gets copied to
rootfs-dbg. Changing this would require more changes.

$ diff -r --no-dereference debug-fs1 debug-fs2
Only in debug-fs1/etc/default: postinst
Only in debug-fs2/etc/init.d: run-postinsts
diff -r --no-dereference debug-fs1/etc/issue debug-fs2/etc/issue
1c1
< Poky (Yocto Project Reference Distro) 4.3+snapshot-
bc66d6ea4b3a6e6c9131295ee0783b88b5a90a02 \n \l
---
> Poky (Yocto Project Reference Distro) 4.3+snapshot-
44ec9356d2d8686189531b55b2bd2b268b5dffa3 \n \l
diff -r --no-dereference debug-fs1/etc/issue.net debug-
fs2/etc/issue.net
1c1
< Poky (Yocto Project Reference Distro) 4.3+snapshot-
bc66d6ea4b3a6e6c9131295ee0783b88b5a90a02 %h
---
> Poky (Yocto Project Reference Distro) 4.3+snapshot-
44ec9356d2d8686189531b55b2bd2b268b5dffa3 %h
Only in debug-fs1/etc: ld.so.cache
Only in debug-fs2/etc/rcS.d: S99run-postinsts
Only in debug-fs1/etc: timestamp
Only in debug-fs1/etc: version
Only in debug-fs2/usr/sbin: run-postinsts
Only in debug-fs2/var/volatile: log
Only in debug-fs2/var/volatile: tmp

Best reagrds,
Adrian

> Cheers,
> 
> Richard
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#190217):
> https://lists.openembedded.org/g/openembedded-core/message/190217
> Mute This Topic: https://lists.openembedded.org/mt/102316026/4454582
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe:
> https://lists.openembedded.org/g/openembedded-core/unsub [
> adrian.freihofer@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [OE-core] [PATCH v8 3/8] image-combined-dbg: make this the default

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 5855 bytes --]

Can you explain your issues with gdb with using a rootfs-dbg, because I don’t see any problem with a filesystem that only contains debug symbols, given how easily gdb can be configured to look into alternate paths. I’ll admit I also don’t know the difference between these modes, and also wonder how IMAGE_GEN_DEBUGFS relates.

--
Christopher Larson
chris_larson@mentor.com, chris.larson@siemens.com, kergoth@gmail.com
Principal Software Engineer, Embedded Linux Solutions, Siemens Digital Industries Software
On Nov 15, 2023 at 7:21 AM -0700, Adrian Freihofer <adrian.freihofer@gmail.com>, wrote:
> On Mon, 2023-11-06 at 13:50 +0000, Richard Purdie wrote:
> > On Wed, 2023-11-01 at 12:01 +0100, Adrian Freihofer wrote:
> > > Remove the image-combined-dbg.bbclass and make this the default
> > > behavior for the rootfs-dbg. A rootfs-dbg with only debug symbols
> > > but
> > > no executable binaries also causes problems with gdb, which is
> > > probably the most common use case for the roofs-dbg. This change
> > > simplifies and improves the user experience for a slightly larger
> > > rootfs-dbg.
> > >
> > > If the rootfs-dbg contains a complete copy of the rootfs, it is
> > > also
> > > usable for booting the target device over the network. This in turn
> > > simplifies other use cases with e.g. the use of perf on a device
> > > booted over the network.
> > >
> > > Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> >
> > I'm being pressured for review on this so I'll just say what the
> > problem is. I struggle to review this as off the top of my head, I
> > can't remember the difference between "rootfs-dbg" or "image-
> > combined-
> > dbg". I understand the patch gets rid of one and the argument appears
> > to be that gdb doesn't work well with the case that is removed.
> >
> > What isn't here is any reminder of what the differences are, or a
> > pointer to the history which lead us to have two different modes in
> > the
> > first place. We presumably had a reason for adding it.
> >
> > That means in order to review it, I'd have to dig into the history
> > and
> > work out the differences, then work out why we added the two modes
> > and
> > then determine if they're still needed.
> >
> > What would help speed up review would be a pointer to the original
> > commits and/or a summary of why the were added. A summary of the
> > differences between the two modes would also help/
> >
>
> We have tried to work through the git history and to search for use
> cases that rely on the old implementation. The git history does not
> explain why it is like it is.
>
> Enguerrand came up with theoretical use cases where the rootfs-dbg
> could be mounted somehow as overlay at run-time. I can't rule out the
> possibility of someone doing something like that.
>
> Maybe we should keep that as it is.
>
> @Ross: do you have a different opinion regarding the discussion we had:
> https://lists.openembedded.org/g/openembedded-core/message/188995
>
>
> Here are my notes. Just in case we would like to come back to this some
> when.
>
> - 7a7c6b021f114c6bedfbdc9afd2bf2925b238d19:
> The first implementation of rootfs-dbg only added the *-dbg packages
> and the package database.
> - 69d3df9169133d5e05eff25019569fb8974d48c2
> - c1ce0d9a9e200e35a9b6f9d537232875683ab9f1
> - e73a85be3e020561db92a197c593afe7fd952919
> - 1800b1ba7ae07cd010e529653eaa1d7a5841e6e5
> Added some workarounds related to opkg and openssl which got finally
> reverted again.
> - 364c4c7d3fc049f2b734a6a235758a75af364ce0
> Support for PACKAGE_DEBUG_SPLIT_STYLE= 'debug-with-srcpkg'
> - 5f30534c47571d199d8655b3da51f7b55fe20c04
> Introduce IMAGE_INSTALL_DEBUGFS
> - 18f080fbe4cf51824e5f1d73a10e06e3a5724423
> Removes the package database later, as it caused problems with RPMs.
>
> Comparison of
> $ tar xvfj core-image-minimal-qemux86-64.rootfs.tar.bz2
> $ tar xvfj core-image-minimal-qemux86-64.rootfs-dbg.tar.bz2
> without this patch verus
> $ tar xvfj core-image-minimal-qemux86-64.rootfs-dbg.tar.bz2
> with this patch applied shows some differences. This is expected
> because
> some of the post rootfs steps run after rootfs gets copied to
> rootfs-dbg. Changing this would require more changes.
>
> $ diff -r --no-dereference debug-fs1 debug-fs2
> Only in debug-fs1/etc/default: postinst
> Only in debug-fs2/etc/init.d: run-postinsts
> diff -r --no-dereference debug-fs1/etc/issue debug-fs2/etc/issue
> 1c1
> < Poky (Yocto Project Reference Distro) 4.3+snapshot-
> bc66d6ea4b3a6e6c9131295ee0783b88b5a90a02 \n \l
> ---
> > Poky (Yocto Project Reference Distro) 4.3+snapshot-
> 44ec9356d2d8686189531b55b2bd2b268b5dffa3 \n \l
> diff -r --no-dereference debug-fs1/etc/issue.net debug-
> fs2/etc/issue.net
> 1c1
> < Poky (Yocto Project Reference Distro) 4.3+snapshot-
> bc66d6ea4b3a6e6c9131295ee0783b88b5a90a02 %h
> ---
> > Poky (Yocto Project Reference Distro) 4.3+snapshot-
> 44ec9356d2d8686189531b55b2bd2b268b5dffa3 %h
> Only in debug-fs1/etc: ld.so.cache
> Only in debug-fs2/etc/rcS.d: S99run-postinsts
> Only in debug-fs1/etc: timestamp
> Only in debug-fs1/etc: version
> Only in debug-fs2/usr/sbin: run-postinsts
> Only in debug-fs2/var/volatile: log
> Only in debug-fs2/var/volatile: tmp
>
> Best reagrds,
> Adrian
>
> > Cheers,
> >
> > Richard
> >
> >
> >
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#190602): https://lists.openembedded.org/g/openembedded-core/message/190602
> Mute This Topic: https://lists.openembedded.org/mt/102316026/3617123
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [kergoth@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

[-- Attachment #2: Type: text/html, Size: 7300 bytes --]

Re: [OE-core] [PATCH v8 3/8] image-combined-dbg: make this the default

[adrian.freihofer]

On Wed, 2023-11-15 at 09:26 -0700, Christopher Larson wrote:
> 
> Can you explain your issues with gdb with using a rootfs-dbg, because
> I don’t see any problem with a filesystem that only contains debug
> symbols, given how easily gdb can be configured to look into
> alternate paths. I’ll admit I also don’t know the difference between
> these modes, and also wonder how IMAGE_GEN_DEBUGFS relates.
> 
> 
It simply does not find the symbols if only the rootfs-dbg is
available. If this class 
https://git.yoctoproject.org/poky/tree/meta/classes-recipe/image-combined-dbg.bbclass
is added to the build it just works.

Since this is not too obvious, I wondered if it wouldn't be better if
the combined debugfs were the default behavior. Then Ross asked the
same question and I developed a patch to simplify this.

But either way, there are too many unanswered questions to change a
well-established default behavior. And there are other ways to deal
with it.
Thank you for pointing this out.

RE: [OE-core] [PATCH v8 3/8] image-combined-dbg: make this the default

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Adrian Freihofer
> Sent: den 15 november 2023 23:33
> To: Christopher Larson <kergoth@gmail.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH v8 3/8] image-combined-dbg: make this the
> default
> 
> On Wed, 2023-11-15 at 09:26 -0700, Christopher Larson wrote:
> >
> > Can you explain your issues with gdb with using a rootfs-dbg, because
> > I don’t see any problem with a filesystem that only contains debug
> > symbols, given how easily gdb can be configured to look into
> > alternate paths. I’ll admit I also don’t know the difference between
> > these modes, and also wonder how IMAGE_GEN_DEBUGFS relates.
> >
> >
> It simply does not find the symbols if only the rootfs-dbg is
> available. If this class
> https://git.yoctoproject.org/poky/tree/meta/classes-recipe/image-combined-dbg.bbclass
> is added to the build it just works.
> 
> Since this is not too obvious, I wondered if it wouldn't be better if
> the combined debugfs were the default behavior. Then Ross asked the
> same question and I developed a patch to simplify this.
> 
> But either way, there are too many unanswered questions to change a
> well-established default behavior. And there are other ways to deal with it.
> Thank you for pointing this out.

For what it's worth, we have local changes that more or less do this 
(predating image-combined-dbg.bbclass), as we have only seen a use 
case for the combined debugfs tar ball. So I was rather hoping this 
would be accepted, so that we can drop our local changes...

//Peter

Re: [PATCH v8 3/8] image-combined-dbg: make this the default

[Enguerrand de Ribaucourt]

While the debug-only rootfs is not usable by itself, changing the way it's currently combined (from the doc) could produce unintended side effects.
Check out this post for explanations: https://lists.openembedded.org/g/openembedded-core/message/190490

Re: [OE-core] [PATCH v8 3/8] image-combined-dbg: make this the default

[adrian.freihofer]

On Mon, 2023-11-20 at 22:53 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-
> > core@lists.openembedded.org> On Behalf Of Adrian Freihofer
> > Sent: den 15 november 2023 23:33
> > To: Christopher Larson <kergoth@gmail.com>
> > Cc: openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH v8 3/8] image-combined-dbg: make this
> > the
> > default
> > 
> > On Wed, 2023-11-15 at 09:26 -0700, Christopher Larson wrote:
> > > 
> > > Can you explain your issues with gdb with using a rootfs-dbg,
> > > because
> > > I don’t see any problem with a filesystem that only contains
> > > debug
> > > symbols, given how easily gdb can be configured to look into
> > > alternate paths. I’ll admit I also don’t know the difference
> > > between
> > > these modes, and also wonder how IMAGE_GEN_DEBUGFS relates.
> > > 
> > > 
> > It simply does not find the symbols if only the rootfs-dbg is
> > available. If this class
> > https://git.yoctoproject.org/poky/tree/meta/classes-recipe/image-combined-dbg.bbclass
> > is added to the build it just works.
> > 
> > Since this is not too obvious, I wondered if it wouldn't be better
> > if
> > the combined debugfs were the default behavior. Then Ross asked the
> > same question and I developed a patch to simplify this.
> > 
> > But either way, there are too many unanswered questions to change a
> > well-established default behavior. And there are other ways to deal
> > with it.
> > Thank you for pointing this out.
> 
> For what it's worth, we have local changes that more or less do this 
> (predating image-combined-dbg.bbclass), as we have only seen a use 
> case for the combined debugfs tar ball. So I was rather hoping this 
> would be accepted, so that we can drop our local changes...
> 
> //Peter
> 

In theory, I completely agree with Christopher. That seems to be the
right way to go. But I have tested this again. I am not able to
configure GDB to find the sources with a split rootf + rootfs-dbg. I
tried adding rootfs and rootfs-dbg in different order and with
different paths to the solib search path. Stepping into libraries
contained in rootfs/rootfs-dbg does not work. Maybe I did something
wrong or we could consider this a bug in GDB. But with a combined
rootfs-dbg this works in all the different variants I have tried.

One approach to handle this is to improve GDB's solib search function
to support a shared rootfs dbg. But it's not only about GDB. There are
other tools which most probably have similar issues. 

From a usability point of view, I would at least like to have an easy
and obvious way to configure a combined rootfs-dbg for an image. It
took me an infinite amount of time to figure out why GDB doesn't work
and that there is an image-combined-dbg.bbclass supposed to fix that. 
Defaulting to a not combined rootfs-dbg and providing code in a hidden
bbclass is misleading.

What could also help a bit is to make the code of the image-combined-
dbg.bbclass more prominent and more official e.g. by moving it into the
image.bbclass and support to enable it via variable which can be
documented and referred in various sections of the documentation but
also in the local.conf.template. The documentation should then point
out that e.g. stepping with GDB does not work until the binaries are
available from the same rootfs folder as the debug symbols are. Such a
patch would be a minor refactoring only. @Peter: Would this help for
your use case?

This would also allow the default to be changed some when in the
future. The question of who needs a shared dbg rootfs for which use
case, as opposed to those who struggle with simple remote debugging
because of this default behavior, still seems valid to me.

But I also have to say that recent documentation points out that the
rootfs.tar must be extracted into the extracted rootfs-dbg which is a
working solution. The issue which I have is with the SDK which tries to
avoid building the tars and extracting them again just for providing
the debug symbols on the host. This use case is probably new.

Best regards,
Adrian

Re: [OE-core] [PATCH v8 3/8] image-combined-dbg: make this the default

[Alexander Kanavin]

On Wed, 22 Nov 2023 at 13:23, Adrian Freihofer
<adrian.freihofer@gmail.com> wrote:

> But I also have to say that recent documentation points out that the
> rootfs.tar must be extracted into the extracted rootfs-dbg which is a
> working solution. The issue which I have is with the SDK which tries to
> avoid building the tars and extracting them again just for providing
> the debug symbols on the host. This use case is probably new.

Sorry for barging in without fully understanding the issue, but I've
added debuginfod support to yocto precisely to avoid having to deal
with large, awkward, slow debug filesystems. It's a lot leaner and
slimmer approach.

Alex

Re: [OE-core] [PATCH v8 3/8] image-combined-dbg: make this the default

[adrian.freihofer]

On Wed, 2023-11-22 at 13:25 +0100, Alexander Kanavin wrote:
> On Wed, 22 Nov 2023 at 13:23, Adrian Freihofer
> <adrian.freihofer@gmail.com> wrote:
> 
> > But I also have to say that recent documentation points out that
> > the
> > rootfs.tar must be extracted into the extracted rootfs-dbg which is
> > a
> > working solution. The issue which I have is with the SDK which
> > tries to
> > avoid building the tars and extracting them again just for
> > providing
> > the debug symbols on the host. This use case is probably new.
> 
> Sorry for barging in without fully understanding the issue, but I've
> added debuginfod support to yocto precisely to avoid having to deal
> with large, awkward, slow debug filesystems. It's a lot leaner and
> slimmer approach.
> 
Thank you Alex. Looking at that is already on my todo list. I think is
is a very interesting approach.
But I hesitated a bit because creating some files seems to be easier
than starting and stopping a daemon behind an IDE. And the fact that
debuginfod is available probably doesn't mean that rootfs-dbg is no
longer supported. So let's just get started and then improve.

Adrian

[PATCH v8 4/8] tests: add a C++ example recipe

[Adrian Freihofer]

This simple C++ project supports compilation with cmake, meson and
autotools.
It's supposed to be used with oe-selftest for the devtool ide plugin.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 meta-selftest/recipes-test/cpp/.gitignore     |  1 +
 .../recipes-test/cpp/autotools-example.bb     | 18 ++++++
 .../cpp/autotools-example/run-ptest           | 10 ++++
 .../recipes-test/cpp/cmake-example.bb         | 17 ++++++
 .../recipes-test/cpp/cmake-example/run-ptest  | 10 ++++
 .../recipes-test/cpp/cpp-example.inc          | 24 ++++++++
 .../recipes-test/cpp/files/CMakeLists.txt     | 60 +++++++++++++++++++
 .../recipes-test/cpp/files/Makefile.am        | 13 ++++
 .../recipes-test/cpp/files/configure.ac       | 11 ++++
 .../cpp/files/cpp-example-lib.cpp             | 17 ++++++
 .../cpp/files/cpp-example-lib.hpp             | 16 +++++
 .../recipes-test/cpp/files/cpp-example.cpp    | 16 +++++
 .../recipes-test/cpp/files/meson.build        | 34 +++++++++++
 .../cpp/files/test-cpp-example.cpp            | 20 +++++++
 .../recipes-test/cpp/meson-example.bb         | 17 ++++++
 .../recipes-test/cpp/meson-example/run-ptest  | 10 ++++
 16 files changed, 294 insertions(+)
 create mode 100644 meta-selftest/recipes-test/cpp/.gitignore
 create mode 100644 meta-selftest/recipes-test/cpp/autotools-example.bb
 create mode 100644 meta-selftest/recipes-test/cpp/autotools-example/run-ptest
 create mode 100644 meta-selftest/recipes-test/cpp/cmake-example.bb
 create mode 100644 meta-selftest/recipes-test/cpp/cmake-example/run-ptest
 create mode 100644 meta-selftest/recipes-test/cpp/cpp-example.inc
 create mode 100644 meta-selftest/recipes-test/cpp/files/CMakeLists.txt
 create mode 100644 meta-selftest/recipes-test/cpp/files/Makefile.am
 create mode 100644 meta-selftest/recipes-test/cpp/files/configure.ac
 create mode 100644 meta-selftest/recipes-test/cpp/files/cpp-example-lib.cpp
 create mode 100644 meta-selftest/recipes-test/cpp/files/cpp-example-lib.hpp
 create mode 100644 meta-selftest/recipes-test/cpp/files/cpp-example.cpp
 create mode 100644 meta-selftest/recipes-test/cpp/files/meson.build
 create mode 100644 meta-selftest/recipes-test/cpp/files/test-cpp-example.cpp
 create mode 100644 meta-selftest/recipes-test/cpp/meson-example.bb
 create mode 100644 meta-selftest/recipes-test/cpp/meson-example/run-ptest

diff --git a/meta-selftest/recipes-test/cpp/.gitignore b/meta-selftest/recipes-test/cpp/.gitignore
new file mode 100644
index 00000000000..30d388a12b7
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/.gitignore
@@ -0,0 +1 @@
+build*
\ No newline at end of file
diff --git a/meta-selftest/recipes-test/cpp/autotools-example.bb b/meta-selftest/recipes-test/cpp/autotools-example.bb
new file mode 100644
index 00000000000..f5d8aa48154
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/autotools-example.bb
@@ -0,0 +1,18 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+SUMMARY = "A C++ example compiled with autotools."
+
+inherit autotools
+
+require cpp-example.inc
+
+SRC_URI += "\
+    file://configure.ac \
+    file://Makefile.am \
+"
+
+FILES:${PN}-ptest += "${bindir}/test-autotools-example"
diff --git a/meta-selftest/recipes-test/cpp/autotools-example/run-ptest b/meta-selftest/recipes-test/cpp/autotools-example/run-ptest
new file mode 100644
index 00000000000..51548259886
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/autotools-example/run-ptest
@@ -0,0 +1,10 @@
+#!/bin/sh
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+testautoexample
+
+# Note: run-ptests exits with exit value from test-cmake-example
diff --git a/meta-selftest/recipes-test/cpp/cmake-example.bb b/meta-selftest/recipes-test/cpp/cmake-example.bb
new file mode 100644
index 00000000000..96d543180b4
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/cmake-example.bb
@@ -0,0 +1,17 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+SUMMARY = "A C++ example compiled with cmake."
+
+inherit cmake
+
+require cpp-example.inc
+
+SRC_URI += "\
+    file://CMakeLists.txt \
+"
+
+FILES:${PN}-ptest += "${bindir}/test-cmake-example"
diff --git a/meta-selftest/recipes-test/cpp/cmake-example/run-ptest b/meta-selftest/recipes-test/cpp/cmake-example/run-ptest
new file mode 100644
index 00000000000..94b620a1984
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/cmake-example/run-ptest
@@ -0,0 +1,10 @@
+#!/bin/sh
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+test-cmake-example
+
+# Note: run-ptests exits with exit value from test-cmake-example
diff --git a/meta-selftest/recipes-test/cpp/cpp-example.inc b/meta-selftest/recipes-test/cpp/cpp-example.inc
new file mode 100644
index 00000000000..39c61cf4ceb
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/cpp-example.inc
@@ -0,0 +1,24 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+inherit ptest
+
+DEPENDS += "json-c"
+
+PV = "1.0"
+
+S = "${WORKDIR}"
+
+SRC_URI = "\
+    file://cpp-example.cpp \
+    file://cpp-example-lib.hpp \
+    file://cpp-example-lib.cpp \
+    file://test-cpp-example.cpp \
+    file://run-ptest \
+"
diff --git a/meta-selftest/recipes-test/cpp/files/CMakeLists.txt b/meta-selftest/recipes-test/cpp/files/CMakeLists.txt
new file mode 100644
index 00000000000..839aa59b5e3
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/files/CMakeLists.txt
@@ -0,0 +1,60 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+cmake_minimum_required(VERSION 3.22)
+
+project(cmake-example
+  VERSION 1.0.0
+  LANGUAGES CXX
+)
+
+option(BUILD_SHARED_LIBS "Build using shared libraries" ON)
+
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD_REQUIRED On)
+set(CMAKE_CXX_EXTENSIONS Off)
+
+include(GNUInstallDirs)
+
+# Find json-c
+# find_package(PkgConfig REQUIRED)
+# pkg_check_modules(JSONC REQUIRED json-c)
+find_package(json-c)
+
+# A simple library linking json-c library found by pkgconfig
+add_library(cmake-example-lib cpp-example-lib.cpp cpp-example-lib.hpp)
+set_target_properties(cmake-example-lib PROPERTIES 
+    VERSION ${PROJECT_VERSION}
+    SOVERSION ${PROJECT_VERSION_MAJOR}
+)
+target_link_libraries(cmake-example-lib PRIVATE json-c::json-c)
+# target_link_libraries(cmake-example-lib ${JSONC_LIBRARIES})
+# target_include_directories(cmake-example-lib PUBLIC ${JSONC_INCLUDE_DIRS})
+# target_compile_options(cmake-example-lib PUBLIC ${JSONC_CFLAGS_OTHER})
+install(TARGETS cmake-example-lib
+    INCLUDES DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
+    ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
+    LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
+)
+
+# A simple executable linking the library
+add_executable(cmake-example cpp-example.cpp)
+target_link_libraries(cmake-example PRIVATE cmake-example-lib)
+
+install(TARGETS cmake-example
+    RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
+)
+
+# A simple test executable for testing the library
+add_executable(test-cmake-example test-cpp-example.cpp)
+target_link_libraries(test-cmake-example PRIVATE cmake-example-lib)
+
+install(TARGETS test-cmake-example
+    RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
+)
+
+include(CTest)
+add_test(NAME test-cmake-example COMMAND test-cmake-example)
diff --git a/meta-selftest/recipes-test/cpp/files/Makefile.am b/meta-selftest/recipes-test/cpp/files/Makefile.am
new file mode 100644
index 00000000000..39cff91d2ac
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/files/Makefile.am
@@ -0,0 +1,13 @@
+noinst_LIBRARIES = libautoexample.a
+libautoexample_a_SOURCES = cpp-example-lib.cpp
+
+bin_PROGRAMS = autoexample testautoexample
+
+autoexample_SOURCES = cpp-example.cpp
+autoexample_LDADD = libautoexample.a
+
+testautoexample_SOURCES = test-cpp-example.cpp
+testautoexample_LDADD = libautoexample.a
+TESTS=testautoexample
+
+AM_LDFLAGS = -ljson-c
diff --git a/meta-selftest/recipes-test/cpp/files/configure.ac b/meta-selftest/recipes-test/cpp/files/configure.ac
new file mode 100644
index 00000000000..29c6d1ca6fc
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/files/configure.ac
@@ -0,0 +1,11 @@
+
+AC_INIT([autoexample], [1.0])
+AM_INIT_AUTOMAKE([-Wall -Werror foreign])
+AC_PROG_CXX
+AX_CXX_COMPILE_STDCXX_17
+AC_PROG_RANLIB
+AM_PROG_AR
+AC_CONFIG_FILES([
+    Makefile
+])
+AC_OUTPUT
diff --git a/meta-selftest/recipes-test/cpp/files/cpp-example-lib.cpp b/meta-selftest/recipes-test/cpp/files/cpp-example-lib.cpp
new file mode 100644
index 00000000000..29fd10fb7e1
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/files/cpp-example-lib.cpp
@@ -0,0 +1,17 @@
+/*
+* Copyright OpenEmbedded Contributors
+*
+* SPDX-License-Identifier: MIT
+*/
+
+#include <string>
+#include <json-c/json.h>
+#include "cpp-example-lib.hpp"
+
+const std::string& CppExample::get_string() {
+    return test_string;
+}
+
+const char* CppExample::get_json_c_version() {
+    return json_c_version();
+}
diff --git a/meta-selftest/recipes-test/cpp/files/cpp-example-lib.hpp b/meta-selftest/recipes-test/cpp/files/cpp-example-lib.hpp
new file mode 100644
index 00000000000..8a87e6b510a
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/files/cpp-example-lib.hpp
@@ -0,0 +1,16 @@
+/*
+* Copyright OpenEmbedded Contributors
+*
+* SPDX-License-Identifier: MIT
+*/
+
+#pragma once
+
+#include <string>
+
+struct CppExample {
+    inline static const std::string test_string = "cpp-example-lib Magic: 123456789";
+
+    const std::string& get_string();
+    const char* get_json_c_version();
+};
diff --git a/meta-selftest/recipes-test/cpp/files/cpp-example.cpp b/meta-selftest/recipes-test/cpp/files/cpp-example.cpp
new file mode 100644
index 00000000000..de7f2106dec
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/files/cpp-example.cpp
@@ -0,0 +1,16 @@
+/*
+* Copyright OpenEmbedded Contributors
+*
+* SPDX-License-Identifier: MIT
+*/
+
+#include "cpp-example-lib.hpp"
+
+#include <iostream>
+
+int main() {
+    auto cpp_example = CppExample();
+    std::cout << "C++ example linking " << cpp_example.get_string() << std::endl;
+    std::cout << "Linking json-c version " << cpp_example.get_json_c_version() << std::endl;
+    return 0;
+}
diff --git a/meta-selftest/recipes-test/cpp/files/meson.build b/meta-selftest/recipes-test/cpp/files/meson.build
new file mode 100644
index 00000000000..7753da69721
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/files/meson.build
@@ -0,0 +1,34 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+project('meson-example', 'cpp',
+    version: '1.0.0',
+    default_options: ['cpp_std=c++17']
+    )
+
+
+jsoncdep = dependency('json-c')
+
+mesonexlib = shared_library('mesonexlib',
+    'cpp-example-lib.cpp', 'cpp-example-lib.hpp',
+	version: meson.project_version(),
+	soversion: meson.project_version().split('.')[0],
+    dependencies : jsoncdep,
+    install : true
+    )
+
+executable('mesonex',
+    'cpp-example.cpp',
+    link_with : mesonexlib,
+    install : true
+    )
+
+test_mesonex = executable('test-mesonex',
+    'test-cpp-example.cpp',
+    link_with : mesonexlib,
+    install : true
+)
+test('meson example test', test_mesonex)
diff --git a/meta-selftest/recipes-test/cpp/files/test-cpp-example.cpp b/meta-selftest/recipes-test/cpp/files/test-cpp-example.cpp
new file mode 100644
index 00000000000..f9257e1aa2f
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/files/test-cpp-example.cpp
@@ -0,0 +1,20 @@
+/*
+* Copyright OpenEmbedded Contributors
+*
+* SPDX-License-Identifier: MIT
+*/
+
+#include "cpp-example-lib.hpp"
+
+#include <iostream>
+
+int main() {
+    auto cpp_example = CppExample();
+    auto ret_string = cpp_example.get_string();
+    if(0 == ret_string.compare(CppExample::test_string)) {
+        std::cout << "PASS: " << ret_string << " = " << CppExample::test_string << std::endl;
+    } else {
+        std::cout << "FAIL: " << ret_string << " != " << CppExample::test_string << std::endl;
+        return 1;
+    }
+}
diff --git a/meta-selftest/recipes-test/cpp/meson-example.bb b/meta-selftest/recipes-test/cpp/meson-example.bb
new file mode 100644
index 00000000000..f15dbb50731
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/meson-example.bb
@@ -0,0 +1,17 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+SUMMARY = "A C++ example compiled with meson."
+
+inherit pkgconfig meson
+
+require cpp-example.inc
+
+SRC_URI += "\
+    file://meson.build \
+"
+
+FILES:${PN}-ptest += "${bindir}/test-mesonex"
diff --git a/meta-selftest/recipes-test/cpp/meson-example/run-ptest b/meta-selftest/recipes-test/cpp/meson-example/run-ptest
new file mode 100644
index 00000000000..b1804f00961
--- /dev/null
+++ b/meta-selftest/recipes-test/cpp/meson-example/run-ptest
@@ -0,0 +1,10 @@
+#!/bin/sh
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+test-mesonex
+
+# Note: run-ptests exits with exit value from test-mesonex
-- 
2.41.0

[PATCH v8 5/8] devtool: refactor deploy-target

[Adrian Freihofer]

Make the deploy function independent from d. This allows to call the
function also from Python code not running in bitbake.
This is needed to for the devtool ide plugin which will call the
do_install task and the code from devtool deploy-target independently
from a bitbake server. This allows a much quicker workflow.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 scripts/lib/devtool/__init__.py |   5 +-
 scripts/lib/devtool/deploy.py   | 230 +++++++++++++++++---------------
 2 files changed, 124 insertions(+), 111 deletions(-)

diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 702db669de3..7d64547616e 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -78,12 +78,15 @@ def exec_fakeroot(d, cmd, **kwargs):
     """Run a command under fakeroot (pseudo, in fact) so that it picks up the appropriate file permissions"""
     # Grab the command and check it actually exists
     fakerootcmd = d.getVar('FAKEROOTCMD')
+    fakerootenv = d.getVar('FAKEROOTENV')
+    exec_fakeroot_no_d(fakerootcmd, fakerootenv, cmd, kwargs)
+
+def exec_fakeroot_no_d(fakerootcmd, fakerootenv, cmd, **kwargs):
     if not os.path.exists(fakerootcmd):
         logger.error('pseudo executable %s could not be found - have you run a build yet? pseudo-native should install this and if you have run any build then that should have been built')
         return 2
     # Set up the appropriate environment
     newenv = dict(os.environ)
-    fakerootenv = d.getVar('FAKEROOTENV')
     for varvalue in fakerootenv.split():
         if '=' in varvalue:
             splitval = varvalue.split('=', 1)
diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py
index e14a5874177..ea7e2cb1ae8 100644
--- a/scripts/lib/devtool/deploy.py
+++ b/scripts/lib/devtool/deploy.py
@@ -16,7 +16,7 @@ import bb.utils
 import argparse_oe
 import oe.types
 
-from devtool import exec_fakeroot, setup_tinfoil, check_workspace_recipe, DevtoolError
+from devtool import exec_fakeroot_no_d, setup_tinfoil, check_workspace_recipe, DevtoolError
 
 logger = logging.getLogger('devtool')
 
@@ -133,16 +133,11 @@ def _prepare_remote_script(deploy, verbose=False, dryrun=False, undeployall=Fals
 
     return '\n'.join(lines)
 
-
-
-def deploy(args, config, basepath, workspace):
-    """Entry point for the devtool 'deploy' subcommand"""
+def deploy_cached(srcdir, workdir, path, strip_cmd, libdir, base_libdir, max_process, fakerootcmd, fakerootenv, args):
     import math
     import oe.recipeutils
     import oe.package
 
-    check_workspace_recipe(workspace, args.recipename, checksrc=False)
-
     try:
         host, destdir = args.target.split(':')
     except ValueError:
@@ -152,116 +147,131 @@ def deploy(args, config, basepath, workspace):
     if not destdir.endswith('/'):
         destdir += '/'
 
+    recipe_outdir = srcdir
+    if not os.path.exists(recipe_outdir) or not os.listdir(recipe_outdir):
+        raise DevtoolError('No files to deploy - have you built the %s '
+                        'recipe? If so, the install step has not installed '
+                        'any files.' % args.recipename)
+
+    if args.strip and not args.dry_run:
+        # Fakeroot copy to new destination
+        srcdir = recipe_outdir
+        recipe_outdir = os.path.join(workdir, 'devtool-deploy-target-stripped')
+        if os.path.isdir(recipe_outdir):
+            exec_fakeroot_no_d(fakerootcmd, fakerootenv, "rm -rf %s" % recipe_outdir, shell=True)
+        exec_fakeroot_no_d(fakerootcmd, fakerootenv, "cp -af %s %s" % (os.path.join(srcdir, '.'), recipe_outdir), shell=True)
+        os.environ['PATH'] = ':'.join([os.environ['PATH'], path or ''])
+        oe.package.strip_execs(args.recipename, recipe_outdir, strip_cmd, libdir, base_libdir, max_process)
+
+    filelist = []
+    inodes = set({})
+    ftotalsize = 0
+    for root, _, files in os.walk(recipe_outdir):
+        for fn in files:
+            fstat = os.lstat(os.path.join(root, fn))
+            # Get the size in kiB (since we'll be comparing it to the output of du -k)
+            # MUST use lstat() here not stat() or getfilesize() since we don't want to
+            # dereference symlinks
+            if fstat.st_ino in inodes:
+                fsize = 0
+            else:
+                fsize = int(math.ceil(float(fstat.st_size)/1024))
+            inodes.add(fstat.st_ino)
+            ftotalsize += fsize
+            # The path as it would appear on the target
+            fpath = os.path.join(destdir, os.path.relpath(root, recipe_outdir), fn)
+            filelist.append((fpath, fsize))
+
+    if args.dry_run:
+        print('Files to be deployed for %s on target %s:' % (args.recipename, args.target))
+        for item, _ in filelist:
+            print('  %s' % item)
+        return 0
+
+    extraoptions = ''
+    if args.no_host_check:
+        extraoptions += '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
+    if not args.show_status:
+        extraoptions += ' -q'
+
+    scp_sshexec = ''
+    ssh_sshexec = 'ssh'
+    if args.ssh_exec:
+        scp_sshexec = "-S %s" % args.ssh_exec
+        ssh_sshexec = args.ssh_exec
+    scp_port = ''
+    ssh_port = ''
+    if args.port:
+        scp_port = "-P %s" % args.port
+        ssh_port = "-p %s" % args.port
+
+    if args.key:
+        extraoptions += ' -i %s' % args.key
+
+    # In order to delete previously deployed files and have the manifest file on
+    # the target, we write out a shell script and then copy it to the target
+    # so we can then run it (piping tar output to it).
+    # (We cannot use scp here, because it doesn't preserve symlinks.)
+    tmpdir = tempfile.mkdtemp(prefix='devtool')
+    try:
+        tmpscript = '/tmp/devtool_deploy.sh'
+        tmpfilelist = os.path.join(os.path.dirname(tmpscript), 'devtool_deploy.list')
+        shellscript = _prepare_remote_script(deploy=True,
+                                            verbose=args.show_status,
+                                            nopreserve=args.no_preserve,
+                                            nocheckspace=args.no_check_space)
+        # Write out the script to a file
+        with open(os.path.join(tmpdir, os.path.basename(tmpscript)), 'w') as f:
+            f.write(shellscript)
+        # Write out the file list
+        with open(os.path.join(tmpdir, os.path.basename(tmpfilelist)), 'w') as f:
+            f.write('%d\n' % ftotalsize)
+            for fpath, fsize in filelist:
+                f.write('%s %d\n' % (fpath, fsize))
+        # Copy them to the target
+        ret = subprocess.call("scp %s %s %s %s/* %s:%s" % (scp_sshexec, scp_port, extraoptions, tmpdir, args.target, os.path.dirname(tmpscript)), shell=True)
+        if ret != 0:
+            raise DevtoolError('Failed to copy script to %s - rerun with -s to '
+                            'get a complete error message' % args.target)
+    finally:
+        shutil.rmtree(tmpdir)
+
+    # Now run the script
+    ret = exec_fakeroot_no_d(fakerootcmd, fakerootenv, 'tar cf - . | %s  %s %s %s \'sh %s %s %s %s\'' % (ssh_sshexec, ssh_port, extraoptions, args.target, tmpscript, args.recipename, destdir, tmpfilelist), cwd=recipe_outdir, shell=True)
+    if ret != 0:
+        raise DevtoolError('Deploy failed - rerun with -s to get a complete '
+                        'error message')
+
+    logger.info('Successfully deployed %s' % recipe_outdir)
+
+    files_list = []
+    for root, _, files in os.walk(recipe_outdir):
+        for filename in files:
+            filename = os.path.relpath(os.path.join(root, filename), recipe_outdir)
+            files_list.append(os.path.join(destdir, filename))
+
+def deploy(args, config, basepath, workspace):
+    """Entry point for the devtool 'deploy' subcommand"""
+    check_workspace_recipe(workspace, args.recipename, checksrc=False)
+
     tinfoil = setup_tinfoil(basepath=basepath)
     try:
         try:
             rd = tinfoil.parse_recipe(args.recipename)
+
+            srcdir = rd.getVar('D')
+            workdir = rd.getVar('WORKDIR')
+            path = rd.getVar('PATH')
+            strip_cmd = rd.getVar('STRIP')
+            libdir = rd.getVar('libdir'),
+            base_libdir = rd.getVar('base_libdir')
+            max_process = int(rd.getVar("BB_NUMBER_THREADS") or os.cpu_count() or 1)
+            fakerootcmd = rd.getVar('FAKEROOTCMD')
+            fakerootenv = rd.getVar('FAKEROOTENV')
+            deploy_cached(srcdir, workdir, path, strip_cmd, libdir, base_libdir, max_process, fakerootcmd, fakerootenv, args)
         except Exception as e:
             raise DevtoolError('Exception parsing recipe %s: %s' %
                             (args.recipename, e))
-        recipe_outdir = rd.getVar('D')
-        if not os.path.exists(recipe_outdir) or not os.listdir(recipe_outdir):
-            raise DevtoolError('No files to deploy - have you built the %s '
-                            'recipe? If so, the install step has not installed '
-                            'any files.' % args.recipename)
-
-        if args.strip and not args.dry_run:
-            # Fakeroot copy to new destination
-            srcdir = recipe_outdir
-            recipe_outdir = os.path.join(rd.getVar('WORKDIR'), 'devtool-deploy-target-stripped')
-            if os.path.isdir(recipe_outdir):
-                exec_fakeroot(rd, "rm -rf %s" % recipe_outdir, shell=True)
-            exec_fakeroot(rd, "cp -af %s %s" % (os.path.join(srcdir, '.'), recipe_outdir), shell=True)
-            os.environ['PATH'] = ':'.join([os.environ['PATH'], rd.getVar('PATH') or ''])
-            oe.package.strip_execs(args.recipename, recipe_outdir, rd.getVar('STRIP'), rd.getVar('libdir'),
-                        rd.getVar('base_libdir'), rd)
-
-        filelist = []
-        inodes = set({})
-        ftotalsize = 0
-        for root, _, files in os.walk(recipe_outdir):
-            for fn in files:
-                fstat = os.lstat(os.path.join(root, fn))
-                # Get the size in kiB (since we'll be comparing it to the output of du -k)
-                # MUST use lstat() here not stat() or getfilesize() since we don't want to
-                # dereference symlinks
-                if fstat.st_ino in inodes:
-                    fsize = 0
-                else:
-                    fsize = int(math.ceil(float(fstat.st_size)/1024))
-                inodes.add(fstat.st_ino)
-                ftotalsize += fsize
-                # The path as it would appear on the target
-                fpath = os.path.join(destdir, os.path.relpath(root, recipe_outdir), fn)
-                filelist.append((fpath, fsize))
-
-        if args.dry_run:
-            print('Files to be deployed for %s on target %s:' % (args.recipename, args.target))
-            for item, _ in filelist:
-                print('  %s' % item)
-            return 0
-
-        extraoptions = ''
-        if args.no_host_check:
-            extraoptions += '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
-        if not args.show_status:
-            extraoptions += ' -q'
-
-        scp_sshexec = ''
-        ssh_sshexec = 'ssh'
-        if args.ssh_exec:
-            scp_sshexec = "-S %s" % args.ssh_exec
-            ssh_sshexec = args.ssh_exec
-        scp_port = ''
-        ssh_port = ''
-        if args.port:
-            scp_port = "-P %s" % args.port
-            ssh_port = "-p %s" % args.port
-
-        if args.key:
-            extraoptions += ' -i %s' % args.key
-
-        # In order to delete previously deployed files and have the manifest file on
-        # the target, we write out a shell script and then copy it to the target
-        # so we can then run it (piping tar output to it).
-        # (We cannot use scp here, because it doesn't preserve symlinks.)
-        tmpdir = tempfile.mkdtemp(prefix='devtool')
-        try:
-            tmpscript = '/tmp/devtool_deploy.sh'
-            tmpfilelist = os.path.join(os.path.dirname(tmpscript), 'devtool_deploy.list')
-            shellscript = _prepare_remote_script(deploy=True,
-                                                verbose=args.show_status,
-                                                nopreserve=args.no_preserve,
-                                                nocheckspace=args.no_check_space)
-            # Write out the script to a file
-            with open(os.path.join(tmpdir, os.path.basename(tmpscript)), 'w') as f:
-                f.write(shellscript)
-            # Write out the file list
-            with open(os.path.join(tmpdir, os.path.basename(tmpfilelist)), 'w') as f:
-                f.write('%d\n' % ftotalsize)
-                for fpath, fsize in filelist:
-                    f.write('%s %d\n' % (fpath, fsize))
-            # Copy them to the target
-            ret = subprocess.call("scp %s %s %s %s/* %s:%s" % (scp_sshexec, scp_port, extraoptions, tmpdir, args.target, os.path.dirname(tmpscript)), shell=True)
-            if ret != 0:
-                raise DevtoolError('Failed to copy script to %s - rerun with -s to '
-                                'get a complete error message' % args.target)
-        finally:
-            shutil.rmtree(tmpdir)
-
-        # Now run the script
-        ret = exec_fakeroot(rd, 'tar cf - . | %s  %s %s %s \'sh %s %s %s %s\'' % (ssh_sshexec, ssh_port, extraoptions, args.target, tmpscript, args.recipename, destdir, tmpfilelist), cwd=recipe_outdir, shell=True)
-        if ret != 0:
-            raise DevtoolError('Deploy failed - rerun with -s to get a complete '
-                            'error message')
-
-        logger.info('Successfully deployed %s' % recipe_outdir)
-
-        files_list = []
-        for root, _, files in os.walk(recipe_outdir):
-            for filename in files:
-                filename = os.path.relpath(os.path.join(root, filename), recipe_outdir)
-                files_list.append(os.path.join(destdir, filename))
     finally:
         tinfoil.shutdown()
 
-- 
2.41.0

Re: [OE-core] [PATCH v8 5/8] devtool: refactor deploy-target

[Richard Purdie]

On Wed, 2023-11-01 at 12:01 +0100, Adrian Freihofer wrote:
> Make the deploy function independent from d. This allows to call the
> function also from Python code not running in bitbake.
> This is needed to for the devtool ide plugin which will call the
> do_install task and the code from devtool deploy-target independently
> from a bitbake server. This allows a much quicker workflow.
> 
> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> ---
>  scripts/lib/devtool/__init__.py |   5 +-
>  scripts/lib/devtool/deploy.py   | 230 +++++++++++++++++---------------
>  2 files changed, 124 insertions(+), 111 deletions(-)

This is much better than some previous versions. The question is 
whether it is easy to review and the answer is that it is still pretty
hard and we're being asked to the the changes on faith that nothing got
broken functionality wise.

For a change like this, ideally it needs to be really clear what the
actual changes are. The way to do that is probably two commits, one
moving the code into a new function but leaving it basically unchanged.
The second commit can then make the functional changes.

The other option might be to force diff to show the change differently,
not sure if that would work or not.

Cheers,

Richard

[PATCH v8 6/8] devtool: new ide plugin

[Adrian Freihofer]

The new devtool ide plugin configures an IDE to work with the eSDK.

With this initial implementation VSCode is the default IDE.
The plugin works for recipes inheriting the cmake or the meson bbclass.
Support for more programming languages and build tools may be added in
the future.

Using the plugin in recipe modes:
$ devtool modify a-recipe
$ devtool ide a-recipe a-image
$ code "$BUILDDIR/workspace/sources/a-recipe"
Work in VSCode, after installing the proposed plugins

Using the plugin without a recipe
$ devtool ide none a-image
vscode where/the/sources/are
Use the cross tool-chain which is provided as a cmake-kit.

The goal of this implementation is to create a configuration for VSCode
(or other IDEs) that allows to work on the code of a recipe completely
independent from bitbake. bitbake is only called if the configuration or
the whole SDK has to be regenerated. But bitbake should not need to be
called while working in the IDE. This has two major advantages over
calling devtool build from the IDE:
- The IDE provides plugins for integration with cmake, for example.
  These features are usable, which would not be the case if bitbake or
  devtool are called from within the IDE.
- It is much faster.

Many thanks to Enguerrand de Ribaucourt for testing and bug fixing.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 scripts/lib/devtool/ide.py                   | 1095 ++++++++++++++++++
 scripts/lib/devtool/ide_handlers/__init__.py |   23 +
 scripts/lib/devtool/ide_handlers/ide_base.py |   46 +
 scripts/lib/devtool/ide_handlers/ide_code.py |  420 +++++++
 scripts/lib/devtool/ide_handlers/ide_none.py |   91 ++
 5 files changed, 1675 insertions(+)
 create mode 100755 scripts/lib/devtool/ide.py
 create mode 100644 scripts/lib/devtool/ide_handlers/__init__.py
 create mode 100644 scripts/lib/devtool/ide_handlers/ide_base.py
 create mode 100644 scripts/lib/devtool/ide_handlers/ide_code.py
 create mode 100644 scripts/lib/devtool/ide_handlers/ide_none.py

diff --git a/scripts/lib/devtool/ide.py b/scripts/lib/devtool/ide.py
new file mode 100755
index 00000000000..77615ac30e5
--- /dev/null
+++ b/scripts/lib/devtool/ide.py
@@ -0,0 +1,1095 @@
+#! /usr/bin/env python3
+#
+# Copyright (C) 2023 Siemens AG
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+"""Devtool ide plugin"""
+
+import os
+import stat
+import sys
+import logging
+import json
+import re
+import shutil
+import subprocess
+from argparse import RawTextHelpFormatter
+from enum import IntEnum, auto
+
+import bb
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError, parse_recipe
+from devtool.standard import get_real_srctree
+import devtool.ide_handlers
+
+SHARED_SYSROOT_RECIPES = ['shared', 'none',
+                          'meta-ide-support', 'build-sysroots']
+
+logger = logging.getLogger('devtool')
+
+
+class DevtoolIdeMode(IntEnum):
+    UNDEFINED = auto()
+    DEVTOOL_MODIFY = auto()
+    SHARED_SYSROOT = auto()
+
+
+class TargetDevice:
+    """SSH remote login parameters"""
+
+    def __init__(self, args):
+        self.extraoptions = ''
+        if args.no_host_check:
+            self.extraoptions += '-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
+        self.ssh_sshexec = 'ssh'
+        if args.ssh_exec:
+            self.ssh_sshexec = args.ssh_exec
+        self.ssh_port = ''
+        if args.port:
+            self.ssh_port = "-p %s" % args.port
+        if args.key:
+            self.extraoptions += ' -i %s' % args.key
+
+        self.target = args.target
+        target_sp = args.target.split('@')
+        if len(target_sp) == 1:
+            self.login = ""
+            self.host = target_sp[0]
+        elif len(target_sp) == 2:
+            self.login = target_sp[0]
+            self.host = target_sp[1]
+        else:
+            logger.error("Invalid target argument: %s" % args.target)
+
+    @staticmethod
+    def get_devtool_deploy_opts(args):
+        """Filter args for devtool deploy-target args"""
+        if not args.target:
+            return None
+        devtool_deploy_opts = [args.target]
+        if args.no_host_check:
+            devtool_deploy_opts += ["-c"]
+        if args.show_status:
+            devtool_deploy_opts += ["-s"]
+        if args.no_preserve:
+            devtool_deploy_opts += ["-p"]
+        if args.no_check_space:
+            devtool_deploy_opts += ["--no-check-space"]
+        if args.ssh_exec:
+            devtool_deploy_opts += ["-e", args.ssh.exec]
+        if args.port:
+            devtool_deploy_opts += ["-P", args.port]
+        if args.key:
+            devtool_deploy_opts += ["-I", args.key]
+        if args.strip is False:
+            devtool_deploy_opts += ["--no-strip"]
+        return devtool_deploy_opts
+
+
+class RecipeNative:
+    """Base class for calling bitbake to provide a -native recipe"""
+
+    def __init__(self, name, target_arch=None):
+        self.name = name
+        self.target_arch = target_arch
+        self.bootstrap_tasks = [self.name + ':do_addto_recipe_sysroot']
+        self.staging_bindir_native = None
+        self.target_sys = None
+        self.__native_bin = None
+
+    def _initialize(self, config, workspace, tinfoil):
+        """Get the parsed recipe"""
+        recipe_d = parse_recipe(
+            config, tinfoil, self.name, appends=True, filter_workspace=False)
+        if not recipe_d:
+            raise DevtoolError("Parsing %s recipe failed" % self.name)
+        self.staging_bindir_native = os.path.realpath(
+            recipe_d.getVar('STAGING_BINDIR_NATIVE'))
+        self.target_sys = recipe_d.getVar('TARGET_SYS')
+        return recipe_d
+
+    def initialize(self, config, workspace, tinfoil):
+        """Basic initialization that can be overridden by a derived class"""
+        self._initialize(config, workspace, tinfoil)
+
+    @property
+    def native_bin(self):
+        if not self.__native_bin:
+            raise DevtoolError("native binary name is not defined.")
+        return self.__native_bin
+
+
+class RecipeGdbCross(RecipeNative):
+    """Handle handle gdb-cross on the host and the gdbserver on the target device"""
+
+    def __init__(self, args, target_arch, target_device, gdbserver_multi=True):
+        super().__init__('gdb-cross-' + target_arch, target_arch)
+        self.target_device = target_device
+        self.gdb = None
+        self.gdbserver_port_next = int(args.gdbserver_port_start)
+        self.gdbserver_multi = gdbserver_multi
+        self.config_db = {}
+
+    def __find_gdbserver(self, config, tinfoil):
+        """Absolute path of the gdbserver"""
+        recipe_d_gdb = parse_recipe(
+            config, tinfoil, 'gdb', appends=True, filter_workspace=False)
+        if not recipe_d_gdb:
+            raise DevtoolError("Parsing gdb recipe failed")
+        return os.path.join(recipe_d_gdb.getVar('bindir'), 'gdbserver')
+
+    def initialize(self, config, workspace, tinfoil):
+        super()._initialize(config, workspace, tinfoil)
+        gdb_bin = self.target_sys + '-gdb'
+        gdb_path = os.path.join(
+            self.staging_bindir_native, self.target_sys, gdb_bin)
+        self.gdb = gdb_path
+        self.gdbserver_path = self.__find_gdbserver(config, tinfoil)
+
+    @property
+    def host(self):
+        return self.target_device.host
+
+    def __gdbserver_start_cmd(self, binary, port):
+        """Returns a shell command starting the gdbserver on the remote device
+
+        GDB supports two modes:
+        multi: gdbserver remains running over several debug sessions
+        once: gdbserver terminates after the debugged process terminates
+        """
+        if self.gdbserver_multi:
+            gdbserver_cmd = "%s --multi :%s" % (
+                self.gdbserver_path, port)
+        else:
+            gdbserver_cmd = "%s --once :%s %s" % (
+                self.gdbserver_path, port, binary)
+        return "%s %s %s %s 'sh -c \"%s\"'" % (
+            self.target_device.ssh_sshexec, self.target_device.ssh_port, self.target_device.extraoptions, self.target_device.target, gdbserver_cmd)
+
+    def setup_gdbserver_config(self, binary, script_dir):
+        """Generate a GDB configuration for a binary on the target device
+
+        This function adds a GDB configuration for the binary to the internal config_db.
+        This allows to allocate a TCP port per binary.
+        Optionally a shell script is generated which starts the gedbserver on the
+        target device as well as the gdb-cross on the host.
+        """
+        if binary in self.config_db:
+            raise DevtoolError(
+                "gdbserver config for binary %s is already generated" % binary)
+
+        port = self.gdbserver_port_next
+        self.gdbserver_port_next += 1
+        config_entry = {
+            "port": port,
+        }
+        if script_dir:
+            cmd_lines = ['#!/bin/sh']
+            cmd_lines.append(self.__gdbserver_start_cmd(binary, port))
+            binary_name_pretty = binary.replace(os.sep, '-')
+            start_script_name = 'gdbserver_start_%d_%s' % (
+                port, binary_name_pretty)
+            if self.gdbserver_multi:
+                start_script_name += "_m"
+            start_script_file = os.path.join(script_dir, start_script_name)
+            config_entry['gdbserver_start_script'] = start_script_file
+            config_entry['pretty_id'] = 'gdbserver start %d %s' % (
+                port, binary)
+
+            bb.utils.mkdirhier(script_dir)
+            with open(start_script_file, 'w') as script_f:
+                script_f.write(os.linesep.join(cmd_lines))
+                script_f.write(os.linesep)
+            st = os.stat(start_script_file)
+            os.chmod(start_script_file, st.st_mode | stat.S_IEXEC)
+
+        self.config_db[binary] = config_entry
+        return config_entry
+
+    def get_gdbserver_pretty_id(self, binary):
+        """Unique ID for the GDB configuration"""
+        return self.config_db[binary]['pretty_id']
+
+    def get_gdbserver_port(self, binary):
+        """TCP port used by gdbserver"""
+        return self.config_db[binary]['port']
+
+    def get_gdbserver_start_script(self, binary):
+        """Path to the script starting the debug session"""
+        return self.config_db[binary]['gdbserver_start_script']
+
+    def get_gdbserver_start_scripts(self):
+        """Get the paths of all debug session start scripts"""
+        for conf in self.config_db.values():
+            yield (conf['pretty_id'], conf['gdbserver_start_script'])
+
+
+class RecipeImage:
+    """Handle some image recipe related properties
+
+    Most workflows require firmware that runs on the target device.
+    This firmware must be consistent with the setup of the host system.
+    In particular, the debug symbols must be compatible. For this, the
+    rootfs must be created as part of the SDK.
+    """
+
+    def __init__(self, name):
+        self.gdbserver_missing = False
+        self.name = name
+        self.rootfs = None
+        self.__rootfs_dbg = None
+        self.bootstrap_tasks = [self.name + ':do_build']
+
+    def initialize(self, config, tinfoil):
+        image_d = parse_recipe(
+            config, tinfoil, self.name, appends=True, filter_workspace=False)
+        if not image_d:
+            raise DevtoolError(
+                "Parsing image recipe %s failed" % self.name)
+
+        workdir = image_d.getVar('WORKDIR')
+        self.rootfs = os.path.join(workdir, 'rootfs')
+        if image_d.getVar('IMAGE_GEN_DEBUGFS') == "1":
+            self.__rootfs_dbg = os.path.join(workdir, 'rootfs-dbg')
+
+        self.gdbserver_missing = 'gdbserver' not in image_d.getVar(
+            'IMAGE_INSTALL')
+
+    @property
+    def debug_support(self):
+        return bool(self.rootfs_dbg)
+
+    @property
+    def rootfs_dbg(self):
+        if self.__rootfs_dbg and os.path.isdir(self.__rootfs_dbg):
+            return self.__rootfs_dbg
+        return None
+
+
+class RecipeMetaIdeSupport:
+    """For the shared sysroots mode meta-ide-support is needed
+
+    For use cases where just a cross tool-chain is required but
+    no recipe is used, devtool ide abstracts calling bitbake meta-ide-support
+    and bitbake build-sysroots. This also allows to expose the cross-toolchains
+    to IDEs. For example VSCode support different tool-chains with e.g. cmake-kits.
+    """
+
+    def __init__(self):
+        self.bootstrap_tasks = ['meta-ide-support:do_build']
+        self.topdir = None
+        self.datadir = None
+        self.deploy_dir_image = None
+        self.build_sys = None
+        # From toolchain-scripts
+        self.real_multimach_target_sys = None
+
+    def initialize(self, config, tinfoil):
+        meta_ide_support_d = parse_recipe(
+            config, tinfoil, 'meta-ide-support', appends=True, filter_workspace=False)
+        if not meta_ide_support_d:
+            raise DevtoolError("Parsing meta-ide-support recipe failed")
+
+        self.topdir = meta_ide_support_d.getVar('TOPDIR')
+        self.datadir = meta_ide_support_d.getVar('datadir')
+        self.deploy_dir_image = meta_ide_support_d.getVar(
+            'DEPLOY_DIR_IMAGE')
+        self.build_sys = meta_ide_support_d.getVar('BUILD_SYS')
+        self.real_multimach_target_sys = meta_ide_support_d.getVar(
+            'REAL_MULTIMACH_TARGET_SYS')
+
+
+class RecipeBuildSysroots:
+    """For the shared sysroots mode build-sysroots is needed"""
+
+    def __init__(self):
+        self.standalone_sysroot = None
+        self.standalone_sysroot_native = None
+        self.bootstrap_tasks = [
+            'build-sysroots:do_build_target_sysroot',
+            'build-sysroots:do_build_native_sysroot'
+        ]
+
+    def initialize(self, config, tinfoil):
+        build_sysroots_d = parse_recipe(
+            config, tinfoil, 'build-sysroots', appends=True, filter_workspace=False)
+        if not build_sysroots_d:
+            raise DevtoolError("Parsing build-sysroots recipe failed")
+        self.standalone_sysroot = build_sysroots_d.getVar(
+            'STANDALONE_SYSROOT')
+        self.standalone_sysroot_native = build_sysroots_d.getVar(
+            'STANDALONE_SYSROOT_NATIVE')
+
+
+class SharedSysrootsEnv:
+    """Handle the shared sysroots based workflow
+
+    Support the workflow with just a tool-chain without a recipe.
+    It's basically like:
+      bitbake some-dependencies
+      bitbake meta-ide-support
+      bitbake build-sysroots
+      Use the environment-* file found in the deploy folder
+    """
+
+    def __init__(self):
+        self.ide_support = None
+        self.build_sysroots = None
+
+    def initialize(self, ide_support, build_sysroots):
+        self.ide_support = ide_support
+        self.build_sysroots = build_sysroots
+
+    def setup_ide(self, ide):
+        ide.setup(self)
+
+
+class BuildTool(IntEnum):
+    UNDEFINED = auto()
+    CMAKE = auto()
+    MESON = auto()
+
+
+class RecipeModified:
+    """Handling af recipes in the workspace created by devtool modify"""
+    OE_INIT_BUILD_ENV = 'oe-init-build-env'
+
+    def __init__(self, name):
+        self.name = name
+        self.bootstrap_tasks = [name + ':do_install']
+        # workspace
+        self.real_srctree = None
+        self.srctree = None
+        self.temp_dir = None
+        self.bbappend = None
+        # recipe variables from d.getVar
+        self.b = None
+        self.base_libdir = None
+        self.bblayers = None
+        self.bpn = None
+        self.d = None
+        self.fakerootcmd = None
+        self.fakerootenv = None
+        self.libdir = None
+        self.max_process = None
+        self.package_arch = None
+        self.package_debug_split_style = None
+        self.path = None
+        self.pn = None
+        self.recipe_sysroot = None
+        self.recipe_sysroot_native = None
+        self.staging_incdir = None
+        self.strip_cmd = None
+        self.target_arch = None
+        self.workdir = None
+        self.recipe_id = None
+        # recipe variables from d.getVarFlags
+        self.f_do_install_cleandirs = None
+        self.f_do_install_dirs = None
+        # replicate bitbake build environment
+        self.exported_vars = None
+        self.cmd_compile = None
+        self.__oe_init_dir = None
+        # main build tool used by this recipe
+        self.build_tool = BuildTool.UNDEFINED
+        # build_tool = cmake
+        self.oecmake_generator = None
+        self.cmake_cache_vars = None
+        # build_tool = meson
+        self.meson_buildtype = None
+        self.meson_wrapper = None
+        self.mesonopts = None
+        self.extra_oemeson = None
+        self.meson_cross_file = None
+
+    def initialize(self, config, workspace, tinfoil):
+        recipe_d = parse_recipe(
+            config, tinfoil, self.name, appends=True, filter_workspace=False)
+        if not recipe_d:
+            raise DevtoolError("Parsing %s recipe failed" % self.name)
+
+        # Verify this recipe is built as externalsrc setup by devtool modify
+        workspacepn = check_workspace_recipe(
+            workspace, self.name, bbclassextend=True)
+        self.srctree = workspace[workspacepn]['srctree']
+        # Need to grab this here in case the source is within a subdirectory
+        self.real_srctree = get_real_srctree(
+            self.srctree, recipe_d.getVar('S'), recipe_d.getVar('WORKDIR'))
+        self.bbappend = workspace[workspacepn]['bbappend']
+
+        self.temp_dir = os.path.join(config.workspace_path, 'temp', self.name)
+        if os.path.exists(self.temp_dir):
+            shutil.rmtree(self.temp_dir)
+
+        self.b = recipe_d.getVar('B')
+        self.base_libdir = recipe_d.getVar('base_libdir')
+        self.bblayers = recipe_d.getVar('BBLAYERS').split()
+        self.bpn = recipe_d.getVar('BPN')
+        self.d = recipe_d.getVar('D')
+        self.fakerootcmd = recipe_d.getVar('FAKEROOTCMD')
+        self.fakerootenv = recipe_d.getVar('FAKEROOTENV')
+        self.libdir = recipe_d.getVar('libdir')
+        self.max_process = int(recipe_d.getVar(
+            "BB_NUMBER_THREADS") or os.cpu_count() or 1)
+        self.package_arch = recipe_d.getVar('PACKAGE_ARCH')
+        self.package_debug_split_style = recipe_d.getVar(
+            'PACKAGE_DEBUG_SPLIT_STYLE')
+        self.path = recipe_d.getVar('PATH')
+        self.pn = recipe_d.getVar('PN')
+        self.recipe_sysroot = os.path.realpath(
+            recipe_d.getVar('RECIPE_SYSROOT'))
+        self.recipe_sysroot_native = os.path.realpath(
+            recipe_d.getVar('RECIPE_SYSROOT_NATIVE'))
+        self.staging_incdir = os.path.realpath(
+            recipe_d.getVar('STAGING_INCDIR'))
+        self.strip_cmd = recipe_d.getVar('STRIP')
+        self.target_arch = recipe_d.getVar('TARGET_ARCH')
+        self.workdir = os.path.realpath(recipe_d.getVar('WORKDIR'))
+
+        self.f_do_install_cleandirs = recipe_d.getVarFlag(
+            'do_install', 'cleandirs').split()
+        self.f_do_install_dirs = recipe_d.getVarFlag(
+            'do_install', 'dirs').split()
+
+        self.__init_exported_variables(recipe_d)
+
+        if bb.data.inherits_class('cmake', recipe_d):
+            self.oecmake_generator = recipe_d.getVar('OECMAKE_GENERATOR')
+            self.__init_cmake_preset_cache(recipe_d)
+            self.build_tool = BuildTool.CMAKE
+        elif bb.data.inherits_class('meson', recipe_d):
+            self.meson_buildtype = recipe_d.getVar('MESON_BUILDTYPE')
+            self.mesonopts = recipe_d.getVar('MESONOPTS')
+            self.extra_oemeson = recipe_d.getVar('EXTRA_OEMESON')
+            self.meson_cross_file = recipe_d.getVar('MESON_CROSS_FILE')
+            self.build_tool = BuildTool.MESON
+
+        # Recipe ID is the identifier for IDE config sections
+        self.recipe_id = self.bpn + "-" + self.package_arch
+        self.recipe_id_pretty = self.bpn + ": " + self.package_arch
+
+    def is_recipe_cross(self):
+        if self.pn.startswith('nativesdk-') or self.pn.endswith('-native'):
+            return False
+        return True
+
+    def append_to_bbappend(self, append_text):
+        with open(self.bbappend, 'a') as bbap:
+            bbap.write(append_text)
+
+    def remove_from_bbappend(self, append_text):
+        with open(self.bbappend, 'r') as bbap:
+            text = bbap.read()
+        new_text = text.replace(append_text, '')
+        with open(self.bbappend, 'w') as bbap:
+            bbap.write(new_text)
+
+    def debug_build_config(self, args):
+        """Explicitely set for example CMAKE_BUILD_TYPE to Debug if not defined otherwise"""
+        if self.build_tool == BuildTool.CMAKE:
+            append_text = os.linesep + \
+                'OECMAKE_ARGS:append = " -DCMAKE_BUILD_TYPE:STRING=Debug"' + os.linesep
+            if args.debug_build_config and not 'CMAKE_BUILD_TYPE' in self.cmake_cache_vars:
+                self.cmake_cache_vars['CMAKE_BUILD_TYPE'] = {
+                    "type": "STRING",
+                    "value": "Debug",
+                }
+                self.append_to_bbappend(append_text)
+            elif 'CMAKE_BUILD_TYPE' in self.cmake_cache_vars:
+                del self.cmake_cache_vars['CMAKE_BUILD_TYPE']
+                self.remove_from_bbappend(append_text)
+        elif self.build_tool == BuildTool.MESON:
+            append_text = os.linesep + 'MESON_BUILDTYPE = "debug"' + os.linesep
+            if args.debug_build_config and self.meson_buildtype != "debug":
+                self.mesonopts.replace(
+                    '--buildtype ' + self.meson_buildtype, '--buildtype debug')
+                self.append_to_bbappend(append_text)
+            elif self.meson_buildtype == "debug":
+                self.mesonopts.replace(
+                    '--buildtype debug', '--buildtype plain')
+                self.remove_from_bbappend(append_text)
+        elif args.debug_build_config:
+            logger.warn(
+                "--debug-build-config is not implemented for this build tool yet.")
+
+    def solib_search_path(self, image):
+        """Search for debug symbols in the rootfs-dbg
+
+        The debug symbols of shared libraries which are provided by other packages
+        are grabbed from the -dbg packages in the rootfs-dbg.
+
+        Note: For the devtool modified recipe compiled from the IDE, the debug
+        symbols are taken from the unstripped binaries in the image folder.
+        The image folder is created by the do_install task.
+        Also, devtool deploy-target takes the files from the image folder.
+        Running the do_package task is not required when working with the IDE.
+        """
+        so_paths = []
+        if self.package_debug_split_style in ['debug-with-srcpkg', '.debug']:
+            so_paths = [
+                self.base_libdir, os.path.join(self.base_libdir, ".debug"),
+                self.libdir, os.path.join(self.libdir, ".debug")
+            ]
+        elif self.package_debug_split_style == 'debug-file-directory':
+            so_paths = ["/usr/lib/debug"]
+        else:
+            logger.warning(
+                "Cannot find solib search path for a rootfs built with PACKAGE_DEBUG_SPLIT_STYLE=%s." % self.package_debug_split_style)
+        return [os.path.join(image.rootfs_dbg, dbgdir.lstrip('/')) for dbgdir in so_paths]
+
+    def solib_search_path_str(self, image):
+        """Return a : separated list of paths usable by GDB's set solib-search-path"""
+        return ':'.join(self.solib_search_path(image))
+
+    def __init_exported_variables(self, d):
+        """Find all variables with export flag set.
+
+        This allows to generate IDE configurations which compile with the same
+        environment as bitbake does. That's at least a reasonable default behavior.
+        """
+        exported_vars = {}
+
+        vars = (key for key in d.keys() if not key.startswith(
+            "__") and not d.getVarFlag(key, "func", False))
+        for var in vars:
+            func = d.getVarFlag(var, "func", False)
+            if d.getVarFlag(var, 'python', False) and func:
+                continue
+            export = d.getVarFlag(var, "export", False)
+            unexport = d.getVarFlag(var, "unexport", False)
+            if not export and not unexport and not func:
+                continue
+            if unexport:
+                continue
+
+            val = d.getVar(var)
+            if val is None:
+                continue
+            if set(var) & set("-.{}+"):
+                logger.warn(
+                    "Warning: Found invalid character in variable name %s", str(var))
+                continue
+            varExpanded = d.expand(var)
+            val = str(val)
+
+            if varExpanded.startswith("BASH_FUNC_"):
+                varExpanded = varExpanded[10:-2]
+                val = val[3:]  # Strip off "() "
+                logger.warn("Warning: BASH_FUNC_ is not exported to cmake presets (%s() %s)" % (
+                    varExpanded, val))
+                continue
+
+            if func:
+                code_line = "line: {0}, file: {1}\n".format(
+                    d.getVarFlag(var, "lineno", False),
+                    d.getVarFlag(var, "filename", False))
+                val = val.rstrip('\n')
+                logger.warn("Warning: exported shell function %s() is not exported (%s)" %
+                            (varExpanded, code_line))
+                continue
+
+            if export:
+                exported_vars[varExpanded] = val.strip()
+                continue
+
+        self.exported_vars = exported_vars
+
+    def __init_cmake_preset_cache(self, d):
+        """Get the arguments passed to cmake
+
+        Replicate the cmake configure arguments with all details to
+        share on build folder between bitbake and SDK.
+        """
+        site_file = os.path.join(self.workdir, 'site-file.cmake')
+        if os.path.exists(site_file):
+            print("Warning: site-file.cmake is not supported")
+
+        cache_vars = {}
+        oecmake_args = d.getVar('OECMAKE_ARGS').split()
+        extra_oecmake = d.getVar('EXTRA_OECMAKE').split()
+        for param in oecmake_args + extra_oecmake:
+            d_pref = "-D"
+            if param.startswith(d_pref):
+                param = param[len(d_pref):]
+            else:
+                print("Error: expected a -D")
+            param_s = param.split('=', 1)
+            param_nt = param_s[0].split(':', 1)
+
+            def handle_undefined_variable(var):
+                if var.startswith('${') and var.endswith('}'):
+                    return ''
+                else:
+                    return var
+            # Example: FOO=ON
+            if len(param_nt) == 1:
+                cache_vars[param_s[0]] = handle_undefined_variable(param_s[1])
+            # Example: FOO:PATH=/tmp
+            elif len(param_nt) == 2:
+                cache_vars[param_nt[0]] = {
+                    "type": param_nt[1],
+                    "value": handle_undefined_variable(param_s[1]),
+                }
+            else:
+                print("Error: cannot parse %s" % param)
+        self.cmake_cache_vars = cache_vars
+
+    def cmake_preset(self):
+        """Create a preset for cmake that mimics how bitbake calls cmake"""
+        toolchain_file = os.path.join(self.workdir, 'toolchain.cmake')
+        cmake_executable = os.path.join(
+            self.recipe_sysroot_native, 'usr', 'bin', 'cmake')
+        self.cmd_compile = cmake_executable + " --build --preset " + self.recipe_id
+
+        preset_dict_configure = {
+            "name": self.recipe_id,
+            "displayName": self.recipe_id_pretty,
+            "description": "Bitbake build environment for the recipe %s compiled for %s" % (self.bpn, self.package_arch),
+            "binaryDir": self.b,
+            "generator": self.oecmake_generator,
+            "toolchainFile": toolchain_file,
+            "cacheVariables": self.cmake_cache_vars,
+            "environment": self.exported_vars,
+            "cmakeExecutable": cmake_executable
+        }
+
+        preset_dict_build = {
+            "name": self.recipe_id,
+            "displayName": self.recipe_id_pretty,
+            "description": "Bitbake build environment for the recipe %s compiled for %s" % (self.bpn, self.package_arch),
+            "configurePreset": self.recipe_id,
+            "inheritConfigureEnvironment": True
+        }
+
+        preset_dict_test = {
+            "name": self.recipe_id,
+            "displayName": self.recipe_id_pretty,
+            "description": "Bitbake build environment for the recipe %s compiled for %s" % (self.bpn, self.package_arch),
+            "configurePreset": self.recipe_id,
+            "inheritConfigureEnvironment": True
+        }
+
+        preset_dict = {
+            "version": 3,  # cmake 3.21, backward compatible with kirkstone
+            "configurePresets": [preset_dict_configure],
+            "buildPresets": [preset_dict_build],
+            "testPresets": [preset_dict_test]
+        }
+
+        logger.info("generating cmake preset for recipe %s" % self.bpn)
+
+        # Finally write the json file
+        json_file = 'CMakeUserPresets.json'
+        json_path = os.path.join(self.real_srctree, json_file)
+        logger.info("Updating CMake preset: %s (%s)" % (json_file, json_path))
+        if not os.path.exists(self.real_srctree):
+            os.makedirs(self.real_srctree)
+        try:
+            with open(json_path) as f:
+                orig_dict = json.load(f)
+        except json.decoder.JSONDecodeError:
+            logger.info(
+                "Decoding %s failed. Probably because of comments in the json file" % json_path)
+            orig_dict = {}
+        except FileNotFoundError:
+            orig_dict = {}
+
+        # Add or update the presets for the recipe and keep other presets
+        for k, v in preset_dict.items():
+            if isinstance(v, list):
+                update_preset = v[0]
+                preset_added = False
+                if k in orig_dict:
+                    for index, orig_preset in enumerate(orig_dict[k]):
+                        if 'name' in orig_preset:
+                            if orig_preset['name'] == update_preset['name']:
+                                logger.debug("Updating preset: %s" %
+                                             orig_preset['name'])
+                                orig_dict[k][index] = update_preset
+                                preset_added = True
+                                break
+                            else:
+                                logger.debug("keeping preset: %s" %
+                                             orig_preset['name'])
+                        else:
+                            logger.warn("preset without a name found")
+                if not preset_added:
+                    if not k in orig_dict:
+                        orig_dict[k] = []
+                    orig_dict[k].append(update_preset)
+                    logger.debug("Added preset: %s" %
+                                 update_preset['name'])
+            else:
+                orig_dict[k] = v
+
+        with open(json_path, 'w') as f:
+            json.dump(orig_dict, f, indent=4)
+
+    def gen_meson_wrapper(self):
+        """Generate a wrapper script to call meson with the cross environment"""
+        bb.utils.mkdirhier(self.temp_dir)
+        meson_wrapper = os.path.join(self.temp_dir, 'meson')
+        meson_real = os.path.join(
+            self.recipe_sysroot_native, 'usr', 'bin', 'meson.real')
+        with open(meson_wrapper, 'w') as mwrap:
+            mwrap.write("#!/bin/sh" + os.linesep)
+            for var, val in self.exported_vars.items():
+                mwrap.write('export %s="%s"' % (var, val) + os.linesep)
+            mwrap.write("unset CC CXX CPP LD AR NM STRIP" + os.linesep)
+            private_temp = os.path.join(self.b, "meson-private", "tmp")
+            mwrap.write('mkdir -p "%s"' % private_temp + os.linesep)
+            mwrap.write('export TMPDIR="%s"' % private_temp + os.linesep)
+            mwrap.write('exec "%s" "$@"' % meson_real + os.linesep)
+        st = os.stat(meson_wrapper)
+        os.chmod(meson_wrapper, st.st_mode | stat.S_IEXEC)
+        self.meson_wrapper = meson_wrapper
+        self.cmd_compile = meson_wrapper + " compile -C " + self.b
+
+    def which(self, executable):
+        bin_path = shutil.which(executable, path=self.path)
+        if not bin_path:
+            raise DevtoolError(
+                'Cannot find %s. Probably the recipe %s is not built yet.' % (executable, self.bpn))
+        return bin_path
+
+    @staticmethod
+    def is_elf_file(file_path):
+        with open(file_path, "rb") as f:
+            data = f.read(4)
+        if data == b'\x7fELF':
+            return True
+        return False
+
+    def find_installed_binaries(self):
+        """find all executable elf files in the image directory"""
+        binaries = []
+        d_len = len(self.d)
+        re_so = re.compile('.*\.so[.0-9]*$')
+        for root, _, files in os.walk(self.d, followlinks=False):
+            for file in files:
+                if os.path.islink(file):
+                    continue
+                if re_so.match(file):
+                    continue
+                abs_name = os.path.join(root, file)
+                if os.access(abs_name, os.X_OK) and RecipeModified.is_elf_file(abs_name):
+                    binaries.append(abs_name[d_len:])
+        return binaries
+
+    def gen_fakeroot_install_script(self):
+        """Generate a helper script to execute make install with pseudo
+
+        For the deployment to the target device the do_install task must be
+        executed out of the IDE as well. This function generates a script which
+        runs the run.do_install script from bitbake under pseudo so that it picks
+        up the appropriate file permissions. Generating a self-contained script
+        is much quicker than calling bitbake or devtool build from an IDE.
+        """
+        cmd_lines = ['#!/bin/sh']
+        # Ensure the do compile step gets always executed without pseuso before do install
+        # Running do_compile always without pseudo is probably better than trying to have
+        # all the paths referred by compiling added to PSEUDO_IGNORE_PATHS.
+        if self.cmd_compile:
+            cmd_compile = "( cd %s && %s)" % (
+                self.real_srctree, self.cmd_compile)
+            cmd_lines.append(cmd_compile)
+        if not os.access(self.fakerootcmd, os.X_OK):
+            raise DevtoolError(
+                "pseudo executable %s could not be found" % self.fakerootcmd)
+        run_do_install = os.path.join(self.workdir, 'temp', 'run.do_install')
+
+        if not os.access(run_do_install, os.X_OK):
+            raise DevtoolError(
+                "run script does not exists: %s" % run_do_install)
+
+        # Set up the appropriate environment
+        newenv = dict(os.environ)
+        for varvalue in self.fakerootenv.split():
+            if '=' in varvalue:
+                splitval = varvalue.split('=', 1)
+                newenv[splitval[0]] = splitval[1]
+
+        # Replicate the environment variables from bitbake
+        for var, val in newenv.items():
+            cmd_lines.append('export %s="%s"' % (var, val))
+
+        # Setup the task environment as bitbake would do it based on the varFlags
+        for d in self.f_do_install_cleandirs:
+            cmd_lines.append('%s rm -rf %s' % (self.fakerootcmd, d))
+        for d in self.f_do_install_dirs:
+            cmd_lines.append('%s mkdir -p %s' % (self.fakerootcmd, d))
+        if len(self.f_do_install_dirs) > 0:
+            cmd = "cd %s" % self.f_do_install_dirs[-1]
+            cmd_lines.append('%s || { "%s failed"; exit 1; }' % (cmd, cmd))
+
+        # Remove the package* folders from TMPDIR. These folders might contain the sources for the -src packages.
+        # This likely breaks pseudo like:
+        # path mismatch [3 links]: ino 79147802 db
+        # .../build/tmp/.../cmake-example/1.0/package/usr/src/debug/cmake-example/1.0-r0/oe-local-files/cpp-example-lib.cpp
+        # .../build/workspace/sources/cmake-example/oe-local-files/cpp-example-lib.cpp
+        # Since the files are anyway outdated lets deleted them (also from pseudo's db) to workaround this issue.
+        pkg_dirs = ' '.join([os.path.join(self.workdir, d) for d in [
+            "package", "packages-split", "pkgdata", "sstate-install-package", "debugsources.list", "*.spec"]])
+        cmd = "%s rm -rf %s" % (self.fakerootcmd, pkg_dirs)
+        cmd_lines.append('%s || { "%s failed"; exit 1; }' % (cmd, cmd))
+
+        # Finally call run.do_install on pseudo
+        cmd = "%s %s" % (self.fakerootcmd, run_do_install)
+        cmd_lines.append('%s || { "%s failed"; exit 1; }' % (cmd, cmd))
+
+        return self.write_script(cmd_lines, 'bb_run_do_install')
+
+    def gen_deploy_target_script(self, args):
+        """Generate a script which does what devtool deploy-target does
+
+        This script is much quicker than devtool target-deploy. Because it
+        does not need to start a bitbake server. All information from tinfoil
+        is hard-coded in the generated script.
+        """
+        cmd_lines = ['#!/usr/bin/env python3']
+        cmd_lines.append('import sys')
+        cmd_lines.append('devtool_sys_path = %s' % str(sys.path))
+        cmd_lines.append('devtool_sys_path.reverse()')
+        cmd_lines.append('for p in devtool_sys_path:')
+        cmd_lines.append('    if p not in sys.path:')
+        cmd_lines.append('        sys.path.insert(0, p)')
+        cmd_lines.append('from devtool.deploy import deploy_cached')
+        args_filter = ['debug', 'dry_run', 'key', 'no_check_space', 'no_host_check',
+                       'no_preserve', 'port', 'show_status', 'ssh_exec', 'strip', 'target']
+        filtered_args_dict = {key: value for key, value in vars(
+            args).items() if key in args_filter}
+        cmd_lines.append('filtered_args_dict = %s' % str(filtered_args_dict))
+        cmd_lines.append('class Dict2Class(object):')
+        cmd_lines.append('    def __init__(self, my_dict):')
+        cmd_lines.append('        for key in my_dict:')
+        cmd_lines.append('            setattr(self, key, my_dict[key])')
+        cmd_lines.append('filtered_args = Dict2Class(filtered_args_dict)')
+        cmd_lines.append(
+            'setattr(filtered_args, "recipename", "%s")' % self.bpn)
+        cmd_lines.append('deploy_cached("%s", "%s", "%s", "%s", "%s", "%s", %d, "%s", "%s", filtered_args)' %
+                         (self.d, self.workdir, self.path, self.strip_cmd,
+                          self.libdir, self.base_libdir, self.max_process,
+                          self.fakerootcmd, self.fakerootenv))
+        return self.write_script(cmd_lines, 'deploy_target')
+
+    def gen_install_deploy_script(self, args):
+        """Generate a script which does install and deploy"""
+        cmd_lines = ['#!/bin/sh -e']
+        cmd_lines.append(self.gen_fakeroot_install_script())
+        cmd_lines.append(self.gen_deploy_target_script(args))
+        return self.write_script(cmd_lines, 'install_and_deploy')
+
+    def write_script(self, cmd_lines, script_name):
+        bb.utils.mkdirhier(self.temp_dir)
+        script_name_arch = script_name + '_' + self.recipe_id
+        script_file = os.path.join(self.temp_dir, script_name_arch)
+        with open(script_file, 'w') as script_f:
+            script_f.write(os.linesep.join(cmd_lines))
+        st = os.stat(script_file)
+        os.chmod(script_file, st.st_mode | stat.S_IEXEC)
+        return script_file
+
+    @property
+    def oe_init_build_env(self):
+        """Find the oe-init-build-env used for this setup"""
+        oe_init_dir = self.oe_init_dir
+        if oe_init_dir:
+            return os.path.join(oe_init_dir, RecipeModified.OE_INIT_BUILD_ENV)
+        return None
+
+    @property
+    def oe_init_dir(self):
+        """Find the directory where the oe-init-build-env is located
+
+        Assumption: There might be a layer with higher priority than poky
+        which provides to oe-init-build-env in the layer's toplevel folder.
+        """
+        if not self.__oe_init_dir:
+            for layer in reversed(self.bblayers):
+                result = subprocess.run(
+                    ['git', 'rev-parse', '--show-toplevel'], cwd=layer, capture_output=True)
+                if result.returncode == 0:
+                    oe_init_dir = result.stdout.decode('utf-8').strip()
+                    oe_init_path = os.path.join(
+                        oe_init_dir, RecipeModified.OE_INIT_BUILD_ENV)
+                    if os.path.exists(oe_init_path):
+                        logger.debug("Using %s from: %s" % (
+                            RecipeModified.OE_INIT_BUILD_ENV, oe_init_path))
+                        self.__oe_init_dir = oe_init_dir
+                        break
+            if not self.__oe_init_dir:
+                logger.error("Cannot find the bitbake top level folder")
+        return self.__oe_init_dir
+
+
+def ide_setup(args, config, basepath, workspace):
+    """Generate the IDE configuration for the workspace"""
+    bootstrap_tasks = []
+    bootstrap_tasks_late = []
+    devtool_ide_mode = DevtoolIdeMode.UNDEFINED
+    ide = devtool.ide_handlers.create_ide(args)
+
+    tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
+    try:
+        # define mode depending on recipes which need to be processed
+        recipes_shared_names = []
+        recipes_image_names = []
+        recipes_modified_names = []
+        for recipe in args.recipenames:
+            if recipe in SHARED_SYSROOT_RECIPES:
+                recipes_shared_names.append(recipe)
+            else:
+                try:
+                    check_workspace_recipe(
+                        workspace, recipe, bbclassextend=True)
+                    recipes_modified_names.append(recipe)
+                except DevtoolError:
+                    recipe_d = parse_recipe(
+                        config, tinfoil, recipe, appends=True, filter_workspace=False)
+                    if not recipe_d:
+                        raise DevtoolError("Parsing recipe %s failed" % recipe)
+                    if bb.data.inherits_class('image', recipe_d):
+                        recipes_image_names.append(recipe)
+                    else:
+                        raise DevtoolError(
+                            "Recipe is not an image and not a recipe in the workspace.")
+
+        if len(recipes_image_names) > 1:
+            raise DevtoolError("Max one image recipe must be passed.")
+        if recipes_image_names and recipes_modified_names:
+            devtool_ide_mode = DevtoolIdeMode.DEVTOOL_MODIFY
+            if recipes_shared_names:
+                raise DevtoolError("Mixing modified recipes (%s) and shared recipes (%s) is not possible."
+                                   % (str(recipes_modified_names), str(recipes_shared_names)))
+            logger.info("Mode: devtool modify")
+        elif recipes_image_names and recipes_shared_names:
+            devtool_ide_mode = DevtoolIdeMode.SHARED_SYSROOT
+        else:
+            raise DevtoolError("Invalid recipes passed.")
+
+        # Provide a rootfs and the corresponding debug symbols via rootfs-dbg
+        if devtool_ide_mode == DevtoolIdeMode.DEVTOOL_MODIFY or devtool_ide_mode == DevtoolIdeMode.SHARED_SYSROOT:
+            logger.info("Using image: %s" % recipes_image_names[0])
+            image_config = RecipeImage(recipes_image_names[0])
+            image_config.initialize(config, tinfoil)
+            bootstrap_tasks += image_config.bootstrap_tasks
+
+        if devtool_ide_mode == DevtoolIdeMode.SHARED_SYSROOT:
+            logger.info("Generating a shared sysroot SDK.")
+            ide_support = RecipeMetaIdeSupport()
+            ide_support.initialize(config, tinfoil)
+            bootstrap_tasks += ide_support.bootstrap_tasks
+
+            build_sysroots = RecipeBuildSysroots()
+            build_sysroots.initialize(config, tinfoil)
+            bootstrap_tasks_late += build_sysroots.bootstrap_tasks
+            shared_env = SharedSysrootsEnv()
+            shared_env.initialize(ide_support, build_sysroots)
+
+        recipes_modified = []
+        if devtool_ide_mode == DevtoolIdeMode.DEVTOOL_MODIFY:
+            logger.info("Using modified recipe: %s" %
+                        recipes_modified_names[0])
+            for modified_recipe_name in recipes_modified_names:
+                recipe_modified = RecipeModified(modified_recipe_name)
+                recipe_modified.initialize(config, workspace, tinfoil)
+                bootstrap_tasks += recipe_modified.bootstrap_tasks
+                recipes_modified.append(recipe_modified)
+
+            target_device = TargetDevice(args)
+            gdb_cross = RecipeGdbCross(
+                args, recipes_modified[0].target_arch, target_device, ide.gdb_multi_mode)
+            gdb_cross.initialize(config, workspace, tinfoil)
+            bootstrap_tasks += gdb_cross.bootstrap_tasks
+    finally:
+        tinfoil.shutdown()
+
+    if devtool_ide_mode == DevtoolIdeMode.DEVTOOL_MODIFY:
+        for recipe_modified in recipes_modified:
+            if not recipe_modified.is_recipe_cross():
+                raise DevtoolError(
+                    "Only cross compiled recipes are currently supported.")
+            recipe_modified.debug_build_config(args)
+
+    if devtool_ide_mode == DevtoolIdeMode.DEVTOOL_MODIFY or devtool_ide_mode == DevtoolIdeMode.SHARED_SYSROOT:
+        if not args.skip_bitbake:
+            bb_cmd = 'bitbake '
+            if args.bitbake_k:
+                bb_cmd += "-k "
+            bb_cmd_early = bb_cmd + ' '.join(bootstrap_tasks)
+            exec_build_env_command(
+                config.init_path, basepath, bb_cmd_early, watch=True)
+            if bootstrap_tasks_late:
+                bb_cmd_late = bb_cmd + ' '.join(bootstrap_tasks_late)
+                exec_build_env_command(
+                    config.init_path, basepath, bb_cmd_late, watch=True)
+
+    if (image_config.gdbserver_missing):
+        logger.warning(
+            "gdbserver not installed in image. Remote debugging will not be available")
+
+    if devtool_ide_mode == DevtoolIdeMode.SHARED_SYSROOT:
+        ide.setup_shared_sysroots(shared_env)
+    elif devtool_ide_mode == DevtoolIdeMode.DEVTOOL_MODIFY:
+        for recipe_modified in recipes_modified:
+            ide.setup_modified_recipe(
+                args, image_config, gdb_cross, recipe_modified)
+    else:
+        raise DevtoolError("Must not end up here.")
+
+
+def register_commands(subparsers, context):
+    """Register devtool subcommands from this plugin"""
+    parser_ide = subparsers.add_parser('ide', help='Setup the IDE (VSCode)',
+                                       description='Configure the IDE to work with the source code of a recipe.',
+                                       group='working', order=50, formatter_class=RawTextHelpFormatter)
+    parser_ide.add_argument(
+        'recipenames', nargs='+', help='Generate an IDE configuration suitable to work on the given recipes.\n'
+        'Different types of recipes lead to different types of IDE configurations.\n'
+        '- devtool modify mode:\n'
+        '  At least one devtool modified recipe + one image recipe are required:\n'
+        '  Usage example:\n'
+        '    devtool modify cmake-example\n'
+        '    devtool ide cmake-example core-image-minimal\n'
+        '  The workspace for all package recipes gets configured to use the corresponding per recipe sysroot(s).\n'
+        '  The image recipes is used to generate the target image, the corresponding remote debug configuration as well as the corresponding SDK.\n'
+        '- Shared sysroot mode:\n'
+        '  Usage example:\n'
+        '    devtool ide ' + SHARED_SYSROOT_RECIPES[0] + '\n'
+        '  This command generates a bare cross-toolchain as well as the corresponding shared sysroot directories.\n'
+        '  To use this tool-chain the environment-* file found in the deploy..image folder needs to be sourced into a shell.\n'
+        '  In case of VSCode and cmake the tool-chain is also exposed as a cmake-kit')
+    parser_ide.add_argument(
+        '-i', '--ide', choices=devtool.ide_handlers.SUPPORTED_IDES, default=devtool.ide_handlers.get_default_ide(),
+        help='Setup the configuration for this IDE')
+    parser_ide.add_argument(
+        '-t', '--target', default='root@192.168.7.2',
+        help='Live target machine running an ssh server: user@hostname.')
+    parser_ide.add_argument(
+        '-G', '--gdbserver-port-start', default="1234", help='port where gdbserver is listening.')
+    parser_ide.add_argument(
+        '-c', '--no-host-check', help='Disable ssh host key checking', action='store_true')
+    parser_ide.add_argument(
+        '-e', '--ssh-exec', help='Executable to use in place of ssh')
+    parser_ide.add_argument(
+        '-P', '--port', help='Specify ssh port to use for connection to the target')
+    parser_ide.add_argument(
+        '-I', '--key', help='Specify ssh private key for connection to the target')
+    parser_ide.add_argument(
+        '--skip-bitbake', help='Generate IDE configuration but skip calling bibtake to update the SDK.', action='store_true')
+    parser_ide.add_argument(
+        '-k', '--bitbake-k', help='Pass -k parameter to bitbake', action='store_true')
+    parser_ide.add_argument(
+        '--no-strip', help='Do not strip executables prior to deploy', dest='strip', action='store_false')
+    parser_ide.add_argument(
+        '-n', '--dry-run', help='List files to be undeployed only', action='store_true')
+    parser_ide.add_argument(
+        '-s', '--show-status', help='Show progress/status output', action='store_true')
+    parser_ide.add_argument(
+        '-p', '--no-preserve', help='Do not preserve existing files', action='store_true')
+    parser_ide.add_argument(
+        '--no-check-space', help='Do not check for available space before deploying', action='store_true')
+    parser_ide.add_argument(
+        '--debug-build-config', help='Use debug build flags, for example set CMAKE_BUILD_TYPE=Debug', action='store_true')
+    parser_ide.set_defaults(func=ide_setup)
diff --git a/scripts/lib/devtool/ide_handlers/__init__.py b/scripts/lib/devtool/ide_handlers/__init__.py
new file mode 100644
index 00000000000..9fcc96837ad
--- /dev/null
+++ b/scripts/lib/devtool/ide_handlers/__init__.py
@@ -0,0 +1,23 @@
+
+import shutil
+from devtool.ide_handlers.ide_none import IdeNone
+from devtool.ide_handlers.ide_code import IdeVSCode
+
+SUPPORTED_IDES = ['code', 'none']
+
+def get_default_ide():
+    """Check which IDEs are installed and return a reasonable default setting"""
+    for an_ide in SUPPORTED_IDES[:-1]:
+        if shutil.which(an_ide):
+            return an_ide
+    return SUPPORTED_IDES[-1]
+
+
+def create_ide(args):
+    """Simple factory for the IDE"""
+    if args.ide == 'code':
+        return IdeVSCode(args.ide)
+    elif args.ide == 'none':
+        return IdeNone(args.ide)
+    else:
+        raise ValueError("ide must be in %s" % str(SUPPORTED_IDES))
diff --git a/scripts/lib/devtool/ide_handlers/ide_base.py b/scripts/lib/devtool/ide_handlers/ide_base.py
new file mode 100644
index 00000000000..b9563166e42
--- /dev/null
+++ b/scripts/lib/devtool/ide_handlers/ide_base.py
@@ -0,0 +1,46 @@
+import os
+import json
+import logging
+
+logger = logging.getLogger('devtool')
+
+
+class IdeBase:
+    """Base class defining an interface for IDE plugins"""
+
+    def __init__(self, ide_name):
+        self.ide_name = ide_name
+        self.gdb_multi_mode = True
+
+    def setup_shared_sysroots(self, shared_env):
+        logger.warn("Shared sysroot mode is not supported for IDE %s" %
+                    self.ide_name)
+
+    def setup_modified_recipe(self, args, image_config, gdb_cross, modified_recipe):
+        logger.warn("Modified recipe mode is not supported for IDE %s" %
+                    self.ide_name)
+
+    @staticmethod
+    def update_json_file(json_dir, json_file, update_dict):
+        """Update a json file
+
+        By default it uses the dict.update function. If this is not sutiable
+        the update function might be passed via update_func parameter.
+        """
+        json_path = os.path.join(json_dir, json_file)
+        logger.info("Updating IDE config file: %s (%s)" %
+                    (json_file, json_path))
+        if not os.path.exists(json_dir):
+            os.makedirs(json_dir)
+        try:
+            with open(json_path) as f:
+                orig_dict = json.load(f)
+        except json.decoder.JSONDecodeError:
+            logger.info(
+                "Decoding %s failed. Probably because of comments in the json file" % json_path)
+            orig_dict = {}
+        except FileNotFoundError:
+            orig_dict = {}
+        orig_dict.update(update_dict)
+        with open(json_path, 'w') as f:
+            json.dump(orig_dict, f, indent=4)
diff --git a/scripts/lib/devtool/ide_handlers/ide_code.py b/scripts/lib/devtool/ide_handlers/ide_code.py
new file mode 100644
index 00000000000..8496f0a5ce5
--- /dev/null
+++ b/scripts/lib/devtool/ide_handlers/ide_code.py
@@ -0,0 +1,420 @@
+import os
+import json
+import logging
+from devtool.ide_handlers.ide_base import IdeBase
+from devtool.ide import BuildTool
+
+logger = logging.getLogger('devtool')
+
+
+class IdeVSCode(IdeBase):
+    """Manage IDE configurations for VSCode
+
+    Recipe mode:
+    - cmake: generates a cmake-preset
+    - meson: workspace settings callin the cross meson are gernerated.
+
+    Shared sysroot mode:
+    In shared sysroot mode, the cross tool-chain is exported to the user's global configuration.
+    A workspace cannot be created because there is no recipe that defines how a workspace could be set up.
+    - cmake: adds a cmake-kit to .local/share/CMakeTools/cmake-tools-kits.json
+             The cmake-kit uses the environment script and the tool-chain file
+             generated by meta-ide-support.
+    - meson: Meson needs manual workspace configuration.
+    """
+
+    def __init__(self, ide_name):
+        self.gdb_multi_mode = False
+
+    def setup_shared_sysroots(self, shared_env):
+        """Expose the toolchain of the dSDK"""
+        datadir = shared_env.ide_support.datadir
+        deploy_dir_image = shared_env.ide_support.deploy_dir_image
+        real_multimach_target_sys = shared_env.ide_support.real_multimach_target_sys
+        standalone_sysroot_native = shared_env.build_sysroots.standalone_sysroot_native
+        vscode_ws_path = os.path.join(
+            os.environ['HOME'], '.local', 'share', 'CMakeTools')
+        cmake_kits_path = os.path.join(vscode_ws_path, 'cmake-tools-kits.json')
+        oecmake_generator = "Ninja"
+        env_script = os.path.join(
+            deploy_dir_image, 'environment-setup-' + real_multimach_target_sys)
+
+        if not os.path.isdir(vscode_ws_path):
+            os.makedirs(vscode_ws_path)
+        cmake_kits_old = []
+        if os.path.exists(cmake_kits_path):
+            with open(cmake_kits_path, 'r', encoding='utf-8') as cmake_kits_file:
+                cmake_kits_old = json.load(cmake_kits_file)
+        cmake_kits = cmake_kits_old.copy()
+
+        cmake_kit_new = {
+            "name": "OE " + real_multimach_target_sys,
+            "environmentSetupScript": env_script,
+            "toolchainFile": standalone_sysroot_native + datadir + "/cmake/OEToolchainConfig.cmake",
+            "preferredGenerator": {
+                "name": oecmake_generator
+            }
+        }
+
+        def merge_kit(cmake_kits, cmake_kit_new):
+            i = 0
+            while i < len(cmake_kits):
+                if 'environmentSetupScript' in cmake_kits[i] and \
+                        cmake_kits[i]['environmentSetupScript'] == cmake_kit_new['environmentSetupScript']:
+                    cmake_kits[i] = cmake_kit_new
+                    return
+                i += 1
+            cmake_kits.append(cmake_kit_new)
+        merge_kit(cmake_kits, cmake_kit_new)
+
+        if cmake_kits != cmake_kits_old:
+            logger.info("Updating: %s" % cmake_kits_path)
+            with open(cmake_kits_path, 'w', encoding='utf-8') as cmake_kits_file:
+                json.dump(cmake_kits, cmake_kits_file, indent=4)
+        else:
+            logger.info("Already up to date: %s" % cmake_kits_path)
+
+        cmake_native = os.path.join(
+            shared_env.build_sysroots.standalone_sysroot_native, 'usr', 'bin', 'cmake')
+        if os.path.isfile(cmake_native):
+            logger.info('cmake-kits call cmake by default. If the cmake provided by this SDK should be used, please add the following line to ".vscode/settings.json" file: "cmake.cmakePath": "%s"' % cmake_native)
+        else:
+            logger.error("Cannot find cmake native at: %s" % cmake_native)
+
+    def dot_code_dir(self, modified_recipe):
+        return os.path.join(modified_recipe.srctree, '.vscode')
+
+    def __vscode_settings_meson(self, settings_dict, modified_recipe):
+        if modified_recipe.build_tool != BuildTool.MESON:
+            return
+        settings_dict["mesonbuild.mesonPath"] = modified_recipe.meson_wrapper
+
+        confopts = modified_recipe.mesonopts.split()
+        confopts += modified_recipe.meson_cross_file.split()
+        confopts += modified_recipe.extra_oemeson.split()
+        settings_dict["mesonbuild.configureOptions"] = confopts
+        settings_dict["mesonbuild.buildFolder"] = modified_recipe.b
+
+    def __vscode_settings_cmake(self, settings_dict, modified_recipe):
+        """Add cmake specific settings to settings.json.
+
+        Note: most settings are passed to the cmake preset.
+        """
+        if modified_recipe.build_tool != BuildTool.CMAKE:
+            return
+        settings_dict["cmake.configureOnOpen"] = True
+        settings_dict["cmake.sourceDirectory"] = modified_recipe.real_srctree
+
+    def vscode_settings(self, modified_recipe):
+        files_excludes = {
+            "**/.git/**": True,
+            "**/oe-logs/**": True,
+            "**/oe-workdir/**": True,
+            "**/source-date-epoch/**": True
+        }
+        python_exclude = [
+            "**/.git/**",
+            "**/oe-logs/**",
+            "**/oe-workdir/**",
+            "**/source-date-epoch/**"
+        ]
+        settings_dict = {
+            "files.watcherExclude": files_excludes,
+            "files.exclude": files_excludes,
+            "python.analysis.exclude": python_exclude
+        }
+        self.__vscode_settings_cmake(settings_dict, modified_recipe)
+        self.__vscode_settings_meson(settings_dict, modified_recipe)
+
+        settings_file = 'settings.json'
+        IdeBase.update_json_file(
+            self.dot_code_dir(modified_recipe), settings_file, settings_dict)
+
+    def __vscode_extensions_cmake(self, modified_recipe, recommendations):
+        if modified_recipe.build_tool != BuildTool.CMAKE:
+            return
+        recommendations += [
+            "twxs.cmake",
+            "ms-vscode.cmake-tools",
+            "ms-vscode.cpptools",
+            "ms-vscode.cpptools-extension-pack",
+            "ms-vscode.cpptools-themes"
+        ]
+
+    def __vscode_extensions_meson(self, modified_recipe, recommendations):
+        if modified_recipe.build_tool != BuildTool.MESON:
+            return
+        recommendations += [
+            'mesonbuild.mesonbuild',
+            "ms-vscode.cpptools",
+            "ms-vscode.cpptools-extension-pack",
+            "ms-vscode.cpptools-themes"
+        ]
+
+    def vscode_extensions(self, modified_recipe):
+        recommendations = []
+        self.__vscode_extensions_cmake(modified_recipe, recommendations)
+        self.__vscode_extensions_meson(modified_recipe, recommendations)
+        extensions_file = 'extensions.json'
+        IdeBase.update_json_file(
+            self.dot_code_dir(modified_recipe), extensions_file, {"recommendations": recommendations})
+
+    def vscode_c_cpp_properties(self, modified_recipe):
+        properties_dict = {
+            "name": modified_recipe.recipe_id_pretty,
+        }
+        if modified_recipe.build_tool == BuildTool.CMAKE:
+            properties_dict["configurationProvider"] = "ms-vscode.cmake-tools"
+        elif modified_recipe.build_tool == BuildTool.MESON:
+            properties_dict["configurationProvider"] = "mesonbuild.mesonbuild"
+        else:  # no C/C++ build
+            return
+
+        properties_dicts = {
+            "configurations": [
+                properties_dict
+            ],
+            "version": 4
+        }
+        prop_file = 'c_cpp_properties.json'
+        IdeBase.update_json_file(
+            self.dot_code_dir(modified_recipe), prop_file, properties_dicts)
+
+    def vscode_launch_bin_dbg(self, modified_recipe, image, gdb_cross, binary):
+        gdb_cross.setup_gdbserver_config(binary, modified_recipe.temp_dir)
+        pretty_id = gdb_cross.get_gdbserver_pretty_id(binary)
+        gdbserver_port = gdb_cross.get_gdbserver_port(binary)
+
+        launch_config = {
+            "name": pretty_id,
+            "type": "cppdbg",
+            "request": "launch",
+            "program": os.path.join(modified_recipe.d, binary.lstrip('/')),
+            "stopAtEntry": True,
+            "cwd": "${workspaceFolder}",
+            "environment": [],
+            "externalConsole": False,
+            "MIMode": "gdb",
+            "preLaunchTask": pretty_id,
+            "miDebuggerPath": gdb_cross.gdb,
+            "miDebuggerServerAddress": "%s:%d" % (gdb_cross.host, gdbserver_port)
+        }
+
+        # Search for header files in recipe-sysroot.
+        src_file_map = {
+            "/usr/include": os.path.join(modified_recipe.recipe_sysroot, "usr", "include")
+        }
+        # First of all search for not stripped binaries in the image folder.
+        # These binaries are copied (and optionally stripped) by deploy-target
+        setup_commands = [
+            {
+                "description": "sysroot",
+                "text": "set sysroot " + modified_recipe.d
+            }
+        ]
+
+        if image.rootfs_dbg:
+            launch_config['additionalSOLibSearchPath'] = modified_recipe.solib_search_path_str(
+                image)
+            src_file_map["/usr/src/debug"] = os.path.join(
+                image.rootfs_dbg, "usr", "src", "debug")
+        else:
+            logger.warning(
+                "Cannot setup debug symbols configuration for GDB. IMAGE_GEN_DEBUGFS is not enabled.")
+
+        launch_config['sourceFileMap'] = src_file_map
+        launch_config['setupCommands'] = setup_commands
+        return launch_config
+
+    def vscode_launch_bin(self, modified_recipe, image, gdb_cross):
+        """GDB Launch configuration for binaries (elf files)"""
+        binaries = modified_recipe.find_installed_binaries()
+        configurations = [self.vscode_launch_bin_dbg(modified_recipe,
+                                                     image, gdb_cross, binary) for binary in binaries]
+        launch_dict = {
+            "version": "0.2.0",
+            "configurations": configurations
+        }
+        launch_file = 'launch.json'
+        IdeBase.update_json_file(
+            self.dot_code_dir(modified_recipe), launch_file, launch_dict)
+
+    def vscode_launch(self, modified_recipe, image, gdb_cross):
+        if modified_recipe.build_tool in [BuildTool.CMAKE, BuildTool.MESON, BuildTool.UNDEFINED]:
+            self.vscode_launch_bin(modified_recipe, image, gdb_cross)
+        else:
+            logger.info(
+                "Generating a launch configuration for this recipe is not yet suported")
+
+    def vscode_tasks_cpp(self, args, modified_recipe, gdb_cross):
+        run_install_deploy = modified_recipe.gen_install_deploy_script(args)
+        install_task_name = "install && deploy-target %s" % modified_recipe.recipe_id_pretty
+        tasks_dict = {
+            "version": "2.0.0",
+            "tasks": [
+                {
+                    "label": install_task_name,
+                    "type": "shell",
+                    "command": run_install_deploy,
+                    "problemMatcher": []
+                }
+            ]
+        }
+        for pretty_id, start_script in gdb_cross.get_gdbserver_start_scripts():
+            tasks_dict['tasks'].append(
+                {
+                    "label": pretty_id,
+                    "type": "shell",
+                    "isBackground": True,
+                    "dependsOn": [
+                        install_task_name
+                    ],
+                    "command": start_script,
+                    "problemMatcher": [
+                        {
+                            "pattern": [
+                                {
+                                    "regexp": ".",
+                                    "file": 1,
+                                    "location": 2,
+                                    "message": 3
+                                }
+                            ],
+                            "background": {
+                                "activeOnStart": True,
+                                "beginsPattern": ".",
+                                "endsPattern": ".",
+                            }
+                        }
+                    ]
+                })
+        tasks_file = 'tasks.json'
+        IdeBase.update_json_file(
+            self.dot_code_dir(modified_recipe), tasks_file, tasks_dict)
+
+    def vscode_tasks_fallback(self, args, modified_recipe, gdb_cross=None):
+        oe_init_dir = modified_recipe.oe_init_dir
+        oe_init = ". %s > /dev/null && " % modified_recipe.oe_init_build_env
+        task_devtool_build = "devtool build %s" % modified_recipe.recipe_id_pretty
+        task_devtool_build_clean = "devtool build %s --clean" % modified_recipe.recipe_id_pretty
+        task_devtool_deploy = "devtool deploy-target %s" % modified_recipe.recipe_id_pretty
+        task_devtool_build_deploy = "devtool build & deploy-target %s" % modified_recipe.recipe_id_pretty
+        deploy_opts = ' '.join(TargetDevice.get_devtool_deploy_opts(args))
+        tasks_dict = {
+            "version": "2.0.0",
+            "tasks": [
+                {
+                    "label": task_devtool_build,
+                    "type": "shell",
+                    "command": "bash",
+                    "linux": {
+                        "options": {
+                            "cwd": oe_init_dir
+                        }
+                    },
+                    "args": [
+                        "--login",
+                        "-c",
+                        "%s%s" % (oe_init, task_devtool_build)
+                    ],
+                    "problemMatcher": []
+                },
+                {
+                    "label": task_devtool_deploy,
+                    "type": "shell",
+                    "command": "bash",
+                    "linux": {
+                        "options": {
+                            "cwd": oe_init_dir
+                        }
+                    },
+                    "args": [
+                        "--login",
+                        "-c",
+                        "%s%s %s" % (
+                            oe_init, task_devtool_deploy, deploy_opts)
+                    ],
+                    "problemMatcher": []
+                },
+                {
+                    "label": task_devtool_build_deploy,
+                    "dependsOrder": "sequence",
+                    "dependsOn": [
+                        task_devtool_build,
+                        task_devtool_deploy
+                    ],
+                    "problemMatcher": [],
+                    "group": {
+                        "kind": "build",
+                        "isDefault": True
+                    }
+                },
+                {
+                    "label": task_devtool_build_clean,
+                    "type": "shell",
+                    "command": "bash",
+                    "linux": {
+                        "options": {
+                            "cwd": oe_init_dir
+                        }
+                    },
+                    "args": [
+                        "--login",
+                        "-c",
+                        "%s%s" % (oe_init, task_devtool_build_clean)
+                    ],
+                    "problemMatcher": []
+                }
+            ]
+        }
+        if gdb_cross:
+            for pretty_id, start_script in gdb_cross.get_gdbserver_start_scripts():
+                tasks_dict['tasks'].append(
+                    {
+                        "label": pretty_id,
+                        "type": "shell",
+                        "isBackground": True,
+                        "dependsOn": [
+                            task_devtool_build_deploy
+                        ],
+                        "command": start_script,
+                        "problemMatcher": [
+                            {
+                                "pattern": [
+                                    {
+                                        "regexp": ".",
+                                        "file": 1,
+                                        "location": 2,
+                                        "message": 3
+                                    }
+                                ],
+                                "background": {
+                                    "activeOnStart": True,
+                                    "beginsPattern": ".",
+                                    "endsPattern": ".",
+                                }
+                            }
+                        ]
+                    })
+        tasks_file = 'tasks.json'
+        IdeBase.update_json_file(
+            self.dot_code_dir(modified_recipe), tasks_file, tasks_dict)
+
+    def vscode_tasks(self, args, modified_recipe, gdb_cross):
+        if modified_recipe.build_tool in [BuildTool.CMAKE, BuildTool.MESON]:
+            self.vscode_tasks_cpp(args, modified_recipe, gdb_cross)
+        else:
+            self.vscode_tasks_fallback(args, modified_recipe, gdb_cross)
+
+    def setup_modified_recipe(self, args, image_config, gdb_cross, modified_recipe):
+        if modified_recipe.build_tool == BuildTool.CMAKE:
+            modified_recipe.cmake_preset()
+        if modified_recipe.build_tool == BuildTool.MESON:
+            modified_recipe.gen_meson_wrapper()
+
+        self.vscode_settings(modified_recipe)
+        self.vscode_extensions(modified_recipe)
+        self.vscode_c_cpp_properties(modified_recipe)
+        if args.target:
+            self.vscode_launch(modified_recipe, image_config, gdb_cross)
+            self.vscode_tasks(args, modified_recipe, gdb_cross)
diff --git a/scripts/lib/devtool/ide_handlers/ide_none.py b/scripts/lib/devtool/ide_handlers/ide_none.py
new file mode 100644
index 00000000000..6a2e9d4a655
--- /dev/null
+++ b/scripts/lib/devtool/ide_handlers/ide_none.py
@@ -0,0 +1,91 @@
+
+import os
+import logging
+from devtool.ide_handlers.ide_base import IdeBase
+from devtool.ide import BuildTool
+
+logger = logging.getLogger('devtool')
+
+
+class IdeNone(IdeBase):
+    """Generate some generic helpers for other IDEs
+
+    Recipe mode:
+    Generate some helper scripts for femote debugging with GDB
+
+    Shared sysroot mode:
+    A wrapper for bitbake meta-ide-support and bitbake build-sysroots
+    """
+
+    def setup_shared_sysroots(self, shared_env):
+        real_multimach_target_sys = shared_env.ide_support.real_multimach_target_sys
+        deploy_dir_image = shared_env.ide_support.deploy_dir_image
+        env_script = os.path.join(
+            deploy_dir_image, 'environment-setup-' + real_multimach_target_sys)
+        logger.info(
+            "To use this SDK please source this: %s" % env_script)
+
+    @staticmethod
+    def get_unique_gdbinit_name(binary):
+        return 'gdbinit' + binary.replace(os.sep, '-')
+
+    def none_launch(self, args, modified_recipe, image, gdb_cross):
+        """generate some helper scripts
+
+        - an install (running on pseudo) and deploy script
+        - a gdbinit file per executable
+        """
+        # install and deploy helper scripts
+        script_path = modified_recipe.gen_install_deploy_script(args)
+        logger.info("Created helper script: %s" % script_path)
+
+        # gdbinit
+        binaries = modified_recipe.find_installed_binaries()
+        for binary in binaries:
+            gdb_cross.setup_gdbserver_config(binary, modified_recipe.temp_dir)
+            gdbserver_port = str(gdb_cross.get_gdbserver_port(binary))
+            if gdb_cross.gdbserver_multi:
+                target_help = '#   gdbserver --multi :' + gdbserver_port
+                remote_cmd = 'target extended-remote '
+            else:
+                target_help = '#   gdbserver :' + gdbserver_port + ' ' + binary
+                remote_cmd = 'target remote '
+            gdbinit_path = os.path.join(
+                modified_recipe.real_srctree, IdeNone.get_unique_gdbinit_name(binary))
+
+            gdbinit_lines = ['# This file is generated by devtool ide']
+            gdbinit_lines.append('# On the remote target:')
+            gdbinit_lines.append(target_help)
+            gdbinit_lines.append('# On the build machine:')
+            gdbinit_lines.append('#   cd ' + modified_recipe.real_srctree)
+            gdbinit_lines.append(
+                '#   ' + gdb_cross.gdb + ' -ix ' + gdbinit_path)
+
+            gdbinit_lines.append('set sysroot ' + modified_recipe.d)
+            gdbinit_lines.append('set substitute-path "/usr/include" "' +
+                                 os.path.join(modified_recipe.recipe_sysroot, 'usr', 'include') + '"')
+            # Disable debuginfod for now, the IDE configuration uses rootfs-dbg from the image workdir.
+            gdbinit_lines.append('set debuginfod enabled off')
+            if image.rootfs_dbg:
+                gdbinit_lines.append(
+                    'set solib-search-path "' + modified_recipe.solib_search_path_str(image) + '"')
+                gdbinit_lines.append('set substitute-path "/usr/src/debug" "' + os.path.join(
+                    image.rootfs_dbg, 'usr', 'src', 'debug') + '"')
+            gdbinit_lines.append(
+                remote_cmd + gdb_cross.host + ':' + gdbserver_port)
+            gdbinit_lines.append('set remote exec-file ' + binary)
+            gdbinit_lines.append(
+                'run ' + os.path.join(modified_recipe.d, binary))
+
+            with open(gdbinit_path, 'w') as gdbinit_file:
+                gdbinit_file.write('\n'.join(gdbinit_lines))
+            logger.info("Created .gdbinit: %s" % gdbinit_path)
+
+    def setup_modified_recipe(self, args, image_config, gdb_cross, modified_recipe):
+        if modified_recipe.build_tool == BuildTool.CMAKE:
+            modified_recipe.cmake_preset()
+        if modified_recipe.build_tool == BuildTool.MESON:
+            modified_recipe.gen_meson_wrapper()
+
+        if modified_recipe.build_tool == BuildTool.CMAKE or modified_recipe.build_tool == BuildTool.MESON:
+            self.none_launch(args, modified_recipe, image_config, gdb_cross)
-- 
2.41.0

Re: [OE-core] [PATCH v8 6/8] devtool: new ide plugin

[Richard Purdie]

On Wed, 2023-11-01 at 12:01 +0100, Adrian Freihofer wrote:
> The new devtool ide plugin configures an IDE to work with the eSDK.
> 
> With this initial implementation VSCode is the default IDE.
> The plugin works for recipes inheriting the cmake or the meson bbclass.
> Support for more programming languages and build tools may be added in
> the future.
> 
> Using the plugin in recipe modes:
> $ devtool modify a-recipe
> $ devtool ide a-recipe a-image
> $ code "$BUILDDIR/workspace/sources/a-recipe"
> Work in VSCode, after installing the proposed plugins
> 
> Using the plugin without a recipe
> $ devtool ide none a-image
> vscode where/the/sources/are
> Use the cross tool-chain which is provided as a cmake-kit.

Firstly, we should think carefully about the namespace. "ide" is in
this patch and I've heard talk about "esdk" elsewhere.

The plugin makes some assumptions since it is effectively behaving more
like an SDK than a full build environment. I did wonder if "ide-sdk"
might be the most appropriate?

Somehow we need to ensure users to be aware this uses a single shared
sysroot which is quite different to a specific single recipe
environment.

> The goal of this implementation is to create a configuration for VSCode
> (or other IDEs) that allows to work on the code of a recipe completely
> independent from bitbake. bitbake is only called if the configuration or
> the whole SDK has to be regenerated. But bitbake should not need to be
> called while working in the IDE. This has two major advantages over
> calling devtool build from the IDE:
> - The IDE provides plugins for integration with cmake, for example.
>   These features are usable, which would not be the case if bitbake or
>   devtool are called from within the IDE.
> - It is much faster.
> 
> Many thanks to Enguerrand de Ribaucourt for testing and bug fixing.
> 
> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>

The main thing worrying me about this plugin/code is the code
effectively copying and emulating bitbake:


> +    def gen_fakeroot_install_script(self):
> +        """Generate a helper script to execute make install with pseudo
> +
> +        For the deployment to the target device the do_install task must be
> +        executed out of the IDE as well. This function generates a script which
> +        runs the run.do_install script from bitbake under pseudo so that it picks
> +        up the appropriate file permissions. If the intent is to keep forking off bits of code which are too slow. Generating a self-contained script
> +        is much quicker than calling bitbake or devtool build from an IDE.
> +        """
> +        cmd_lines = ['#!/bin/sh']
> +        # Ensure the do compile step gets always executed without pseuso before do install
> +        # Running do_compile always without pseudo is probably better than trying to have
> +        # all the paths referred by compiling added to PSEUDO_IGNORE_PATHS.
> +        if self.cmd_compile:
> +            cmd_compile = "( cd %s && %s)" % (
> +                self.real_srctree, self.cmd_compile)
> +            cmd_lines.append(cmd_compile)
> +        if not os.access(self.fakerootcmd, os.X_OK):
> +            raise DevtoolError(
> +                "pseudo executable %s could not be found" % self.fakerootcmd)
> +        run_do_install = os.path.join(self.workdir, 'temp', 'run.do_install')
> +
> +        if not os.access(run_do_install, os.X_OK):
> +            raise DevtoolError(
> +                "run script does not exists: %s" % run_do_install)
> +
> +        # Set up the appropriate environment
> +        newenv = dict(os.environ)
> +        for varvalue in self.fakerootenv.split():
> +            if '=' in varvalue:
> +                splitval = varvalue.split('=', 1)
> +                newenv[splitval[0]] = splitval[1]
> +
> +        # Replicate the environment variables from bitbake
> +        for var, val in newenv.items():
> +            cmd_lines.append('export %s="%s"' % (var, val))
> +
> +        # Setup the task environment as bitbake would do it based on the varFlags
> +        for d in self.f_do_install_cleandirs:
> +            cmd_lines.append('%s rm -rf %s' % (self.fakerootcmd, d))
> +        for d in self.f_do_install_dirs:
> +            cmd_lines.append('%s mkdir -p %s' % (self.fakerootcmd, d))
> +        if len(self.f_do_install_dirs) > 0:
> +            cmd = "cd %s" % self.f_do_install_dirs[-1]
> +            cmd_lines.append('%s || { "%s failed"; exit 1; }' % (cmd, cmd))
> +
> +        # Remove the package* folders from TMPDIR. These folders might contain the sources for the -src packages.
> +        # This likely breaks pseudo like:
> +        # path mismatch [3 links]: ino 79147802 db
> +        # .../build/tmp/.../cmake-example/1.0/package/usr/src/debug/cmake-example/1.0-r0/oe-local-files/cpp-example-lib.cpp
> +        # .../build/workspace/sources/cmake-example/oe-local-files/cpp-example-lib.cpp
> +        # Since the files are anyway outdated lets deleted them (also from pseudo's db) to workaround this issue.
> +        pkg_dirs = ' '.join([os.path.join(self.workdir, d) for d in [
> +            "package", "packages-split", "pkgdata", "sstate-install-package", "debugsources.list", "*.spec"]])
> +        cmd = "%s rm -rf %s" % (self.fakerootcmd, pkg_dirs)
> +        cmd_lines.append('%s || { "%s failed"; exit 1; }' % (cmd, cmd))
> +
> +        # Finally call run.do_install on pseudo
> +        cmd = "%s %s" % (self.fakerootcmd, run_do_install)
> +        cmd_lines.append('%s || { "%s failed"; exit 1; }' % (cmd, cmd))
> +
> +        return self.write_script(cmd_lines, 'bb_run_do_install')
> +
> +    def gen_deploy_target_script(self, args):
> +        """Generate a script which does what devtool deploy-target does
> +
> +        This script is much quicker than devtool target-deploy. Because it
> +        does not need to start a bitbake server. All information from tinfoil
> +        is hard-coded in the generated script.
> +        """
> +        cmd_lines = ['#!/usr/bin/env python3']
> +        cmd_lines.append('import sys')
> +        cmd_lines.append('devtool_sys_path = %s' % str(sys.path))
> +        cmd_lines.append('devtool_sys_path.reverse()')
> +        cmd_lines.append('for p in devtool_sys_path:')
> +        cmd_lines.append('    if p not in sys.path:')
> +        cmd_lines.append('        sys.path.insert(0, p)')
> +        cmd_lines.append('from devtool.deploy import deploy_cached')
> +        args_filter = ['debug', 'dry_run', 'key', 'no_check_space', 'no_host_check',
> +                       'no_preserve', 'port', 'show_status', 'ssh_exec', 'strip', 'target']
> +        filtered_args_dict = {key: value for key, value in vars(
> +            args).items() if key in args_filter}
> +        cmd_lines.append('filtered_args_dict = %s' % str(filtered_args_dict))
> +        cmd_lines.append('class Dict2Class(object):')
> +        cmd_lines.append('    def __init__(self, my_dict):')
> +        cmd_lines.append('        for key in my_dict:')
> +        cmd_lines.append('            setattr(self, key, my_dict[key])')
> +        cmd_lines.append('filtered_args = Dict2Class(filtered_args_dict)')
> +        cmd_lines.append(
> +            'setattr(filtered_args, "recipename", "%s")' % self.bpn)
> +        cmd_lines.append('deploy_cached("%s", "%s", "%s", "%s", "%s", "%s", %d, "%s", "%s", filtered_args)' %
> +                         (self.d, self.work If the intent is to keep forking off bits of code which are too slow.dir, self.path, self.strip_cmd,
> +                          self.libdir, self.base_libdir, self.max_process,
> +                          self.fakerootcmd, self.fakerootenv))
> +        return self.write_script(cmd_lines, 'deploy_target')
> +
> +    def gen_install_deploy_script(self, args):
> +        """Generate a script which does install and deploy"""
> +        cmd_lines = ['#!/bin/sh -e']
> +        cmd_lines.append(self.gen_fakeroot_install_script())
> +        cmd_lines.append(self.gen_deploy_target_script(args))
> +        return self.write_script(cmd_lines, 'install_and_deploy')
> +
> +    def write_script(self, cmd_lines, script_name):
> +        bb.utils.mkdirhier(self.temp_dir)
> +        script_name_arch = script_name + '_' + self.recipe_id
> +        script_file = os.path.join(self.temp_dir, script_name_arch)
> +        with open(script_file, 'w') as script_f:
> +            script_f.write(os.linesep.join(cmd_lines))
> +        st = os.stat(script_file)
> +        os.chmod(script_file, st.st_mode | stat.S_IEXEC)
> +        return script_file

You say devtool is too slow as it has to start a bitbake server. Are
there ways we could get bitbake to behave in a fast enough way to make
this work well enough for the IDE so we don't have to duplicate this?

There are a few things which can be done:

a) memory resident bitbake means the start/stop of the server is no
longer needed

b) bitbake -b specifying a specific recipe is extremely fast as it
bypasses any need to parse multiple recipes

c) could a tinfoil client/daemon persist as a helper for the plugin in
the background to avoid having to fork these bits of code?


I suspect a lot comes down to intent with this code. If the intent is
that this is "done" and once merged, will stay like this or even
duplicate more bits as needed in future, I don't think I'm prepared to
take it. If we document clearly that this approach is a WIP and
ultimately we want to use something directly with bitbake, there are
probably paths to merging it.

I do want to have a clear understanding on what the ultimate goal and
code would look like.

Cheers,

Richard

Re: [OE-core] [PATCH v8 6/8] devtool: new ide plugin

[adrian.freihofer]

Hi Richard

First of all, thank you for your review.

On Mon, 2023-11-06 at 14:24 +0000, Richard Purdie wrote:
> On Wed, 2023-11-01 at 12:01 +0100, Adrian Freihofer wrote:
> > The new devtool ide plugin configures an IDE to work with the eSDK.
> > 
> > With this initial implementation VSCode is the default IDE.
> > The plugin works for recipes inheriting the cmake or the meson
> > bbclass.
> > Support for more programming languages and build tools may be added
> > in
> > the future.
> > 
> > Using the plugin in recipe modes:
> > $ devtool modify a-recipe
> > $ devtool ide a-recipe a-image
> > $ code "$BUILDDIR/workspace/sources/a-recipe"
> > Work in VSCode, after installing the proposed plugins
> > 
> > Using the plugin without a recipe
> > $ devtool ide none a-image
> > vscode where/the/sources/are
> > Use the cross tool-chain which is provided as a cmake-kit.
> 
> Firstly, we should think carefully about the namespace. "ide" is in
> this patch and I've heard talk about "esdk" elsewhere.
> 
> The plugin makes some assumptions since it is effectively behaving
> more
> like an SDK than a full build environment. I did wonder if "ide-sdk"
> might be the most appropriate?
> 
> Somehow we need to ensure users to be aware this uses a single shared
> sysroot which is quite different to a specific single recipe
> environment.

Renamed the plugin to devtool ide-sdk.
> 
> > The goal of this implementation is to create a configuration for
> > VSCode
> > (or other IDEs) that allows to work on the code of a recipe
> > completely
> > independent from bitbake. bitbake is only called if the
> > configuration or
> > the whole SDK has to be regenerated. But bitbake should not need to
> > be
> > called while working in the IDE. This has two major advantages over
> > calling devtool build from the IDE:
> > - The IDE provides plugins for integration with cmake, for example.
> >   These features are usable, which would not be the case if bitbake
> > or
> >   devtool are called from within the IDE.
> > - It is much faster.
> > 
> > Many thanks to Enguerrand de Ribaucourt for testing and bug fixing.
> > 
> > Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> 
> The main thing worrying me about this plugin/code is the code
> effectively copying and emulating bitbake:
> 
I see 2 different use cases:
   1. A Yocto developer needs to quickly fix a few lines of code.
      That's where devtool modify, devtool build works perfectly fine.
   2. An application developer needs to develop and maintain a cross
      compiled application as a full time job for many years. That's
      where bitbake is not the right tool for me.

When working with the IDE as an application developer, the IDE should
control all the build steps for the recipe which is in the workspace.
My understanding is that a cross development workflow usually requires
the following steps:
- Setup the SDK (bitbake, devtool)
- configure (IDE)
- compile (IDE)
- run unit tests (on the host with Qemu) (IDE)
- remote debuggging on the target device which includes (IDE)
- do_install (IDE,pseudo)
- deploying the artifacts to the target device (IDE,pseudo)
- start gdbserver on the target device (IDE)
- start GDB on the host device (IDE)
- Update the SDK (bitbake, devtool)

This bullet list shows where I think bitbake should be used and where I
think the IDE should do the job without calling bitbake for the
application development workflow. It's somewhat about separating two
parts of the workflow:
- Application development with an IDE and a static SDK
- Updating the SDK with bitbake.

The main problem with calling bitbake from the IDE is that all the IDE
plugins for handling e.g. cmake, meson, Python, Rust and other build
tools will not work if bitbake is between the IDE and the build tool.
That's why at least the configure and the compile steps must be
controlled by the IDE without bitbake. And that is also the main reason
for a design like devtool ide-sdk has.

When it comes to the do_install and other remote debugging related
tasks, the IDE anyway does not more than calling shell commands. There
could be a script which calls bitbake -c install and devtool deploy-
target. It would be slower but still usable. And bitbake could probably
be optimized to handle this workflow quicker. I agree. I also changed
my implementation back to work like this.

But it is more complicated. When calling bitbake -c install it runs a
task queue for all the tasks (e.g. do_configure, do_compile..) of the
recipe. For example, there is no guarantee that only the do_install
task will be executed. Bitbake always tries to find out if a previous
task needs to be executed and automatically does so. Since bitbake has
no information about which step has already be executed by the IDE
there is a great chance that bitbake does more than necessary. It could
also happen, that bitbake ignores changes done by the developer in the
IDE and overules some manual steps. An example of this is that bitbake
re-compiles everything when a file listed in the CONFIGURE_FILES
variable is changed. This makes no sense in the IDE context but can be
important otherwise.

Since you asked, I reworked my code and went through all these details
again. But at some point my conclusion was again that the complexity of
resolving many build dependencies should not be part of an application
development workflow (no bitbake). The SDK should be static by itself
but easily updated on request (bitbake). The only reaming argument for
calling bitbake which I can see is that this could theoretically become
the one fits all solution some when. But this idea is old and hasn't
materialized yet, which reinforces my assumption that I want to try
something simpler by replicating some lines of code into independent
generated scripts. Maybe later on we could return back to improve
bitbake for this use case as well.

When I look at all this together, I come to the following conclusion:
 * devtool ide-sdk is not ready for merging into poky. There are still
   some open questions. But poky is not the right repository for
   experimental stuff. I'll move the plugin to a separate layer in its
   own git repository and hopefully come back to upstreaming later.
 * Covering the Yocto-centric workflow with devtool modify, devtool
   build and devtool deploy-target is useful for Yocto developers. This
   could be covered by the VSCode bitbake plugin independently of the
   devtool ide-sdk plugin. However, I think that the bitbake plugin
   should not attempt to implement the more complicated application
   development workflow that the devtool ide-sdk plugin attempts to
   address. At some point, an implementation that queries details via
   Tinfoil from Bitbake should be used. But a Tinfoil based
   implementation should be covered by oe-selftest. Otherwise it will
   probably fall apart pretty quickly.
 * To avoid conflicts between the devtool ide-sdk plugin and the VSCode
   plugin, I will remove the generation of devtool build tasks. This
   would allow to have the VSCode plugin generating these tasks without
   conflicts.

> 
> You say devtool is too slow as it has to start a bitbake server. Are
> there ways we could get bitbake to behave in a fast enough way to
> make
> this work well enough for the IDE so we don't have to duplicate this?
> 
What we would need is a bitbake option which does not deal with
dependencies. Something like bitbake --run-only-this-task install a-
recipe. But this would be a very strange option for a tool which was
invented to deal with complex dependency situations.

> There are a few things which can be done:
> 
> a) memory resident bitbake means the start/stop of the server is no
> longer needed

It helps against re-parsing. But it comes with negative side effects: A
bitbake server in mem-res mode does not re-parse the recipes when they
are changed. Is this the expected behavior?

When an IDE starts a bitbake server in mem-res mode, changes to recipes
are ignored by all bitbake instances started in that build folder. This
is very confusing, especially for a user who did not intentionally
start the server in this mode.

> 
> b) bitbake -b specifying a specific recipe is extremely fast as it
> bypasses any need to parse multiple recipes

The -b also skips the parsing of bbappends. But bbappends are very
important for the use case in general and in particular because the
externalsrc class is injected by a bbappend created by devtool modify.

> 
> c) could a tinfoil client/daemon persist as a helper for the plugin
> in
> the background to avoid having to fork these bits of code?

Such a client would block the bitbake socket. But the idea is to have a
fast workflow in the IDE and support an SDK update workflow via
bitbake. I am not able to see a simple implementation based on this
idea.

> 
> I suspect a lot comes down to intent with this code. If the intent is
> that this is "done" and once merged, will stay like this or even
> duplicate more bits as needed in future, I don't think I'm prepared
> to
> take it. If we document clearly that this approach is a WIP and
> ultimately we want to use something directly with bitbake, there are
> probably paths to merging it.
> 
> I do want to have a clear understanding on what the ultimate goal and
> code would look like.

I will get back to you after I have gained more experience.
Unfortunately, I can't promise that this will be the big breakthrough
at the moment.

Best regards,
Adrian

> 
> Cheers,
> 
> Richard
> 

Re: [OE-core] [PATCH v8 6/8] devtool: new ide plugin

[Richard Purdie]

On Wed, 2023-11-15 at 15:01 +0100, adrian.freihofer@gmail.com wrote:
> When it comes to the do_install and other remote debugging related
> tasks, the IDE anyway does not more than calling shell commands. There
> could be a script which calls bitbake -c install and devtool deploy-
> target. It would be slower but still usable. And bitbake could probably
> be optimized to handle this workflow quicker. I agree. I also changed
> my implementation back to work like this.
> 
> But it is more complicated. When calling bitbake -c install it runs a
> task queue for all the tasks (e.g. do_configure, do_compile..) of the
> recipe. For example, there is no guarantee that only the do_install
> task will be executed. Bitbake always tries to find out if a previous
> task needs to be executed and automatically does so. Since bitbake has
> no information about which step has already be executed by the IDE
> there is a great chance that bitbake does more than necessary. It could
> also happen, that bitbake ignores changes done by the developer in the
> IDE and overules some manual steps. An example of this is that bitbake
> re-compiles everything when a file listed in the CONFIGURE_FILES
> variable is changed. This makes no sense in the IDE context but can be
> important otherwise.

FWIW I think it would be fine to add an API to tinfoil to execute a
task and only that task ignoring dependencies.

> Since you asked, I reworked my code and went through all these details
> again. But at some point my conclusion was again that the complexity of
> resolving many build dependencies should not be part of an application
> development workflow (no bitbake). The SDK should be static by itself
> but easily updated on request (bitbake). The only reaming argument for
> calling bitbake which I can see is that this could theoretically become
> the one fits all solution some when. But this idea is old and hasn't
> materialized yet, which reinforces my assumption that I want to try
> something simpler by replicating some lines of code into independent
> generated scripts. Maybe later on we could return back to improve
> bitbake for this use case as well.
> 
> When I look at all this together, I come to the following conclusion:
>  * devtool ide-sdk is not ready for merging into poky. There are still
>    some open questions. But poky is not the right repository for
>    experimental stuff. I'll move the plugin to a separate layer in its
>    own git repository and hopefully come back to upstreaming later.
>  * Covering the Yocto-centric workflow with devtool modify, devtool
>    build and devtool deploy-target is useful for Yocto developers. This
>    could be covered by the VSCode bitbake plugin independently of the
>    devtool ide-sdk plugin. However, I think that the bitbake plugin
>    should not attempt to implement the more complicated application
>    development workflow that the devtool ide-sdk plugin attempts to
>    address. At some point, an implementation that queries details via
>    Tinfoil from Bitbake should be used. But a Tinfoil based
>    implementation should be covered by oe-selftest. Otherwise it will
>    probably fall apart pretty quickly.
>  * To avoid conflicts between the devtool ide-sdk plugin and the VSCode
>    plugin, I will remove the generation of devtool build tasks. This
>    would allow to have the VSCode plugin generating these tasks without
>    conflicts.
> 
> > 
> > You say devtool is too slow as it has to start a bitbake server. Are
> > there ways we could get bitbake to behave in a fast enough way to
> > make
> > this work well enough for the IDE so we don't have to duplicate this?
> > 
> What we would need is a bitbake option which does not deal with
> dependencies. Something like bitbake --run-only-this-task install a-
> recipe. But this would be a very strange option for a tool which was
> invented to deal with complex dependency situations.

We could however implement it via tinfoil, which should also work for
this scenario?

> > There are a few things which can be done:
> > 
> > a) memory resident bitbake means the start/stop of the server is no
> > longer needed
> 
> It helps against re-parsing. But it comes with negative side effects: A
> bitbake server in mem-res mode does not re-parse the recipes when they
> are changed. Is this the expected behavior?

Until recently we used to monitor the filesystem with inotify however
there are no ordering guarantees with it so we can't order the
modifications data stream with our own parsing one. We recently ended
up removing it. The IDE can ask bitbake to validate it's caches and if
bitbake is in memory, that is a fast operation.

> When an IDE starts a bitbake server in mem-res mode, changes to recipes
> are ignored by all bitbake instances started in that build folder. This
> is very confusing, especially for a user who did not intentionally
> start the server in this mode.

Can the IDE provide some hints about when bitbake needs to do
something?

> > b) bitbake -b specifying a specific recipe is extremely fast as it
> > bypasses any need to parse multiple recipes
> 
> The -b also skips the parsing of bbappends. But bbappends are very
> important for the use case in general and in particular because the
> externalsrc class is injected by a bbappend created by devtool modify.

That is a valid concern, we should probably open a bug and think about
that. -b existed long before bbappends did!

> > c) could a tinfoil client/daemon persist as a helper for the plugin
> > in
> > the background to avoid having to fork these bits of code?
> 
> Such a client would block the bitbake socket. But the idea is to have a
> fast workflow in the IDE and support an SDK update workflow via
> bitbake. I am not able to see a simple implementation based on this
> idea.

I think it would depend on how the client was written. It could close
it's connection let leave the server memory resident?
Cheers,

Richard

Re: [OE-core] [PATCH v8 6/8] devtool: new ide plugin

[adrian.freihofer]

Hi Richard,

> FWIW I think it would be fine to add an API to tinfoil to execute a
> task and only that task ignoring dependencies.

I'm going to try to develop a patch for a Tinfoil API extension today.
I'm not sure if I'm up to it, but I'm sure I'm motivated to try. :-)
I think that this could lead to a very useful compromise.

Thank you!
Adrian

[PATCH v8 7/8] oe-selftest devtool: ide tests

[Adrian Freihofer]

Add some oe-selftests for the new devtool ide plugin. Most of the
workflows are covered.

Many thanks to Enguerrand de Ribaucourt for testing and bug fixing.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 meta/lib/oeqa/selftest/cases/devtool.py | 274 ++++++++++++++++++++++++
 1 file changed, 274 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index b5c488be8e8..8407465efaa 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -11,6 +11,7 @@ import tempfile
 import glob
 import fnmatch
 import unittest
+import json
 
 from oeqa.selftest.case import OESelftestTestCase
 from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer
@@ -2199,3 +2200,276 @@ class DevtoolUpgradeTests(DevtoolBase):
 
         #Step 4.5
         runCmd("grep %s %s" % (modconfopt, codeconfigfile))
+
+
+
+class DevtoolIdeTests(DevtoolBase):
+    def __write_bb_config(self, recipe_names):
+        """Helper to write the bitbake local.conf file"""
+        conf_lines = [
+            'IMAGE_GEN_DEBUGFS = "1"',
+            'IMAGE_INSTALL:append = " gdbserver %s"' % ' '.join([r + '-ptest' for r in recipe_names])
+        ]
+        self.write_config("\n".join(conf_lines))
+
+    def __devtool_ide_recipe(self, recipe_name, build_file, testimage):
+        """Setup a recipe for working with devtool ide
+
+        Basically devtool modify -x followed by some tests
+        """
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake -c clean %s' % recipe_name)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+
+        result = runCmd('devtool modify %s -x %s' % (recipe_name, tempdir))
+        self.assertExists(os.path.join(tempdir, build_file),
+                        'Extracted source could not be found')
+        self.assertExists(os.path.join(self.workspacedir, 'conf',
+                        'layer.conf'), 'Workspace directory not created')
+        matches = glob.glob(os.path.join(self.workspacedir,
+                            'appends', recipe_name + '.bbappend'))
+        self.assertTrue(matches, 'bbappend not created %s' % result.output)
+
+        # Test devtool status
+        result = runCmd('devtool status')
+        self.assertIn(recipe_name, result.output)
+        self.assertIn(tempdir, result.output)
+        self._check_src_repo(tempdir)
+
+        # Usually devtool ide would initiate the build of the SDK.
+        # But there is a circular dependency with starting Qemu and passing the IP of runqemu to devtool ide.
+        bitbake("%s qemu-native qemu-helper-native" % testimage)
+        deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE')
+        self.add_command_to_tearDown('bitbake -c clean %s' % testimage)
+        self.add_command_to_tearDown('rm -f %s/%s*' % (deploy_dir_image, testimage))
+
+        return tempdir
+
+    def __get_recipe_ids(self, recipe_name):
+        bb_vars = get_bb_vars(['PACKAGE_ARCH'])
+        package_arch = bb_vars['PACKAGE_ARCH']
+        recipe_id = recipe_name + "-" + package_arch
+        recipe_id_pretty = recipe_name + ": " + package_arch
+        return (recipe_id, recipe_id_pretty)
+
+    def __verify_install_script_code(self, tempdir, recipe_name):
+        """Verify the scripts referred by the tasks.json file are fine.
+
+        This function does not depend on Qemu. Therefore it verifies the scripts
+        exists and the install step works as expected. But it does not try to
+        deploy to Qemu.
+        """
+        recipe_id, recipe_id_pretty = self.__get_recipe_ids(recipe_name)
+        with open(os.path.join(tempdir, '.vscode', 'tasks.json')) as tasks_j:
+            tasks_d = json.load(tasks_j)
+        tasks = tasks_d["tasks"]
+        task_install = next((task for task in tasks if task["label"] == "install && deploy-target %s"  % recipe_id_pretty), None)
+        self.assertIsNot(task_install, None)
+        install_deploy_cmd = task_install["command"]
+        # execute only the bb_run_do_install script since the deploy would require e.g. Qemu running.
+        workspace_temp = os.path.join(self.builddir, 'workspace', 'temp', recipe_name)
+        i_and_d_script = "install_and_deploy_" + recipe_id
+        i_and_d_script_path = os.path.join(workspace_temp, i_and_d_script)
+        self.assertExists(i_and_d_script_path)
+        i_script = "bb_run_do_install_" + recipe_id
+        install_cmd = install_deploy_cmd.replace(i_and_d_script, i_script)
+        install_cmd_path = os.path.join(workspace_temp, install_cmd)
+        self.assertExists(install_cmd_path)
+        runCmd(install_cmd, cwd=tempdir)
+
+    def __devtool_ide_qemu(self, tempdir, qemu, recipe_name, example_exe):
+        """Verify deployment and execution in Qemu system work for one recipe.
+
+        This function checks the entire SDK workflow: changing the code, recompiling
+        it and deploying it back to Qemu, and checking that the changes have been
+        incorporated into the provided binaries. It also runs the tests of the recipe.
+        """
+        recipe_id, _ = self.__get_recipe_ids(recipe_name)
+        i_and_d_script = "install_and_deploy_" + recipe_id
+        install_deploy_cmd = os.path.join(self.builddir, 'workspace', 'temp', recipe_name, i_and_d_script)
+        self.assertExists(install_deploy_cmd, '%s script not found' % install_deploy_cmd)
+        runCmd(install_deploy_cmd)
+
+        MAGIC_STRING_ORIG = "Magic: 123456789"
+        MAGIC_STRING_NEW = "Magic: 987654321"
+        ptest_cmd = "ptest-runner " + recipe_name
+
+        # validate that SSH is working
+        status, _ = qemu.run("uname")
+        self.assertEqual(status, 0, msg="Failed to connect to the SSH server on Qemu")
+
+        # Verify the unmodified example prints the magic string
+        status, output = qemu.run(example_exe)
+        self.assertEqual(status, 0, msg="%s failed: %s" % (example_exe, output))
+        self.assertIn(MAGIC_STRING_ORIG, output)
+
+        # Verify the unmodified ptests work
+        status, output = qemu.run(ptest_cmd)
+        self.assertEqual(status, 0, msg="%s failed: %s" % (ptest_cmd, output))
+        self.assertIn("PASS: cpp-example-lib", output)
+
+        # Replace the Magic String in the code, compile and deploy to Qemu
+        cpp_example_lib_hpp = os.path.join(tempdir, 'cpp-example-lib.hpp')
+        with open(cpp_example_lib_hpp, 'r') as file:
+            cpp_code = file.read()
+            cpp_code = cpp_code.replace(MAGIC_STRING_ORIG, MAGIC_STRING_NEW)
+        with open(cpp_example_lib_hpp, 'w') as file:
+            file.write(cpp_code)
+        runCmd(install_deploy_cmd, cwd=tempdir)
+
+        # Verify the modified example prints the modified magic string
+        status, output = qemu.run(example_exe)
+        self.assertEqual(status, 0, msg="%s failed: %s" % (example_exe, output))
+        self.assertNotIn(MAGIC_STRING_ORIG, output)
+        self.assertIn(MAGIC_STRING_NEW, output)
+
+        # Verify the modified example ptests work
+        status, output = qemu.run(ptest_cmd)
+        self.assertEqual(status, 0, msg="%s failed: %s" % (ptest_cmd, output))
+        self.assertIn("PASS: cpp-example-lib", output)
+
+    def __verify_cmake_preset(self, tempdir):
+        """Verify the generated cmake preset works as expected
+
+        Check if compiling works
+        Check if unit tests can be executed in qemu (not qemu-system)
+        """
+        with open(os.path.join(tempdir, 'CMakeUserPresets.json')) as cmake_preset_j:
+            cmake_preset_d = json.load(cmake_preset_j)
+        config_presets = cmake_preset_d["configurePresets"]
+        self.assertEqual(len(config_presets), 1)
+        cmake_exe = config_presets[0]["cmakeExecutable"]
+        preset_name = config_presets[0]["name"]
+
+        # Verify the wrapper for cmake native is available
+        self.assertExists(cmake_exe)
+
+        # Verify the cmake preset generated by devtool ide is available
+        result = runCmd('%s --list-presets' % cmake_exe, cwd=tempdir)
+        self.assertIn(preset_name, result.output)
+
+        # Verify cmake re-uses the o files compiled by bitbake
+        result = runCmd('%s --build --preset %s' % (cmake_exe, preset_name), cwd=tempdir)
+        self.assertIn("ninja: no work to do.", result.output)
+
+        # Verify the unit tests work (in Qemu user mode)
+        result = runCmd('%s --build --preset %s --target test' % (cmake_exe, preset_name), cwd=tempdir)
+        self.assertIn("100% tests passed", result.output)
+
+        # Verify re-building and testing works again
+        result = runCmd('%s --build --preset %s --target clean' % (cmake_exe, preset_name), cwd=tempdir)
+        self.assertIn("Cleaning", result.output)
+        result = runCmd('%s --build --preset %s' % (cmake_exe, preset_name), cwd=tempdir)
+        self.assertIn("Building", result.output)
+        self.assertIn("Linking", result.output)
+        result = runCmd('%s --build --preset %s --target test' % (cmake_exe, preset_name), cwd=tempdir)
+        self.assertIn("Running tests...", result.output)
+        self.assertIn("100% tests passed", result.output)
+
+    @OETestTag("runqemu")
+    def test_devtool_ide_none_qemu(self):
+        """Start qemu-system and run tests for multiple recipes. ide=none is used."""
+        recipe_names = ["cmake-example", "meson-example"]
+        testimage = "oe-selftest-image"
+
+        self.__write_bb_config(recipe_names)
+        self._check_runqemu_prerequisites()
+
+        # Verify deployment to Qemu (system mode) works
+        bitbake(testimage)
+        with runqemu(testimage, runqemuparams="nographic") as qemu:
+            # cmake-example recipe
+            recipe_name = "cmake-example"
+            example_exe = "cmake-example"
+            build_file = "CMakeLists.txt"
+            tempdir = self.__devtool_ide_recipe(recipe_name, build_file, testimage)
+            bitbake_sdk_cmd = 'devtool ide %s %s -t root@%s -c --ide=none' % (recipe_name, testimage, qemu.ip)
+            runCmd(bitbake_sdk_cmd)
+            self.__verify_cmake_preset(tempdir)
+            self.__devtool_ide_qemu(tempdir, qemu, recipe_name, example_exe)
+
+            # meson-example recipe
+            recipe_name = "meson-example"
+            example_exe = "mesonex"
+            build_file = "meson.build"
+            tempdir = self.__devtool_ide_recipe(recipe_name, build_file, testimage)
+            bitbake_sdk_cmd = 'devtool ide %s %s -t root@%s -c --ide=none' % (recipe_name, testimage, qemu.ip)
+            runCmd(bitbake_sdk_cmd)
+            self.__devtool_ide_qemu(tempdir, qemu, recipe_name, example_exe)
+
+
+    def test_devtool_ide_code_cmake(self):
+        """Verify a cmake recipe works with ide=code mode"""
+        recipe_name = "cmake-example"
+        build_file = "CMakeLists.txt"
+        testimage = "oe-selftest-image"
+
+        self.__write_bb_config([recipe_name])
+        tempdir = self.__devtool_ide_recipe(recipe_name, build_file, testimage)
+        bitbake_sdk_cmd = 'devtool ide %s %s -t root@192.168.17.17 -c --ide=code' % (recipe_name, testimage)
+        runCmd(bitbake_sdk_cmd)
+        self.__verify_cmake_preset(tempdir)
+        self.__verify_install_script_code(tempdir,  recipe_name)
+
+    def test_devtool_ide_code_meson(self):
+        """Verify a meson recipe works with ide=code mode"""
+        recipe_name = "meson-example"
+        build_file = "meson.build"
+        testimage = "oe-selftest-image"
+
+        self.__write_bb_config([recipe_name])
+        tempdir = self.__devtool_ide_recipe(recipe_name, build_file, testimage)
+        bitbake_sdk_cmd = 'devtool ide %s %s -t root@192.168.17.17 -c --ide=code' % (recipe_name, testimage)
+        runCmd(bitbake_sdk_cmd)
+
+        with open(os.path.join(tempdir, '.vscode', 'settings.json')) as settings_j:
+            settings_d = json.load(settings_j)
+        meson_exe = settings_d["mesonbuild.mesonPath"]
+        meson_build_folder = settings_d["mesonbuild.buildFolder"]
+
+        # Verify the wrapper for meson native is available
+        self.assertExists(meson_exe)
+
+        # Verify meson re-uses the o files compiled by bitbake
+        result = runCmd('%s compile -C  %s' % (meson_exe, meson_build_folder), cwd=tempdir)
+        self.assertIn("ninja: no work to do.", result.output)
+
+        # Verify the unit tests work (in Qemu)
+        runCmd('%s test -C  %s' % (meson_exe, meson_build_folder), cwd=tempdir)
+
+        # Verify re-building and testing works again
+        result = runCmd('%s compile -C  %s --clean' % (meson_exe, meson_build_folder), cwd=tempdir)
+        self.assertIn("Cleaning...", result.output)
+        result = runCmd('%s compile -C  %s' % (meson_exe, meson_build_folder), cwd=tempdir)
+        self.assertIn("Linking target", result.output)
+        runCmd('%s test -C  %s' % (meson_exe, meson_build_folder), cwd=tempdir)
+
+        self.__verify_install_script_code(tempdir,  recipe_name)
+
+    def test_devtool_ide_shared_sysroots(self):
+        """Verify the shared sysroot SDK without a recipe works fine."""
+        result = runCmd('devtool ide none oe-selftest-image --ide=code')
+        bb_vars = get_bb_vars(['MULTIMACH_TARGET_SYS', 'DEPLOY_DIR_IMAGE', 'COREBASE'])
+        environment_script = 'environment-setup-%s' % bb_vars['MULTIMACH_TARGET_SYS']
+        deploydir = bb_vars['DEPLOY_DIR_IMAGE']
+        environment_script_path = os.path.join(deploydir, environment_script)
+
+        # Verify the printed note really referres to a cmake executable
+        cmake_native = ""
+        for line in result.output.splitlines():
+            m = re.search(r'"cmake.cmakePath": "(.*)"', line)
+            if m:
+                cmake_native = m.group(1)
+                break
+
+        self.assertExists(cmake_native)
+        self.assertExists(environment_script_path)
+
+        # Verify building the cmake-example works
+        cmake_example_src = os.path.join(bb_vars['COREBASE'], 'meta-selftest', 'recipes-test', 'cpp', 'files')
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        result = runCmd('%s %s' % (cmake_native, cmake_example_src), cwd=tempdir)
+        result = runCmd('%s --build %s' % (cmake_native, tempdir), cwd=tempdir)
-- 
2.41.0

[PATCH v8 8/8] docs: cover devtool ide

[Adrian Freihofer]

Cover the new devtool ide plugin in the extensible sdk section.

Many thanks to Enguerrand de Ribaucourt for his re-view and
contributions.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 documentation/sdk-manual/extensible.rst | 153 +++++++++++++++++++++++-
 1 file changed, 152 insertions(+), 1 deletion(-)

diff --git a/documentation/sdk-manual/extensible.rst b/documentation/sdk-manual/extensible.rst
index 355c6cb0e4a..51d9fff2638 100644
--- a/documentation/sdk-manual/extensible.rst
+++ b/documentation/sdk-manual/extensible.rst
@@ -63,6 +63,8 @@ their own pros and cons:
    need to provide a well-functioning binary artefact cache over the network
    for developers with underpowered laptops.
 
+.. _setting_up_ext_sdk_in_build:
+
 Setting up the Extensible SDK environment directly in a Yocto build
 -------------------------------------------------------------------
 
@@ -230,13 +232,15 @@ all the commands.
    See the ":doc:`/ref-manual/devtool-reference`"
    section in the Yocto Project Reference Manual.
 
-Three ``devtool`` subcommands provide entry-points into development:
+Four ``devtool`` subcommands provide entry-points into development:
 
 -  *devtool add*: Assists in adding new software to be built.
 
 -  *devtool modify*: Sets up an environment to enable you to modify
    the source of an existing component.
 
+-  *devtool ide*: Generates a configuration for an IDE.
+
 -  *devtool upgrade*: Updates an existing recipe so that you can
    build it for an updated set of source files.
 
@@ -614,6 +618,153 @@ command:
       decide you do not want to proceed with your work. If you do use this
       command, realize that the source tree is preserved.
 
+Use ``devtool ide`` to generate a configuration for the IDE
+-----------------------------------------------------------
+
+``devtool ide`` automatically configures IDEs for cross-compiling and remote debugging.
+
+Two different use cases are supported:
+
+#. *Recipe mode*: Generate the IDE configuration for a workspace created by ``devtool modify``.
+
+   In order to use the tool, a few settings must be made.
+   As a starting example, the following lines of code can be added to the ``local.conf`` file::
+
+      # Build the companion debug file system
+      IMAGE_GEN_DEBUGFS = "1"
+      # Optimize build time: with devtool ide the dbg tar is not needed
+      IMAGE_FSTYPES_DEBUGFS = ""
+
+      # ssh is mandatory, no password simplifies the usage
+      EXTRA_IMAGE_FEATURES += "\
+         ssh-server-openssh \
+         debug-tweaks \
+      "
+
+      # Remote debugging needs the gdbserver on the target device
+      IMAGE_INSTALL:append = " gdbserver"
+
+   Assuming the development environment is set up correctly and a workspace has been created
+   for the recipe using ``devtool modify recipe``, the following command can create the
+   configuration for VSCode in the recipe workspace::
+
+      $ devtool ide recipe core-image-minimal --target root@192.168.7.2
+
+   What this command does exactly depends on the recipe or the build tool used by the recipe.
+   Currently, only CMake and Meson are supported natively.
+   Here is what it does for a recipe which inherits the :ref:`ref-classes-cmake` class:
+
+   - Prepare the SDK by calling ``bitbake core-image-minimal``, ``gdb-cross``, ``qemu-native``...
+
+   - Generate a CMake preset with configures CMake to use exactly the same environent and
+     the same CMmake cache configuration as used by ``bitbake recipe``. The CMake preset referres
+     to the per-recipe-sysroot of the recipe.
+
+     Currently Configure, Build and Test presets are supported. Test presets execute the test
+     binaries with Qemu (qemu-user not qemu-system).
+
+   - Generate a helper script to handle the ``do_install`` with pseudo.
+
+   - Generate a helper script which does the same as ``devtool deploy-target`` does.
+     Therefore ``devtool ide`` supports the same set of target related command line parameters
+     as ``devtool deploy-target`` does.
+     The ``--target`` parameter in the example above is just an example for that. It is optional.
+
+   - Generate some helper scripts to start ``gdbserver`` on the target device.
+
+   - Generate the ``.vscode`` folder containing the following files:
+
+   - ``c_ccp_properties.json``: Configure IntelliSense for code navigation.
+
+   - ``extensions.json``: Recommend the extensions which are used.
+
+   - ``launch.json``: Provide a configuration for remote debugging with ``gdb-cross`` and ``gdbserver``.
+     The debug-symbols are searched in the build-folder and in the rootfs-dbg folder
+     which is provided by the image recipe passed to the ``devtool ide`` command.
+
+   - ``settings.json``: Configure the code indexers to ignore the build folders.
+     Without reasonable exclude settings, it is very likely that some indexer plugins will run
+     with 100% CPU load until an OOM exception occurs. In the case of VSCode, the indexers started
+     immediately and do not stop or reconfigure when the ignore list is updated. This means that the
+     settings.json file must be available before VSCode is started in the workspace folder or before
+     ``devtool modify`` has added the ``oe-workdir`` symlink to the large build folder of ``bitbake``.
+
+   - ``tasks.json``: Provide some helpers for running
+
+      - :ref:`ref-tasks-install` and ``devtool deploy-target``
+
+      - start ``gdbserver`` via ssh
+
+   For a recipe which inherits the :ref:`ref-classes-meson` a similar configuration is generated.
+   Because there is nothing like a Meson preset a wrapper script for Meson is generated.
+
+   It's possible to pass multiple recipes to the ``devtool ide`` command.
+   ``devtool ide`` tries to handle the recipes in a reasonable way if possible.
+
+   ``devtool ide`` aims to support multiple programming languages and multiple IDEs natively.
+   Native means that the IDE is configured to call the build tool (e.g. CMake or Meson) directly.
+   This has several advantages. First of all, it is much faster than ``devtool build``, for example.
+   But it also allows to use the very good integration of tools like CMake or GDB directly with VSCode or other IDEs.
+   However, supporting many programming languages and multiple IDEs is quite an elaborate and constantly evolving thing.
+   To handle combinations that are not natively supported, ``devtool ide`` creates a configuration that falls back
+   to ``devtool build`` and ``devtool deploy-target`` if there is no native support available.
+
+   The default IDE is VSCode. Some hints about using VSCode:
+
+   - To work with CMake press ``Ctrl + Shift + p``, type ``cmake``.
+     This will show some possible commands like selecting a CMake preset, compiling or running CTest.
+
+   - To work with Meson press ``Ctrl + Shift + p``, type ``meson``.
+     This will show some possible commands like compiling or executing the unit tests.
+
+   - For the deployment to the target device, just press ``Ctrl + Shift + p``, type ``task``.
+     Select the ``install & deploy task``.
+
+   - For remote debugging, switch to the debugging view by pressing the "play" button with the ``bug icon`` on the left side.
+     This will provide a green play button with a drop-down list where a debug configuration can be selected.
+     After selecting one of the generated configurations, press the "play" button.
+
+     Starting a remote debugging sesssion automatically initiates the deployment to the target device.
+     If this is not desired, the ``"dependsOn": ["install && deploy-target...]`` parameter of the tasks
+     with ``"label": "gdbserver start...`` can be removed from the ``tasks.json`` file.
+
+   Additionally ``--ide=none`` is supported.
+   With the none IDE parameter some generic configurations files like ``.gdbinit`` files and some helper scripts
+   starting the gdbserver remotely on the target device as well as the gdb client on the host are generated.
+
+   .. note::
+
+      To ensure that the debug symbols on the build machine match the binaries running on the target system,
+      it is essential that the image built by ``devtool ide`` is running on the target system.
+
+#. *Shared sysroots mode*: Generate the IDE configuration for using a cross-toolchain as provided by
+   ``bitbake meta-ide-support build-sysroots``.
+
+   For some special recipes and use cases a per-recipe-sysroot based SDK is not suitable.
+   Therefore ``devtool ide`` also supports setting up the shared sysroots environment and generating
+   IDE configurations referring to the shared sysroots. Recipes leading to a shared sysroot
+   are for example ``meta-ide-support`` or ``shared-sysroots``.
+   Also passing ``none`` as a recipe name leads to a shared sysroot SDK::
+
+      $ devtool ide none core-image-minimal
+
+   For VSCode the cross-tool-chain is exposed as a CMake kit. CMake kits are defined in
+   ``~/.local/share/CMakeTools/cmake-tools-kits.json``.
+   The following minimalistic example shows how the cross-toolchain can be selected in VSCode.
+   Fist of all we create a minimalistic CMake project and start VSCode::
+
+      mkdir kit-test
+      echo "project(foo VERSION 1.0)" > kit-test/CMakeLists.txt
+      code kit-test
+
+   If there is a CMake project in the workspace cross-compilation is supported:
+
+   - Press ``Ctrl + Shift + P``, type ``CMake: Scan for Kits``
+   - Press ``Ctrl + Shift + P``, type ``CMake: Select a Kit``
+
+   For other IDEs than VSCode ``devtool ide none ...`` is just a simple wrapper for the
+   setup of the extensible SDK as decribed in :ref:`setting_up_ext_sdk_in_build`.
+
 Use ``devtool upgrade`` to Create a Version of the Recipe that Supports a Newer Version of the Software
 -------------------------------------------------------------------------------------------------------
 
-- 
2.41.0

Re: [OE-core] [PATCH v8 8/8] docs: cover devtool ide

[Quentin Schulz]

Hi Adrian,

Please at least Cc the docs mailing list :) The docs is actually handled 
in a separate repo (well the yocto docs, for bitbake it's in the bitbake 
mono-repo along side the code), by different people.

Cc'ing docs.

First, thanks for sending documentation related to a new feature while 
it's being developed, this is very much appreciated.

As an overall comment, I feel like the documentation about the internal 
working is not necessary. The user shouldn't care about what the tools 
does internally, only how to use it and configure it. If somehow it is 
broken, not supporting a specific use-case or feature, then a look at 
the code is warranted, where the documentation of the internals is desired.
The part I'm talking about is the one between
+   Here is what it does for a recipe which inherits the 
:ref:`ref-classes-cmake` class:
and
+   It's possible to pass multiple recipes to the ``devtool ide`` command.

Other comments inline, only typos. I have never used VSCode and I'm not 
knowledgeable in Meson nor Cmake nor SDKs so that all i could contribute 
to this patch review :)

On 11/1/23 12:01, Adrian Freihofer via lists.openembedded.org wrote:
> Cover the new devtool ide plugin in the extensible sdk section.
> 
> Many thanks to Enguerrand de Ribaucourt for his re-view and
> contributions.
> 
> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> ---
>   documentation/sdk-manual/extensible.rst | 153 +++++++++++++++++++++++-
>   1 file changed, 152 insertions(+), 1 deletion(-)
> 
> diff --git a/documentation/sdk-manual/extensible.rst b/documentation/sdk-manual/extensible.rst
> index 355c6cb0e4a..51d9fff2638 100644
> --- a/documentation/sdk-manual/extensible.rst
> +++ b/documentation/sdk-manual/extensible.rst
> @@ -63,6 +63,8 @@ their own pros and cons:
>      need to provide a well-functioning binary artefact cache over the network
>      for developers with underpowered laptops.
>   
> +.. _setting_up_ext_sdk_in_build:
> +
>   Setting up the Extensible SDK environment directly in a Yocto build
>   -------------------------------------------------------------------
>   
> @@ -230,13 +232,15 @@ all the commands.
>      See the ":doc:`/ref-manual/devtool-reference`"
>      section in the Yocto Project Reference Manual.
>   
> -Three ``devtool`` subcommands provide entry-points into development:
> +Four ``devtool`` subcommands provide entry-points into development:
>   

Just remove the number so we don't have to maintain it.

>   -  *devtool add*: Assists in adding new software to be built.
>   
>   -  *devtool modify*: Sets up an environment to enable you to modify
>      the source of an existing component.
>   
> +-  *devtool ide*: Generates a configuration for an IDE.
> +
>   -  *devtool upgrade*: Updates an existing recipe so that you can
>      build it for an updated set of source files.
>   
> @@ -614,6 +618,153 @@ command:
>         decide you do not want to proceed with your work. If you do use this
>         command, realize that the source tree is preserved.
>   
> +Use ``devtool ide`` to generate a configuration for the IDE
> +-----------------------------------------------------------
> +
> +``devtool ide`` automatically configures IDEs for cross-compiling and remote debugging.
> +
> +Two different use cases are supported:
> +

Ditto.

> +#. *Recipe mode*: Generate the IDE configuration for a workspace created by ``devtool modify``.
> +
> +   In order to use the tool, a few settings must be made.
> +   As a starting example, the following lines of code can be added to the ``local.conf`` file::
> +
> +      # Build the companion debug file system
> +      IMAGE_GEN_DEBUGFS = "1"
> +      # Optimize build time: with devtool ide the dbg tar is not needed
> +      IMAGE_FSTYPES_DEBUGFS = ""
> +
> +      # ssh is mandatory, no password simplifies the usage
> +      EXTRA_IMAGE_FEATURES += "\
> +         ssh-server-openssh \
> +         debug-tweaks \
> +      "
> +
> +      # Remote debugging needs the gdbserver on the target device
> +      IMAGE_INSTALL:append = " gdbserver"
> +
> +   Assuming the development environment is set up correctly and a workspace has been created
> +   for the recipe using ``devtool modify recipe``, the following command can create the
> +   configuration for VSCode in the recipe workspace::
> +
> +      $ devtool ide recipe core-image-minimal --target root@192.168.7.2
> +
> +   What this command does exactly depends on the recipe or the build tool used by the recipe.
> +   Currently, only CMake and Meson are supported natively.
> +   Here is what it does for a recipe which inherits the :ref:`ref-classes-cmake` class:
> +
> +   - Prepare the SDK by calling ``bitbake core-image-minimal``, ``gdb-cross``, ``qemu-native``...
> +
> +   - Generate a CMake preset with configures CMake to use exactly the same environent and

s/with/which/

s/environent/environment/

> +     the same CMmake cache configuration as used by ``bitbake recipe``. The CMake preset referres

s/referres/refers/

> +     to the per-recipe-sysroot of the recipe.
> +

s/per-recipe-sysroot/per-recipe sysroot/

> +     Currently Configure, Build and Test presets are supported. Test presets execute the test
> +     binaries with Qemu (qemu-user not qemu-system).
> +
> +   - Generate a helper script to handle the ``do_install`` with pseudo.
> +

Please use a reference for do_install. :ref:`ref-tasks-install`

> +   - Generate a helper script which does the same as ``devtool deploy-target`` does.
> +     Therefore ``devtool ide`` supports the same set of target related command line parameters
> +     as ``devtool deploy-target`` does.
> +     The ``--target`` parameter in the example above is just an example for that. It is optional.
> +
> +   - Generate some helper scripts to start ``gdbserver`` on the target device.
> +
> +   - Generate the ``.vscode`` folder containing the following files:
> +
> +   - ``c_ccp_properties.json``: Configure IntelliSense for code navigation.
> +
> +   - ``extensions.json``: Recommend the extensions which are used.
> +
> +   - ``launch.json``: Provide a configuration for remote debugging with ``gdb-cross`` and ``gdbserver``.
> +     The debug-symbols are searched in the build-folder and in the rootfs-dbg folder

s/debug-symbols/debug symbols/
s/build-folder/build folder/ maybe actually provide the variable name 
(if there's one) so we know where to look (because I don't really know 
what this actually means in terms of paths).

> +     which is provided by the image recipe passed to the ``devtool ide`` command.
> +
> +   - ``settings.json``: Configure the code indexers to ignore the build folders.
> +     Without reasonable exclude settings, it is very likely that some indexer plugins will run
> +     with 100% CPU load until an OOM exception occurs. In the case of VSCode, the indexers started
> +     immediately and do not stop or reconfigure when the ignore list is updated. This means that the
> +     settings.json file must be available before VSCode is started in the workspace folder or before
> +     ``devtool modify`` has added the ``oe-workdir`` symlink to the large build folder of ``bitbake``.
> +
> +   - ``tasks.json``: Provide some helpers for running
> +

s/running//

> +      - :ref:`ref-tasks-install` and ``devtool deploy-target``
> +

+running
at the beginning of the bullet point

> +      - start ``gdbserver`` via ssh
> +

s/start/starting/


> +   For a recipe which inherits the :ref:`ref-classes-meson` a similar configuration is generated.
> +   Because there is nothing like a Meson preset a wrapper script for Meson is generated.
> +
> +   It's possible to pass multiple recipes to the ``devtool ide`` command.
> +   ``devtool ide`` tries to handle the recipes in a reasonable way if possible.
> +
> +   ``devtool ide`` aims to support multiple programming languages and multiple IDEs natively.
> +   Native means that the IDE is configured to call the build tool (e.g. CMake or Meson) directly.
> +   This has several advantages. First of all, it is much faster than ``devtool build``, for example.
> +   But it also allows to use the very good integration of tools like CMake or GDB directly with VSCode or other IDEs.
> +   However, supporting many programming languages and multiple IDEs is quite an elaborate and constantly evolving thing.
> +   To handle combinations that are not natively supported, ``devtool ide`` creates a configuration that falls back
> +   to ``devtool build`` and ``devtool deploy-target`` if there is no native support available.
> +
> +   The default IDE is VSCode. Some hints about using VSCode:
> +
> +   - To work with CMake press ``Ctrl + Shift + p``, type ``cmake``.
> +     This will show some possible commands like selecting a CMake preset, compiling or running CTest.
> +
> +   - To work with Meson press ``Ctrl + Shift + p``, type ``meson``.
> +     This will show some possible commands like compiling or executing the unit tests.
> +
> +   - For the deployment to the target device, just press ``Ctrl + Shift + p``, type ``task``.
> +     Select the ``install & deploy task``.
> +
> +   - For remote debugging, switch to the debugging view by pressing the "play" button with the ``bug icon`` on the left side.
> +     This will provide a green play button with a drop-down list where a debug configuration can be selected.
> +     After selecting one of the generated configurations, press the "play" button.
> +
> +     Starting a remote debugging sesssion automatically initiates the deployment to the target device.
> +     If this is not desired, the ``"dependsOn": ["install && deploy-target...]`` parameter of the tasks
> +     with ``"label": "gdbserver start...`` can be removed from the ``tasks.json`` file.
> +
> +   Additionally ``--ide=none`` is supported.
> +   With the none IDE parameter some generic configurations files like ``.gdbinit`` files and some helper scripts
> +   starting the gdbserver remotely on the target device as well as the gdb client on the host are generated.
> +
> +   .. note::
> +
> +      To ensure that the debug symbols on the build machine match the binaries running on the target system,
> +      it is essential that the image built by ``devtool ide`` is running on the target system.
> +
> +#. *Shared sysroots mode*: Generate the IDE configuration for using a cross-toolchain as provided by
> +   ``bitbake meta-ide-support build-sysroots``.
> +
> +   For some special recipes and use cases a per-recipe-sysroot based SDK is not suitable.

s/per-recipe-sysroot/per-recipe sysroot/

> +   Therefore ``devtool ide`` also supports setting up the shared sysroots environment and generating
> +   IDE configurations referring to the shared sysroots. Recipes leading to a shared sysroot
> +   are for example ``meta-ide-support`` or ``shared-sysroots``.
> +   Also passing ``none`` as a recipe name leads to a shared sysroot SDK::
> +
> +      $ devtool ide none core-image-minimal
> +
> +   For VSCode the cross-tool-chain is exposed as a CMake kit. CMake kits are defined in
> +   ``~/.local/share/CMakeTools/cmake-tools-kits.json``.
> +   The following minimalistic example shows how the cross-toolchain can be selected in VSCode.

s/minimalistic/minimalist/ (or just remove the word entirely)

> +   Fist of all we create a minimalistic CMake project and start VSCode::
> +

Ditto.

> +      mkdir kit-test
> +      echo "project(foo VERSION 1.0)" > kit-test/CMakeLists.txt
> +      code kit-test
> +
> +   If there is a CMake project in the workspace cross-compilation is supported:
> +
> +   - Press ``Ctrl + Shift + P``, type ``CMake: Scan for Kits``
> +   - Press ``Ctrl + Shift + P``, type ``CMake: Select a Kit``
> +
> +   For other IDEs than VSCode ``devtool ide none ...`` is just a simple wrapper for the
> +   setup of the extensible SDK as decribed in :ref:`setting_up_ext_sdk_in_build`.
> +

s/decribed/described/

Cheers,
Quentin

Re: [OE-core] [PATCH v8 8/8] docs: cover devtool ide

[adrian.freihofer]

Hi Michael, hi Quentin

Sent a new version to the docs mailing list:
https://lists.yoctoproject.org/g/docs/message/4578

I hope this will address all your findings and I also hope that the v8
of the devtool ide will be accepted soon.

Adrian