[PATCH 0/4] Add meson wrap fallback for slirp & dtc

[PATCH 0/4] Add meson wrap fallback for slirp & dtc

[marcandre.lureau]

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Hi,

Meson "wrap" is a mechanism to build dependencies that doesn't rely on git
submodules and integrate external dependencies as subproject()s.

This offers developpers a simpler way to build QEMU with missing system
dependencies (ex, libslirp in my case), but also simplify the fallback build
definition of dtc/libfdt.

In constrast with QEMU configure submodule handling, the subprojects are not
downloaded automatically, and the user has to call "meson subprojects download"
himself prior to running configure/meson.

Marc-André Lureau (4):
  mtest2make.py: teach suite name that are just "PROJECT"
  build-sys: prevent meson from downloading wrapped subprojects
  build-sys: add slirp.wrap
  build-sys: replace dtc submodule with dtc.wrap

 configure                     | 25 +++-------------
 meson.build                   | 56 +++++------------------------------
 .gitignore                    |  3 ++
 .gitmodules                   |  3 --
 dtc                           |  1 -
 meson_options.txt             |  5 ++--
 scripts/meson-buildoptions.sh |  4 +--
 scripts/mtest2make.py         |  9 +++---
 subprojects/dtc.wrap          |  6 ++++
 subprojects/slirp.wrap        |  6 ++++
 10 files changed, 35 insertions(+), 83 deletions(-)
 delete mode 160000 dtc
 create mode 100644 subprojects/dtc.wrap
 create mode 100644 subprojects/slirp.wrap

-- 
2.39.2

[PATCH 1/4] mtest2make.py: teach suite name that are just "PROJECT"

[marcandre.lureau]

From: Marc-André Lureau <marcandre.lureau@redhat.com>

A subproject test may be simply in the "PROJECT" suite (such as
"qemu-common" with the following patches)

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/mtest2make.py | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
index 0fe81efbbc..179dd54871 100644
--- a/scripts/mtest2make.py
+++ b/scripts/mtest2make.py
@@ -51,10 +51,11 @@ def process_tests(test, targets, suites):
 
     test_suites = test['suite'] or ['default']
     for s in test_suites:
-        # The suite name in the introspection info is "PROJECT:SUITE"
-        s = s.split(':')[1]
-        if s == 'slow' or s == 'thorough':
-            continue
+        # The suite name in the introspection info is "PROJECT" or "PROJECT:SUITE"
+        if ':' in s:
+            s = s.split(':')[1]
+            if s == 'slow' or s == 'thorough':
+                continue
         if s.endswith('-slow'):
             s = s[:-5]
             suites[s].speeds.append('slow')
-- 
2.39.2

[PATCH 2/4] build-sys: prevent meson from downloading wrapped subprojects

[marcandre.lureau]

From: Marc-André Lureau <marcandre.lureau@redhat.com>

The following patches are going to introduce meson wrap dependencies,
which is a solution to download and build missing dependencies.

The QEMU build-system will do network access with no way to avoid the
fallback. As a start, hardcode "--wrap-mode=nodownload" in configure, so
that wraps would be used only after a conscious decision of the user to
use "meson subprojects download" (before running configure).

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 configure | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/configure b/configure
index 2a8a9be8a1..02b4aa4ce9 100755
--- a/configure
+++ b/configure
@@ -2585,6 +2585,10 @@ if test "$skip_meson" = no; then
 
   rm -rf meson-private meson-info meson-logs
 
+  # Prevent meson from automatically downloading wrapped subprojects when missing.
+  # You can use 'meson subprojects download' before running configure.
+  meson_option_add "--wrap-mode=nodownload"
+
   # Built-in options
   test "$bindir" != "bin" && meson_option_add "-Dbindir=$bindir"
   test "$default_feature" = no && meson_option_add -Dauto_features=disabled
-- 
2.39.2

[PATCH 3/4] build-sys: add slirp.wrap

[marcandre.lureau]

From: Marc-André Lureau <marcandre.lureau@redhat.com>

This allows to build with --enable-slirp / -D slirp=enabled, even when
libslirp is not installed on the system. Meson will pull it from git in
that case.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 .gitignore             | 2 ++
 subprojects/slirp.wrap | 6 ++++++
 2 files changed, 8 insertions(+)
 create mode 100644 subprojects/slirp.wrap

diff --git a/.gitignore b/.gitignore
index 61fa39967b..1ea59f4819 100644
--- a/.gitignore
+++ b/.gitignore
@@ -20,3 +20,5 @@ GTAGS
 *.swp
 *.patch
 *.gcov
+
+/subprojects/slirp
diff --git a/subprojects/slirp.wrap b/subprojects/slirp.wrap
new file mode 100644
index 0000000000..ace4f26102
--- /dev/null
+++ b/subprojects/slirp.wrap
@@ -0,0 +1,6 @@
+[wrap-git]
+url = https://gitlab.freedesktop.org/slirp/libslirp
+revision = 15c52d697529eb3e78c5d8aa324d61715bce33b6
+
+[provide]
+slirp = libslirp_dep
-- 
2.39.2

[PATCH 4/4] build-sys: replace dtc submodule with dtc.wrap

[marcandre.lureau]

From: Marc-André Lureau <marcandre.lureau@redhat.com>

Use meson wrap fallback for libfdt. Simplify handling of
fallback/internal build and get rid of a submodule.

This drops support for ./configure --enable-fdt=git/--enable-fdt=internal:
if the system version is enough, it will use it, otherwise it can fall back
on the wrapped subproject.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 configure                     | 21 -------------
 meson.build                   | 56 +++++------------------------------
 .gitignore                    |  1 +
 .gitmodules                   |  3 --
 dtc                           |  1 -
 meson_options.txt             |  5 ++--
 scripts/meson-buildoptions.sh |  4 +--
 subprojects/dtc.wrap          |  6 ++++
 8 files changed, 18 insertions(+), 79 deletions(-)
 delete mode 160000 dtc
 create mode 100644 subprojects/dtc.wrap

diff --git a/configure b/configure
index 02b4aa4ce9..051a5cc69b 100755
--- a/configure
+++ b/configure
@@ -297,8 +297,6 @@ vfio_user_server="disabled"
 # are included in the automatically generated help message)
 
 # 1. Track which submodules are needed
-fdt="auto"
-
 # 2. Automatically enable/disable other options
 tcg="auto"
 cfi="false"
@@ -860,14 +858,6 @@ for opt do
   ;;
   --disable-cfi) cfi="false"
   ;;
-  --disable-fdt) fdt="disabled"
-  ;;
-  --enable-fdt) fdt="enabled"
-  ;;
-  --enable-fdt=git) fdt="internal"
-  ;;
-  --enable-fdt=*) fdt="$optarg"
-  ;;
   --with-coroutine=*) coroutine="$optarg"
   ;;
   --with-git=*) git="$optarg"
@@ -1546,16 +1536,6 @@ if ! compile_prog "$glib_cflags -Werror" "$glib_libs" ; then
     fi
 fi
 
-##########################################
-# fdt probe
-
-case "$fdt" in
-  auto | enabled | internal)
-    # Simpler to always update submodule, even if not needed.
-    git_submodules="${git_submodules} dtc"
-    ;;
-esac
-
 ##########################################
 # check and set a backend for coroutine
 
@@ -2597,7 +2577,6 @@ if test "$skip_meson" = no; then
 
   # QEMU options
   test "$cfi" != false && meson_option_add "-Dcfi=$cfi"
-  test "$fdt" != auto && meson_option_add "-Dfdt=$fdt"
   test -n "${LIB_FUZZING_ENGINE+xxx}" && meson_option_add "-Dfuzzing_engine=$LIB_FUZZING_ENGINE"
   test "$qemu_suffix" != qemu && meson_option_add "-Dqemu_suffix=$qemu_suffix"
   test "$smbd" != '' && meson_option_add "-Dsmbd=$smbd"
diff --git a/meson.build b/meson.build
index 77d2ae87e4..9017141d7d 100644
--- a/meson.build
+++ b/meson.build
@@ -2759,54 +2759,14 @@ endif
 
 fdt = not_found
 if have_system
-  fdt_opt = get_option('fdt')
-  if fdt_opt in ['enabled', 'auto', 'system']
-    have_internal = fs.exists(meson.current_source_dir() / 'dtc/libfdt/Makefile.libfdt')
-    fdt = cc.find_library('fdt', kwargs: static_kwargs,
-                          required: fdt_opt == 'system' or
-                                    fdt_opt == 'enabled' and not have_internal)
-    if fdt.found() and cc.links('''
-       #include <libfdt.h>
-       #include <libfdt_env.h>
-       int main(void) { fdt_find_max_phandle(NULL, NULL); return 0; }''',
-         dependencies: fdt)
-      fdt_opt = 'system'
-    elif fdt_opt == 'system'
-       error('system libfdt requested, but it is too old (1.5.1 or newer required)')
-    elif have_internal
-      fdt_opt = 'internal'
-    else
-      fdt_opt = 'disabled'
-      fdt = not_found
+    # libfdt is not always shipped with a pkgconfig file.
+    fdt = cc.find_library('fdt', kwargs: static_kwargs, required: false)
+    if not fdt.found() or not cc.has_function('fdt_find_max_phandle', dependencies: fdt)
+      fdt = dependency('fdt', kwargs: static_kwargs,
+                method: 'pkg-config', version: '>=1.5.1',
+                required: (get_option('fdt').auto() and fdt_required.length() > 0) or get_option('fdt').enabled(),
+                default_options: ['python=disabled', 'tools=false'])
     endif
-  endif
-  if fdt_opt == 'internal'
-    fdt_files = files(
-      'dtc/libfdt/fdt.c',
-      'dtc/libfdt/fdt_ro.c',
-      'dtc/libfdt/fdt_wip.c',
-      'dtc/libfdt/fdt_sw.c',
-      'dtc/libfdt/fdt_rw.c',
-      'dtc/libfdt/fdt_strerror.c',
-      'dtc/libfdt/fdt_empty_tree.c',
-      'dtc/libfdt/fdt_addresses.c',
-      'dtc/libfdt/fdt_overlay.c',
-      'dtc/libfdt/fdt_check.c',
-    )
-
-    fdt_inc = include_directories('dtc/libfdt')
-    libfdt = static_library('fdt',
-                            build_by_default: false,
-                            sources: fdt_files,
-                            include_directories: fdt_inc)
-    fdt = declare_dependency(link_with: libfdt,
-                             include_directories: fdt_inc)
-  endif
-else
-  fdt_opt = 'disabled'
-endif
-if not fdt.found() and fdt_required.length() > 0
-  error('fdt not available but required by targets ' + ', '.join(fdt_required))
 endif
 
 config_host_data.set('CONFIG_CAPSTONE', capstone.found())
@@ -3989,7 +3949,7 @@ summary_info += {'Linux io_uring support': linux_io_uring}
 summary_info += {'ATTR/XATTR support': libattr}
 summary_info += {'RDMA support':      rdma}
 summary_info += {'PVRDMA support':    have_pvrdma}
-summary_info += {'fdt support':       fdt_opt == 'disabled' ? false : fdt_opt}
+summary_info += {'fdt support':       fdt}
 summary_info += {'libcap-ng support': libcap_ng}
 summary_info += {'bpf support':       libbpf}
 summary_info += {'spice protocol support': spice_protocol}
diff --git a/.gitignore b/.gitignore
index 1ea59f4819..13662d4f24 100644
--- a/.gitignore
+++ b/.gitignore
@@ -21,4 +21,5 @@ GTAGS
 *.patch
 *.gcov
 
+/subprojects/dtc
 /subprojects/slirp
diff --git a/.gitmodules b/.gitmodules
index 6ce5bf49c5..410a00d566 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -13,9 +13,6 @@
 [submodule "roms/qemu-palcode"]
 	path = roms/qemu-palcode
 	url = https://gitlab.com/qemu-project/qemu-palcode.git
-[submodule "dtc"]
-	path = dtc
-	url = https://gitlab.com/qemu-project/dtc.git
 [submodule "roms/u-boot"]
 	path = roms/u-boot
 	url = https://gitlab.com/qemu-project/u-boot.git
diff --git a/dtc b/dtc
deleted file mode 160000
index b6910bec11..0000000000
--- a/dtc
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit b6910bec11614980a21e46fbccc35934b671bd81
diff --git a/meson_options.txt b/meson_options.txt
index fc9447d267..493100ba4c 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -279,9 +279,8 @@ option('vduse_blk_export', type: 'feature', value: 'auto',
 
 option('capstone', type: 'feature', value: 'auto',
        description: 'Whether and how to find the capstone library')
-option('fdt', type: 'combo', value: 'auto',
-       choices: ['disabled', 'enabled', 'auto', 'system', 'internal'],
-       description: 'Whether and how to find the libfdt library')
+option('fdt', type: 'feature', value: 'auto',
+       description: 'libfdt device tree library')
 
 option('selinux', type: 'feature', value: 'auto',
        description: 'SELinux support in qemu-nbd')
diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
index 009fab1515..2a8e91a60c 100644
--- a/scripts/meson-buildoptions.sh
+++ b/scripts/meson-buildoptions.sh
@@ -24,8 +24,6 @@ meson_options_help() {
   printf "%s\n" '  --enable-debug-mutex     mutex debugging support'
   printf "%s\n" '  --enable-debug-stack-usage'
   printf "%s\n" '                           measure coroutine stack usage'
-  printf "%s\n" '  --enable-fdt[=CHOICE]    Whether and how to find the libfdt library'
-  printf "%s\n" '                           (choices: auto/disabled/enabled/internal/system)'
   printf "%s\n" '  --enable-fuzzing         build fuzzing targets'
   printf "%s\n" '  --enable-gcov            Enable coverage tracking.'
   printf "%s\n" '  --enable-gprof           QEMU profiling with gprof'
@@ -91,6 +89,7 @@ meson_options_help() {
   printf "%s\n" '  dmg             dmg image format support'
   printf "%s\n" '  docs            Documentations build support'
   printf "%s\n" '  dsound          DirectSound sound support'
+  printf "%s\n" '  fdt             libfdt device tree library'
   printf "%s\n" '  fuse            FUSE block device export'
   printf "%s\n" '  fuse-lseek      SEEK_HOLE/SEEK_DATA support for FUSE exports'
   printf "%s\n" '  gcrypt          libgcrypt cryptography support'
@@ -262,7 +261,6 @@ _meson_option_parse() {
     --disable-dsound) printf "%s" -Ddsound=disabled ;;
     --enable-fdt) printf "%s" -Dfdt=enabled ;;
     --disable-fdt) printf "%s" -Dfdt=disabled ;;
-    --enable-fdt=*) quote_sh "-Dfdt=$2" ;;
     --enable-fuse) printf "%s" -Dfuse=enabled ;;
     --disable-fuse) printf "%s" -Dfuse=disabled ;;
     --enable-fuse-lseek) printf "%s" -Dfuse_lseek=enabled ;;
diff --git a/subprojects/dtc.wrap b/subprojects/dtc.wrap
new file mode 100644
index 0000000000..da789704d0
--- /dev/null
+++ b/subprojects/dtc.wrap
@@ -0,0 +1,6 @@
+[wrap-git]
+url = https://github.com/dgibson/dtc.git
+revision = 72fc810c3025f07de718f5f32a290a6cd5d1f4ee
+
+[provide]
+fdt = libfdt_dep
-- 
2.39.2

Re: [PATCH 0/4] Add meson wrap fallback for slirp & dtc

[Daniel P. Berrangé]

On Thu, Mar 02, 2023 at 05:18:44PM +0400, marcandre.lureau@redhat.com wrote:
> From: Marc-André Lureau <marcandre.lureau@redhat.com>
> 
> Hi,
> 
> Meson "wrap" is a mechanism to build dependencies that doesn't rely on git
> submodules and integrate external dependencies as subproject()s.
> 
> This offers developpers a simpler way to build QEMU with missing system
> dependencies (ex, libslirp in my case), but also simplify the fallback build
> definition of dtc/libfdt.

Do we actually need this facility though ? We've already determined
that every platform we need has libslirp now, and IIUC Thomas determined
recently that dtc is also available everywhere we need it to be.

So why would we want to continue to special case these two libraries,
out of all the many many many other libraries we also have deps on.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/4] Add meson wrap fallback for slirp & dtc

[Peter Maydell]

On Mon, 6 Mar 2023 at 10:06, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, Mar 02, 2023 at 05:18:44PM +0400, marcandre.lureau@redhat.com wrote:
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > Hi,
> >
> > Meson "wrap" is a mechanism to build dependencies that doesn't rely on git
> > submodules and integrate external dependencies as subproject()s.
> >
> > This offers developpers a simpler way to build QEMU with missing system
> > dependencies (ex, libslirp in my case), but also simplify the fallback build
> > definition of dtc/libfdt.
>
> Do we actually need this facility though ? We've already determined
> that every platform we need has libslirp now, and IIUC Thomas determined
> recently that dtc is also available everywhere we need it to be.
>
> So why would we want to continue to special case these two libraries,
> out of all the many many many other libraries we also have deps on.

Also, it looks like the wrap mechanism is still basically "we have
a file that indicates what the external git URL of the dependency
is and specifies a commit hash to use", it's just changing the
mechanism we use to get the source from git submodules to this
new thing. Maybe the new thing really is better -- certainly
git submodules are absolutely awful -- but we should have one
mechanism, not two.

thanks
-- PMM

Re: [PATCH 0/4] Add meson wrap fallback for slirp & dtc

[Marc-André Lureau]

[-- Attachment #1: Type: text/plain, Size: 1405 bytes --]

Hi

On Mon, Mar 6, 2023 at 2:06 PM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Thu, Mar 02, 2023 at 05:18:44PM +0400, marcandre.lureau@redhat.com
> wrote:
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > Hi,
> >
> > Meson "wrap" is a mechanism to build dependencies that doesn't rely on
> git
> > submodules and integrate external dependencies as subproject()s.
> >
> > This offers developpers a simpler way to build QEMU with missing system
> > dependencies (ex, libslirp in my case), but also simplify the fallback
> build
> > definition of dtc/libfdt.
>
> Do we actually need this facility though ? We've already determined
> that every platform we need has libslirp now, and IIUC Thomas determined
> recently that dtc is also available everywhere we need it to be.
>

The main benefit is for developers: you have the source code of
QEMU-related projects with the source tree. Code navigation, debugging, or
various build tests are easier (compilation flags, static build etc). You
don't have to "pollute" your system with (what could be) QEMU-specific
libraries.


> So why would we want to continue to special case these two libraries,
> out of all the many many many other libraries we also have deps on.
>

They are more often updated, or developped with QEMU? For the reasons I
listed, I would welcome more wrapped subprojects.

[-- Attachment #2: Type: text/html, Size: 2134 bytes --]

Re: [PATCH 0/4] Add meson wrap fallback for slirp & dtc

[Marc-André Lureau]

[-- Attachment #1: Type: text/plain, Size: 2008 bytes --]

Hi

On Mon, Mar 6, 2023 at 2:18 PM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Mon, 6 Mar 2023 at 10:06, Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> >
> > On Thu, Mar 02, 2023 at 05:18:44PM +0400, marcandre.lureau@redhat.com
> wrote:
> > > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> > >
> > > Hi,
> > >
> > > Meson "wrap" is a mechanism to build dependencies that doesn't rely on
> git
> > > submodules and integrate external dependencies as subproject()s.
> > >
> > > This offers developpers a simpler way to build QEMU with missing system
> > > dependencies (ex, libslirp in my case), but also simplify the fallback
> build
> > > definition of dtc/libfdt.
> >
> > Do we actually need this facility though ? We've already determined
> > that every platform we need has libslirp now, and IIUC Thomas determined
> > recently that dtc is also available everywhere we need it to be.
> >
> > So why would we want to continue to special case these two libraries,
> > out of all the many many many other libraries we also have deps on.
>
> Also, it looks like the wrap mechanism is still basically "we have
> a file that indicates what the external git URL of the dependency
> is and specifies a commit hash to use", it's just changing the
> mechanism we use to get the source from git submodules to this
> new thing. Maybe the new thing really is better -- certainly
> git submodules are absolutely awful -- but we should have one
> mechanism, not two.
>

I think we all experienced git submodules are more intrusive and may break
various git workflows.

wrap files may indeed point to a git URL with a revision, but they are more
versatile than that and can download from released tarball instead, for
example.

Some projects, like glib, use both submodules and wrap files (
https://gitlab.gnome.org/GNOME/glib/-/tree/main/subprojects), there is no
incompatibility with both solutions. As often, use what fits best your
needs.

[-- Attachment #2: Type: text/html, Size: 2826 bytes --]

Re: [PATCH 0/4] Add meson wrap fallback for slirp & dtc

[Daniel P. Berrangé]

On Mon, Mar 06, 2023 at 02:19:25PM +0400, Marc-André Lureau wrote:
> Hi
> 
> On Mon, Mar 6, 2023 at 2:06 PM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> 
> > On Thu, Mar 02, 2023 at 05:18:44PM +0400, marcandre.lureau@redhat.com
> > wrote:
> > > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> > >
> > > Hi,
> > >
> > > Meson "wrap" is a mechanism to build dependencies that doesn't rely on
> > git
> > > submodules and integrate external dependencies as subproject()s.
> > >
> > > This offers developpers a simpler way to build QEMU with missing system
> > > dependencies (ex, libslirp in my case), but also simplify the fallback
> > build
> > > definition of dtc/libfdt.
> >
> > Do we actually need this facility though ? We've already determined
> > that every platform we need has libslirp now, and IIUC Thomas determined
> > recently that dtc is also available everywhere we need it to be.
> >
> 
> The main benefit is for developers: you have the source code of
> QEMU-related projects with the source tree. Code navigation, debugging, or
> various build tests are easier (compilation flags, static build etc). You
> don't have to "pollute" your system with (what could be) QEMU-specific
> libraries.

That's pushing developers to use builds of 3rd party libararies
that don't actually match what our users are going to end up
deploying with though.

> > So why would we want to continue to special case these two libraries,
> > out of all the many many many other libraries we also have deps on.
> >
> 
> They are more often updated, or developped with QEMU? For the reasons I
> listed, I would welcome more wrapped subprojects.

I don't think that they actually have more frequent updates that other
libraries. In any case from QEMU's POV it doesn't matter how often upstream
does releases. We're targetting the existing versions available in the OS
and so don't want to use bleeding edge upstream releases.

This also significantly expands our testing matrix. For each library
we have the possiblity that users have the distro version vs the wrapped
version. That is many new combinations users are exposed to, that we are
realistically never going to have the bandwidth to test in CI.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/4] Add meson wrap fallback for slirp & dtc

[Marc-André Lureau]

Hi

On Mon, Mar 6, 2023 at 2:33 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Mon, Mar 06, 2023 at 02:19:25PM +0400, Marc-André Lureau wrote:
> > Hi
> >
> > On Mon, Mar 6, 2023 at 2:06 PM Daniel P. Berrangé <berrange@redhat.com>
> > wrote:
> >
> > > On Thu, Mar 02, 2023 at 05:18:44PM +0400, marcandre.lureau@redhat.com
> > > wrote:
> > > > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> > > >
> > > > Hi,
> > > >
> > > > Meson "wrap" is a mechanism to build dependencies that doesn't rely on
> > > git
> > > > submodules and integrate external dependencies as subproject()s.
> > > >
> > > > This offers developpers a simpler way to build QEMU with missing system
> > > > dependencies (ex, libslirp in my case), but also simplify the fallback
> > > build
> > > > definition of dtc/libfdt.
> > >
> > > Do we actually need this facility though ? We've already determined
> > > that every platform we need has libslirp now, and IIUC Thomas determined
> > > recently that dtc is also available everywhere we need it to be.
> > >
> >
> > The main benefit is for developers: you have the source code of
> > QEMU-related projects with the source tree. Code navigation, debugging, or
> > various build tests are easier (compilation flags, static build etc). You
> > don't have to "pollute" your system with (what could be) QEMU-specific
> > libraries.
>
> That's pushing developers to use builds of 3rd party libararies
> that don't actually match what our users are going to end up
> deploying with though.

The combinations of versions used by developers is already virtually
limitless. I don't think adding a specific upstream version/revision
to the mix is making the situation worse. It could be even the
opposite, if we manage to build an "official" static version of qemu
for example (with specific dependencies).

>
> > > So why would we want to continue to special case these two libraries,
> > > out of all the many many many other libraries we also have deps on.
> > >
> >
> > They are more often updated, or developped with QEMU? For the reasons I
> > listed, I would welcome more wrapped subprojects.
>
> I don't think that they actually have more frequent updates that other
> libraries. In any case from QEMU's POV it doesn't matter how often upstream
> does releases. We're targetting the existing versions available in the OS
> and so don't want to use bleeding edge upstream releases.

That doesn't change that policy.

>
> This also significantly expands our testing matrix. For each library
> we have the possiblity that users have the distro version vs the wrapped
> version. That is many new combinations users are exposed to, that we are
> realistically never going to have the bandwidth to test in CI.

yes, I would consider this a tier 2 support, since it aims at
developers, and not having to cover it in CI.

-- 
Marc-André Lureau

Re: [PATCH 0/4] Add meson wrap fallback for slirp & dtc

[Paolo Bonzini]

On 3/6/23 11:06, Daniel P. Berrangé wrote:
>> This offers developpers a simpler way to build QEMU with missing system
>> dependencies (ex, libslirp in my case), but also simplify the fallback build
>> definition of dtc/libfdt.
> Do we actually need this facility though ? We've already determined
> that every platform we need has libslirp now, and IIUC Thomas determined
> recently that dtc is also available everywhere we need it to be.

libvfio-user can use Meson subprojects instead of submodules, too; and 
with Pip support probably coming in 8.1, we can remove the meson 
submodule.  Then, the only mostly-mandatory submodule would be 
keycodemapdb; SLOF requires a cross-compiler and the pre-built is binary 
is shipped binary, while softfloat/testfloat are test only and maybe 
could even be embedded.  So there is a path towards getting rid of 
submodules at least for the main QEMU build process.

Also, Windows installer builds could benefit from having wrapdb support 
for the mandatory dependencies in Windows (pixman, zlib, glib, possibly 
SDL).

Since they are unintrusive and easy to revert, I think we could include 
patches 1-3 as experimental in 8.0, though I'm happy to oblige if people 
disagree (and only include patch 1).

Paolo

Re: [PATCH 0/4] Add meson wrap fallback for slirp & dtc

[Daniel P. Berrangé]

On Mon, Mar 06, 2023 at 12:36:25PM +0100, Paolo Bonzini wrote:
> On 3/6/23 11:06, Daniel P. Berrangé wrote:
> > > This offers developpers a simpler way to build QEMU with missing system
> > > dependencies (ex, libslirp in my case), but also simplify the fallback build
> > > definition of dtc/libfdt.
> > Do we actually need this facility though ? We've already determined
> > that every platform we need has libslirp now, and IIUC Thomas determined
> > recently that dtc is also available everywhere we need it to be.
> 
> libvfio-user can use Meson subprojects instead of submodules, too; and with
> Pip support probably coming in 8.1, we can remove the meson submodule.
> Then, the only mostly-mandatory submodule would be keycodemapdb;

On the topic of keycodemapdb, we designed it as a copylib thing because
it seemed like the simplest approach, but I have wondered whether we
should change tack and promote it as installable data package.

Basically the keycodemap-gen tool ino /usr/bin, and CSV data file into
/usr/share, and perhaps add a pkg-config file to enable apps to query
its existance / path to keycodemap-gen ?

The main downside is we would need to promise back compat of the CLI
tool. The upside is that consumers of it would eliminate a git submodule
and in theory be more up2date, though that depends on the distros actually
updating the datafile periodically.  I'm really on the fence about whether
its worth it vs carrying on as a submodule (or meson subproject if that's
nicer long term).

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|