* [PATCH 1/6] remove remaining traces of meson submodule
2023-05-19 8:56 [PATCH v2 0/6] meson: use subprojects for bundled projects Paolo Bonzini
@ 2023-05-19 8:56 ` Paolo Bonzini
2023-05-19 9:02 ` Daniel P. Berrangé
2023-05-19 8:56 ` [PATCH 2/6] meson: simplify logic for -Dfdt Paolo Bonzini
` (6 subsequent siblings)
7 siblings, 1 reply; 22+ messages in thread
From: Paolo Bonzini @ 2023-05-19 8:56 UTC (permalink / raw)
To: qemu-devel; +Cc: marcandre.lureau, berrange, richard.henderson, peter.maydell
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
.gitlab-ci.d/buildtest-template.yml | 4 ----
scripts/archive-source.sh | 2 +-
2 files changed, 1 insertion(+), 5 deletions(-)
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 7edb50b76035..c9f2e737c09a 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -5,10 +5,6 @@
before_script:
- JOBS=$(expr $(nproc) + 1)
script:
- - if test -n "$LD_JOBS";
- then
- scripts/git-submodule.sh update meson ;
- fi
- mkdir build
- cd build
- ../configure --enable-werror --disable-docs --enable-fdt=system
diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index 23e042dacdb3..c03532915471 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -26,7 +26,7 @@ sub_file="${sub_tdir}/submodule.tar"
# independent of what the developer currently has initialized
# in their checkout, because the build environment is completely
# different to the host OS.
-submodules="dtc meson ui/keycodemapdb"
+submodules="dtc ui/keycodemapdb"
submodules="$submodules tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
sub_deinit=""
--
2.40.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH 1/6] remove remaining traces of meson submodule
2023-05-19 8:56 ` [PATCH 1/6] remove remaining traces of meson submodule Paolo Bonzini
@ 2023-05-19 9:02 ` Daniel P. Berrangé
0 siblings, 0 replies; 22+ messages in thread
From: Daniel P. Berrangé @ 2023-05-19 9:02 UTC (permalink / raw)
To: Paolo Bonzini
Cc: qemu-devel, marcandre.lureau, richard.henderson, peter.maydell
On Fri, May 19, 2023 at 10:56:41AM +0200, Paolo Bonzini wrote:
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> .gitlab-ci.d/buildtest-template.yml | 4 ----
> scripts/archive-source.sh | 2 +-
> 2 files changed, 1 insertion(+), 5 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 2/6] meson: simplify logic for -Dfdt
2023-05-19 8:56 [PATCH v2 0/6] meson: use subprojects for bundled projects Paolo Bonzini
2023-05-19 8:56 ` [PATCH 1/6] remove remaining traces of meson submodule Paolo Bonzini
@ 2023-05-19 8:56 ` Paolo Bonzini
2023-05-19 9:04 ` Daniel P. Berrangé
2023-05-19 8:56 ` [PATCH 3/6] meson: use subproject for internal libfdt Paolo Bonzini
` (5 subsequent siblings)
7 siblings, 1 reply; 22+ messages in thread
From: Paolo Bonzini @ 2023-05-19 8:56 UTC (permalink / raw)
To: qemu-devel; +Cc: marcandre.lureau, berrange, richard.henderson, peter.maydell
fdt_opt == 'disabled' is going to give an error if libfdt is required
by any target, so catch that immediately. For fdt_opt == 'enabled',
instead, do not check immediately whether the internal libfdt is present.
Instead do the check after ascertaining that libfdt is absent or too old.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
meson.build | 29 ++++++++++++++++-------------
1 file changed, 16 insertions(+), 13 deletions(-)
diff --git a/meson.build b/meson.build
index 41c87c441f8e..0de7a858bce1 100644
--- a/meson.build
+++ b/meson.build
@@ -3041,13 +3041,14 @@ if have_system and vfio_user_server_allowed
endif
fdt = not_found
-if have_system
- fdt_opt = get_option('fdt')
+fdt_opt = get_option('fdt')
+if fdt_required.length() > 0 or fdt_opt == 'enabled'
+ if fdt_opt == 'disabled'
+ error('fdt disabled but required by targets ' + ', '.join(fdt_required))
+ endif
+
if fdt_opt in ['enabled', 'auto', 'system']
- have_internal = fs.exists(meson.current_source_dir() / 'dtc/libfdt/Makefile.libfdt')
- fdt = cc.find_library('fdt',
- required: fdt_opt == 'system' or
- fdt_opt == 'enabled' and not have_internal)
+ fdt = cc.find_library('fdt', required: fdt_opt == 'system')
if fdt.found() and cc.links('''
#include <libfdt.h>
#include <libfdt_env.h>
@@ -3056,14 +3057,19 @@ if have_system
fdt_opt = 'system'
elif fdt_opt == 'system'
error('system libfdt requested, but it is too old (1.5.1 or newer required)')
- elif have_internal
- fdt_opt = 'internal'
else
- fdt_opt = 'disabled'
+ fdt_opt = 'internal'
fdt = not_found
endif
endif
- if fdt_opt == 'internal'
+ if not fdt.found()
+ assert(fdt_opt == 'internal')
+ have_internal = fs.exists(meson.current_source_dir() / 'subprojects/dtc/meson.build')
+
+ if not have_internal
+ error('libfdt source not found - please pull git submodule')
+ endif
+
fdt_files = files(
'dtc/libfdt/fdt.c',
'dtc/libfdt/fdt_ro.c',
@@ -3088,9 +3094,6 @@ if have_system
else
fdt_opt = 'disabled'
endif
-if not fdt.found() and fdt_required.length() > 0
- error('fdt not available but required by targets ' + ', '.join(fdt_required))
-endif
config_host_data.set('CONFIG_CAPSTONE', capstone.found())
config_host_data.set('CONFIG_FDT', fdt.found())
--
2.40.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH 2/6] meson: simplify logic for -Dfdt
2023-05-19 8:56 ` [PATCH 2/6] meson: simplify logic for -Dfdt Paolo Bonzini
@ 2023-05-19 9:04 ` Daniel P. Berrangé
0 siblings, 0 replies; 22+ messages in thread
From: Daniel P. Berrangé @ 2023-05-19 9:04 UTC (permalink / raw)
To: Paolo Bonzini
Cc: qemu-devel, marcandre.lureau, richard.henderson, peter.maydell
On Fri, May 19, 2023 at 10:56:42AM +0200, Paolo Bonzini wrote:
> fdt_opt == 'disabled' is going to give an error if libfdt is required
> by any target, so catch that immediately. For fdt_opt == 'enabled',
> instead, do not check immediately whether the internal libfdt is present.
> Instead do the check after ascertaining that libfdt is absent or too old.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> meson.build | 29 ++++++++++++++++-------------
> 1 file changed, 16 insertions(+), 13 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 3/6] meson: use subproject for internal libfdt
2023-05-19 8:56 [PATCH v2 0/6] meson: use subprojects for bundled projects Paolo Bonzini
2023-05-19 8:56 ` [PATCH 1/6] remove remaining traces of meson submodule Paolo Bonzini
2023-05-19 8:56 ` [PATCH 2/6] meson: simplify logic for -Dfdt Paolo Bonzini
@ 2023-05-19 8:56 ` Paolo Bonzini
2023-05-19 9:05 ` Daniel P. Berrangé
2023-05-19 8:56 ` [PATCH 4/6] meson: use subproject for keycodemapdb Paolo Bonzini
` (4 subsequent siblings)
7 siblings, 1 reply; 22+ messages in thread
From: Paolo Bonzini @ 2023-05-19 8:56 UTC (permalink / raw)
To: qemu-devel; +Cc: marcandre.lureau, berrange, richard.henderson, peter.maydell
Recent dtc/libfdt can use either Make or meson as the build system.
By using a subproject, our own meson.build can remove the hard
coded list of source files.
This is also the first step towards managing downloads with .wrap
files instead of submodule.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
.gitmodules | 4 ++--
configure | 2 +-
meson.build | 24 ++++--------------------
scripts/archive-source.sh | 2 +-
dtc => subprojects/dtc | 0
5 files changed, 8 insertions(+), 24 deletions(-)
rename dtc => subprojects/dtc (100%)
diff --git a/.gitmodules b/.gitmodules
index 2a3a12033c4b..3ed5d073d630 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -13,8 +13,8 @@
[submodule "roms/qemu-palcode"]
path = roms/qemu-palcode
url = https://gitlab.com/qemu-project/qemu-palcode.git
-[submodule "dtc"]
- path = dtc
+[submodule "subprojects/dtc"]
+ path = subprojects/dtc
url = https://gitlab.com/qemu-project/dtc.git
[submodule "roms/u-boot"]
path = roms/u-boot
diff --git a/configure b/configure
index 31a787153e38..5bbca83d9a31 100755
--- a/configure
+++ b/configure
@@ -1188,7 +1188,7 @@ fi
case "$fdt" in
auto | enabled | internal)
# Simpler to always update submodule, even if not needed.
- git_submodules="${git_submodules} dtc"
+ git_submodules="${git_submodules} subprojects/dtc"
;;
esac
diff --git a/meson.build b/meson.build
index 0de7a858bce1..1a55039ff7c3 100644
--- a/meson.build
+++ b/meson.build
@@ -3070,26 +3070,10 @@ if fdt_required.length() > 0 or fdt_opt == 'enabled'
error('libfdt source not found - please pull git submodule')
endif
- fdt_files = files(
- 'dtc/libfdt/fdt.c',
- 'dtc/libfdt/fdt_ro.c',
- 'dtc/libfdt/fdt_wip.c',
- 'dtc/libfdt/fdt_sw.c',
- 'dtc/libfdt/fdt_rw.c',
- 'dtc/libfdt/fdt_strerror.c',
- 'dtc/libfdt/fdt_empty_tree.c',
- 'dtc/libfdt/fdt_addresses.c',
- 'dtc/libfdt/fdt_overlay.c',
- 'dtc/libfdt/fdt_check.c',
- )
-
- fdt_inc = include_directories('dtc/libfdt')
- libfdt = static_library('fdt',
- build_by_default: false,
- sources: fdt_files,
- include_directories: fdt_inc)
- fdt = declare_dependency(link_with: libfdt,
- include_directories: fdt_inc)
+ libfdt_proj = subproject('dtc',
+ default_options: ['tools=false', 'yaml=disabled',
+ 'python=disabled', 'default_library=static'])
+ fdt = libfdt_proj.get_variable('libfdt_dep')
endif
else
fdt_opt = 'disabled'
diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index c03532915471..a7c2886334f0 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -26,7 +26,7 @@ sub_file="${sub_tdir}/submodule.tar"
# independent of what the developer currently has initialized
# in their checkout, because the build environment is completely
# different to the host OS.
-submodules="dtc ui/keycodemapdb"
+submodules="subprojects/dtc ui/keycodemapdb"
submodules="$submodules tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
sub_deinit=""
diff --git a/dtc b/subprojects/dtc
similarity index 100%
rename from dtc
rename to subprojects/dtc
--
2.40.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH 3/6] meson: use subproject for internal libfdt
2023-05-19 8:56 ` [PATCH 3/6] meson: use subproject for internal libfdt Paolo Bonzini
@ 2023-05-19 9:05 ` Daniel P. Berrangé
0 siblings, 0 replies; 22+ messages in thread
From: Daniel P. Berrangé @ 2023-05-19 9:05 UTC (permalink / raw)
To: Paolo Bonzini
Cc: qemu-devel, marcandre.lureau, richard.henderson, peter.maydell
On Fri, May 19, 2023 at 10:56:43AM +0200, Paolo Bonzini wrote:
> Recent dtc/libfdt can use either Make or meson as the build system.
> By using a subproject, our own meson.build can remove the hard
> coded list of source files.
>
> This is also the first step towards managing downloads with .wrap
> files instead of submodule.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> .gitmodules | 4 ++--
> configure | 2 +-
> meson.build | 24 ++++--------------------
> scripts/archive-source.sh | 2 +-
> dtc => subprojects/dtc | 0
> 5 files changed, 8 insertions(+), 24 deletions(-)
> rename dtc => subprojects/dtc (100%)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 4/6] meson: use subproject for keycodemapdb
2023-05-19 8:56 [PATCH v2 0/6] meson: use subprojects for bundled projects Paolo Bonzini
` (2 preceding siblings ...)
2023-05-19 8:56 ` [PATCH 3/6] meson: use subproject for internal libfdt Paolo Bonzini
@ 2023-05-19 8:56 ` Paolo Bonzini
2023-05-19 9:06 ` Daniel P. Berrangé
2023-05-19 8:56 ` [PATCH 5/6] configure: rename --enable-pypi to --enable-download, control subprojects too Paolo Bonzini
` (3 subsequent siblings)
7 siblings, 1 reply; 22+ messages in thread
From: Paolo Bonzini @ 2023-05-19 8:56 UTC (permalink / raw)
To: qemu-devel; +Cc: marcandre.lureau, berrange, richard.henderson, peter.maydell
By using a subproject, our own meson.build can use variables from
the subproject instead of hard-coded paths. This is also the first step
towards managing downloads with .wrap files instead of submodule.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
.gitmodules | 4 ++--
configure | 4 ++--
scripts/archive-source.sh | 2 +-
{ui => subprojects}/keycodemapdb | 0
ui/meson.build | 8 ++++----
5 files changed, 9 insertions(+), 9 deletions(-)
rename {ui => subprojects}/keycodemapdb (100%)
diff --git a/.gitmodules b/.gitmodules
index 3ed5d073d630..f8b2ddf3877c 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -25,8 +25,8 @@
[submodule "roms/QemuMacDrivers"]
path = roms/QemuMacDrivers
url = https://gitlab.com/qemu-project/QemuMacDrivers.git
-[submodule "ui/keycodemapdb"]
- path = ui/keycodemapdb
+[submodule "subprojects/keycodemapdb"]
+ path = subprojects/keycodemapdb
url = https://gitlab.com/qemu-project/keycodemapdb.git
[submodule "roms/seabios-hppa"]
path = roms/seabios-hppa
diff --git a/configure b/configure
index 5bbca83d9a31..2b6995e16756 100755
--- a/configure
+++ b/configure
@@ -254,7 +254,7 @@ else
git_submodules_action="ignore"
fi
-git_submodules="ui/keycodemapdb"
+git_submodules="subprojects/keycodemapdb"
git="git"
debug_tcg="no"
docs="auto"
@@ -806,7 +806,7 @@ case $git_submodules_action in
fi
;;
ignore)
- if ! test -f "$source_path/ui/keycodemapdb/README"
+ if ! test -f "$source_path/subprojects/keycodemapdb/README"
then
echo
echo "ERROR: missing GIT submodules"
diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index a7c2886334f0..dba5ae05b67e 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -26,7 +26,7 @@ sub_file="${sub_tdir}/submodule.tar"
# independent of what the developer currently has initialized
# in their checkout, because the build environment is completely
# different to the host OS.
-submodules="subprojects/dtc ui/keycodemapdb"
+submodules="subprojects/dtc subprojects/keycodemapdb"
submodules="$submodules tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
sub_deinit=""
diff --git a/ui/keycodemapdb b/subprojects/keycodemapdb
similarity index 100%
rename from ui/keycodemapdb
rename to subprojects/keycodemapdb
diff --git a/ui/meson.build b/ui/meson.build
index 330369707dd7..e09b616a66bc 100644
--- a/ui/meson.build
+++ b/ui/meson.build
@@ -162,15 +162,15 @@ keymaps = [
]
if have_system or xkbcommon.found()
+ keycodemapdb_proj = subproject('keycodemapdb', required: true)
foreach e : keymaps
output = 'input-keymap-@0@-to-@1@.c.inc'.format(e[0], e[1])
genh += custom_target(output,
output: output,
capture: true,
- input: files('keycodemapdb/data/keymaps.csv'),
- command: [python, files('keycodemapdb/tools/keymap-gen'),
- 'code-map',
- '--lang', 'glib2',
+ input: keycodemapdb_proj.get_variable('keymaps_csv'),
+ command: [python, keycodemapdb_proj.get_variable('keymap_gen').full_path(),
+ 'code-map', '--lang', 'glib2',
'--varname', 'qemu_input_map_@0@_to_@1@'.format(e[0], e[1]),
'@INPUT0@', e[0], e[1]])
endforeach
--
2.40.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH 4/6] meson: use subproject for keycodemapdb
2023-05-19 8:56 ` [PATCH 4/6] meson: use subproject for keycodemapdb Paolo Bonzini
@ 2023-05-19 9:06 ` Daniel P. Berrangé
0 siblings, 0 replies; 22+ messages in thread
From: Daniel P. Berrangé @ 2023-05-19 9:06 UTC (permalink / raw)
To: Paolo Bonzini
Cc: qemu-devel, marcandre.lureau, richard.henderson, peter.maydell
On Fri, May 19, 2023 at 10:56:44AM +0200, Paolo Bonzini wrote:
> By using a subproject, our own meson.build can use variables from
> the subproject instead of hard-coded paths. This is also the first step
> towards managing downloads with .wrap files instead of submodule.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> .gitmodules | 4 ++--
> configure | 4 ++--
> scripts/archive-source.sh | 2 +-
> {ui => subprojects}/keycodemapdb | 0
> ui/meson.build | 8 ++++----
> 5 files changed, 9 insertions(+), 9 deletions(-)
> rename {ui => subprojects}/keycodemapdb (100%)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 5/6] configure: rename --enable-pypi to --enable-download, control subprojects too
2023-05-19 8:56 [PATCH v2 0/6] meson: use subprojects for bundled projects Paolo Bonzini
` (3 preceding siblings ...)
2023-05-19 8:56 ` [PATCH 4/6] meson: use subproject for keycodemapdb Paolo Bonzini
@ 2023-05-19 8:56 ` Paolo Bonzini
2023-05-19 9:07 ` Daniel P. Berrangé
2023-05-19 8:56 ` [PATCH 6/6] meson: subprojects: replace submodules with wrap files Paolo Bonzini
` (2 subsequent siblings)
7 siblings, 1 reply; 22+ messages in thread
From: Paolo Bonzini @ 2023-05-19 8:56 UTC (permalink / raw)
To: qemu-devel; +Cc: marcandre.lureau, berrange, richard.henderson, peter.maydell
The behavior of --{enable,disable}-pypi is similar to that of
-Dwrapmode={default,nodownload} respectively. In particular,
in both cases a feature needs to be explicitly enabled for the
dependency to be downloaded.
So, use a single option to control both cases. Now, --enable-slirp
will trigger cloning and building of libslirp if the .pc file
is not found on the machine.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
.gitignore | 2 --
configure | 17 +++++++----------
subprojects/.gitignore | 3 +++
3 files changed, 10 insertions(+), 12 deletions(-)
create mode 100644 subprojects/.gitignore
diff --git a/.gitignore b/.gitignore
index 1ea59f481921..61fa39967b54 100644
--- a/.gitignore
+++ b/.gitignore
@@ -20,5 +20,3 @@ GTAGS
*.swp
*.patch
*.gcov
-
-/subprojects/slirp
diff --git a/configure b/configure
index 2b6995e16756..43620b80c8df 100755
--- a/configure
+++ b/configure
@@ -267,7 +267,7 @@ bsd_user=""
plugins="$default_feature"
ninja=""
python=
-pypi="enabled"
+download="enabled"
bindir="bin"
skip_meson=no
vfio_user_server="disabled"
@@ -765,9 +765,9 @@ for opt do
--with-git-submodules=*)
git_submodules_action="$optarg"
;;
- --disable-pypi) pypi="disabled"
+ --disable-download) download="disabled"
;;
- --enable-pypi) pypi="enabled"
+ --enable-download) download="enabled"
;;
--enable-plugins) if test "$mingw32" = "yes"; then
error_exit "TCG plugins not currently supported on Windows platforms"
@@ -978,7 +978,7 @@ python="$(command -v "$python")"
# - venv is allowed to use system packages;
# - all setup can be performed offline;
# - missing packages may be fetched from PyPI,
-# unless --disable-pypi is passed.
+# unless --disable-download is passed.
# - pip is not installed into the venv when possible,
# but ensurepip is called as a fallback when necessary.
@@ -995,7 +995,7 @@ python="$python -B"
mkvenv="$python ${source_path}/python/scripts/mkvenv.py"
mkvenv_flags=""
-if test "$pypi" = "enabled" ; then
+if test "$download" = "enabled" ; then
mkvenv_flags="--online"
fi
@@ -1018,7 +1018,7 @@ meson="$(cd pyvenv/bin; pwd)/meson"
# Conditionally ensure Sphinx is installed.
mkvenv_flags=""
-if test "$pypi" = "enabled" -a "$docs" = "enabled" ; then
+if test "$download" = "enabled" -a "$docs" = "enabled" ; then
mkvenv_flags="--online"
fi
@@ -1960,11 +1960,8 @@ if test "$skip_meson" = no; then
rm -rf meson-private meson-info meson-logs
- # Prevent meson from automatically downloading wrapped subprojects when missing.
- # You can use 'meson subprojects download' before running configure.
- meson_option_add "--wrap-mode=nodownload"
-
# Built-in options
+ test "$download" = "disabled" && meson_option_add "--wrap-mode=nodownload"
test "$bindir" != "bin" && meson_option_add "-Dbindir=$bindir"
test "$default_feature" = no && meson_option_add -Dauto_features=disabled
test "$static" = yes && meson_option_add -Dprefer_static=true
diff --git a/subprojects/.gitignore b/subprojects/.gitignore
new file mode 100644
index 000000000000..7560ebb0b1a0
--- /dev/null
+++ b/subprojects/.gitignore
@@ -0,0 +1,3 @@
+/packagecache
+
+/slirp
--
2.40.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH 5/6] configure: rename --enable-pypi to --enable-download, control subprojects too
2023-05-19 8:56 ` [PATCH 5/6] configure: rename --enable-pypi to --enable-download, control subprojects too Paolo Bonzini
@ 2023-05-19 9:07 ` Daniel P. Berrangé
0 siblings, 0 replies; 22+ messages in thread
From: Daniel P. Berrangé @ 2023-05-19 9:07 UTC (permalink / raw)
To: Paolo Bonzini
Cc: qemu-devel, marcandre.lureau, richard.henderson, peter.maydell
On Fri, May 19, 2023 at 10:56:45AM +0200, Paolo Bonzini wrote:
> The behavior of --{enable,disable}-pypi is similar to that of
> -Dwrapmode={default,nodownload} respectively. In particular,
> in both cases a feature needs to be explicitly enabled for the
> dependency to be downloaded.
>
> So, use a single option to control both cases. Now, --enable-slirp
> will trigger cloning and building of libslirp if the .pc file
> is not found on the machine.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> .gitignore | 2 --
> configure | 17 +++++++----------
> subprojects/.gitignore | 3 +++
> 3 files changed, 10 insertions(+), 12 deletions(-)
> create mode 100644 subprojects/.gitignore
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 6/6] meson: subprojects: replace submodules with wrap files
2023-05-19 8:56 [PATCH v2 0/6] meson: use subprojects for bundled projects Paolo Bonzini
` (4 preceding siblings ...)
2023-05-19 8:56 ` [PATCH 5/6] configure: rename --enable-pypi to --enable-download, control subprojects too Paolo Bonzini
@ 2023-05-19 8:56 ` Paolo Bonzini
2023-05-19 9:25 ` Daniel P. Berrangé
2023-05-19 9:21 ` [PATCH v2 0/6] meson: use subprojects for bundled projects Peter Maydell
2023-05-19 9:30 ` Peter Maydell
7 siblings, 1 reply; 22+ messages in thread
From: Paolo Bonzini @ 2023-05-19 8:56 UTC (permalink / raw)
To: qemu-devel; +Cc: marcandre.lureau, berrange, richard.henderson, peter.maydell
Compared to submodules, .wrap files have several advantages:
* option parsing and downloading is delegated to meson
* the commit is stored in a text file instead of a magic entry in the
git tree object
* we could stop shipping external dependencies that are only used as a
fallback, but not break compilation on platforms that lack them.
For example it may make sense to download dtc at build time,
controlled by --enable-download, even when building from a tarball.
This is _not_ done in this patch.
dependency() can fall back to a wrap automatically. However, this
is only possible for libraries that come with a .pc file, and this
is not very common for libfdt even though the upstream project in
principle provides it; it also removes the control that we provide with
--enable-fdt={system,internal}. Therefore, the logic to pick system
vs. internal libfdt is left untouched.
It is possible to use subprojects also for berkeley-softfloat-3
and berkeley-testfloat-3, but this is left for later.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
.gitlab-ci.d/buildtest-template.yml | 1 +
.gitmodules | 9 -----
configure | 54 +++++------------------------
meson.build | 17 ++-------
meson_options.txt | 1 +
scripts/archive-source.sh | 11 ++++--
scripts/make-release | 5 +++
subprojects/.gitignore | 3 ++
subprojects/dtc | 1 -
subprojects/dtc.wrap | 4 +++
subprojects/keycodemapdb | 1 -
subprojects/keycodemapdb.wrap | 4 +++
subprojects/libvfio-user | 1 -
subprojects/libvfio-user.wrap | 4 +++
14 files changed, 41 insertions(+), 75 deletions(-)
delete mode 160000 subprojects/dtc
create mode 100644 subprojects/dtc.wrap
delete mode 160000 subprojects/keycodemapdb
create mode 100644 subprojects/keycodemapdb.wrap
delete mode 160000 subprojects/libvfio-user
create mode 100644 subprojects/libvfio-user.wrap
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index c9f2e737c09a..3c997d7265b2 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -44,6 +44,7 @@
script:
- scripts/git-submodule.sh update
$(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
+ - meson subprojects download $(cd build/subprojects && echo *)
- cd build
- find . -type f -exec touch {} +
# Avoid recompiling by hiding ninja with NINJA=":"
diff --git a/.gitmodules b/.gitmodules
index f8b2ddf3877c..a934c85e4e40 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -13,9 +13,6 @@
[submodule "roms/qemu-palcode"]
path = roms/qemu-palcode
url = https://gitlab.com/qemu-project/qemu-palcode.git
-[submodule "subprojects/dtc"]
- path = subprojects/dtc
- url = https://gitlab.com/qemu-project/dtc.git
[submodule "roms/u-boot"]
path = roms/u-boot
url = https://gitlab.com/qemu-project/u-boot.git
@@ -25,9 +22,6 @@
[submodule "roms/QemuMacDrivers"]
path = roms/QemuMacDrivers
url = https://gitlab.com/qemu-project/QemuMacDrivers.git
-[submodule "subprojects/keycodemapdb"]
- path = subprojects/keycodemapdb
- url = https://gitlab.com/qemu-project/keycodemapdb.git
[submodule "roms/seabios-hppa"]
path = roms/seabios-hppa
url = https://gitlab.com/qemu-project/seabios-hppa.git
@@ -55,6 +49,3 @@
[submodule "tests/lcitool/libvirt-ci"]
path = tests/lcitool/libvirt-ci
url = https://gitlab.com/libvirt/libvirt-ci.git
-[submodule "subprojects/libvfio-user"]
- path = subprojects/libvfio-user
- url = https://gitlab.com/qemu-project/libvfio-user.git
diff --git a/configure b/configure
index 43620b80c8df..1e737f322360 100755
--- a/configure
+++ b/configure
@@ -254,7 +254,7 @@ else
git_submodules_action="ignore"
fi
-git_submodules="subprojects/keycodemapdb"
+git_submodules=""
git="git"
debug_tcg="no"
docs="auto"
@@ -270,7 +270,6 @@ python=
download="enabled"
bindir="bin"
skip_meson=no
-vfio_user_server="disabled"
use_containers="yes"
gdb_bin=$(command -v "gdb-multiarch" || command -v "gdb")
gdb_arches=""
@@ -282,16 +281,13 @@ unset target_list_exclude
# The following Meson options are handled manually (still they
# are included in the automatically generated help message)
-
-# 1. Track which submodules are needed
-fdt="auto"
-
-# 2. Automatically enable/disable other options
+# because they automatically enable/disable other options
tcg="auto"
cfi="false"
-# 3. Need to check for -static-pie before Meson runs. Also,
-# Meson has PIE as a boolean rather than enabled/disabled/auto.
+# Meson has PIE as a boolean rather than enabled/disabled/auto,
+# and we also need to check for -static-pie before Meson runs
+# which requires knowing whether --static is enabled.
pie=""
static="no"
@@ -752,14 +748,6 @@ for opt do
;;
--disable-cfi) cfi="false"
;;
- --disable-fdt) fdt="disabled"
- ;;
- --enable-fdt) fdt="enabled"
- ;;
- --enable-fdt=git) fdt="internal"
- ;;
- --enable-fdt=*) fdt="$optarg"
- ;;
--with-git=*) git="$optarg"
;;
--with-git-submodules=*)
@@ -783,10 +771,6 @@ for opt do
;;
--gdb=*) gdb_bin="$optarg"
;;
- --enable-vfio-user-server) vfio_user_server="enabled"
- ;;
- --disable-vfio-user-server) vfio_user_server="disabled"
- ;;
# everything else has the same name in configure and meson
--*) meson_option_parse "$opt" "$optarg"
;;
@@ -806,10 +790,11 @@ case $git_submodules_action in
fi
;;
ignore)
- if ! test -f "$source_path/subprojects/keycodemapdb/README"
+ if ! test -f "$source_path/subprojects/keycodemapdb/README" \
+ || ! test -f "$source_path/tests/fp/berkeley-testfloat-3/README.md"
then
echo
- echo "ERROR: missing GIT submodules"
+ echo "ERROR: missing GIT submodules or subprojects"
echo
if test -e "$source_path/.git"; then
echo "--with-git-submodules=ignore specified but submodules were not"
@@ -1182,16 +1167,6 @@ EOF
fi
fi
-##########################################
-# fdt probe
-
-case "$fdt" in
- auto | enabled | internal)
- # Simpler to always update submodule, even if not needed.
- git_submodules="${git_submodules} subprojects/dtc"
- ;;
-esac
-
########################################
# check if ccache is interfering with
# semantic analysis of macros
@@ -1627,17 +1602,6 @@ write_target_makefile() {
fi
}
-##########################################
-# check for vfio_user_server
-
-case "$vfio_user_server" in
- enabled )
- if test "$git_submodules_action" != "ignore"; then
- git_submodules="${git_submodules} subprojects/libvfio-user"
- fi
- ;;
-esac
-
#######################################
# cross-compiled firmware targets
@@ -1971,12 +1935,10 @@ if test "$skip_meson" = no; then
# QEMU options
test "$cfi" != false && meson_option_add "-Dcfi=$cfi"
test "$docs" != auto && meson_option_add "-Ddocs=$docs"
- test "$fdt" != auto && meson_option_add "-Dfdt=$fdt"
test -n "${LIB_FUZZING_ENGINE+xxx}" && meson_option_add "-Dfuzzing_engine=$LIB_FUZZING_ENGINE"
test "$qemu_suffix" != qemu && meson_option_add "-Dqemu_suffix=$qemu_suffix"
test "$smbd" != '' && meson_option_add "-Dsmbd=$smbd"
test "$tcg" != enabled && meson_option_add "-Dtcg=$tcg"
- test "$vfio_user_server" != auto && meson_option_add "-Dvfio_user_server=$vfio_user_server"
run_meson() {
NINJA=$ninja $meson setup --prefix "$prefix" "$@" $cross_arg "$PWD" "$source_path"
}
diff --git a/meson.build b/meson.build
index 1a55039ff7c3..aa4ab1390fd2 100644
--- a/meson.build
+++ b/meson.build
@@ -3029,14 +3029,7 @@ endif
libvfio_user_dep = not_found
if have_system and vfio_user_server_allowed
- have_internal = fs.exists(meson.current_source_dir() / 'subprojects/libvfio-user/meson.build')
-
- if not have_internal
- error('libvfio-user source not found - please pull git submodule')
- endif
-
- libvfio_user_proj = subproject('libvfio-user')
-
+ libvfio_user_proj = subproject('libvfio-user', required: true)
libvfio_user_dep = libvfio_user_proj.get_variable('libvfio_user_dep')
endif
@@ -3064,13 +3057,7 @@ if fdt_required.length() > 0 or fdt_opt == 'enabled'
endif
if not fdt.found()
assert(fdt_opt == 'internal')
- have_internal = fs.exists(meson.current_source_dir() / 'subprojects/dtc/meson.build')
-
- if not have_internal
- error('libfdt source not found - please pull git submodule')
- endif
-
- libfdt_proj = subproject('dtc',
+ libfdt_proj = subproject('dtc', required: true,
default_options: ['tools=false', 'yaml=disabled',
'python=disabled', 'default_library=static'])
fdt = libfdt_proj.get_variable('libfdt_dep')
diff --git a/meson_options.txt b/meson_options.txt
index 972c458b80b4..635a2bfd351f 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -296,6 +296,7 @@ option('capstone', type: 'feature', value: 'auto',
description: 'Whether and how to find the capstone library')
option('fdt', type: 'combo', value: 'auto',
choices: ['disabled', 'enabled', 'auto', 'system', 'internal'],
+ deprecated: { 'git': 'internal' },
description: 'Whether and how to find the libfdt library')
option('selinux', type: 'feature', value: 'auto',
diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index dba5ae05b67e..b99cb66e4122 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -26,8 +26,8 @@ sub_file="${sub_tdir}/submodule.tar"
# independent of what the developer currently has initialized
# in their checkout, because the build environment is completely
# different to the host OS.
-submodules="subprojects/dtc subprojects/keycodemapdb"
-submodules="$submodules tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
+subprojects="dtc keycodemapdb libvfio-user"
+submodules="tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
sub_deinit=""
function cleanup() {
@@ -70,4 +70,11 @@ for sm in $submodules; do
tar --concatenate --file "$tar_file" "$sub_file"
test $? -ne 0 && error "failed append submodule $sm to $tar_file"
done
+
+for sp in $subprojects; do
+ meson subprojects download $sp
+ test $? -ne 0 && error "failed to download subproject $sp"
+ tar --append --file "$tar_file" --exclude=.git subprojects/$sp
+ test $? -ne 0 && error "failed to append subproject $sp to $tar_file"
+done
exit 0
diff --git a/scripts/make-release b/scripts/make-release
index 44a9d86a04a7..0604e61b8143 100755
--- a/scripts/make-release
+++ b/scripts/make-release
@@ -16,6 +16,9 @@ if [ $# -ne 2 ]; then
exit 0
fi
+# Only include wraps that are invoked with subproject()
+SUBPROJECTS="dtc libvfio-user keycodemapdb"
+
src="$1"
version="$2"
destination=qemu-${version}
@@ -26,6 +29,8 @@ git clone --single-branch -b "v${version}" -c advice.detachedHead=false \
pushd ${destination}
git submodule update --init --single-branch
+meson subprojects download $SUBPROJECTS
+
(cd roms/seabios && git describe --tags --long --dirty > .version)
(cd roms/skiboot && ./make_version.sh > .version)
# Fetch edk2 submodule's submodules, since it won't have access to them via
diff --git a/subprojects/.gitignore b/subprojects/.gitignore
index 7560ebb0b1a0..935e9a72a8f9 100644
--- a/subprojects/.gitignore
+++ b/subprojects/.gitignore
@@ -1,3 +1,6 @@
/packagecache
+/dtc
+/keycodemapdb
+/libvfio-user
/slirp
diff --git a/subprojects/dtc b/subprojects/dtc
deleted file mode 160000
index b6910bec1161..000000000000
--- a/subprojects/dtc
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit b6910bec11614980a21e46fbccc35934b671bd81
diff --git a/subprojects/dtc.wrap b/subprojects/dtc.wrap
new file mode 100644
index 000000000000..d1bc9174e913
--- /dev/null
+++ b/subprojects/dtc.wrap
@@ -0,0 +1,4 @@
+[wrap-git]
+url = https://gitlab.com/qemu-project/dtc.git
+revision = b6910bec11614980a21e46fbccc35934b671bd81
+depth = 1
diff --git a/subprojects/keycodemapdb b/subprojects/keycodemapdb
deleted file mode 160000
index f5772a62ec52..000000000000
--- a/subprojects/keycodemapdb
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit f5772a62ec52591ff6870b7e8ef32482371f22c6
diff --git a/subprojects/keycodemapdb.wrap b/subprojects/keycodemapdb.wrap
new file mode 100644
index 000000000000..dda7b0e5716f
--- /dev/null
+++ b/subprojects/keycodemapdb.wrap
@@ -0,0 +1,4 @@
+[wrap-git]
+url = https://gitlab.com/qemu-project/keycodemapdb.git
+revision = f5772a62ec52591ff6870b7e8ef32482371f22c6
+depth = 1
diff --git a/subprojects/libvfio-user b/subprojects/libvfio-user
deleted file mode 160000
index 0b28d205572c..000000000000
--- a/subprojects/libvfio-user
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit 0b28d205572c80b568a1003db2c8f37ca333e4d7
diff --git a/subprojects/libvfio-user.wrap b/subprojects/libvfio-user.wrap
new file mode 100644
index 000000000000..416955ca4515
--- /dev/null
+++ b/subprojects/libvfio-user.wrap
@@ -0,0 +1,4 @@
+[wrap-git]
+url = https://gitlab.com/qemu-project/libvfio-user.git
+revision = 0b28d205572c80b568a1003db2c8f37ca333e4d7
+depth = 1
--
2.40.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH 6/6] meson: subprojects: replace submodules with wrap files
2023-05-19 8:56 ` [PATCH 6/6] meson: subprojects: replace submodules with wrap files Paolo Bonzini
@ 2023-05-19 9:25 ` Daniel P. Berrangé
2023-05-19 9:36 ` Paolo Bonzini
0 siblings, 1 reply; 22+ messages in thread
From: Daniel P. Berrangé @ 2023-05-19 9:25 UTC (permalink / raw)
To: Paolo Bonzini
Cc: qemu-devel, marcandre.lureau, richard.henderson, peter.maydell
On Fri, May 19, 2023 at 10:56:46AM +0200, Paolo Bonzini wrote:
> Compared to submodules, .wrap files have several advantages:
>
> * option parsing and downloading is delegated to meson
>
> * the commit is stored in a text file instead of a magic entry in the
> git tree object
This avoids one of the big pain points with submodules, in that
when switching branches you often end up with dirty submodules,
which then get accidentally added to the commit. IIUC, with wrap,
when switching branches, meson should automatically update the
wrap to the right git hash and re-buld as needed. I think most
people would consider this along to be sufficient reason to
drop submodules.
> * we could stop shipping external dependencies that are only used as a
> fallback, but not break compilation on platforms that lack them.
> For example it may make sense to download dtc at build time,
> controlled by --enable-download, even when building from a tarball.
> This is _not_ done in this patch.
IIUC, you actually added code to make-release to avoid doing this.
Considering each lib
* dtc - the distro should always have it anyway
* libvfio-user - don't think this has found its way
into any distros yet. So if we don't
bundle it, distros have to pacakge it
or re-bundle with their QEMU bulds
* keycodemapdb - has always been intended to be a copylib,
not packaged separately by distros. In
retrospect I think this wa probably a
mistake. None the less today, it will
need to be re-bundled by distros if we
omitted it.
I think there's a decent case to be made for 'dtc' to be dropped
from 'make-release', but keep the other two.
> dependency() can fall back to a wrap automatically. However, this
> is only possible for libraries that come with a .pc file, and this
> is not very common for libfdt even though the upstream project in
> principle provides it; it also removes the control that we provide with
> --enable-fdt={system,internal}. Therefore, the logic to pick system
> vs. internal libfdt is left untouched.
>
> It is possible to use subprojects also for berkeley-softfloat-3
> and berkeley-testfloat-3, but this is left for later.
>mak
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> .gitlab-ci.d/buildtest-template.yml | 1 +
> .gitmodules | 9 -----
> configure | 54 +++++------------------------
> meson.build | 17 ++-------
> meson_options.txt | 1 +
> scripts/archive-source.sh | 11 ++++--
> scripts/make-release | 5 +++
> subprojects/.gitignore | 3 ++
> subprojects/dtc | 1 -
> subprojects/dtc.wrap | 4 +++
> subprojects/keycodemapdb | 1 -
> subprojects/keycodemapdb.wrap | 4 +++
> subprojects/libvfio-user | 1 -
> subprojects/libvfio-user.wrap | 4 +++
> 14 files changed, 41 insertions(+), 75 deletions(-)
> delete mode 160000 subprojects/dtc
> create mode 100644 subprojects/dtc.wrap
> delete mode 160000 subprojects/keycodemapdb
> create mode 100644 subprojects/keycodemapdb.wrap
> delete mode 160000 subprojects/libvfio-user
> create mode 100644 subprojects/libvfio-user.wrap
>
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index c9f2e737c09a..3c997d7265b2 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -44,6 +44,7 @@
> script:
> - scripts/git-submodule.sh update
> $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
> + - meson subprojects download $(cd build/subprojects && echo *)
Why is this addition needed ? Isn't meson supposd to automatically
download the wrapped subprojects it needs without explicit user
action ?
> diff --git a/meson_options.txt b/meson_options.txt
> index 972c458b80b4..635a2bfd351f 100644
> --- a/meson_options.txt
> +++ b/meson_options.txt
> @@ -296,6 +296,7 @@ option('capstone', type: 'feature', value: 'auto',
> description: 'Whether and how to find the capstone library')
> option('fdt', type: 'combo', value: 'auto',
> choices: ['disabled', 'enabled', 'auto', 'system', 'internal'],
> + deprecated: { 'git': 'internal' },
> description: 'Whether and how to find the libfdt library')
Do we need to deprecate this, as opposed to removing it ?
We've considered build time options to not be subject to a deprecation
process, just documenting changes in the release notes.
>
> option('selinux', type: 'feature', value: 'auto',
> diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
> index dba5ae05b67e..b99cb66e4122 100755
> --- a/scripts/archive-source.sh
> +++ b/scripts/archive-source.sh
> @@ -26,8 +26,8 @@ sub_file="${sub_tdir}/submodule.tar"
> # independent of what the developer currently has initialized
> # in their checkout, because the build environment is completely
> # different to the host OS.
> -submodules="subprojects/dtc subprojects/keycodemapdb"
> -submodules="$submodules tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
> +subprojects="dtc keycodemapdb libvfio-user"
> +submodules="tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
> sub_deinit=""
>
> function cleanup() {
> @@ -70,4 +70,11 @@ for sm in $submodules; do
> tar --concatenate --file "$tar_file" "$sub_file"
> test $? -ne 0 && error "failed append submodule $sm to $tar_file"
> done
> +
> +for sp in $subprojects; do
> + meson subprojects download $sp
> + test $? -ne 0 && error "failed to download subproject $sp"
> + tar --append --file "$tar_file" --exclude=.git subprojects/$sp
> + test $? -ne 0 && error "failed to append subproject $sp to $tar_file"
> +done
> exit 0
> diff --git a/scripts/make-release b/scripts/make-release
> index 44a9d86a04a7..0604e61b8143 100755
> --- a/scripts/make-release
> +++ b/scripts/make-release
> @@ -16,6 +16,9 @@ if [ $# -ne 2 ]; then
> exit 0
> fi
>
> +# Only include wraps that are invoked with subproject()
> +SUBPROJECTS="dtc libvfio-user keycodemapdb"
> +
> src="$1"
> version="$2"
> destination=qemu-${version}
> @@ -26,6 +29,8 @@ git clone --single-branch -b "v${version}" -c advice.detachedHead=false \
> pushd ${destination}
>
> git submodule update --init --single-branch
> +meson subprojects download $SUBPROJECTS
> +
> (cd roms/seabios && git describe --tags --long --dirty > .version)
> (cd roms/skiboot && ./make_version.sh > .version)
> # Fetch edk2 submodule's submodules, since it won't have access to them via
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH 6/6] meson: subprojects: replace submodules with wrap files
2023-05-19 9:25 ` Daniel P. Berrangé
@ 2023-05-19 9:36 ` Paolo Bonzini
0 siblings, 0 replies; 22+ messages in thread
From: Paolo Bonzini @ 2023-05-19 9:36 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, marcandre.lureau, richard.henderson, peter.maydell
On 5/19/23 11:25, Daniel P. Berrangé wrote:
> * dtc - the distro should always have it anyway
> * libvfio-user - don't think this has found its way
> into any distros yet. So if we don't
> bundle it, distros have to pacakge it
> or re-bundle with their QEMU bulds
I think it's not ABI-stable either.
> * keycodemapdb - has always been intended to be a copylib,
> not packaged separately by distros. In
> retrospect I think this wa probably a
> mistake. None the less today, it will
> need to be re-bundled by distros if we
> omitted it.
>
> I think there's a decent case to be made for 'dtc' to be dropped
> from 'make-release', but keep the other two.
Sure.
>>
>> script:
>> - scripts/git-submodule.sh update
>> $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
>> + - meson subprojects download $(cd build/subprojects && echo *)
>
> Why is this addition needed ? Isn't meson supposd to automatically
> download the wrapped subprojects it needs without explicit user
> action ?
Because this is the script for the test jobs; they don't get
subprojects/ from the build artifacts, and they don't rerun "meson
setup" either. So "build/subprojects" is already populated but the
source tree does not have the downloads.
"meson test" also runs submodule tests, and they fail. See
https://gitlab.com/bonzini/qemu/-/jobs/4309651149#L305 for an example.
>> @@ -296,6 +296,7 @@ option('capstone', type: 'feature', value: 'auto',
>> description: 'Whether and how to find the capstone library')
>> option('fdt', type: 'combo', value: 'auto',
>> choices: ['disabled', 'enabled', 'auto', 'system', 'internal'],
>> + deprecated: { 'git': 'internal' },
>> description: 'Whether and how to find the libfdt library')
>
> Do we need to deprecate this, as opposed to removing it ?
>
> We've considered build time options to not be subject to a deprecation
> process, just documenting changes in the release notes.
No issue with that. In this case it's so easy that I didn't bother.
Paolo
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v2 0/6] meson: use subprojects for bundled projects
2023-05-19 8:56 [PATCH v2 0/6] meson: use subprojects for bundled projects Paolo Bonzini
` (5 preceding siblings ...)
2023-05-19 8:56 ` [PATCH 6/6] meson: subprojects: replace submodules with wrap files Paolo Bonzini
@ 2023-05-19 9:21 ` Peter Maydell
2023-05-19 9:28 ` Paolo Bonzini
2023-05-19 9:32 ` Daniel P. Berrangé
2023-05-19 9:30 ` Peter Maydell
7 siblings, 2 replies; 22+ messages in thread
From: Peter Maydell @ 2023-05-19 9:21 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, marcandre.lureau, berrange, richard.henderson
On Fri, 19 May 2023 at 09:56, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> QEMU bundles a copy of dtc and keycodemapdb. They both support meson,
> so we can run their build system via subproject() instead of hardcoding
> their contents.
>
> In fact, now that QEMU's configure script knows how to install missing
> Python dependencies, we can replace submodules altogether with .wrap
> files, which have several advantages, either immediate or potential:
So do we have a path for getting rid of submodules entirely?
I'd be all in favour of that, but I'm somewhat less in favour
of "some things are submodules and some things are this new
.wrap file setup"...
thanks
-- PMM
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH v2 0/6] meson: use subprojects for bundled projects
2023-05-19 9:21 ` [PATCH v2 0/6] meson: use subprojects for bundled projects Peter Maydell
@ 2023-05-19 9:28 ` Paolo Bonzini
2023-05-19 9:32 ` Daniel P. Berrangé
1 sibling, 0 replies; 22+ messages in thread
From: Paolo Bonzini @ 2023-05-19 9:28 UTC (permalink / raw)
To: Peter Maydell; +Cc: qemu-devel, marcandre.lureau, berrange, richard.henderson
On Fri, May 19, 2023 at 11:21 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Fri, 19 May 2023 at 09:56, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> > QEMU bundles a copy of dtc and keycodemapdb. They both support meson,
> > so we can run their build system via subproject() instead of hardcoding
> > their contents.
> >
> > In fact, now that QEMU's configure script knows how to install missing
> > Python dependencies, we can replace submodules altogether with .wrap
> > files, which have several advantages, either immediate or potential:
>
> So do we have a path for getting rid of submodules entirely?
For building QEMU, yes.
The firmware submodules would not be affected, but those are really
just a way to ensure we respect the GPL when distributing firmware
blobs. We could keep them, as they mostly stay out of the way, or we
could move the information to files that are then consulted by
scripts/make-release. And those files could very well reuse the same
".ini" format that is used by .wrap files.
Paolo
> I'd be all in favour of that, but I'm somewhat less in favour
> of "some things are submodules and some things are this new
> .wrap file setup"...
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v2 0/6] meson: use subprojects for bundled projects
2023-05-19 9:21 ` [PATCH v2 0/6] meson: use subprojects for bundled projects Peter Maydell
2023-05-19 9:28 ` Paolo Bonzini
@ 2023-05-19 9:32 ` Daniel P. Berrangé
2023-05-19 9:38 ` Peter Maydell
` (2 more replies)
1 sibling, 3 replies; 22+ messages in thread
From: Daniel P. Berrangé @ 2023-05-19 9:32 UTC (permalink / raw)
To: Peter Maydell
Cc: Paolo Bonzini, qemu-devel, marcandre.lureau, richard.henderson
On Fri, May 19, 2023 at 10:21:03AM +0100, Peter Maydell wrote:
> On Fri, 19 May 2023 at 09:56, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> > QEMU bundles a copy of dtc and keycodemapdb. They both support meson,
> > so we can run their build system via subproject() instead of hardcoding
> > their contents.
> >
> > In fact, now that QEMU's configure script knows how to install missing
> > Python dependencies, we can replace submodules altogether with .wrap
> > files, which have several advantages, either immediate or potential:
>
> So do we have a path for getting rid of submodules entirely?
> I'd be all in favour of that, but I'm somewhat less in favour
> of "some things are submodules and some things are this new
> .wrap file setup"...
I think we can ignore 'roms' submodules on the basis they are never used
during a normal build, they're mostly just a maintainer convenience and
for release archive source bundling. That would leave us with
$ grep submodule .gitmodules | sort | grep -v roms
[submodule "dtc"]
[submodule "subprojects/libvfio-user"]
[submodule "tests/fp/berkeley-softfloat-3"]
[submodule "tests/fp/berkeley-testfloat-3"]
[submodule "tests/lcitool/libvirt-ci"]
[submodule "ui/keycodemapdb"]
as submodules which are commonly used. This series handles dtc, libvfio-user
and keycodemapdb leaving the float tests and the libvirt-ci module. The
libvirt-ci module isn't used by meson, so I guess that doesn't directly
fit into its wrap handling.
Maybe we can none the less wire up libvirt-ci into wrap, or alternatively
since its python, could we make our new venv setup tools put libvirt-ci
into the venv ?
Paolo mentioned the float stuff as future work, but not sure what his
plans are specifically?
Feels like we should be able to figure out some way to get rid of all
the submodules though, except for the roms, which are special and ok to
leave IMHO
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH v2 0/6] meson: use subprojects for bundled projects
2023-05-19 9:32 ` Daniel P. Berrangé
@ 2023-05-19 9:38 ` Peter Maydell
2023-05-19 9:40 ` Paolo Bonzini
2023-05-19 11:48 ` Paolo Bonzini
2 siblings, 0 replies; 22+ messages in thread
From: Peter Maydell @ 2023-05-19 9:38 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Paolo Bonzini, qemu-devel, marcandre.lureau, richard.henderson
On Fri, 19 May 2023 at 10:32, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, May 19, 2023 at 10:21:03AM +0100, Peter Maydell wrote:
> > On Fri, 19 May 2023 at 09:56, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > >
> > > QEMU bundles a copy of dtc and keycodemapdb. They both support meson,
> > > so we can run their build system via subproject() instead of hardcoding
> > > their contents.
> > >
> > > In fact, now that QEMU's configure script knows how to install missing
> > > Python dependencies, we can replace submodules altogether with .wrap
> > > files, which have several advantages, either immediate or potential:
> >
> > So do we have a path for getting rid of submodules entirely?
> > I'd be all in favour of that, but I'm somewhat less in favour
> > of "some things are submodules and some things are this new
> > .wrap file setup"...
>
> I think we can ignore 'roms' submodules on the basis they are never used
> during a normal build, they're mostly just a maintainer convenience and
> for release archive source bundling.
Yeah, I agree -- or at least I agree they're a distinct use of
submodules. We might at some point want to look at whether there's
a better way to deal with ROM blobs than what we do today, especially
as the size of the source archives for things like UEFI is much
bigger than I expect anybody anticipated when they set the rom
blob machinery up. I don't think the way we do ROM blobs is
very convenient for our distro downstreams either.
-- PMM
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v2 0/6] meson: use subprojects for bundled projects
2023-05-19 9:32 ` Daniel P. Berrangé
2023-05-19 9:38 ` Peter Maydell
@ 2023-05-19 9:40 ` Paolo Bonzini
2023-05-19 11:48 ` Paolo Bonzini
2 siblings, 0 replies; 22+ messages in thread
From: Paolo Bonzini @ 2023-05-19 9:40 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Peter Maydell, qemu-devel, marcandre.lureau, richard.henderson
On Fri, May 19, 2023 at 11:32 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> Maybe we can none the less wire up libvirt-ci into wrap, or alternatively
> since its python, could we make our new venv setup tools put libvirt-ci
> into the venv ?
libvirt-ci is used from the source tree so I think it doesn't even fit
into the venv. It's not unlike the roms.
> Paolo mentioned the float stuff as future work, but not sure what his
> plans are specifically?
No plans in particular---after all the whole meson conversion was like
"let's try to find a day every month to proceed", and these two
submodules are not particularly big. I know it's doable but I heard
there were plans of embedding it too, so I left it aside and focused
on the more interesting ones.
Paolo
> Feels like we should be able to figure out some way to get rid of all
> the submodules though, except for the roms, which are special and ok to
> leave IMHO
>
> With regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v2 0/6] meson: use subprojects for bundled projects
2023-05-19 9:32 ` Daniel P. Berrangé
2023-05-19 9:38 ` Peter Maydell
2023-05-19 9:40 ` Paolo Bonzini
@ 2023-05-19 11:48 ` Paolo Bonzini
2023-05-24 7:51 ` Thomas Huth
2 siblings, 1 reply; 22+ messages in thread
From: Paolo Bonzini @ 2023-05-19 11:48 UTC (permalink / raw)
To: Daniel P. Berrangé, Peter Maydell
Cc: qemu-devel, marcandre.lureau, richard.henderson, Thomas Huth
On 5/19/23 11:32, Daniel P. Berrangé wrote:
> Feels like we should be able to figure out some way to get rid of all
> the submodules though, except for the roms, which are special and ok to
> leave IMHO
Hmm, almost. roms/SLOF is used to build
pc-bios/s390-ccw/s390-netboot.img, so right now GIT_SUBMODULE and
GIT_SUBMODULE_ACTION could not be removed even without counting the
tests/fp submodules. That's about 10,000 lines of code, or about 20% of
all of SLOF; it might be a bit too much for vendoring.
Adding Thomas for ideas...
Paolo
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v2 0/6] meson: use subprojects for bundled projects
2023-05-19 11:48 ` Paolo Bonzini
@ 2023-05-24 7:51 ` Thomas Huth
0 siblings, 0 replies; 22+ messages in thread
From: Thomas Huth @ 2023-05-24 7:51 UTC (permalink / raw)
To: Paolo Bonzini, Daniel P. Berrangé, Peter Maydell
Cc: qemu-devel, marcandre.lureau, richard.henderson, qemu-s390x,
Christian Borntraeger
On 19/05/2023 13.48, Paolo Bonzini wrote:
> On 5/19/23 11:32, Daniel P. Berrangé wrote:
>> Feels like we should be able to figure out some way to get rid of all
>> the submodules though, except for the roms, which are special and ok to
>> leave IMHO
>
> Hmm, almost. roms/SLOF is used to build pc-bios/s390-ccw/s390-netboot.img,
> so right now GIT_SUBMODULE and GIT_SUBMODULE_ACTION could not be removed
> even without counting the tests/fp submodules. That's about 10,000 lines of
> code, or about 20% of all of SLOF; it might be a bit too much for vendoring.
>
> Adding Thomas for ideas...
Sorry, I also don't have any good ideas here. We could maybe copy the libs
from the SLOF repository over into the s390-ccw folder, but that's kind of
ugly, too (there aren't many updates in SLOF anymore, so it wouldn't at
least hurt too much from a synchronization point of view) ... CC:-ing
qemu-s390x mailing list, maybe somebody else there has a better idea.
Thomas
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v2 0/6] meson: use subprojects for bundled projects
2023-05-19 8:56 [PATCH v2 0/6] meson: use subprojects for bundled projects Paolo Bonzini
` (6 preceding siblings ...)
2023-05-19 9:21 ` [PATCH v2 0/6] meson: use subprojects for bundled projects Peter Maydell
@ 2023-05-19 9:30 ` Peter Maydell
7 siblings, 0 replies; 22+ messages in thread
From: Peter Maydell @ 2023-05-19 9:30 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, marcandre.lureau, berrange, richard.henderson
On Fri, 19 May 2023 at 09:56, Paolo Bonzini <pbonzini@redhat.com> wrote:
> It is possible to use subprojects also for berkeley-softfloat-3
> and berkeley-testfloat-3. This would require moving the corresponding
> parts of tests/fp/meson.build to an overlay file.
I think we also talked about simply putting those directly in
the main qemu repo, since we've forked the tests anyway.
-- PMM
^ permalink raw reply [flat|nested] 22+ messages in thread