[PATCH v2] configure: add support for pseudo-"in source tree" builds

[PATCH v2] configure: add support for pseudo-"in source tree" builds

[Daniel P. Berrangé]

Meson requires the build dir to be separate from the source tree. Many
people are used to just running "./configure && make" though and the
meson conversion breaks that.

This introduces some backcompat support to make it appear as if an
"in source tree" build is being done, but with the the results in the
"build/" directory. This allows "./configure && make" to work as it
did historically, albeit with the output binaries staying under build/.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---

This is a simple integration of Eric's proposal from

  https://lists.gnu.org/archive/html/qemu-devel/2020-03/msg07290.html

with a bit of configure magic. It is enough to enable

   ./configure
   make
   make check

I've not tested it beyond that. Note it blows away the "build/"
dir each time ./configure is run so it is pristine each time.

We could optionally symlink binaries from build/ into $PWD
if people think that is important, eg by changing GNUmakefile
to have:

recurse: all
        for bin in `find build -maxdepth 1 -type f -executable | grep -v -E '(ninjatool|config.status)'`; \
        do \
          ln -f -s $$bin . ; \
        done

and some cleanup logic to purge the symlinks for "make clean"

This goes on top of Paolo's most recent meson port v175 posting,
or whatever number it is upto now :-)

In v2:

 - Use a marker file so we don't blow away a build/ dir
   we didn't create
 - Silence the distclean rule
 - Fix broken use of error_exit that's not defined yet
 - Add comment to GNUmakefile

 .gitignore |  2 ++
 configure  | 48 +++++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 47 insertions(+), 3 deletions(-)

diff --git a/.gitignore b/.gitignore
index 92311284ef..4ccb9ed975 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,3 +1,5 @@
+/GNUmakefile
+/build/
 /.doctrees
 /config-devices.*
 /config-all-devices.*
diff --git a/configure b/configure
index cc5f58f31a..cdffe221c7 100755
--- a/configure
+++ b/configure
@@ -11,6 +11,51 @@ unset CLICOLOR_FORCE GREP_OPTIONS
 # Don't allow CCACHE, if present, to use cached results of compile tests!
 export CCACHE_RECACHE=yes
 
+# make source path absolute
+source_path=$(cd "$(dirname -- "$0")"; pwd)
+
+if test "$PWD" == "$source_path"
+then
+    echo "Using './build' as the directory for build output"
+
+    MARKER=build/auto-created-by-configure
+
+    if test -e build
+    then
+	if test -f $MARKER
+	then
+	   rm -rf build
+	else
+	    echo "ERROR: ./build dir already exists and was not previously created by configure"
+	    exit 1
+	fi
+    fi
+
+    mkdir build
+    touch $MARKER
+
+    cat > GNUmakefile <<EOF
+# This file is auto-generated by configure to support in-source tree
+# 'make' command invokation
+
+ifeq (\$(MAKECMDGOALS),)
+recurse: all
+endif
+
+.NOTPARALLEL: %
+%: force
+	@echo 'changing dir to build for \$(MAKE) "\$(MAKECMDGOALS)"...'
+	@\$(MAKE) -C build -f Makefile \$(MAKECMDGOALS)
+	@if test "\$(MAKECMDGOALS)" = "distclean" && test -e build/auto-created-by-configure ; then rm -rf build GNUmakefile ; fi
+force: ;
+.PHONY: force
+GNUmakefile: ;
+
+EOF
+    cd build
+    exec $source_path/configure "$@"
+fi
+
 # Temporary directory used for files created while
 # configure runs. Since it is in the build directory
 # we can safely blow away any previous version of it
@@ -297,9 +342,6 @@ ld_has() {
     $ld --help 2>/dev/null | grep ".$1" >/dev/null 2>&1
 }
 
-# make source path absolute
-source_path=$(cd "$(dirname -- "$0")"; pwd)
-
 if printf %s\\n "$source_path" "$PWD" | grep -q "[[:space:]:]";
 then
   error_exit "main directory cannot contain spaces nor colons"
-- 
2.26.2

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Philippe Mathieu-Daudé]

On 8/20/20 7:31 PM, Daniel P. Berrangé wrote:
> Meson requires the build dir to be separate from the source tree. Many
> people are used to just running "./configure && make" though and the
> meson conversion breaks that.
> 
> This introduces some backcompat support to make it appear as if an
> "in source tree" build is being done, but with the the results in the
> "build/" directory. This allows "./configure && make" to work as it
> did historically, albeit with the output binaries staying under build/.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> 
> This is a simple integration of Eric's proposal from
> 
>   https://lists.gnu.org/archive/html/qemu-devel/2020-03/msg07290.html
> 
> with a bit of configure magic. It is enough to enable
> 
>    ./configure
>    make
>    make check
> 
> I've not tested it beyond that. Note it blows away the "build/"
> dir each time ./configure is run so it is pristine each time.
> 
> We could optionally symlink binaries from build/ into $PWD
> if people think that is important, eg by changing GNUmakefile
> to have:
> 
> recurse: all
>         for bin in `find build -maxdepth 1 -type f -executable | grep -v -E '(ninjatool|config.status)'`; \
>         do \
>           ln -f -s $$bin . ; \
>         done
> 
> and some cleanup logic to purge the symlinks for "make clean"
> 
> This goes on top of Paolo's most recent meson port v175 posting,
> or whatever number it is upto now :-)
> 
> In v2:
> 
>  - Use a marker file so we don't blow away a build/ dir
>    we didn't create
>  - Silence the distclean rule
>  - Fix broken use of error_exit that's not defined yet
>  - Add comment to GNUmakefile
> 
>  .gitignore |  2 ++
>  configure  | 48 +++++++++++++++++++++++++++++++++++++++++++++---
>  2 files changed, 47 insertions(+), 3 deletions(-)
> 
> diff --git a/.gitignore b/.gitignore
> index 92311284ef..4ccb9ed975 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -1,3 +1,5 @@
> +/GNUmakefile
> +/build/
>  /.doctrees
>  /config-devices.*
>  /config-all-devices.*
> diff --git a/configure b/configure
> index cc5f58f31a..cdffe221c7 100755
> --- a/configure
> +++ b/configure
> @@ -11,6 +11,51 @@ unset CLICOLOR_FORCE GREP_OPTIONS
>  # Don't allow CCACHE, if present, to use cached results of compile tests!
>  export CCACHE_RECACHE=yes
>  
> +# make source path absolute
> +source_path=$(cd "$(dirname -- "$0")"; pwd)
> +
> +if test "$PWD" == "$source_path"
> +then
> +    echo "Using './build' as the directory for build output"
> +
> +    MARKER=build/auto-created-by-configure
> +
> +    if test -e build
> +    then
> +	if test -f $MARKER
> +	then
> +	   rm -rf build
> +	else
> +	    echo "ERROR: ./build dir already exists and was not previously created by configure"
> +	    exit 1

Tested normal build in fresh clone, then in my current workdir:

./configure
Using './build' as the directory for build output
ERROR: ./build dir already exists and was not previously created by
configure

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> +	fi
> +    fi
> +
> +    mkdir build
> +    touch $MARKER
> +
> +    cat > GNUmakefile <<EOF
> +# This file is auto-generated by configure to support in-source tree
> +# 'make' command invokation
> +
> +ifeq (\$(MAKECMDGOALS),)
> +recurse: all
> +endif
> +
> +.NOTPARALLEL: %
> +%: force
> +	@echo 'changing dir to build for \$(MAKE) "\$(MAKECMDGOALS)"...'
> +	@\$(MAKE) -C build -f Makefile \$(MAKECMDGOALS)
> +	@if test "\$(MAKECMDGOALS)" = "distclean" && test -e build/auto-created-by-configure ; then rm -rf build GNUmakefile ; fi
> +force: ;
> +.PHONY: force
> +GNUmakefile: ;
> +
> +EOF
> +    cd build
> +    exec $source_path/configure "$@"
> +fi
> +
>  # Temporary directory used for files created while
>  # configure runs. Since it is in the build directory
>  # we can safely blow away any previous version of it
> @@ -297,9 +342,6 @@ ld_has() {
>      $ld --help 2>/dev/null | grep ".$1" >/dev/null 2>&1
>  }
>  
> -# make source path absolute
> -source_path=$(cd "$(dirname -- "$0")"; pwd)
> -
>  if printf %s\\n "$source_path" "$PWD" | grep -q "[[:space:]:]";
>  then
>    error_exit "main directory cannot contain spaces nor colons"
> 

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Eric Blake]

On 8/20/20 12:31 PM, Daniel P. Berrangé wrote:
> Meson requires the build dir to be separate from the source tree. Many
> people are used to just running "./configure && make" though and the
> meson conversion breaks that.
> 
> This introduces some backcompat support to make it appear as if an
> "in source tree" build is being done, but with the the results in the

s/the the/the/

> "build/" directory. This allows "./configure && make" to work as it
> did historically, albeit with the output binaries staying under build/.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> 

> 
> In v2:
> 
>   - Use a marker file so we don't blow away a build/ dir
>     we didn't create
>   - Silence the distclean rule
>   - Fix broken use of error_exit that's not defined yet
>   - Add comment to GNUmakefile
> 
>   .gitignore |  2 ++
>   configure  | 48 +++++++++++++++++++++++++++++++++++++++++++++---
>   2 files changed, 47 insertions(+), 3 deletions(-)
> 
> diff --git a/.gitignore b/.gitignore
> index 92311284ef..4ccb9ed975 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -1,3 +1,5 @@
> +/GNUmakefile
> +/build/
>   /.doctrees
>   /config-devices.*
>   /config-all-devices.*
> diff --git a/configure b/configure
> index cc5f58f31a..cdffe221c7 100755
> --- a/configure
> +++ b/configure
> @@ -11,6 +11,51 @@ unset CLICOLOR_FORCE GREP_OPTIONS
>   # Don't allow CCACHE, if present, to use cached results of compile tests!
>   export CCACHE_RECACHE=yes
>   
> +# make source path absolute
> +source_path=$(cd "$(dirname -- "$0")"; pwd)

Our mails crossed; my 1 comment about CDPATH impacting this is still valid.

> +
> +if test "$PWD" == "$source_path"

and s/==/=/ for non-bash

> +then
> +    echo "Using './build' as the directory for build output"
> +
> +    MARKER=build/auto-created-by-configure
> +
> +    if test -e build
> +    then
> +	if test -f $MARKER
> +	then
> +	   rm -rf build
> +	else
> +	    echo "ERROR: ./build dir already exists and was not previously created by configure"
> +	    exit 1

TAB damage.

> +	fi
> +    fi
> +
> +    mkdir build
> +    touch $MARKER
> +
> +    cat > GNUmakefile <<EOF
> +# This file is auto-generated by configure to support in-source tree
> +# 'make' command invokation

invocation

> +
> +ifeq (\$(MAKECMDGOALS),)
> +recurse: all
> +endif
> +
> +.NOTPARALLEL: %
> +%: force
> +	@echo 'changing dir to build for \$(MAKE) "\$(MAKECMDGOALS)"...'
> +	@\$(MAKE) -C build -f Makefile \$(MAKECMDGOALS)
> +	@if test "\$(MAKECMDGOALS)" = "distclean" && test -e build/auto-created-by-configure ; then rm -rf build GNUmakefile ; fi
> +force: ;
> +.PHONY: force
> +GNUmakefile: ;
> +
> +EOF

Quote the first EOF to avoid having to quote all $ in the heredoc

> +    cd build
> +    exec $source_path/configure "$@"
> +fi
> +
>   # Temporary directory used for files created while
>   # configure runs. Since it is in the build directory
>   # we can safely blow away any previous version of it
> @@ -297,9 +342,6 @@ ld_has() {
>       $ld --help 2>/dev/null | grep ".$1" >/dev/null 2>&1
>   }
>   
> -# make source path absolute
> -source_path=$(cd "$(dirname -- "$0")"; pwd)
> -
>   if printf %s\\n "$source_path" "$PWD" | grep -q "[[:space:]:]";
>   then
>     error_exit "main directory cannot contain spaces nor colons"
> 

With corrections, this is looking good; I'm okay if v3 includes:
Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Eric Blake]

On 8/20/20 12:31 PM, Daniel P. Berrangé wrote:
> Meson requires the build dir to be separate from the source tree. Many
> people are used to just running "./configure && make" though and the
> meson conversion breaks that.
> 
> This introduces some backcompat support to make it appear as if an
> "in source tree" build is being done, but with the the results in the
> "build/" directory. This allows "./configure && make" to work as it
> did historically, albeit with the output binaries staying under build/.

An observation:

If you already had out-of-tree builds, this does not change anything. 
You can do an incremental build, where a tree that builds pre-merge 
should continue to build incrementally with 'make -C dir' post-merge.

If you are used to in-tree builds, and do a fresh checkout, this lets 
you maintain the illusion of an in-tree build although binaries may be 
located differently than you are used to.

But if you do an incremental update to an in-tree build, this will cause 
some odd behaviors, starting with the git update:

$ git merge $paolos_tag
error: The following untracked working tree files would be overwritten 
by merge:
	accel/kvm/trace.h
...
	util/trace.h
Please move or remove them before you merge.
Aborting
$ find -name trace.h -delete
$ git merge $paolos_tag
$ git am $this_patch
$ make
config-host.mak is out-of-date, running configure
Using './build' as the directory for build output
Submodule 'meson' (https://github.com/mesonbuild/meson/) registered for 
path 'meson'
Cloning into '/home/eblake/qemu-tmp2/meson'...
...
Command line for building ['libcommon.fa'] is long, using a response file
./ninjatool -t ninja2make --omit clean dist uninstall < build.ninja > 
Makefile.ninja
/bin/sh: build.ninja: No such file or directory
   GEN     tests/test-qapi-gen
make: Nothing to be done for 'all'.
$ echo $?
0
$ make
changing dir to build for make ""...
make[1]: Entering directory '/home/eblake/qemu-tmp2/build'
Makefile:84: *** This is an out of tree build but your source tree 
(/home/eblake/qemu-tmp2) seems to have been used for an in-tree build. 
You can fix this by running "make distclean && rm -rf *-linux-user 
*-softmmu" in your source tree.  Stop.
make[1]: Leaving directory '/home/eblake/qemu-tmp2/build'
make: *** [GNUmakefile:11: all] Error 2
$ echo $?
2

So I'm not sure why the first build gets as far as it does, but does NOT 
complete things and yet does not fail make, but my advice is that you 
should NOT try to an incremental build on in-tree build when crossing 
the meson epoch.  If you are a fan of in-tree convenience, you need a 
ONE-TIME distclean when pulling in these changes (the fact that you HAVE 
to clean up trace.h files to merge in the meson stuff should be a hint 
for that).  After that has been done, you can go back to pretending 
meson supports in-tree.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Daniel P. Berrangé]

On Thu, Aug 20, 2020 at 04:15:59PM -0500, Eric Blake wrote:
> On 8/20/20 12:31 PM, Daniel P. Berrangé wrote:
> > Meson requires the build dir to be separate from the source tree. Many
> > people are used to just running "./configure && make" though and the
> > meson conversion breaks that.
> > 
> > This introduces some backcompat support to make it appear as if an
> > "in source tree" build is being done, but with the the results in the
> > "build/" directory. This allows "./configure && make" to work as it
> > did historically, albeit with the output binaries staying under build/.
> 
> An observation:
> 
> If you already had out-of-tree builds, this does not change anything. You
> can do an incremental build, where a tree that builds pre-merge should
> continue to build incrementally with 'make -C dir' post-merge.
> 
> If you are used to in-tree builds, and do a fresh checkout, this lets you
> maintain the illusion of an in-tree build although binaries may be located
> differently than you are used to.
> 
> But if you do an incremental update to an in-tree build, this will cause
> some odd behaviors, starting with the git update:

snip.

> So I'm not sure why the first build gets as far as it does, but does NOT
> complete things and yet does not fail make, but my advice is that you should
> NOT try to an incremental build on in-tree build when crossing the meson
> epoch.  If you are a fan of in-tree convenience, you need a ONE-TIME
> distclean when pulling in these changes (the fact that you HAVE to clean up
> trace.h files to merge in the meson stuff should be a hint for that).  After
> that has been done, you can go back to pretending meson supports in-tree.

Yeah, I think that's something we'll have to live with. The amount of
extra complexity that would be required to try to make the incremental
build work across the meson introduction would be quite large. A more
efficient use of time is to just clean the dir manually, rather than
debugging some makefile rules that will only be needed once.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Kevin Wolf]

Am 20.08.2020 um 23:15 hat Eric Blake geschrieben:
> On 8/20/20 12:31 PM, Daniel P. Berrangé wrote:
> > Meson requires the build dir to be separate from the source tree. Many
> > people are used to just running "./configure && make" though and the
> > meson conversion breaks that.
> > 
> > This introduces some backcompat support to make it appear as if an
> > "in source tree" build is being done, but with the the results in the
> > "build/" directory. This allows "./configure && make" to work as it
> > did historically, albeit with the output binaries staying under build/.
> 
> An observation:
> 
> If you already had out-of-tree builds, this does not change anything. You
> can do an incremental build, where a tree that builds pre-merge should
> continue to build incrementally with 'make -C dir' post-merge.
> 
> If you are used to in-tree builds, and do a fresh checkout, this lets you
> maintain the illusion of an in-tree build although binaries may be located
> differently than you are used to.
> 
> But if you do an incremental update to an in-tree build, this will cause
> some odd behaviors, starting with the git update:
> 
> $ git merge $paolos_tag
> error: The following untracked working tree files would be overwritten by
> merge:
> 	accel/kvm/trace.h
> ...
> 	util/trace.h
> Please move or remove them before you merge.
> Aborting
> $ find -name trace.h -delete
> $ git merge $paolos_tag
> $ git am $this_patch
> $ make
> config-host.mak is out-of-date, running configure
> Using './build' as the directory for build output
> Submodule 'meson' (https://github.com/mesonbuild/meson/) registered for path
> 'meson'
> Cloning into '/home/eblake/qemu-tmp2/meson'...
> ...
> Command line for building ['libcommon.fa'] is long, using a response file
> ./ninjatool -t ninja2make --omit clean dist uninstall < build.ninja >
> Makefile.ninja
> /bin/sh: build.ninja: No such file or directory
>   GEN     tests/test-qapi-gen
> make: Nothing to be done for 'all'.
> $ echo $?
> 0
> $ make
> changing dir to build for make ""...
> make[1]: Entering directory '/home/eblake/qemu-tmp2/build'
> Makefile:84: *** This is an out of tree build but your source tree
> (/home/eblake/qemu-tmp2) seems to have been used for an in-tree build. You
> can fix this by running "make distclean && rm -rf *-linux-user *-softmmu" in
> your source tree.  Stop.
> make[1]: Leaving directory '/home/eblake/qemu-tmp2/build'
> make: *** [GNUmakefile:11: all] Error 2
> $ echo $?
> 2
> 
> So I'm not sure why the first build gets as far as it does, but does NOT
> complete things and yet does not fail make, but my advice is that you should
> NOT try to an incremental build on in-tree build when crossing the meson
> epoch.  If you are a fan of in-tree convenience, you need a ONE-TIME
> distclean when pulling in these changes (the fact that you HAVE to clean up
> trace.h files to merge in the meson stuff should be a hint for that).  After
> that has been done, you can go back to pretending meson supports in-tree.

Sounds like it will be painful to switch between branches based on make
and branches based on meson. By extension, it will also be painful to
check out and build old versions for comparison, or doing that even more
than once during git bisect. :-(

Kevin

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Paolo Bonzini]

On 21/08/20 12:04, Kevin Wolf wrote:
>> So I'm not sure why the first build gets as far as it does, but does NOT
>> complete things and yet does not fail make, but my advice is that you should
>> NOT try to an incremental build on in-tree build when crossing the meson
>> epoch.  If you are a fan of in-tree convenience, you need a ONE-TIME
>> distclean when pulling in these changes (the fact that you HAVE to clean up
>> trace.h files to merge in the meson stuff should be a hint for that).  After
>> that has been done, you can go back to pretending meson supports in-tree.
> Sounds like it will be painful to switch between branches based on make
> and branches based on meson. By extension, it will also be painful to
> check out and build old versions for comparison, or doing that even more
> than once during git bisect. :-(

Not if you switch to out-of-tree builds...

Paolo

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Kevin Wolf]

Am 21.08.2020 um 12:15 hat Paolo Bonzini geschrieben:
> On 21/08/20 12:04, Kevin Wolf wrote:
> >> So I'm not sure why the first build gets as far as it does, but does NOT
> >> complete things and yet does not fail make, but my advice is that you should
> >> NOT try to an incremental build on in-tree build when crossing the meson
> >> epoch.  If you are a fan of in-tree convenience, you need a ONE-TIME
> >> distclean when pulling in these changes (the fact that you HAVE to clean up
> >> trace.h files to merge in the meson stuff should be a hint for that).  After
> >> that has been done, you can go back to pretending meson supports in-tree.
> > Sounds like it will be painful to switch between branches based on make
> > and branches based on meson. By extension, it will also be painful to
> > check out and build old versions for comparison, or doing that even more
> > than once during git bisect. :-(
> 
> Not if you switch to out-of-tree builds...

I don't see what out-of-tree builds change about this unless I delete
and reconfigure them after each step? Even in those cases where I use
out-of-tree builds, I often reuse the same directory.

Kevin

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Paolo Bonzini]

On 21/08/20 12:54, Kevin Wolf wrote:
>>> Sounds like it will be painful to switch between branches based on make
>>> and branches based on meson. By extension, it will also be painful to
>>> check out and build old versions for comparison, or doing that even more
>>> than once during git bisect. :-(
>> Not if you switch to out-of-tree builds...
> I don't see what out-of-tree builds change about this unless I delete
> and reconfigure them after each step? Even in those cases where I use
> out-of-tree builds, I often reuse the same directory.

In two ways:

1) bisection will rarely jump around the point where Meson was
introduced.  First, because if it coincides with the 5.1.0 release, then
you know in advance if the problem was introduced before or after the
merge.  Second, because the bisection will quickly land on a merge
commit, and then you can (just for going down the merged branch) use a
separate out-of-tree build directory.

2) for the rare case of working on a branch that was developed pre-Meson
and has not been rebased (or need not be rebased), you again use a
separate out-of-tree build directory just for that one, and you don't
have to worry about doing distclean or an incremental builds.

Paolo

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Daniel P. Berrangé]

On Fri, Aug 21, 2020 at 12:04:26PM +0200, Kevin Wolf wrote:
> Am 20.08.2020 um 23:15 hat Eric Blake geschrieben:
> > On 8/20/20 12:31 PM, Daniel P. Berrangé wrote:
> > > Meson requires the build dir to be separate from the source tree. Many
> > > people are used to just running "./configure && make" though and the
> > > meson conversion breaks that.
> > > 
> > > This introduces some backcompat support to make it appear as if an
> > > "in source tree" build is being done, but with the the results in the
> > > "build/" directory. This allows "./configure && make" to work as it
> > > did historically, albeit with the output binaries staying under build/.
> > 
> > An observation:
> > 
> > If you already had out-of-tree builds, this does not change anything. You
> > can do an incremental build, where a tree that builds pre-merge should
> > continue to build incrementally with 'make -C dir' post-merge.
> > 
> > If you are used to in-tree builds, and do a fresh checkout, this lets you
> > maintain the illusion of an in-tree build although binaries may be located
> > differently than you are used to.
> > 
> > But if you do an incremental update to an in-tree build, this will cause
> > some odd behaviors, starting with the git update:
> > 
> > $ git merge $paolos_tag
> > error: The following untracked working tree files would be overwritten by
> > merge:
> > 	accel/kvm/trace.h
> > ...
> > 	util/trace.h
> > Please move or remove them before you merge.
> > Aborting
> > $ find -name trace.h -delete
> > $ git merge $paolos_tag
> > $ git am $this_patch
> > $ make
> > config-host.mak is out-of-date, running configure
> > Using './build' as the directory for build output
> > Submodule 'meson' (https://github.com/mesonbuild/meson/) registered for path
> > 'meson'
> > Cloning into '/home/eblake/qemu-tmp2/meson'...
> > ...
> > Command line for building ['libcommon.fa'] is long, using a response file
> > ./ninjatool -t ninja2make --omit clean dist uninstall < build.ninja >
> > Makefile.ninja
> > /bin/sh: build.ninja: No such file or directory
> >   GEN     tests/test-qapi-gen
> > make: Nothing to be done for 'all'.
> > $ echo $?
> > 0
> > $ make
> > changing dir to build for make ""...
> > make[1]: Entering directory '/home/eblake/qemu-tmp2/build'
> > Makefile:84: *** This is an out of tree build but your source tree
> > (/home/eblake/qemu-tmp2) seems to have been used for an in-tree build. You
> > can fix this by running "make distclean && rm -rf *-linux-user *-softmmu" in
> > your source tree.  Stop.
> > make[1]: Leaving directory '/home/eblake/qemu-tmp2/build'
> > make: *** [GNUmakefile:11: all] Error 2
> > $ echo $?
> > 2
> > 
> > So I'm not sure why the first build gets as far as it does, but does NOT
> > complete things and yet does not fail make, but my advice is that you should
> > NOT try to an incremental build on in-tree build when crossing the meson
> > epoch.  If you are a fan of in-tree convenience, you need a ONE-TIME
> > distclean when pulling in these changes (the fact that you HAVE to clean up
> > trace.h files to merge in the meson stuff should be a hint for that).  After
> > that has been done, you can go back to pretending meson supports in-tree.
> 
> Sounds like it will be painful to switch between branches based on make
> and branches based on meson. By extension, it will also be painful to
> check out and build old versions for comparison, or doing that even more
> than once during git bisect. :-(

Such is life when using in-source-dir builds.

Even with our current build system, using a separate build directory
makes it easier to work across multiple branches, as you can trivially
keep the builds from each branch separate and they won't interfere
with each other.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Paolo Bonzini]

On 20/08/20 19:31, Daniel P. Berrangé wrote:
> Meson requires the build dir to be separate from the source tree. Many
> people are used to just running "./configure && make" though and the
> meson conversion breaks that.
> 
> This introduces some backcompat support to make it appear as if an
> "in source tree" build is being done, but with the the results in the
> "build/" directory. This allows "./configure && make" to work as it
> did historically, albeit with the output binaries staying under build/.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> 
> This is a simple integration of Eric's proposal from
> 
>   https://lists.gnu.org/archive/html/qemu-devel/2020-03/msg07290.html
> 
> with a bit of configure magic. It is enough to enable
> 
>    ./configure
>    make
>    make check
> 
> I've not tested it beyond that. Note it blows away the "build/"
> dir each time ./configure is run so it is pristine each time.
> 
> We could optionally symlink binaries from build/ into $PWD
> if people think that is important, eg by changing GNUmakefile
> to have:
> 
> recurse: all
>         for bin in `find build -maxdepth 1 -type f -executable | grep -v -E '(ninjatool|config.status)'`; \
>         do \
>           ln -f -s $$bin . ; \
>         done
> 
> and some cleanup logic to purge the symlinks for "make clean"
> 
> This goes on top of Paolo's most recent meson port v175 posting,
> or whatever number it is upto now :-)
> 
> In v2:
> 
>  - Use a marker file so we don't blow away a build/ dir
>    we didn't create
>  - Silence the distclean rule
>  - Fix broken use of error_exit that's not defined yet
>  - Add comment to GNUmakefile
> 
>  .gitignore |  2 ++
>  configure  | 48 +++++++++++++++++++++++++++++++++++++++++++++---
>  2 files changed, 47 insertions(+), 3 deletions(-)
> 
> diff --git a/.gitignore b/.gitignore
> index 92311284ef..4ccb9ed975 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -1,3 +1,5 @@
> +/GNUmakefile
> +/build/
>  /.doctrees
>  /config-devices.*
>  /config-all-devices.*
> diff --git a/configure b/configure
> index cc5f58f31a..cdffe221c7 100755
> --- a/configure
> +++ b/configure
> @@ -11,6 +11,51 @@ unset CLICOLOR_FORCE GREP_OPTIONS
>  # Don't allow CCACHE, if present, to use cached results of compile tests!
>  export CCACHE_RECACHE=yes
>  
> +# make source path absolute
> +source_path=$(cd "$(dirname -- "$0")"; pwd)
> +
> +if test "$PWD" == "$source_path"
> +then
> +    echo "Using './build' as the directory for build output"
> +
> +    MARKER=build/auto-created-by-configure
> +
> +    if test -e build
> +    then
> +	if test -f $MARKER
> +	then
> +	   rm -rf build
> +	else
> +	    echo "ERROR: ./build dir already exists and was not previously created by configure"
> +	    exit 1
> +	fi
> +    fi
> +
> +    mkdir build
> +    touch $MARKER
> +
> +    cat > GNUmakefile <<EOF
> +# This file is auto-generated by configure to support in-source tree
> +# 'make' command invokation
> +
> +ifeq (\$(MAKECMDGOALS),)
> +recurse: all
> +endif
> +
> +.NOTPARALLEL: %
> +%: force
> +	@echo 'changing dir to build for \$(MAKE) "\$(MAKECMDGOALS)"...'
> +	@\$(MAKE) -C build -f Makefile \$(MAKECMDGOALS)
> +	@if test "\$(MAKECMDGOALS)" = "distclean" && test -e build/auto-created-by-configure ; then rm -rf build GNUmakefile ; fi
> +force: ;
> +.PHONY: force
> +GNUmakefile: ;
> +
> +EOF
> +    cd build
> +    exec $source_path/configure "$@"
> +fi
> +
>  # Temporary directory used for files created while
>  # configure runs. Since it is in the build directory
>  # we can safely blow away any previous version of it
> @@ -297,9 +342,6 @@ ld_has() {
>      $ld --help 2>/dev/null | grep ".$1" >/dev/null 2>&1
>  }
>  
> -# make source path absolute
> -source_path=$(cd "$(dirname -- "$0")"; pwd)
> -
>  if printf %s\\n "$source_path" "$PWD" | grep -q "[[:space:]:]";
>  then
>    error_exit "main directory cannot contain spaces nor colons"
> 

Queued, thanks!

Paolo

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Kevin Wolf]

Am 20.08.2020 um 19:31 hat Daniel P. Berrangé geschrieben:
> Meson requires the build dir to be separate from the source tree. Many
> people are used to just running "./configure && make" though and the
> meson conversion breaks that.
> 
> This introduces some backcompat support to make it appear as if an
> "in source tree" build is being done, but with the the results in the
> "build/" directory. This allows "./configure && make" to work as it
> did historically, albeit with the output binaries staying under build/.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> 
> This is a simple integration of Eric's proposal from
> 
>   https://lists.gnu.org/archive/html/qemu-devel/2020-03/msg07290.html
> 
> with a bit of configure magic. It is enough to enable
> 
>    ./configure
>    make
>    make check
> 
> I've not tested it beyond that. Note it blows away the "build/"
> dir each time ./configure is run so it is pristine each time.

I guess "make install" is the only other one that normal users would
care about. We shoud make sure that it works (I don't see why it
wouldn't, but worth testing anyway).

Kevin

Re: [PATCH v2] configure: add support for pseudo-"in source tree" builds

[Daniel P. Berrangé]

On Fri, Aug 21, 2020 at 12:17:53PM +0200, Kevin Wolf wrote:
> Am 20.08.2020 um 19:31 hat Daniel P. Berrangé geschrieben:
> > Meson requires the build dir to be separate from the source tree. Many
> > people are used to just running "./configure && make" though and the
> > meson conversion breaks that.
> > 
> > This introduces some backcompat support to make it appear as if an
> > "in source tree" build is being done, but with the the results in the
> > "build/" directory. This allows "./configure && make" to work as it
> > did historically, albeit with the output binaries staying under build/.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> > 
> > This is a simple integration of Eric's proposal from
> > 
> >   https://lists.gnu.org/archive/html/qemu-devel/2020-03/msg07290.html
> > 
> > with a bit of configure magic. It is enough to enable
> > 
> >    ./configure
> >    make
> >    make check
> > 
> > I've not tested it beyond that. Note it blows away the "build/"
> > dir each time ./configure is run so it is pristine each time.
> 
> I guess "make install" is the only other one that normal users would
> care about. We shoud make sure that it works (I don't see why it
> wouldn't, but worth testing anyway).

I've just tested  "make instal DESTDIR=`pwd`/vroot" in order to
validate that variables are passed through too, and it appears to
be succesful.

$  make install DESTDIR=`pwd`/vroot
...snip...
$ find vroot/ | head -30
vroot/
vroot/usr
vroot/usr/local
vroot/usr/local/bin
vroot/usr/local/bin/qemu-nbd
vroot/usr/local/bin/qemu-keymap
vroot/usr/local/bin/qemu-pr-helper
vroot/usr/local/bin/qemu-storage-daemon
vroot/usr/local/bin/elf2dmp
vroot/usr/local/bin/qemu-img
vroot/usr/local/bin/qemu-edid
vroot/usr/local/bin/qemu-system-x86_64
vroot/usr/local/bin/qemu-ga
vroot/usr/local/bin/qemu-io
vroot/usr/local/var
vroot/usr/local/var/run
vroot/usr/local/libexec
vroot/usr/local/libexec/vhost-user-gpu
vroot/usr/local/libexec/qemu-bridge-helper
vroot/usr/local/libexec/virtfs-proxy-helper
vroot/usr/local/libexec/virtiofsd
vroot/usr/local/share
vroot/usr/local/share/icons
vroot/usr/local/share/icons/hicolor
vroot/usr/local/share/icons/hicolor/256x256
vroot/usr/local/share/icons/hicolor/256x256/apps
vroot/usr/local/share/icons/hicolor/256x256/apps/qemu.png
vroot/usr/local/share/icons/hicolor/48x48
vroot/usr/local/share/icons/hicolor/48x48/apps
vroot/usr/local/share/icons/hicolor/48x48/apps/qemu.png
...snip...

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|