[Qemu-devel] [PATCH 0/3] Fix various issues with Sphinx build machinery

[Qemu-devel] [PATCH 0/3] Fix various issues with Sphinx build machinery

[Peter Maydell]

This patchset fixes some bugs with the Sphinx build machinery.
Most importantly, it fixes in-tree builds, which were broken
if you had sphinx-build available. The other two patches fix
some more minor problems that I noticed in the course of
fixing the first one.

thanks
-- PMM

Peter Maydell (3):
  Makefile: Fix Sphinx documentation builds for in-tree builds
  Makefile: Fix 'make distclean'
  Makefile: Don't install non-sphinx files in sphinx docs install

 Makefile | 34 ++++++++++++++++++++++------------
 1 file changed, 22 insertions(+), 12 deletions(-)

-- 
2.20.1

[Qemu-devel] [PATCH 1/3] Makefile: Fix Sphinx documentation builds for in-tree builds

[Peter Maydell]

The Sphinx build-sphinx tool does not permit building a manual
into the same directory as its source files. This meant that
commit 5f71eac06e15b9a3fa1134d446f broke QEMU in-source-tree
builds, which would fail with:
  Error: source directory and destination directory are same.

Fix this by making in-tree builds build the Sphinx manuals
into a subdirectory of docs/.

Fixes: 5f71eac06e15b9a3fa1134d446f ("Makefile, configure: Support building rST documentation")
Reported-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 Makefile | 30 +++++++++++++++++++-----------
 1 file changed, 19 insertions(+), 11 deletions(-)

diff --git a/Makefile b/Makefile
index d2463c92371..0194ef7fa55 100644
--- a/Makefile
+++ b/Makefile
@@ -655,12 +655,20 @@ dist: qemu-$(VERSION).tar.bz2
 qemu-%.tar.bz2:
 	$(SRC_PATH)/scripts/make-release "$(SRC_PATH)" "$(patsubst qemu-%.tar.bz2,%,$@)"
 
-# Note that these commands assume that there are no HTML files in
-# the docs subdir in the source tree! If there are then this will
-# blow them away for an in-source-tree 'make clean'.
+# Sphinx does not allow building manuals into the same directory as
+# the source files, so if we're doing an in-tree QEMU build we must
+# build the manuals into a subdirectory (and then install them from
+# there for 'make install'). For an out-of-tree build we can just
+# use the docs/ subdirectory in the build tree as normal.
+ifeq ($(realpath $(SRC_PATH)),$(realpath .))
+MANUAL_BUILDDIR := docs/built
+else
+MANUAL_BUILDDIR := docs
+endif
+
 define clean-manual =
-rm -rf docs/$1/_static
-rm -f docs/$1/objects.inv docs/$1/searchindex.js docs/$1/*.html
+rm -rf $(MANUAL_BUILDDIR)/$1/_static
+rm -f $(MANUAL_BUILDDIR)/$1/objects.inv $(MANUAL_BUILDDIR)/$1/searchindex.js $(MANUAL_BUILDDIR)/$1/*.html
 endef
 
 distclean: clean
@@ -720,8 +728,8 @@ BLOBS=
 endif
 
 define install-manual =
-for d in $$(cd docs && find $1 -type d); do $(INSTALL_DIR) "$(DESTDIR)$(qemu_docdir)/$$d"; done
-for f in $$(cd docs && find $1 -type f); do $(INSTALL_DATA) "docs/$$f" "$(DESTDIR)$(qemu_docdir)/$$f"; done
+for d in $$(cd $(MANUAL_BUILDDIR) && find $1 -type d); do $(INSTALL_DIR) "$(DESTDIR)$(qemu_docdir)/$$d"; done
+for f in $$(cd $(MANUAL_BUILDDIR) && find $1 -type f); do $(INSTALL_DATA) "$(MANUAL_BUILDDIR)/$$f" "$(DESTDIR)$(qemu_docdir)/$$f"; done
 endef
 
 # Note that we deliberately do not install the "devel" manual: it is
@@ -885,17 +893,17 @@ docs/version.texi: $(SRC_PATH)/VERSION
 # and handles "don't rebuild things unless necessary" itself.
 # The '.doctrees' files are cached information to speed this up.
 .PHONY: sphinxdocs
-sphinxdocs: docs/devel/index.html docs/interop/index.html
+sphinxdocs: $(MANUAL_BUILDDIR)/devel/index.html $(MANUAL_BUILDDIR)/interop/index.html
 
 # Canned command to build a single manual
-build-manual = $(call quiet-command,sphinx-build $(if $(V),,-q) -b html -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 docs/$1 ,"SPHINX","docs/$1")
+build-manual = $(call quiet-command,sphinx-build $(if $(V),,-q) -b html -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
 # We assume all RST files in the manual's directory are used in it
 manual-deps = $(wildcard $(SRC_PATH)/docs/$1/*.rst) $(SRC_PATH)/docs/$1/conf.py $(SRC_PATH)/docs/conf.py
 
-docs/devel/index.html: $(call manual-deps,devel)
+$(MANUAL_BUILDDIR)/devel/index.html: $(call manual-deps,devel)
 	$(call build-manual,devel)
 
-docs/interop/index.html: $(call manual-deps,interop)
+$(MANUAL_BUILDDIR)/interop/index.html: $(call manual-deps,interop)
 	$(call build-manual,interop)
 
 qemu-options.texi: $(SRC_PATH)/qemu-options.hx $(SRC_PATH)/scripts/hxtool
-- 
2.20.1

[Qemu-devel] [PATCH 2/3] Makefile: Fix 'make distclean'

[Peter Maydell]

We forgot the '-r' option on the rm command to clean up the
Sphinx .doctrees working directory, which meant that
"make distclean" fails:
 rm: cannot remove '.doctrees': Is a directory

Add the missing option.

Fixes: 5f71eac06e15b9a3fa1134d446f ("Makefile, configure: Support building rST documentation")
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 0194ef7fa55..9fd7f3f9bfb 100644
--- a/Makefile
+++ b/Makefile
@@ -691,7 +691,7 @@ distclean: clean
 	rm -f docs/interop/qemu-qmp-ref.html docs/interop/qemu-ga-ref.html
 	rm -f docs/qemu-block-drivers.7
 	rm -f docs/qemu-cpu-models.7
-	rm -f .doctrees
+	rm -rf .doctrees
 	$(call clean-manual,devel)
 	$(call clean-manual,interop)
 	for d in $(TARGET_DIRS); do \
-- 
2.20.1

[Qemu-devel] [PATCH 3/3] Makefile: Don't install non-sphinx files in sphinx docs install

[Peter Maydell]

If we're doing an out-of-tree build of Sphinx, then we
copy some extra spurious files to the install directory
as part of 'make install':
qemu-ga-qapi.texi
qemu-ga-ref.7
qemu-ga-ref.7.pod
qemu-ga-ref.html
qemu-ga-ref.txt
qemu-qmp-qapi.texi
qemu-qmp-ref.7
qemu-qmp-ref.7.pod
qemu-qmp-ref.html
qemu-qmp-ref.txt

because these have been built into build/docs/interop along
with the Sphinx interop documents. Filter them out of the
set of files we install when we're installing the Sphinx-built
manual files. (They are installed into their correct locations
as part of the main install-doc target already.)

Fixes: 5f71eac06e15b9a3fa1134d446f ("Makefile, configure: Support building rST documentation")
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 Makefile | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 9fd7f3f9bfb..6ccb8639b08 100644
--- a/Makefile
+++ b/Makefile
@@ -727,9 +727,11 @@ else
 BLOBS=
 endif
 
+# Note that we manually filter-out the non-Sphinx documentation which
+# is currently built into the docs/interop directory in the build tree.
 define install-manual =
 for d in $$(cd $(MANUAL_BUILDDIR) && find $1 -type d); do $(INSTALL_DIR) "$(DESTDIR)$(qemu_docdir)/$$d"; done
-for f in $$(cd $(MANUAL_BUILDDIR) && find $1 -type f); do $(INSTALL_DATA) "$(MANUAL_BUILDDIR)/$$f" "$(DESTDIR)$(qemu_docdir)/$$f"; done
+for f in $$(cd $(MANUAL_BUILDDIR) && find $1 -type f -a '!' '(' -name 'qemu-*-qapi.*' -o -name 'qemu-*-ref.*' ')' ); do $(INSTALL_DATA) "$(MANUAL_BUILDDIR)/$$f" "$(DESTDIR)$(qemu_docdir)/$$f"; done
 endef
 
 # Note that we deliberately do not install the "devel" manual: it is
-- 
2.20.1

Re: [Qemu-devel] [PATCH 1/3] Makefile: Fix Sphinx documentation builds for in-tree builds

[Philippe Mathieu-Daudé]

On 3/8/19 2:57 PM, Peter Maydell wrote:
> The Sphinx build-sphinx tool does not permit building a manual
> into the same directory as its source files. This meant that
> commit 5f71eac06e15b9a3fa1134d446f broke QEMU in-source-tree
> builds, which would fail with:
>   Error: source directory and destination directory are same.
> 
> Fix this by making in-tree builds build the Sphinx manuals
> into a subdirectory of docs/.
> 
> Fixes: 5f71eac06e15b9a3fa1134d446f ("Makefile, configure: Support building rST documentation")
> Reported-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> ---
>  Makefile | 30 +++++++++++++++++++-----------
>  1 file changed, 19 insertions(+), 11 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index d2463c92371..0194ef7fa55 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -655,12 +655,20 @@ dist: qemu-$(VERSION).tar.bz2
>  qemu-%.tar.bz2:
>  	$(SRC_PATH)/scripts/make-release "$(SRC_PATH)" "$(patsubst qemu-%.tar.bz2,%,$@)"
>  
> -# Note that these commands assume that there are no HTML files in
> -# the docs subdir in the source tree! If there are then this will
> -# blow them away for an in-source-tree 'make clean'.
> +# Sphinx does not allow building manuals into the same directory as
> +# the source files, so if we're doing an in-tree QEMU build we must
> +# build the manuals into a subdirectory (and then install them from
> +# there for 'make install'). For an out-of-tree build we can just
> +# use the docs/ subdirectory in the build tree as normal.
> +ifeq ($(realpath $(SRC_PATH)),$(realpath .))
> +MANUAL_BUILDDIR := docs/built
> +else
> +MANUAL_BUILDDIR := docs
> +endif
> +
>  define clean-manual =
> -rm -rf docs/$1/_static
> -rm -f docs/$1/objects.inv docs/$1/searchindex.js docs/$1/*.html
> +rm -rf $(MANUAL_BUILDDIR)/$1/_static
> +rm -f $(MANUAL_BUILDDIR)/$1/objects.inv $(MANUAL_BUILDDIR)/$1/searchindex.js $(MANUAL_BUILDDIR)/$1/*.html
>  endef
>  
>  distclean: clean
> @@ -720,8 +728,8 @@ BLOBS=
>  endif
>  
>  define install-manual =
> -for d in $$(cd docs && find $1 -type d); do $(INSTALL_DIR) "$(DESTDIR)$(qemu_docdir)/$$d"; done
> -for f in $$(cd docs && find $1 -type f); do $(INSTALL_DATA) "docs/$$f" "$(DESTDIR)$(qemu_docdir)/$$f"; done
> +for d in $$(cd $(MANUAL_BUILDDIR) && find $1 -type d); do $(INSTALL_DIR) "$(DESTDIR)$(qemu_docdir)/$$d"; done
> +for f in $$(cd $(MANUAL_BUILDDIR) && find $1 -type f); do $(INSTALL_DATA) "$(MANUAL_BUILDDIR)/$$f" "$(DESTDIR)$(qemu_docdir)/$$f"; done
>  endef
>  
>  # Note that we deliberately do not install the "devel" manual: it is
> @@ -885,17 +893,17 @@ docs/version.texi: $(SRC_PATH)/VERSION
>  # and handles "don't rebuild things unless necessary" itself.
>  # The '.doctrees' files are cached information to speed this up.
>  .PHONY: sphinxdocs
> -sphinxdocs: docs/devel/index.html docs/interop/index.html
> +sphinxdocs: $(MANUAL_BUILDDIR)/devel/index.html $(MANUAL_BUILDDIR)/interop/index.html
>  
>  # Canned command to build a single manual
> -build-manual = $(call quiet-command,sphinx-build $(if $(V),,-q) -b html -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 docs/$1 ,"SPHINX","docs/$1")
> +build-manual = $(call quiet-command,sphinx-build $(if $(V),,-q) -b html -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
>  # We assume all RST files in the manual's directory are used in it
>  manual-deps = $(wildcard $(SRC_PATH)/docs/$1/*.rst) $(SRC_PATH)/docs/$1/conf.py $(SRC_PATH)/docs/conf.py
>  
> -docs/devel/index.html: $(call manual-deps,devel)
> +$(MANUAL_BUILDDIR)/devel/index.html: $(call manual-deps,devel)
>  	$(call build-manual,devel)
>  
> -docs/interop/index.html: $(call manual-deps,interop)
> +$(MANUAL_BUILDDIR)/interop/index.html: $(call manual-deps,interop)
>  	$(call build-manual,interop)
>  
>  qemu-options.texi: $(SRC_PATH)/qemu-options.hx $(SRC_PATH)/scripts/hxtool
> 

Re: [Qemu-devel] [PATCH 2/3] Makefile: Fix 'make distclean'

[Philippe Mathieu-Daudé]

On 3/8/19 2:57 PM, Peter Maydell wrote:
> We forgot the '-r' option on the rm command to clean up the
> Sphinx .doctrees working directory, which meant that
> "make distclean" fails:
>  rm: cannot remove '.doctrees': Is a directory
> 
> Add the missing option.
> 
> Fixes: 5f71eac06e15b9a3fa1134d446f ("Makefile, configure: Support building rST documentation")
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> ---
>  Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 0194ef7fa55..9fd7f3f9bfb 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -691,7 +691,7 @@ distclean: clean
>  	rm -f docs/interop/qemu-qmp-ref.html docs/interop/qemu-ga-ref.html
>  	rm -f docs/qemu-block-drivers.7
>  	rm -f docs/qemu-cpu-models.7
> -	rm -f .doctrees
> +	rm -rf .doctrees
>  	$(call clean-manual,devel)
>  	$(call clean-manual,interop)
>  	for d in $(TARGET_DIRS); do \
> 

Re: [Qemu-devel] [PATCH 3/3] Makefile: Don't install non-sphinx files in sphinx docs install

[Philippe Mathieu-Daudé]

On 3/8/19 2:57 PM, Peter Maydell wrote:
> If we're doing an out-of-tree build of Sphinx, then we
> copy some extra spurious files to the install directory
> as part of 'make install':
> qemu-ga-qapi.texi
> qemu-ga-ref.7
> qemu-ga-ref.7.pod
> qemu-ga-ref.html
> qemu-ga-ref.txt
> qemu-qmp-qapi.texi
> qemu-qmp-ref.7
> qemu-qmp-ref.7.pod
> qemu-qmp-ref.html
> qemu-qmp-ref.txt
> 
> because these have been built into build/docs/interop along
> with the Sphinx interop documents. Filter them out of the
> set of files we install when we're installing the Sphinx-built
> manual files. (They are installed into their correct locations
> as part of the main install-doc target already.)
> 
> Fixes: 5f71eac06e15b9a3fa1134d446f ("Makefile, configure: Support building rST documentation")
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> ---
>  Makefile | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 9fd7f3f9bfb..6ccb8639b08 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -727,9 +727,11 @@ else
>  BLOBS=
>  endif
>  
> +# Note that we manually filter-out the non-Sphinx documentation which
> +# is currently built into the docs/interop directory in the build tree.
>  define install-manual =
>  for d in $$(cd $(MANUAL_BUILDDIR) && find $1 -type d); do $(INSTALL_DIR) "$(DESTDIR)$(qemu_docdir)/$$d"; done
> -for f in $$(cd $(MANUAL_BUILDDIR) && find $1 -type f); do $(INSTALL_DATA) "$(MANUAL_BUILDDIR)/$$f" "$(DESTDIR)$(qemu_docdir)/$$f"; done
> +for f in $$(cd $(MANUAL_BUILDDIR) && find $1 -type f -a '!' '(' -name 'qemu-*-qapi.*' -o -name 'qemu-*-ref.*' ')' ); do $(INSTALL_DATA) "$(MANUAL_BUILDDIR)/$$f" "$(DESTDIR)$(qemu_docdir)/$$f"; done
>  endef
>  
>  # Note that we deliberately do not install the "devel" manual: it is
> 

Re: [Qemu-devel] [PATCH 0/3] Fix various issues with Sphinx build machinery

[Peter Maydell]

On Fri, 8 Mar 2019 at 13:57, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> This patchset fixes some bugs with the Sphinx build machinery.
> Most importantly, it fixes in-tree builds, which were broken
> if you had sphinx-build available. The other two patches fix
> some more minor problems that I noticed in the course of
> fixing the first one.
>
> thanks
> -- PMM
>
> Peter Maydell (3):
>   Makefile: Fix Sphinx documentation builds for in-tree builds
>   Makefile: Fix 'make distclean'
>   Makefile: Don't install non-sphinx files in sphinx docs install

Applied to master as a buildfix, thanks. (Hopefully this will
fix the Travis builds.)

-- PMM