[PATCH] sphinx: Use separate doctree directories for different builders

[PATCH] sphinx: Use separate doctree directories for different builders

[Eduardo Habkost]

sphinx-build is buggy when multiple processes are using the same
doctree directory in parallel.  See the 3-year-old Sphinx bug
report at: https://github.com/sphinx-doc/sphinx/issues/2946

Instead of avoiding parallel builds or adding some kind of
locking, I'm using the simplest solution: just using a different
doctree cache for each builder.

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 Makefile | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 30f0abfb42..d20e7ffce3 100644
--- a/Makefile
+++ b/Makefile
@@ -983,7 +983,10 @@ sphinxdocs: $(MANUAL_BUILDDIR)/devel/index.html $(MANUAL_BUILDDIR)/interop/index
 
 # Canned command to build a single manual
 # Arguments: $1 = manual name, $2 = Sphinx builder ('html' or 'man')
-build-manual = $(call quiet-command,CONFDIR="$(qemu_confdir)" sphinx-build $(if $(V),,-q) -W -n -b $2 -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
+# Note the use of different doctree for each (manual, builder) tuple;
+# this works around Sphinx not handling parallel invocation on
+# a single doctree: https://github.com/sphinx-doc/sphinx/issues/2946
+build-manual = $(call quiet-command,CONFDIR="$(qemu_confdir)" sphinx-build $(if $(V),,-q) -W -n -b $2 -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1-$2 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
 # We assume all RST files in the manual's directory are used in it
 manual-deps = $(wildcard $(SRC_PATH)/docs/$1/*.rst) $(SRC_PATH)/docs/$1/conf.py $(SRC_PATH)/docs/conf.py
 
-- 
2.21.0

Re: [PATCH] sphinx: Use separate doctree directories for different builders

[John Snow]

On 10/14/19 11:01 AM, Eduardo Habkost wrote:
> sphinx-build is buggy when multiple processes are using the same
> doctree directory in parallel.  See the 3-year-old Sphinx bug
> report at: https://github.com/sphinx-doc/sphinx/issues/2946
> 
> Instead of avoiding parallel builds or adding some kind of
> locking, I'm using the simplest solution: just using a different
> doctree cache for each builder.
> 
> Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

Reviewed-by: John Snow <jsnow@redhat.com>

> ---
>  Makefile | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 30f0abfb42..d20e7ffce3 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -983,7 +983,10 @@ sphinxdocs: $(MANUAL_BUILDDIR)/devel/index.html $(MANUAL_BUILDDIR)/interop/index
>  
>  # Canned command to build a single manual
>  # Arguments: $1 = manual name, $2 = Sphinx builder ('html' or 'man')
> -build-manual = $(call quiet-command,CONFDIR="$(qemu_confdir)" sphinx-build $(if $(V),,-q) -W -n -b $2 -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
> +# Note the use of different doctree for each (manual, builder) tuple;
> +# this works around Sphinx not handling parallel invocation on
> +# a single doctree: https://github.com/sphinx-doc/sphinx/issues/2946
> +build-manual = $(call quiet-command,CONFDIR="$(qemu_confdir)" sphinx-build $(if $(V),,-q) -W -n -b $2 -D version=$(VERSION) -D release="$(FULL_VERSION)" -d .doctrees/$1-$2 $(SRC_PATH)/docs/$1 $(MANUAL_BUILDDIR)/$1 ,"SPHINX","$(MANUAL_BUILDDIR)/$1")
>  # We assume all RST files in the manual's directory are used in it
>  manual-deps = $(wildcard $(SRC_PATH)/docs/$1/*.rst) $(SRC_PATH)/docs/$1/conf.py $(SRC_PATH)/docs/conf.py
>  
> 

Thanks for this!

Re: [PATCH] sphinx: Use separate doctree directories for different builders

[Peter Maydell]

On Mon, 14 Oct 2019 at 16:01, Eduardo Habkost <ehabkost@redhat.com> wrote:
>
> sphinx-build is buggy when multiple processes are using the same
> doctree directory in parallel.  See the 3-year-old Sphinx bug
> report at: https://github.com/sphinx-doc/sphinx/issues/2946
>
> Instead of avoiding parallel builds or adding some kind of
> locking, I'm using the simplest solution: just using a different
> doctree cache for each builder.
>
> Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>  Makefile | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)


Applied to master as a buildfix, thanks.

-- PMM