[PATCH] docs: Use make invocation's -j argument for parallelism

[PATCH] docs: Use make invocation's -j argument for parallelism

[Kees Cook]

While sphinx 1.7 and later supports "-jauto" for parallelism, this
effectively ignores the "-j" flag used in the "make" invocation, which
may cause confusion for build systems. Instead, extract the available
parallelism from "make"'s job server (since it is not exposed in any
special variables) and use that for the "sphinx-build" run. Now things
work correctly for builds where -j is specified at the top-level:

	make -j16 htmldocs

Signed-off-by: Kees Cook <keescook@chromium.org>
---
 Documentation/Makefile  |  3 +--
 scripts/jobserver-count | 46 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 47 insertions(+), 2 deletions(-)
 create mode 100755 scripts/jobserver-count

diff --git a/Documentation/Makefile b/Documentation/Makefile
index e145e4db508b..4408eeaf2891 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -33,8 +33,6 @@ ifeq ($(HAVE_SPHINX),0)
 
 else # HAVE_SPHINX
 
-export SPHINXOPTS = $(shell perl -e 'open IN,"sphinx-build --version 2>&1 |"; while (<IN>) { if (m/([\d\.]+)/) { print "-jauto" if ($$1 >= "1.7") } ;} close IN')
-
 # User-friendly check for pdflatex and latexmk
 HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
 HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi)
@@ -68,6 +66,7 @@ quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
 	PYTHONDONTWRITEBYTECODE=1 \
 	BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(srctree)/$(src)/$5/$(SPHINX_CONF)) \
 	$(SPHINXBUILD) \
+	-j $(shell $(srctree)/scripts/jobserver-count) \
 	-b $2 \
 	-c $(abspath $(srctree)/$(src)) \
 	-d $(abspath $(BUILDDIR)/.doctrees/$3) \
diff --git a/scripts/jobserver-count b/scripts/jobserver-count
new file mode 100755
index 000000000000..5fc9d2fd5254
--- /dev/null
+++ b/scripts/jobserver-count
@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# This determines how many parallel tasks "make" is expecting, as it is
+# not exposed via an special variables.
+# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
+import os, sys, fcntl
+
+# Set non-blocking for a given file descriptor.
+def nonblock(fd):
+	flags = fcntl.fcntl(fd, fcntl.F_GETFL)
+	fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
+	return fd
+
+# Fetch the make environment options.
+flags = os.environ.get('MAKEFLAGS', None)
+if flags == None:
+	print("1")
+	sys.exit(0)
+
+# Look for "--jobserver=R,W"
+opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
+if len(opts) != 1:
+	print("1")
+	sys.exit(0)
+
+# Parse out R,W file descriptor numbers and set them nonblocking.
+fds = opts[0].split("=", 1)[1]
+reader, writer = [nonblock(int(x)) for x in fds.split(",", 1)]
+
+# Read out as many jobserver slots as possible.
+jobs = b""
+while True:
+	try:
+		slot = os.read(reader, 1)
+		jobs += slot
+	except:
+		break
+# Return all the reserved slots.
+os.write(writer, jobs)
+
+# Report available slots (with a bump for our caller's reserveration).
+if len(jobs) < 1:
+	print("1")
+else:
+	print(len(jobs) + 1)
-- 
2.17.1


-- 
Kees Cook

Re: [PATCH] docs: Use make invocation's -j argument for parallelism

[Mauro Carvalho Chehab]

Em Thu, 19 Sep 2019 11:18:52 -0700
Kees Cook <keescook@chromium.org> escreveu:

> While sphinx 1.7 and later supports "-jauto" for parallelism, this
> effectively ignores the "-j" flag used in the "make" invocation, which
> may cause confusion for build systems. Instead, extract the available
> parallelism from "make"'s job server (since it is not exposed in any
> special variables) and use that for the "sphinx-build" run. Now things
> work correctly for builds where -j is specified at the top-level:
> 
> 	make -j16 htmldocs
> 
> Signed-off-by: Kees Cook <keescook@chromium.org>
> ---
>  Documentation/Makefile  |  3 +--
>  scripts/jobserver-count | 46 +++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 47 insertions(+), 2 deletions(-)
>  create mode 100755 scripts/jobserver-count
> 
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index e145e4db508b..4408eeaf2891 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -33,8 +33,6 @@ ifeq ($(HAVE_SPHINX),0)
>  
>  else # HAVE_SPHINX
>  
> -export SPHINXOPTS = $(shell perl -e 'open IN,"sphinx-build --version 2>&1 |"; while (<IN>) { if (m/([\d\.]+)/) { print "-jauto" if ($$1 >= "1.7") } ;} close IN')
> -
>  # User-friendly check for pdflatex and latexmk
>  HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
>  HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi)
> @@ -68,6 +66,7 @@ quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
>  	PYTHONDONTWRITEBYTECODE=1 \
>  	BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(srctree)/$(src)/$5/$(SPHINX_CONF)) \
>  	$(SPHINXBUILD) \
> +	-j $(shell $(srctree)/scripts/jobserver-count) \
>  	-b $2 \
>  	-c $(abspath $(srctree)/$(src)) \
>  	-d $(abspath $(BUILDDIR)/.doctrees/$3) \
> diff --git a/scripts/jobserver-count b/scripts/jobserver-count
> new file mode 100755
> index 000000000000..5fc9d2fd5254
> --- /dev/null
> +++ b/scripts/jobserver-count
> @@ -0,0 +1,46 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# This determines how many parallel tasks "make" is expecting, as it is
> +# not exposed via an special variables.
> +# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
> +import os, sys, fcntl
> +
> +# Set non-blocking for a given file descriptor.
> +def nonblock(fd):
> +	flags = fcntl.fcntl(fd, fcntl.F_GETFL)
> +	fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
> +	return fd
> +


> +# Fetch the make environment options.
> +flags = os.environ.get('MAKEFLAGS', None)
> +if flags == None:
> +	print("1")
> +	sys.exit(0)
> +
> +# Look for "--jobserver=R,W"
> +opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
> +if len(opts) != 1:
> +	print("1")
> +	sys.exit(0)

Using "1" as default if -j is not specified doesn't sound a good idea. I 
mean, Sphinx is very slow without any parallelism. I would keep "auto" as
default here, if version >= 1.7 (and if there's no explicit SPHINXOPTS).


> +
> +# Parse out R,W file descriptor numbers and set them nonblocking.
> +fds = opts[0].split("=", 1)[1]
> +reader, writer = [nonblock(int(x)) for x in fds.split(",", 1)]
> +
> +# Read out as many jobserver slots as possible.
> +jobs = b""
> +while True:
> +	try:
> +		slot = os.read(reader, 1)
> +		jobs += slot
> +	except:
> +		break
> +# Return all the reserved slots.
> +os.write(writer, jobs)
> +
> +# Report available slots (with a bump for our caller's reserveration).
> +if len(jobs) < 1:
> +	print("1")
> +else:
> +	print(len(jobs) + 1)



Thanks,
Mauro

Re: [PATCH] docs: Use make invocation's -j argument for parallelism

[Kees Cook]

On Thu, Sep 19, 2019 at 04:25:49PM -0300, Mauro Carvalho Chehab wrote:
> Em Thu, 19 Sep 2019 11:18:52 -0700
> Kees Cook <keescook@chromium.org> escreveu:
> > [...]
> > +# Fetch the make environment options.
> > +flags = os.environ.get('MAKEFLAGS', None)
> > +if flags == None:
> > +	print("1")
> > +	sys.exit(0)
> > +
> > +# Look for "--jobserver=R,W"
> > +opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
> > +if len(opts) != 1:
> > +	print("1")
> > +	sys.exit(0)
> 
> Using "1" as default if -j is not specified doesn't sound a good idea. I
> mean, Sphinx is very slow without any parallelism. I would keep "auto" as
> default here, if version >= 1.7 (and if there's no explicit SPHINXOPTS).

Okay, that's fair. This does introduce a behavioral regression from that
perspective. Let me spin a v2 that'll DTRT.

-- 
Kees Cook