[PATCH] meson: Pass -j option to sphinx

[PATCH] meson: Pass -j option to sphinx

[Fabiano Rosas]

Save a bit of build time by passing the number of jobs option to
sphinx.

To avoid warnings from sphinx, alter our plugins to inform whether
they support parallelism. The two plugins touched are quite simple and
I don't see anything that would indicate they do not support being
called in parallel, so return True for both reads and writes.

before:
 $ time make -j16 man html
 ...
 [1/2] Generating docs/QEMU manual with a custom command
 [2/2] Generating docs/QEMU man pages with a custom command

 real    0m49.770s
 user    0m49.425s
 sys     0m0.716s

after:
 $ time make -j16 man html
 ...
 [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
 [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
 real    0m30.153s
 user    1m5.945s
 sys     0m2.440s

Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
 docs/meson.build           | 12 ++++++++++++
 docs/sphinx/fakedbusdoc.py |  5 +++++
 docs/sphinx/qmp_lexer.py   |  5 +++++
 3 files changed, 22 insertions(+)

diff --git a/docs/meson.build b/docs/meson.build
index f220800e3e..9e4bed6fa0 100644
--- a/docs/meson.build
+++ b/docs/meson.build
@@ -10,6 +10,18 @@ if sphinx_build.found()
     SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
   endif
 
+  sphinx_version = run_command(SPHINX_ARGS + ['--version'],
+                               check: false).stdout().split()[1]
+  if sphinx_version.version_compare('>=5.1.2')
+    SPHINX_ARGS += ['-j', 'auto']
+  else
+    nproc = find_program('nproc')
+    if nproc.found()
+      jobs = run_command(nproc, check:false).stdout()
+      SPHINX_ARGS += ['-j', jobs]
+    endif
+  endif
+
   # This is a bit awkward but works: create a trivial document and
   # try to run it with our configuration file (which enforces a
   # version requirement). This will fail if sphinx-build is too old.
diff --git a/docs/sphinx/fakedbusdoc.py b/docs/sphinx/fakedbusdoc.py
index d2c5079046..2d2e6ef640 100644
--- a/docs/sphinx/fakedbusdoc.py
+++ b/docs/sphinx/fakedbusdoc.py
@@ -23,3 +23,8 @@ def run(self):
 def setup(app: Sphinx) -> Dict[str, Any]:
     """Register a fake dbus-doc directive with Sphinx"""
     app.add_directive("dbus-doc", FakeDBusDocDirective)
+
+    return dict(
+        parallel_read_safe = True,
+        parallel_write_safe = True
+    )
diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py
index f7e4c0e198..a59de8a079 100644
--- a/docs/sphinx/qmp_lexer.py
+++ b/docs/sphinx/qmp_lexer.py
@@ -41,3 +41,8 @@ def setup(sphinx):
         sphinx.add_lexer('QMP', QMPExampleLexer)
     except errors.VersionRequirementError:
         sphinx.add_lexer('QMP', QMPExampleLexer())
+
+    return dict(
+        parallel_read_safe = True,
+        parallel_write_safe = True
+    )
-- 
2.35.3

Re: [PATCH] meson: Pass -j option to sphinx

[Daniel P. Berrangé]

On Wed, Apr 26, 2023 at 01:03:46PM -0300, Fabiano Rosas wrote:
> Save a bit of build time by passing the number of jobs option to
> sphinx.
> 
> To avoid warnings from sphinx, alter our plugins to inform whether
> they support parallelism. The two plugins touched are quite simple and
> I don't see anything that would indicate they do not support being
> called in parallel, so return True for both reads and writes.
> 
> before:
>  $ time make -j16 man html
>  ...
>  [1/2] Generating docs/QEMU manual with a custom command
>  [2/2] Generating docs/QEMU man pages with a custom command
> 
>  real    0m49.770s
>  user    0m49.425s
>  sys     0m0.716s
> 
> after:
>  $ time make -j16 man html
>  ...
>  [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
>  [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
>  real    0m30.153s
>  user    1m5.945s
>  sys     0m2.440s

When I test this on Fedora 38, the docs build hangs forever.

I ran sphinx directly and see it prints a traceback and then
fails to exit after this error

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 105, in join
    if not self._join_one():
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 129, in _join_one
    self._result_funcs.pop(tid)(self._args.pop(tid), result)
  File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 478, in merge
    self.env.merge_info_from(docs, env, self.app)
  File "/usr/lib/python3.11/site-packages/sphinx/environment/__init__.py", line 366, in merge_info_from
    domain.merge_domaindata(docnames, other.domaindata[domainname])
  File "/usr/lib/python3.11/site-packages/sphinx/domains/__init__.py", line 295, in merge_domaindata
    raise NotImplementedError('merge_domaindata must be implemented in %s '
NotImplementedError: merge_domaindata must be implemented in <class 'dbusdomain.DBusDomain'> to be able to do parallel builds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/sphinx/cmd/build.py", line 281, in build_main
    app.build(args.force_all, args.filenames)
  File "/usr/lib/python3.11/site-packages/sphinx/application.py", line 347, in build
    self.builder.build_update()
  File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 310, in build_update
    self.build(to_build,


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] meson: Pass -j option to sphinx

[Thomas Huth]

On 26/04/2023 18.03, Fabiano Rosas wrote:
> Save a bit of build time by passing the number of jobs option to
> sphinx.
> 
> To avoid warnings from sphinx, alter our plugins to inform whether
> they support parallelism. The two plugins touched are quite simple and
> I don't see anything that would indicate they do not support being
> called in parallel, so return True for both reads and writes.
> 
> before:
>   $ time make -j16 man html
>   ...
>   [1/2] Generating docs/QEMU manual with a custom command
>   [2/2] Generating docs/QEMU man pages with a custom command
> 
>   real    0m49.770s
>   user    0m49.425s
>   sys     0m0.716s
> 
> after:
>   $ time make -j16 man html
>   ...
>   [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
>   [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
>   real    0m30.153s
>   user    1m5.945s
>   sys     0m2.440s
> 
> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
>   docs/meson.build           | 12 ++++++++++++
>   docs/sphinx/fakedbusdoc.py |  5 +++++
>   docs/sphinx/qmp_lexer.py   |  5 +++++
>   3 files changed, 22 insertions(+)
> 
> diff --git a/docs/meson.build b/docs/meson.build
> index f220800e3e..9e4bed6fa0 100644
> --- a/docs/meson.build
> +++ b/docs/meson.build
> @@ -10,6 +10,18 @@ if sphinx_build.found()
>       SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
>     endif
>   
> +  sphinx_version = run_command(SPHINX_ARGS + ['--version'],
> +                               check: false).stdout().split()[1]
> +  if sphinx_version.version_compare('>=5.1.2')
> +    SPHINX_ARGS += ['-j', 'auto']
> +  else
> +    nproc = find_program('nproc')
> +    if nproc.found()
> +      jobs = run_command(nproc, check:false).stdout()
> +      SPHINX_ARGS += ['-j', jobs]
> +    endif
> +  endif

Wouldn't it be better to use the value from "make -jXX" instead of always 
running with the maximum number of processors here?

  Thomas

Re: [PATCH] meson: Pass -j option to sphinx

[Daniel P. Berrangé]

On Thu, Apr 27, 2023 at 02:24:59PM +0200, Thomas Huth wrote:
> On 26/04/2023 18.03, Fabiano Rosas wrote:
> > Save a bit of build time by passing the number of jobs option to
> > sphinx.
> > 
> > To avoid warnings from sphinx, alter our plugins to inform whether
> > they support parallelism. The two plugins touched are quite simple and
> > I don't see anything that would indicate they do not support being
> > called in parallel, so return True for both reads and writes.
> > 
> > before:
> >   $ time make -j16 man html
> >   ...
> >   [1/2] Generating docs/QEMU manual with a custom command
> >   [2/2] Generating docs/QEMU man pages with a custom command
> > 
> >   real    0m49.770s
> >   user    0m49.425s
> >   sys     0m0.716s
> > 
> > after:
> >   $ time make -j16 man html
> >   ...
> >   [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
> >   [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
> >   real    0m30.153s
> >   user    1m5.945s
> >   sys     0m2.440s
> > 
> > Signed-off-by: Fabiano Rosas <farosas@suse.de>
> > ---
> >   docs/meson.build           | 12 ++++++++++++
> >   docs/sphinx/fakedbusdoc.py |  5 +++++
> >   docs/sphinx/qmp_lexer.py   |  5 +++++
> >   3 files changed, 22 insertions(+)
> > 
> > diff --git a/docs/meson.build b/docs/meson.build
> > index f220800e3e..9e4bed6fa0 100644
> > --- a/docs/meson.build
> > +++ b/docs/meson.build
> > @@ -10,6 +10,18 @@ if sphinx_build.found()
> >       SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
> >     endif
> > +  sphinx_version = run_command(SPHINX_ARGS + ['--version'],
> > +                               check: false).stdout().split()[1]
> > +  if sphinx_version.version_compare('>=5.1.2')
> > +    SPHINX_ARGS += ['-j', 'auto']
> > +  else
> > +    nproc = find_program('nproc')
> > +    if nproc.found()
> > +      jobs = run_command(nproc, check:false).stdout()
> > +      SPHINX_ARGS += ['-j', jobs]
> > +    endif
> > +  endif
> 
> Wouldn't it be better to use the value from "make -jXX" instead of always
> running with the maximum number of processors here?

IIUC, this is not exposed by meson.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] meson: Pass -j option to sphinx

[Fabiano Rosas]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Wed, Apr 26, 2023 at 01:03:46PM -0300, Fabiano Rosas wrote:
>> Save a bit of build time by passing the number of jobs option to
>> sphinx.
>> 
>> To avoid warnings from sphinx, alter our plugins to inform whether
>> they support parallelism. The two plugins touched are quite simple and
>> I don't see anything that would indicate they do not support being
>> called in parallel, so return True for both reads and writes.
>> 
>> before:
>>  $ time make -j16 man html
>>  ...
>>  [1/2] Generating docs/QEMU manual with a custom command
>>  [2/2] Generating docs/QEMU man pages with a custom command
>> 
>>  real    0m49.770s
>>  user    0m49.425s
>>  sys     0m0.716s
>> 
>> after:
>>  $ time make -j16 man html
>>  ...
>>  [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
>>  [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
>>  real    0m30.153s
>>  user    1m5.945s
>>  sys     0m2.440s
>
> When I test this on Fedora 38, the docs build hangs forever.
>
> I ran sphinx directly and see it prints a traceback and then
> fails to exit after this error
>
> Traceback (most recent call last):
>   File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 105, in join
>     if not self._join_one():
>            ^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 129, in _join_one
>     self._result_funcs.pop(tid)(self._args.pop(tid), result)
>   File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 478, in merge
>     self.env.merge_info_from(docs, env, self.app)
>   File "/usr/lib/python3.11/site-packages/sphinx/environment/__init__.py", line 366, in merge_info_from
>     domain.merge_domaindata(docnames, other.domaindata[domainname])
>   File "/usr/lib/python3.11/site-packages/sphinx/domains/__init__.py", line 295, in merge_domaindata
>     raise NotImplementedError('merge_domaindata must be implemented in %s '
> NotImplementedError: merge_domaindata must be implemented in <class 'dbusdomain.DBusDomain'> to be able to do parallel builds!
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File "/usr/lib/python3.11/site-packages/sphinx/cmd/build.py", line 281, in build_main
>     app.build(args.force_all, args.filenames)
>   File "/usr/lib/python3.11/site-packages/sphinx/application.py", line 347, in build
>     self.builder.build_update()
>   File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 310, in build_update
>     self.build(to_build,
>

Ah it seems the DBus plugin is only in effect when using sphinx > 4.
I'll try to reproduce and see what it takes to implement this
merge_domaindata. Thanks for testing.