* [PATCH] meson: Pass -j option to sphinx
@ 2023-04-26 16:03 Fabiano Rosas
2023-04-27 11:58 ` Daniel P. Berrangé
2023-04-27 12:24 ` Thomas Huth
0 siblings, 2 replies; 5+ messages in thread
From: Fabiano Rosas @ 2023-04-26 16:03 UTC (permalink / raw)
To: qemu-devel
Cc: Daniel P . Berrangé, Paolo Bonzini, Marc-André Lureau,
Peter Maydell
Save a bit of build time by passing the number of jobs option to
sphinx.
To avoid warnings from sphinx, alter our plugins to inform whether
they support parallelism. The two plugins touched are quite simple and
I don't see anything that would indicate they do not support being
called in parallel, so return True for both reads and writes.
before:
$ time make -j16 man html
...
[1/2] Generating docs/QEMU manual with a custom command
[2/2] Generating docs/QEMU man pages with a custom command
real 0m49.770s
user 0m49.425s
sys 0m0.716s
after:
$ time make -j16 man html
...
[1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
[2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
real 0m30.153s
user 1m5.945s
sys 0m2.440s
Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
docs/meson.build | 12 ++++++++++++
docs/sphinx/fakedbusdoc.py | 5 +++++
docs/sphinx/qmp_lexer.py | 5 +++++
3 files changed, 22 insertions(+)
diff --git a/docs/meson.build b/docs/meson.build
index f220800e3e..9e4bed6fa0 100644
--- a/docs/meson.build
+++ b/docs/meson.build
@@ -10,6 +10,18 @@ if sphinx_build.found()
SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
endif
+ sphinx_version = run_command(SPHINX_ARGS + ['--version'],
+ check: false).stdout().split()[1]
+ if sphinx_version.version_compare('>=5.1.2')
+ SPHINX_ARGS += ['-j', 'auto']
+ else
+ nproc = find_program('nproc')
+ if nproc.found()
+ jobs = run_command(nproc, check:false).stdout()
+ SPHINX_ARGS += ['-j', jobs]
+ endif
+ endif
+
# This is a bit awkward but works: create a trivial document and
# try to run it with our configuration file (which enforces a
# version requirement). This will fail if sphinx-build is too old.
diff --git a/docs/sphinx/fakedbusdoc.py b/docs/sphinx/fakedbusdoc.py
index d2c5079046..2d2e6ef640 100644
--- a/docs/sphinx/fakedbusdoc.py
+++ b/docs/sphinx/fakedbusdoc.py
@@ -23,3 +23,8 @@ def run(self):
def setup(app: Sphinx) -> Dict[str, Any]:
"""Register a fake dbus-doc directive with Sphinx"""
app.add_directive("dbus-doc", FakeDBusDocDirective)
+
+ return dict(
+ parallel_read_safe = True,
+ parallel_write_safe = True
+ )
diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py
index f7e4c0e198..a59de8a079 100644
--- a/docs/sphinx/qmp_lexer.py
+++ b/docs/sphinx/qmp_lexer.py
@@ -41,3 +41,8 @@ def setup(sphinx):
sphinx.add_lexer('QMP', QMPExampleLexer)
except errors.VersionRequirementError:
sphinx.add_lexer('QMP', QMPExampleLexer())
+
+ return dict(
+ parallel_read_safe = True,
+ parallel_write_safe = True
+ )
--
2.35.3
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] meson: Pass -j option to sphinx
2023-04-26 16:03 [PATCH] meson: Pass -j option to sphinx Fabiano Rosas
@ 2023-04-27 11:58 ` Daniel P. Berrangé
2023-04-27 13:12 ` Fabiano Rosas
2023-04-27 12:24 ` Thomas Huth
1 sibling, 1 reply; 5+ messages in thread
From: Daniel P. Berrangé @ 2023-04-27 11:58 UTC (permalink / raw)
To: Fabiano Rosas
Cc: qemu-devel, Paolo Bonzini, Marc-André Lureau, Peter Maydell
On Wed, Apr 26, 2023 at 01:03:46PM -0300, Fabiano Rosas wrote:
> Save a bit of build time by passing the number of jobs option to
> sphinx.
>
> To avoid warnings from sphinx, alter our plugins to inform whether
> they support parallelism. The two plugins touched are quite simple and
> I don't see anything that would indicate they do not support being
> called in parallel, so return True for both reads and writes.
>
> before:
> $ time make -j16 man html
> ...
> [1/2] Generating docs/QEMU manual with a custom command
> [2/2] Generating docs/QEMU man pages with a custom command
>
> real 0m49.770s
> user 0m49.425s
> sys 0m0.716s
>
> after:
> $ time make -j16 man html
> ...
> [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
> [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
> real 0m30.153s
> user 1m5.945s
> sys 0m2.440s
When I test this on Fedora 38, the docs build hangs forever.
I ran sphinx directly and see it prints a traceback and then
fails to exit after this error
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 105, in join
if not self._join_one():
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 129, in _join_one
self._result_funcs.pop(tid)(self._args.pop(tid), result)
File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 478, in merge
self.env.merge_info_from(docs, env, self.app)
File "/usr/lib/python3.11/site-packages/sphinx/environment/__init__.py", line 366, in merge_info_from
domain.merge_domaindata(docnames, other.domaindata[domainname])
File "/usr/lib/python3.11/site-packages/sphinx/domains/__init__.py", line 295, in merge_domaindata
raise NotImplementedError('merge_domaindata must be implemented in %s '
NotImplementedError: merge_domaindata must be implemented in <class 'dbusdomain.DBusDomain'> to be able to do parallel builds!
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/sphinx/cmd/build.py", line 281, in build_main
app.build(args.force_all, args.filenames)
File "/usr/lib/python3.11/site-packages/sphinx/application.py", line 347, in build
self.builder.build_update()
File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 310, in build_update
self.build(to_build,
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] meson: Pass -j option to sphinx
2023-04-27 11:58 ` Daniel P. Berrangé
@ 2023-04-27 13:12 ` Fabiano Rosas
0 siblings, 0 replies; 5+ messages in thread
From: Fabiano Rosas @ 2023-04-27 13:12 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, Paolo Bonzini, Marc-André Lureau, Peter Maydell
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Wed, Apr 26, 2023 at 01:03:46PM -0300, Fabiano Rosas wrote:
>> Save a bit of build time by passing the number of jobs option to
>> sphinx.
>>
>> To avoid warnings from sphinx, alter our plugins to inform whether
>> they support parallelism. The two plugins touched are quite simple and
>> I don't see anything that would indicate they do not support being
>> called in parallel, so return True for both reads and writes.
>>
>> before:
>> $ time make -j16 man html
>> ...
>> [1/2] Generating docs/QEMU manual with a custom command
>> [2/2] Generating docs/QEMU man pages with a custom command
>>
>> real 0m49.770s
>> user 0m49.425s
>> sys 0m0.716s
>>
>> after:
>> $ time make -j16 man html
>> ...
>> [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
>> [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
>> real 0m30.153s
>> user 1m5.945s
>> sys 0m2.440s
>
> When I test this on Fedora 38, the docs build hangs forever.
>
> I ran sphinx directly and see it prints a traceback and then
> fails to exit after this error
>
> Traceback (most recent call last):
> File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 105, in join
> if not self._join_one():
> ^^^^^^^^^^^^^^^^
> File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 129, in _join_one
> self._result_funcs.pop(tid)(self._args.pop(tid), result)
> File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 478, in merge
> self.env.merge_info_from(docs, env, self.app)
> File "/usr/lib/python3.11/site-packages/sphinx/environment/__init__.py", line 366, in merge_info_from
> domain.merge_domaindata(docnames, other.domaindata[domainname])
> File "/usr/lib/python3.11/site-packages/sphinx/domains/__init__.py", line 295, in merge_domaindata
> raise NotImplementedError('merge_domaindata must be implemented in %s '
> NotImplementedError: merge_domaindata must be implemented in <class 'dbusdomain.DBusDomain'> to be able to do parallel builds!
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "/usr/lib/python3.11/site-packages/sphinx/cmd/build.py", line 281, in build_main
> app.build(args.force_all, args.filenames)
> File "/usr/lib/python3.11/site-packages/sphinx/application.py", line 347, in build
> self.builder.build_update()
> File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 310, in build_update
> self.build(to_build,
>
Ah it seems the DBus plugin is only in effect when using sphinx > 4.
I'll try to reproduce and see what it takes to implement this
merge_domaindata. Thanks for testing.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] meson: Pass -j option to sphinx
2023-04-26 16:03 [PATCH] meson: Pass -j option to sphinx Fabiano Rosas
2023-04-27 11:58 ` Daniel P. Berrangé
@ 2023-04-27 12:24 ` Thomas Huth
2023-04-27 12:29 ` Daniel P. Berrangé
1 sibling, 1 reply; 5+ messages in thread
From: Thomas Huth @ 2023-04-27 12:24 UTC (permalink / raw)
To: Fabiano Rosas, qemu-devel
Cc: Daniel P . Berrangé, Paolo Bonzini, Marc-André Lureau,
Peter Maydell
On 26/04/2023 18.03, Fabiano Rosas wrote:
> Save a bit of build time by passing the number of jobs option to
> sphinx.
>
> To avoid warnings from sphinx, alter our plugins to inform whether
> they support parallelism. The two plugins touched are quite simple and
> I don't see anything that would indicate they do not support being
> called in parallel, so return True for both reads and writes.
>
> before:
> $ time make -j16 man html
> ...
> [1/2] Generating docs/QEMU manual with a custom command
> [2/2] Generating docs/QEMU man pages with a custom command
>
> real 0m49.770s
> user 0m49.425s
> sys 0m0.716s
>
> after:
> $ time make -j16 man html
> ...
> [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
> [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
> real 0m30.153s
> user 1m5.945s
> sys 0m2.440s
>
> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
> docs/meson.build | 12 ++++++++++++
> docs/sphinx/fakedbusdoc.py | 5 +++++
> docs/sphinx/qmp_lexer.py | 5 +++++
> 3 files changed, 22 insertions(+)
>
> diff --git a/docs/meson.build b/docs/meson.build
> index f220800e3e..9e4bed6fa0 100644
> --- a/docs/meson.build
> +++ b/docs/meson.build
> @@ -10,6 +10,18 @@ if sphinx_build.found()
> SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
> endif
>
> + sphinx_version = run_command(SPHINX_ARGS + ['--version'],
> + check: false).stdout().split()[1]
> + if sphinx_version.version_compare('>=5.1.2')
> + SPHINX_ARGS += ['-j', 'auto']
> + else
> + nproc = find_program('nproc')
> + if nproc.found()
> + jobs = run_command(nproc, check:false).stdout()
> + SPHINX_ARGS += ['-j', jobs]
> + endif
> + endif
Wouldn't it be better to use the value from "make -jXX" instead of always
running with the maximum number of processors here?
Thomas
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] meson: Pass -j option to sphinx
2023-04-27 12:24 ` Thomas Huth
@ 2023-04-27 12:29 ` Daniel P. Berrangé
0 siblings, 0 replies; 5+ messages in thread
From: Daniel P. Berrangé @ 2023-04-27 12:29 UTC (permalink / raw)
To: Thomas Huth
Cc: Fabiano Rosas, qemu-devel, Paolo Bonzini, Marc-André Lureau,
Peter Maydell
On Thu, Apr 27, 2023 at 02:24:59PM +0200, Thomas Huth wrote:
> On 26/04/2023 18.03, Fabiano Rosas wrote:
> > Save a bit of build time by passing the number of jobs option to
> > sphinx.
> >
> > To avoid warnings from sphinx, alter our plugins to inform whether
> > they support parallelism. The two plugins touched are quite simple and
> > I don't see anything that would indicate they do not support being
> > called in parallel, so return True for both reads and writes.
> >
> > before:
> > $ time make -j16 man html
> > ...
> > [1/2] Generating docs/QEMU manual with a custom command
> > [2/2] Generating docs/QEMU man pages with a custom command
> >
> > real 0m49.770s
> > user 0m49.425s
> > sys 0m0.716s
> >
> > after:
> > $ time make -j16 man html
> > ...
> > [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
> > [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
> > real 0m30.153s
> > user 1m5.945s
> > sys 0m2.440s
> >
> > Signed-off-by: Fabiano Rosas <farosas@suse.de>
> > ---
> > docs/meson.build | 12 ++++++++++++
> > docs/sphinx/fakedbusdoc.py | 5 +++++
> > docs/sphinx/qmp_lexer.py | 5 +++++
> > 3 files changed, 22 insertions(+)
> >
> > diff --git a/docs/meson.build b/docs/meson.build
> > index f220800e3e..9e4bed6fa0 100644
> > --- a/docs/meson.build
> > +++ b/docs/meson.build
> > @@ -10,6 +10,18 @@ if sphinx_build.found()
> > SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
> > endif
> > + sphinx_version = run_command(SPHINX_ARGS + ['--version'],
> > + check: false).stdout().split()[1]
> > + if sphinx_version.version_compare('>=5.1.2')
> > + SPHINX_ARGS += ['-j', 'auto']
> > + else
> > + nproc = find_program('nproc')
> > + if nproc.found()
> > + jobs = run_command(nproc, check:false).stdout()
> > + SPHINX_ARGS += ['-j', jobs]
> > + endif
> > + endif
>
> Wouldn't it be better to use the value from "make -jXX" instead of always
> running with the maximum number of processors here?
IIUC, this is not exposed by meson.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-04-27 13:12 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-04-26 16:03 [PATCH] meson: Pass -j option to sphinx Fabiano Rosas
2023-04-27 11:58 ` Daniel P. Berrangé
2023-04-27 13:12 ` Fabiano Rosas
2023-04-27 12:24 ` Thomas Huth
2023-04-27 12:29 ` Daniel P. Berrangé
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).