Re: [PATCH v2] meson: Pass -j option to sphinx

["Daniel P. Berrangé" <berrange@redhat.com>]

On Thu, Apr 27, 2023 at 02:25:16PM -0300, Fabiano Rosas wrote:
> Save a bit of build time by passing the number of jobs option to
> sphinx.
> 
> We cannot use the -j option from make because meson does not support
> setting build time parameters for custom targets. Use nproc instead or
> the equivalent sphinx option "-j auto", if that is available.
> 
> Also make sure our plugins support parallelism and report it properly
> to sphinx. Particularly, implement the merge_domaindata method in
> DBusDomain that is used to merge in data from other subprocesses.
> 
> before:
>   $ time make man html
>   ...
>   [1/2] Generating docs/QEMU manual with a custom command
>   [2/2] Generating docs/QEMU man pages with a custom command
> 
>   real    0m43.157s
>   user    0m42.642s
>   sys     0m0.576s
> 
> after:
>   $ time make man html
>   ...
>   [1/2] Generating docs/QEMU manual with a custom command
>   [2/2] Generating docs/QEMU man pages with a custom command
> 
>   real    0m25.014s
>   user    0m51.288s
>   sys     0m2.085s

On my 12 CPU laptop I see a similar magnitude benefit - about
20 seconds is cut from the docs build time - 50 down to 30 secs.

Watching the CPU usage I see sphinx is not very good at keeping
all CPUs busy. For perhaps 2 seconds I'll see  8 sphinx processes
burning CPUs, but the majority of the time it'll only be 1 or 2
sphinx processes.

IOW, we do get a benefit, but it is not nearly as good as one
might hope for given the number of CPUs potentially available.

> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
>  docs/meson.build           | 12 ++++++++++++
>  docs/sphinx/dbusdomain.py  |  4 ++++
>  docs/sphinx/fakedbusdoc.py |  5 +++++
>  docs/sphinx/qmp_lexer.py   |  5 +++++
>  4 files changed, 26 insertions(+)

Tested-by: Daniel P. Berrangé <berrange@redhat.com>

> 
> diff --git a/docs/meson.build b/docs/meson.build
> index f220800e3e..9e4bed6fa0 100644
> --- a/docs/meson.build
> +++ b/docs/meson.build
> @@ -10,6 +10,18 @@ if sphinx_build.found()
>      SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
>    endif
>  
> +  sphinx_version = run_command(SPHINX_ARGS + ['--version'],
> +                               check: false).stdout().split()[1]
> +  if sphinx_version.version_compare('>=5.1.2')
> +    SPHINX_ARGS += ['-j', 'auto']
> +  else
> +    nproc = find_program('nproc')
> +    if nproc.found()
> +      jobs = run_command(nproc, check:false).stdout()
> +      SPHINX_ARGS += ['-j', jobs]
> +    endif
> +  endif

ANy reason for check: false in these 2 run_command calls ?

They'll both return 0 on success, so I would have though
'check: true' was more robust at error reporting ?


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|