[PATCH v4 0/2] docs: Speedup docs build

[Fabiano Rosas <farosas@suse.de>]

We currently have two documentation targets to build:
- 'man' for the man pages;
- 'html' for the web page.

There are two bottlenecks in the process:

1) sphinx runs with a single process;
2) the two targets are serialized.

For (1), we can just add the "-j auto" to sphinx_build and that should
bring some speed gains.

For (2) it's a little trickier because the reason the builds are
serialized is that Sphinx keeps track of changed files, but we still
need a way to tell meson whether a target needs to re-execute, so we
added dependency tracking and timestamp checking for (only) the 'html'
build via the depfile.py extension. Since the sources for both builds
are the same, we made the 'man' build dependent on 'html' so that it
would rebuild when needed.

So patch 1 adds the -j option to Sphinx and patch 2 adds depfile
support to the 'man' build. We can now run the two in parallel (ninja
will take care of that).

On my 16 core machine,
the -j change saves about 20s
the depfile change saves about 10s

===
On master:
 $ ../configure --enable-docs ...
 $ time make -j$(nproc) html man
   GIT     ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
 /usr/bin/ninja  build.ninja && touch build.ninja.stamp
 ninja: no work to do.
 /usr/bin/python3 -B .../qemu/meson/meson.py introspect --targets --tests --benchmarks | /usr/bin/python3 -B scripts/mtest2make.py > Makefile.mtest
   GIT     ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
 [1/2] Generating docs/QEMU manual with a custom command
 [2/2] Generating docs/QEMU man pages with a custom command
 make: Nothing to be done for 'man'.

 real    0m50.155s
 user    0m49.759s
 sys     0m0.761s

 $ mv docs ../saved-docs

This series:
 $ ../configure --enable-docs ...
 $ time make -j$(nproc) html man
   GIT     ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
 /usr/bin/ninja  build.ninja && touch build.ninja.stamp
 ninja: no work to do.
 /usr/bin/python3 -B .../qemu/meson/meson.py introspect --targets --tests --benchmarks | /usr/bin/python3 -B scripts/mtest2make.py > Makefile.mtest
   GIT     ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
 [1/2] Generating docs/QEMU man pages with a custom command
 [2/2] Generating docs/QEMU manual with a custom command
 make: Nothing to be done for 'man'.

 real    0m21.708s
 user    1m12.317s
 sys     0m2.056s

Diff sanity check:
 $ diff -rq docs/ ../saved-docs/
 Only in ../saved-docs/: docs.d      # now manual.dep
 Only in ../saved-docs/: docs.stamp  # now manual.stamp
 Only in docs/: man.dep
 Only in docs/: man.p
 Only in docs/: man.stamp
 Only in docs/: manual.dep
 Only in docs/: manual.stamp
 Files docs/manual.p/about/build-platforms.doctree and ../saved-docs/manual.p/about/build-platforms.doctree differ
 ...  # sphinx cache files, a bunch of these^

Rebuilding (here I show that a man file and an html file are
unchanged, change their source .rst and rebuild each target):

 $ ninja -d explain html
 ninja: no work to do.
 $ ninja -d explain man
 ninja: no work to do.
 $ man -l docs/qemu.1 | grep foobar
 $ grep foobar docs/manual/system/i386/pc.html
 $ vi ../docs/system/target-i386-desc.rst.inc    #add the 'foobar' string

 $ ninja -d explain man
 ninja explain: restat of output docs/man.stamp older than most recent input docs/system/target-i386-desc.rst.inc (1683122999140339620 vs 1683123032492362281)
 ninja explain: docs/man.stamp is dirty
 ninja explain: docs/qemu-block-drivers.7 is dirty
 ninja explain: docs/qemu-cpu-models.7 is dirty
 ninja explain: docs/qemu-ga-ref.7 is dirty
 ninja explain: docs/qemu-ga.8 is dirty
 ninja explain: docs/qemu-img.1 is dirty
 ninja explain: docs/qemu-nbd.8 is dirty
 ninja explain: docs/qemu-pr-helper.8 is dirty
 ninja explain: docs/qemu-qmp-ref.7 is dirty
 ninja explain: docs/qemu-storage-daemon-qmp-ref.7 is dirty
 ninja explain: docs/qemu-storage-daemon.1 is dirty
 ninja explain: docs/qemu-trace-stap.1 is dirty
 ninja explain: docs/qemu.1 is dirty
 ninja explain: docs/virtfs-proxy-helper.1 is dirty
 [1/1] Generating docs/QEMU man pages with a custom command

 $ man -l docs/qemu.1 | grep foobar
        The QEMU PC System emulator simulates the following foobar peripherals:
 $ grep foobar docs/manual/system/i386/pc.html  #html files unchanged

 $ ninja -d explain html
 ninja explain: restat of output docs/manual.stamp older than most recent input docs/system/target-i386-desc.rst.inc (1683122995876337403 vs 1683123032492362281)
 ninja explain: docs/manual.stamp is dirty
 [1/1] Generating docs/QEMU manual with a custom command

 $ man -l docs/qemu.1 | grep foobar
        The QEMU PC System emulator simulates the following foobar peripherals:
 $ grep foobar docs/manual/system/i386/pc.html
 <p>The QEMU PC System emulator simulates the following foobar peripherals:</p>

===

Fabiano Rosas (2):
  meson: Pass -j option to sphinx
  meson: Deserialize the man pages and html builds

 docs/meson.build           | 48 +++++++++++++++++++++++++-------------
 docs/sphinx/dbusdomain.py  |  4 ++++
 docs/sphinx/fakedbusdoc.py |  5 ++++
 docs/sphinx/qmp_lexer.py   |  5 ++++
 4 files changed, 46 insertions(+), 16 deletions(-)

-- 
2.35.3