[Performance Regression since v6.15] empty docs build with O= option

[Performance Regression since v6.15] empty docs build with O= option

[Akira Yokosawa]

Hi all,

Here is a follow-up to my message at:
https://lore.kernel.org/d12fb63e-b346-4094-b9d6-aa6823aea541@gmail.com/

Quoting relevant parts for your convenience:

Akira wrote:
> As a matter of fact, I'm seeing weird performance regression of empty
> documentation builds when the O=<somedir> option is used.
>
> It appeared in v6.15, which has your conversion of get_abi.pl into
> get_abi.py.  Will send a report once the time-consuming bisection
> completes.

Answer to Mauro's question at:
https://lore.kernel.org/20250711130446.1b761538@foz.lan/

Mauro wrote:
> Did you try with docs-next instead? I remember Jon caught one
> issue causing the doctree cache to expire. Can't remember if this
> was on 6.15 or at docs-next, but the fixes should be applied there.

Jon's complaint at https://lore.kernel.org/87frfsdfgc.fsf@trenco.lwn.net/
was against Mauro's v2 series.

Jon has applied the v3 series with the issue fixed at:
https://lore.kernel.org/cover.1750571906.git.mchehab+huawei@kernel.org/

, so the issue has never been in docs-next.

I'm not a python person and bisection is all I could do.

TL;DR

2nd run of "make O=$HOME/output htmldocs" takes much longer since kernel
release v6.15.

2nd run is an empty build and should finish in a matter of 10 seconds.

"make htmldocs" without the O= option doesn't show this symptom.

Here is a summary table.

Note:
    Numbers shown as (x) [y] mean 2nd "make O=$HOME/output htmldocs" run
    takes x seconds, while 2nd "make htmldocs" run takes y seconds
    (rounded up).

       ----------------------------------------
Sphinx      8.2.3 venv       8.1.3     7.2.6 
       ------------------- --------- ----------
Distro    f42      noble      f42      noble 
       --------- --------- --------- ----------
Good   (13) [13] (16) [16] ( 6) [ 6] (  9) [ 9] v6.14
Bad            ---         (72) [ 6] (349) [11] v6.15
Bad            ---         (72) [ 6] (365) [11] v6.16-rc1 + [1]
Bad            ---         (72) [ 7] (342) [10] docs-next (2025.07.11) + [1]
Bad    (77) [14] (82) [15] (73) [ 8] (347) [11] docs-next (2025.07.11) + [1,2,3]
       --------- --------- --------- ----------

[1]: cherry-pick 553ab30a1810 ("Documentation: nouveau: Update GSP message
     queue kernel-doc reference"), to suppress noise of error handling within
     Sphinx.
[2]: Mauro's ynl series v9 at https://lore.kernel.org/cover.1752076293.git.mchehab+huawei@kernel.org/
     (14/13 applied)
[3]: Jon's kdoc series v2 at https://lore.kernel.org/20250710233142.246524-1-corbet@lwn.net/
     (12/12 as is)

f42: python 3.13.5, noble: 3.12.3

* Bisection result

Sphinx 8.2.3 (venv) on top of ubuntu:noble (python 3.12.3):

Last good commit:
(16) 6b48bea16848 ("scripts/get_abi.py: add support for symbol search")

First bad commit:
(48) 9d7ec8867960 ("docs: use get_abi.py for ABI generation")

Merged into Jon's docs-mw branch at:
(82) 1ce8294cfefb ("Merge branch 'mauro' into docs-mw")

Pre merge commits of 1ce8294cfefb:
(15) 2b087edf588c ("docs/zh_CN: Add secrets index Chinese translation")
(82) 1c7e66bc5d20 ("scripts/get_abi.pl: drop now obsoleted script")

Again, The numbers enclosed in () indicate the elapsed times (in seconds,
rounded up) of 2nd "make O=$HOME/output htmldocs" runs.

Sphinx 8.2.3 takes longer than 8.1.3 for empty builds, but it is likely
an independent issue on Sphinx side.

Mauro, I'd like you to have a look into this regression.

        Thanks, Akira

Re: [Performance Regression since v6.15] empty docs build with O= option

[Mauro Carvalho Chehab]

Hi Akira,

Em Mon, 14 Jul 2025 19:42:49 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> Hi all,
> 
> Here is a follow-up to my message at:
> https://lore.kernel.org/d12fb63e-b346-4094-b9d6-aa6823aea541@gmail.com/
> 
> Quoting relevant parts for your convenience:
> 
> Akira wrote:
> > As a matter of fact, I'm seeing weird performance regression of empty
> > documentation builds when the O=<somedir> option is used.
> >
> > It appeared in v6.15, which has your conversion of get_abi.pl into
> > get_abi.py.  Will send a report once the time-consuming bisection
> > completes.  
> 
> Answer to Mauro's question at:
> https://lore.kernel.org/20250711130446.1b761538@foz.lan/
> 
> Mauro wrote:
> > Did you try with docs-next instead? I remember Jon caught one
> > issue causing the doctree cache to expire. Can't remember if this
> > was on 6.15 or at docs-next, but the fixes should be applied there.  
> 
> Jon's complaint at https://lore.kernel.org/87frfsdfgc.fsf@trenco.lwn.net/
> was against Mauro's v2 series.
> 
> Jon has applied the v3 series with the issue fixed at:
> https://lore.kernel.org/cover.1750571906.git.mchehab+huawei@kernel.org/
> 
> , so the issue has never been in docs-next.
> 
> I'm not a python person and bisection is all I could do.

This is probably not a python issue itself, but instead, some
weirdness at the way Sphinx cache is stored.

> TL;DR
> 
> 2nd run of "make O=$HOME/output htmldocs" takes much longer since kernel
> release v6.15.
> 
> 2nd run is an empty build and should finish in a matter of 10 seconds.
> 
> "make htmldocs" without the O= option doesn't show this symptom.
> 
> Here is a summary table.
> 
> Note:
>     Numbers shown as (x) [y] mean 2nd "make O=$HOME/output htmldocs" run
>     takes x seconds, while 2nd "make htmldocs" run takes y seconds
>     (rounded up).
> 
>        ----------------------------------------
> Sphinx      8.2.3 venv       8.1.3     7.2.6 
>        ------------------- --------- ----------
> Distro    f42      noble      f42      noble 
>        --------- --------- --------- ----------
> Good   (13) [13] (16) [16] ( 6) [ 6] (  9) [ 9] v6.14
> Bad            ---         (72) [ 6] (349) [11] v6.15
> Bad            ---         (72) [ 6] (365) [11] v6.16-rc1 + [1]
> Bad            ---         (72) [ 7] (342) [10] docs-next (2025.07.11) + [1]
> Bad    (77) [14] (82) [15] (73) [ 8] (347) [11] docs-next (2025.07.11) + [1,2,3]
>        --------- --------- --------- ----------
> 
> [1]: cherry-pick 553ab30a1810 ("Documentation: nouveau: Update GSP message
>      queue kernel-doc reference"), to suppress noise of error handling within
>      Sphinx.
> [2]: Mauro's ynl series v9 at https://lore.kernel.org/cover.1752076293.git.mchehab+huawei@kernel.org/
>      (14/13 applied)
> [3]: Jon's kdoc series v2 at https://lore.kernel.org/20250710233142.246524-1-corbet@lwn.net/
>      (12/12 as is)
> 
> f42: python 3.13.5, noble: 3.12.3
> 
> * Bisection result
> 
> Sphinx 8.2.3 (venv) on top of ubuntu:noble (python 3.12.3):
> 
> Last good commit:
> (16) 6b48bea16848 ("scripts/get_abi.py: add support for symbol search")
> 
> First bad commit:
> (48) 9d7ec8867960 ("docs: use get_abi.py for ABI generation")
> 
> Merged into Jon's docs-mw branch at:
> (82) 1ce8294cfefb ("Merge branch 'mauro' into docs-mw")
> 
> Pre merge commits of 1ce8294cfefb:
> (15) 2b087edf588c ("docs/zh_CN: Add secrets index Chinese translation")
> (82) 1c7e66bc5d20 ("scripts/get_abi.pl: drop now obsoleted script")
> 
> Again, The numbers enclosed in () indicate the elapsed times (in seconds,
> rounded up) of 2nd "make O=$HOME/output htmldocs" runs.
> 
> Sphinx 8.2.3 takes longer than 8.1.3 for empty builds, but it is likely
> an independent issue on Sphinx side.
> 
> Mauro, I'd like you to have a look into this regression.
> 
>         Thanks, Akira

To debug it, I added the enclosed code to conf.py. See enclosed. The
logic there uses this from https://www.sphinx-doc.org/en/master/extdev/event_callbacks.html:

	env-get-outdated(app, env, added, changed, removed)

	Emitted when the environment determines which source files have
	changed and should be re-read. added, changed and removed are 
	sets of docnames that the environment has determined. You can 
	return a list of docnames to re-read in addition to these.


Assuming that the issue is during ABI generation, we can check with that
what is the root case.

Running the second time without O= doeesn't require rebuild ABI:

    $ time make SPHINXDIRS=admin-guide htmldocs
    Python version: 3.13.5
    Docutils version: 0.21.2
    make[3]: Nothing to be done for 'html'.
    Using alabaster theme
    source directory: admin-guide
    Using Python kernel-doc

    [OUTDATED]
    Added: set()
    Changed: {'gpio/obsolete', 'gpio/index'}
    Removed: set()
    All docs count: 385
    Found docs count: 385
    /root/Documentation/admin-guide/gpio/index.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/chardev' [toc.not_readable]
    /root/Documentation/admin-guide/gpio/obsolete.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/chardev_v1' [toc.not_readable]
    /root/Documentation/admin-guide/gpio/obsolete.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/sysfs' [toc.not_readable]

    real    0m1,489s
    user    0m1,305s
    sys     0m0,263s

Here, it seems that the root cause is because of a non-resolved
dependency at gpio/obsolete.

Now, running the second time with O=, we have:


    $ make clean; time make O=/tmp/foo SPHINXDIRS=admin-guide htmldocs
    ...

    [OUTDATED]
    Added: set()
    Changed: {'abi-stable', 'abi-testing-files', 'abi-removed', 'gpio/index', 'abi-obsolete-files', 'abi-testing', 'abi-stable-files', 'abi-obsolete', 'gpio/obsolete', 'abi-removed-files', 'abi'}
    Removed: set()
    All docs count: 385
    Found docs count: 385

Basically, Sphinx thinks that abi* files had some changes, but
only when O= is used.

At Documentation/sphinx/kernel_abi.py, the only part we touch the
dependency chain is here:

            if f != old_f:
                # Add the file to Sphinx build dependencies
                env.note_dependency(os.path.abspath(f))

If we comment such code or change the logic to:

    diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
    index db6f0380de94..9f2198eb7af1 100644
    --- a/Documentation/sphinx/kernel_abi.py
    +++ b/Documentation/sphinx/kernel_abi.py
    @@ -146,8 +146,9 @@ class KernelCmd(Directive):
                 n += 1
 
             if f != old_f:
-                # Add the file to Sphinx build dependencies
-                env.note_dependency(os.path.abspath(f))
+                # Add the file to Sphinx build dependencies if the file exists
+                if os.path.isfile(f):
+                    env.note_dependency(os.path.abspath(f))
 
                 old_f = f
 
We get a quick build again:

    [OUTDATED]
    Added: set()
    Changed: {'gpio/index', 'gpio/obsolete'}
    Removed: set()
    All docs count: 385
    Found docs count: 385
    /root/Documentation/admin-guide/gpio/index.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/chardev' [toc.not_readable]
    /root/Documentation/admin-guide/gpio/obsolete.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/chardev_v1' [toc.not_readable]
    /root/Documentation/admin-guide/gpio/obsolete.rst:7: WARNING: toctree contains reference to nonexisting document 'userspace-api/gpio/sysfs' [toc.not_readable]
    make[1]: Leaving directory '/tmp/foo'

   real    0m1,444s
   user    0m1,267s
   sys     0m0,252s

However, that's because it won't add any dependency at all when
O= is used. The problem seems to be related to relative/absolute
patches when adding such dependencies.

Let me try to write a proper fix.

Jon,

Btw, should we add something like the code below to be optionally
used at conf.py when we need to identify cache issues?

Regards,
Mauro


---

This is what I added at the end of conf.py, at setup(app):

Thanks,
Mauro