[PATCH v2] sphinx: kernel_abi: fix performance regression with O=<dir>

[PATCH v2] sphinx: kernel_abi: fix performance regression with O=<dir>

[Mauro Carvalho Chehab]

The logic there which adds a dependency note to Sphinx cache
is not taking into account that the build dir may not be
the source dir. This causes a performance regression:

$ time make O=/tmp/foo SPHINXDIRS=admin-guide htmldocs

	[OUTDATED]
	Added: set()
	Changed: {'abi-obsolete', 'abi-removed', 'abi-stable-files', 'abi-obsolete-files', 'abi-stable', 'abi', 'abi-removed-files', 'abi-testing-files', 'abi-testing', 'gpio/index', 'gpio/obsolete'}
	Removed: set()
	All docs count: 385
	Found docs count: 385

	real    0m11,324s
	user    0m15,783s
	sys     0m1,164s

To get the root cause of the problem (ABI files reported as changed),
I used this changeset:

	diff --git a/Documentation/conf.py b/Documentation/conf.py
	index e8766e689c1b..ab486623bd8b 100644
	--- a/Documentation/conf.py
	+++ b/Documentation/conf.py
	@@ -571,3 +571,16 @@ def setup(app):
	     """Patterns need to be updated at init time on older Sphinx versions"""

	     app.connect('config-inited', update_patterns)
	+    app.connect('env-get-outdated', on_outdated)
	+
	+def on_outdated(app, env, added, changed, removed):
	+    """Track cache outdated due to added/changed/removed files"""
	+    print("\n[OUTDATED]")
	+    print(f"Added: {added}")
	+    print(f"Changed: {changed}")
	+    print(f"Removed: {removed}")
	+    print(f"All docs count: {len(env.all_docs)}")
	+    print(f"Found docs count: {len(env.found_docs)}")
	+
	+    # Just return what we have
	+    return added | changed | removed

Reported-by: Akira Yokosawa <akiyks@gmail.com>
Closes: https://lore.kernel.org/linux-doc/c174f7c5-ec21-4eae-b1c3-f643cca90d9d@gmail.com/
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
v2: updated description. No changes at the diff itself

 Documentation/sphinx/kernel_abi.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index db6f0380de94..4c4375201b9e 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -146,8 +146,10 @@ class KernelCmd(Directive):
                 n += 1
 
             if f != old_f:
-                # Add the file to Sphinx build dependencies
-                env.note_dependency(os.path.abspath(f))
+                # Add the file to Sphinx build dependencies if the file exists
+                fname = os.path.join(srctree, f)
+                if os.path.isfile(fname):
+                    env.note_dependency(fname)
 
                 old_f = f
 
-- 
2.50.1

Re: [PATCH v2] sphinx: kernel_abi: fix performance regression with O=<dir>

[Akira Yokosawa]

On Thu, 17 Jul 2025 13:37:19 +0200, Mauro Carvalho Chehab wrote:
> The logic there which adds a dependency note to Sphinx cache
> is not taking into account that the build dir may not be
> the source dir. This causes a performance regression:
> 
> $ time make O=/tmp/foo SPHINXDIRS=admin-guide htmldocs
> 
> 	[OUTDATED]
> 	Added: set()
> 	Changed: {'abi-obsolete', 'abi-removed', 'abi-stable-files', 'abi-obsolete-files', 'abi-stable', 'abi', 'abi-removed-files', 'abi-testing-files', 'abi-testing', 'gpio/index', 'gpio/obsolete'}
> 	Removed: set()
> 	All docs count: 385
> 	Found docs count: 385
> 
> 	real    0m11,324s
> 	user    0m15,783s
> 	sys     0m1,164s
> 
> To get the root cause of the problem (ABI files reported as changed),
> I used this changeset:
> 
> 	diff --git a/Documentation/conf.py b/Documentation/conf.py
> 	index e8766e689c1b..ab486623bd8b 100644
> 	--- a/Documentation/conf.py
> 	+++ b/Documentation/conf.py
> 	@@ -571,3 +571,16 @@ def setup(app):
> 	     """Patterns need to be updated at init time on older Sphinx versions"""
> 
> 	     app.connect('config-inited', update_patterns)
> 	+    app.connect('env-get-outdated', on_outdated)
> 	+
> 	+def on_outdated(app, env, added, changed, removed):
> 	+    """Track cache outdated due to added/changed/removed files"""
> 	+    print("\n[OUTDATED]")
> 	+    print(f"Added: {added}")
> 	+    print(f"Changed: {changed}")
> 	+    print(f"Removed: {removed}")
> 	+    print(f"All docs count: {len(env.all_docs)}")
> 	+    print(f"Found docs count: {len(env.found_docs)}")
> 	+
> 	+    # Just return what we have
> 	+    return added | changed | removed
> 
> Reported-by: Akira Yokosawa <akiyks@gmail.com>
> Closes: https://lore.kernel.org/linux-doc/c174f7c5-ec21-4eae-b1c3-f643cca90d9d@gmail.com/
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

Tested-by: Akira Yokosawa <akiyks@gmail.com>

Thanks!

> ---
> v2: updated description. No changes at the diff itself
> 
[...]

Re: [PATCH v2] sphinx: kernel_abi: fix performance regression with O=<dir>

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> The logic there which adds a dependency note to Sphinx cache
> is not taking into account that the build dir may not be
> the source dir. This causes a performance regression:
>
> $ time make O=/tmp/foo SPHINXDIRS=admin-guide htmldocs
>
> 	[OUTDATED]
> 	Added: set()
> 	Changed: {'abi-obsolete', 'abi-removed', 'abi-stable-files', 'abi-obsolete-files', 'abi-stable', 'abi', 'abi-removed-files', 'abi-testing-files', 'abi-testing', 'gpio/index', 'gpio/obsolete'}
> 	Removed: set()
> 	All docs count: 385
> 	Found docs count: 385
>
> 	real    0m11,324s
> 	user    0m15,783s
> 	sys     0m1,164s
>
> To get the root cause of the problem (ABI files reported as changed),
> I used this changeset:
>
> 	diff --git a/Documentation/conf.py b/Documentation/conf.py
> 	index e8766e689c1b..ab486623bd8b 100644
> 	--- a/Documentation/conf.py
> 	+++ b/Documentation/conf.py
> 	@@ -571,3 +571,16 @@ def setup(app):
> 	     """Patterns need to be updated at init time on older Sphinx versions"""
>
> 	     app.connect('config-inited', update_patterns)
> 	+    app.connect('env-get-outdated', on_outdated)
> 	+
> 	+def on_outdated(app, env, added, changed, removed):
> 	+    """Track cache outdated due to added/changed/removed files"""
> 	+    print("\n[OUTDATED]")
> 	+    print(f"Added: {added}")
> 	+    print(f"Changed: {changed}")
> 	+    print(f"Removed: {removed}")
> 	+    print(f"All docs count: {len(env.all_docs)}")
> 	+    print(f"Found docs count: {len(env.found_docs)}")
> 	+
> 	+    # Just return what we have
> 	+    return added | changed | removed
>
> Reported-by: Akira Yokosawa <akiyks@gmail.com>
> Closes: https://lore.kernel.org/linux-doc/c174f7c5-ec21-4eae-b1c3-f643cca90d9d@gmail.com/
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
> v2: updated description. No changes at the diff itself
>
>  Documentation/sphinx/kernel_abi.py | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
> index db6f0380de94..4c4375201b9e 100644
> --- a/Documentation/sphinx/kernel_abi.py
> +++ b/Documentation/sphinx/kernel_abi.py
> @@ -146,8 +146,10 @@ class KernelCmd(Directive):
>                  n += 1
>  
>              if f != old_f:
> -                # Add the file to Sphinx build dependencies
> -                env.note_dependency(os.path.abspath(f))
> +                # Add the file to Sphinx build dependencies if the file exists
> +                fname = os.path.join(srctree, f)
> +                if os.path.isfile(fname):
> +                    env.note_dependency(fname)
>  

Applied, thanks.

jon