[PATCH 0/1] docs: kernel-doc: doesn't suppress error/warnings

[PATCH 0/1] docs: kernel-doc: doesn't suppress error/warnings

[Mauro Carvalho Chehab]

As discussed at:
   https://lore.kernel.org/all/20250516200350.63be46cd@canb.auug.org.au/

Sphinx logger is set in a way that errors/warnings from kernel-doc are
suppressed, except when V=1.

Finding the root cause is not trivial, as we need to go into the Sphinx
overlays on the top of Python's default logger class. Instead, let's take
the trivial approach: use logger class directly. As we can later revisit
such decision, let's add a comment at the code.

Mauro Carvalho Chehab (1):
  docs: kerneldoc.py: don't use Sphinx logger

 Documentation/sphinx/kerneldoc.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

-- 
2.49.0

[PATCH 1/1] docs: kerneldoc.py: don't use Sphinx logger

[Mauro Carvalho Chehab]

Unfortunately, currently Sphinx logger is suppressing too much, not
allowing warnings to be displayed. Disable it.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/sphinx/kerneldoc.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/Documentation/sphinx/kerneldoc.py b/Documentation/sphinx/kerneldoc.py
index b713a2c4a615..314479718a01 100644
--- a/Documentation/sphinx/kerneldoc.py
+++ b/Documentation/sphinx/kerneldoc.py
@@ -311,7 +311,11 @@ def setup_kfiles(app):
     if kerneldoc_bin and kerneldoc_bin.endswith("kernel-doc.py"):
         print("Using Python kernel-doc")
         out_style = RestFormat()
-        kfiles = KernelFiles(out_style=out_style, logger=logger)
+
+        # Ideally, we should be using Sphinx logger here, but its filtering
+        # rules ending filtering out warnings and errors. So, let's use
+        # Python default logger instead.
+        kfiles = KernelFiles(out_style=out_style)
     else:
         print(f"Using {kerneldoc_bin}")
 
-- 
2.49.0

Re: [PATCH 1/1] docs: kerneldoc.py: don't use Sphinx logger

[Randy Dunlap]

On 5/19/25 10:47 PM, Mauro Carvalho Chehab wrote:
> Unfortunately, currently Sphinx logger is suppressing too much, not
> allowing warnings to be displayed. Disable it.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

On linux-next-20250516, this gives me:

Cannot find file ../drivers/gpio/gpiolib-acpi.c
Cannot find file ../drivers/gpio/gpiolib-acpi.c
...
Sphinx parallel build error!

Versions
========

* Platform:         linux; (Linux-6.14.4-1-default-x86_64-with-glibc2.41)
* Python version:   3.13.3 (CPython)
* Sphinx version:   8.2.3
* Docutils version: 0.21.2
* Jinja2 version:   3.1.6
* Pygments version: 2.19.1

Last Messages
=============


    reading sources... [ 75%]
    translations/it_IT/subsystem-apis .. translations/zh_CN/admin-guide/mm/damon/reclaim

    reading sources... [ 77%]
    translations/zh_CN/admin-guide/mm/damon/start .. translations/zh_CN/core-api/symbol-namespaces

    reading sources... [ 79%]
    translations/zh_CN/core-api/this_cpu_ops .. translations/zh_CN/kernel-hacking/index

Loaded Extensions
=================

* sphinx.ext.mathjax (8.2.3)
* alabaster (1.0.0)
* sphinxcontrib.applehelp (2.0.0)
* sphinxcontrib.devhelp (1.0.6)
* sphinxcontrib.htmlhelp (2.1.0)
* sphinxcontrib.serializinghtml (1.1.10)
* sphinxcontrib.qthelp (2.0.0)
* kerneldoc (1.0)
* rstFlatTable (1.0)
* kernel_include (1.0)
* kfigure (1.0.0)
* sphinx.ext.ifconfig (8.2.3)
* automarkup (unknown version)
* maintainers_include (1.0)
* sphinx.ext.autosectionlabel (8.2.3)
* kernel_abi (1.0)
* kernel_feat (1.0)
* translations (unknown version)

Traceback
=========

      File "/usr/lib/python3.13/site-packages/sphinx/util/parallel.py", line 137, in _join_one
        raise SphinxParallelError(*result)
    sphinx.errors.SphinxParallelError: KeyError: '../drivers/gpio/gpiolib-acpi.c'

and then it's finished (not a normal finish).
So IMHO this patch is not sufficient.


> ---
>  Documentation/sphinx/kerneldoc.py | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/sphinx/kerneldoc.py b/Documentation/sphinx/kerneldoc.py
> index b713a2c4a615..314479718a01 100644
> --- a/Documentation/sphinx/kerneldoc.py
> +++ b/Documentation/sphinx/kerneldoc.py
> @@ -311,7 +311,11 @@ def setup_kfiles(app):
>      if kerneldoc_bin and kerneldoc_bin.endswith("kernel-doc.py"):
>          print("Using Python kernel-doc")
>          out_style = RestFormat()
> -        kfiles = KernelFiles(out_style=out_style, logger=logger)
> +
> +        # Ideally, we should be using Sphinx logger here, but its filtering
> +        # rules ending filtering out warnings and errors. So, let's use
> +        # Python default logger instead.
> +        kfiles = KernelFiles(out_style=out_style)
>      else:
>          print(f"Using {kerneldoc_bin}")
>  

-- 
~Randy

Re: [PATCH 1/1] docs: kerneldoc.py: don't use Sphinx logger

[Mauro Carvalho Chehab]

Hi Randy,

Em Mon, 19 May 2025 22:55:08 -0700
Randy Dunlap <rdunlap@infradead.org> escreveu:

> On 5/19/25 10:47 PM, Mauro Carvalho Chehab wrote:
> > Unfortunately, currently Sphinx logger is suppressing too much, not
> > allowing warnings to be displayed. Disable it.
> > 
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>  
> 
> On linux-next-20250516, this gives me:
> 
> Cannot find file ../drivers/gpio/gpiolib-acpi.c
> Cannot find file ../drivers/gpio/gpiolib-acpi.c
> ...
> Sphinx parallel build error!
> 
> Versions
> ========
> 
> * Platform:         linux; (Linux-6.14.4-1-default-x86_64-with-glibc2.41)
> * Python version:   3.13.3 (CPython)
> * Sphinx version:   8.2.3
> * Docutils version: 0.21.2
> * Jinja2 version:   3.1.6
> * Pygments version: 2.19.1
> 
> Last Messages
> =============
> 
> 
>     reading sources... [ 75%]
>     translations/it_IT/subsystem-apis .. translations/zh_CN/admin-guide/mm/damon/reclaim
> 
>     reading sources... [ 77%]
>     translations/zh_CN/admin-guide/mm/damon/start .. translations/zh_CN/core-api/symbol-namespaces
> 
>     reading sources... [ 79%]
>     translations/zh_CN/core-api/this_cpu_ops .. translations/zh_CN/kernel-hacking/index
> 
> Loaded Extensions
> =================
> 
> * sphinx.ext.mathjax (8.2.3)
> * alabaster (1.0.0)
> * sphinxcontrib.applehelp (2.0.0)
> * sphinxcontrib.devhelp (1.0.6)
> * sphinxcontrib.htmlhelp (2.1.0)
> * sphinxcontrib.serializinghtml (1.1.10)
> * sphinxcontrib.qthelp (2.0.0)
> * kerneldoc (1.0)
> * rstFlatTable (1.0)
> * kernel_include (1.0)
> * kfigure (1.0.0)
> * sphinx.ext.ifconfig (8.2.3)
> * automarkup (unknown version)
> * maintainers_include (1.0)
> * sphinx.ext.autosectionlabel (8.2.3)
> * kernel_abi (1.0)
> * kernel_feat (1.0)
> * translations (unknown version)
> 
> Traceback
> =========
> 
>       File "/usr/lib/python3.13/site-packages/sphinx/util/parallel.py", line 137, in _join_one
>         raise SphinxParallelError(*result)
>     sphinx.errors.SphinxParallelError: KeyError: '../drivers/gpio/gpiolib-acpi.c'
> 
> and then it's finished (not a normal finish).
> So IMHO this patch is not sufficient.

Well, on next-20250516:

	$ ls drivers/gpio/gpiolib-acpi.c
	ls: cannot access 'drivers/gpio/gpiolib-acpi.c': No such file or directory

Avoiding the script to abort is a matter of adding a try/except
block at kerneldoc.py, but I'd say that, if an include file (or any other
file needed for the build) is not found, "make" should abort anyway for
the affected target (Kernel compilation or when doc building).

The only thing that bothers me is the "Sphinx parallel build error!"
message. I'll check if there is a way to suppress it.

Regards,
Mauro

Re: [PATCH 1/1] docs: kerneldoc.py: don't use Sphinx logger

[Akira Yokosawa]

On Tue, 20 May 2025 09:50:37 +0200, Mauro Carvalho Chehab wrote:
> Hi Randy,
> 
> Em Mon, 19 May 2025 22:55:08 -0700
> Randy Dunlap <rdunlap@infradead.org> escreveu:
> 
[...]

>> Traceback
>> =========
>>
>>       File "/usr/lib/python3.13/site-packages/sphinx/util/parallel.py", line 137, in _join_one
>>         raise SphinxParallelError(*result)
>>     sphinx.errors.SphinxParallelError: KeyError: '../drivers/gpio/gpiolib-acpi.c'
>>
>> and then it's finished (not a normal finish).
>> So IMHO this patch is not sufficient.
> 
> Well, on next-20250516:
> 
> 	$ ls drivers/gpio/gpiolib-acpi.c
> 	ls: cannot access 'drivers/gpio/gpiolib-acpi.c': No such file or directory
> 
> Avoiding the script to abort is a matter of adding a try/except
> block at kerneldoc.py, but I'd say that, if an include file (or any other
> file needed for the build) is not found, "make" should abort anyway for
> the affected target (Kernel compilation or when doc building).

Interesting ...

So, it sounds to me you think you have the right to break Stephen's (and
possibly other devs') workflow of test-building kernel docs, aren't you?

I don't buy such an argument.

An innocent typo in pathname somewhere in the doc tree deserves a friendly
warning at most, not a fatal crash within Sphinx.  That would need another
run of "make htmldocs" after fixing the fatal error to see for other
innocuous warnings.

And your change has no effect on exposing those innocuous warnings.

On current docs-next + your change above, running:

    make cleandocs; make KERNELDOC=$PWD/scripts/kernel-doc.pl htmldocs

produces these 3 warnings:

----------------------------------------------------------------------
./drivers/gpu/drm/amd/include/amd_shared.h:369: warning: Incorrect use of kernel-doc format:          * @DC_HDCP_LC_ENABLE_SW_FALLBACK If set, upon HDCP Locality Check FW
./drivers/gpu/drm/amd/include/amd_shared.h:369: warning: Incorrect use of kernel-doc format:          * @DC_HDCP_LC_ENABLE_SW_FALLBACK If set, upon HDCP Locality Check FW
./drivers/gpu/drm/amd/include/amd_shared.h:373: warning: Enum value 'DC_HDCP_LC_ENABLE_SW_FALLBACK' not described in enum 'DC_DEBUG_MASK'
----------------------------------------------------------------------

, while running:

    make cleandocs; make htmldocs

or:

    make cleandocs; make KERNELDOC=$PWD/scripts/kerneldoc htmldocs

doesn't produce them.

Sorry, but I believe you have run out of time.

And let me remind you of the expectation for backward-compatibility widely
accepted our community.

Akira

Re: [PATCH 1/1] docs: kerneldoc.py: don't use Sphinx logger

[Mauro Carvalho Chehab]

Em Tue, 20 May 2025 17:23:30 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> On Tue, 20 May 2025 09:50:37 +0200, Mauro Carvalho Chehab wrote:
> > Hi Randy,
> > 
> > Em Mon, 19 May 2025 22:55:08 -0700
> > Randy Dunlap <rdunlap@infradead.org> escreveu:
> >   
> [...]
> 
> >> Traceback
> >> =========
> >>
> >>       File "/usr/lib/python3.13/site-packages/sphinx/util/parallel.py", line 137, in _join_one
> >>         raise SphinxParallelError(*result)
> >>     sphinx.errors.SphinxParallelError: KeyError: '../drivers/gpio/gpiolib-acpi.c'
> >>
> >> and then it's finished (not a normal finish).
> >> So IMHO this patch is not sufficient.  
> > 
> > Well, on next-20250516:
> > 
> > 	$ ls drivers/gpio/gpiolib-acpi.c
> > 	ls: cannot access 'drivers/gpio/gpiolib-acpi.c': No such file or directory
> > 
> > Avoiding the script to abort is a matter of adding a try/except
> > block at kerneldoc.py, but I'd say that, if an include file (or any other
> > file needed for the build) is not found, "make" should abort anyway for
> > the affected target (Kernel compilation or when doc building).  
> 
> Interesting ...
> 
> So, it sounds to me you think you have the right to break Stephen's (and
> possibly other devs') workflow of test-building kernel docs, aren't you?
> 
> I don't buy such an argument.
> 
> An innocent typo in pathname somewhere in the doc tree deserves a friendly
> warning at most, not a fatal crash within Sphinx.  That would need another
> run of "make htmldocs" after fixing the fatal error to see for other
> innocuous warnings.

If by "innocent typo" you mean removing/renaming/moving a file from
the Kernel tree without updating Makefiles, or adding them with wrong
names, then yes: I do think this should be fatal for the affected "make" 
targets. The quickest such build issues are caught, the best for everyone.

Yet, this is is a separate discussion: if there are bugs at the doc
build system, they should be fixed. In this specific case, the missing
file produces a KeyError when the messages that were supposed to be
generated by it aren't stored on a dict. I sent already some patches
addressing it:

	https://lore.kernel.org/linux-doc/cover.1747730982.git.mchehab+huawei@kernel.org/T/#t

Plus the logger fix to prevent Sphinx to suppress warnings/errors:
	https://lore.kernel.org/linux-doc/cover.1747719873.git.mchehab+huawei@kernel.org/T/#t

> 
> And your change has no effect on exposing those innocuous warnings.
> 
> On current docs-next + your change above, running:
> 
>     make cleandocs; make KERNELDOC=$PWD/scripts/kernel-doc.pl htmldocs
> 
> produces these 3 warnings:
> 
> ----------------------------------------------------------------------
> ./drivers/gpu/drm/amd/include/amd_shared.h:369: warning: Incorrect use of kernel-doc format:          * @DC_HDCP_LC_ENABLE_SW_FALLBACK If set, upon HDCP Locality Check FW
> ./drivers/gpu/drm/amd/include/amd_shared.h:369: warning: Incorrect use of kernel-doc format:          * @DC_HDCP_LC_ENABLE_SW_FALLBACK If set, upon HDCP Locality Check FW
> ./drivers/gpu/drm/amd/include/amd_shared.h:373: warning: Enum value 'DC_HDCP_LC_ENABLE_SW_FALLBACK' not described in enum 'DC_DEBUG_MASK'
> ----------------------------------------------------------------------
> 
> , while running:
> 
>     make cleandocs; make htmldocs
> 
> or:
> 
>     make cleandocs; make KERNELDOC=$PWD/scripts/kerneldoc htmldocs
> 
> doesn't produce them.

Testing here with next-20250516:

	$ ./scripts/kernel-doc.pl -Wall -none ./drivers/gpu/drm/amd/include/amd_shared.h
	$

	$ ./scripts/kernel-doc.py -Wall -none ./drivers/gpu/drm/amd/include/amd_shared.h
	$

Also, if I rename the value 'DC_HDCP_LC_ENABLE_SW_FALLBACK_XXX',
both produces identical warnings:

	$ ./scripts/kernel-doc.pl -Wall -none ./drivers/gpu/drm/amd/include/amd_shared.h
	./drivers/gpu/drm/amd/include/amd_shared.h:373: warning: Enum value 'DC_HDCP_LC_ENABLE_SW_FALLBACK_XXX' not described in enum 'DC_DEBUG_MASK'
	./drivers/gpu/drm/amd/include/amd_shared.h:373: warning: Excess enum value 'DC_HDCP_LC_ENABLE_SW_FALLBACK' description in 'DC_DEBUG_MASK'

	$ ./scripts/kernel-doc.py -Wall -none ./drivers/gpu/drm/amd/include/amd_shared.h
	Warning: ./drivers/gpu/drm/amd/include/amd_shared.h:372 Enum value 'DC_HDCP_LC_ENABLE_SW_FALLBACK_XXX' not described in enum 'DC_DEBUG_MASK'
	Warning: ./drivers/gpu/drm/amd/include/amd_shared.h:372 Excess enum value '%DC_HDCP_LC_ENABLE_SW_FALLBACK' description in 'DC_DEBUG_MASK'

So, it sounds to me that things are the way they're expected,
after applying both series.

Regards,
Mauro

Re: [PATCH 1/1] docs: kerneldoc.py: don't use Sphinx logger

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]

Hi Mauro,

I do appreciate your work on the document system.  Particularly you
have made the process much faster.  However, see below.

On Tue, 20 May 2025 11:19:56 +0200 Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>
> If by "innocent typo" you mean removing/renaming/moving a file from
> the Kernel tree without updating Makefiles, or adding them with wrong
> names, then yes: I do think this should be fatal for the affected "make" 
> targets. The quickest such build issues are caught, the best for everyone.

They are found if you leave them as warnings and just continue on.  I
have been finding them by when the current python version just crashes
(without any useful message before your current change) I rerun the
htmldocs using the perl version to get the name of the missing file and
then report it so it can be fixed.  But the perl version continues on
and produces all the other warnings so I can compare with the previous
and only report new errors/warnigs.  Sometime these reports are not
foxed for days or weeks (or months).

So, please, now that you are producing more warnings, make a missing
file just a warning (or at least continue on).  I have enough to do
every day without having to run "make htmldocs" more than once.

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 1/1] docs: kerneldoc.py: don't use Sphinx logger

[Mauro Carvalho Chehab]

Em Tue, 20 May 2025 21:39:06 +1000
Stephen Rothwell <sfr@canb.auug.org.au> escreveu:

> Hi Mauro,
> 
> I do appreciate your work on the document system.  Particularly you
> have made the process much faster.  However, see below.

Hi Stephen,

> On Tue, 20 May 2025 11:19:56 +0200 Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> >
> > If by "innocent typo" you mean removing/renaming/moving a file from
> > the Kernel tree without updating Makefiles, or adding them with wrong
> > names, then yes: I do think this should be fatal for the affected "make" 
> > targets. The quickest such build issues are caught, the best for everyone.  
> 
> They are found if you leave them as warnings and just continue on.  I
> have been finding them by when the current python version just crashes
> (without any useful message before your current change) I rerun the
> htmldocs using the perl version to get the name of the missing file and
> then report it so it can be fixed.  But the perl version continues on
> and produces all the other warnings so I can compare with the previous
> and only report new errors/warnigs.  Sometime these reports are not
> foxed for days or weeks (or months).
> 

The issues we're seeing weren't due to the conversion from perl to
python, but, instead, to the new logic that calls the Python methods
directly, instead of opening a shell to call kernel-doc.

Those caused some unintended changes, due to the way Sphinx works:

1. Sphinx has an overlay over the Python logger class. Such overlay
   has some logic to output colored messages, which is interesting.
   However, it also has a filtering logic which is affected by the
   flags passed when sphinx-build is called. That was actually
   suppressing not only verbose level logs, but also warnings and
   errors. The fix is a single-line change:

   -	kfiles = KernelFiles(out_style=out_style, logger=logger)
   +	kfiles = KernelFiles(out_style=out_style)

   The patch is at:
	https://lore.kernel.org/linux-doc/6b81b1aaa8446b4d850064dd38ffffa1a1cb6254.1747719873.git.mchehab+huawei@kernel.org/

2. Now that we're not opening a shell anymore, if kernel-doc.py
   dies, it could cause a fatal error on Sphinx. I sent today
   two patches fixing it:

   2.1 Don't stop Documentation/sphinx/kerneldoc.py if a crash
       happens:

	https://lore.kernel.org/linux-doc/064bac2f462c13f56154891d8f3fb788db94f325.1747730982.git.mchehab+huawei@kernel.org/

   2.2 Don't crash trying to retrieve kernel-doc messages from
       a file that doesn't exist. This is patch 1/2 on this
       series:
	https://lore.kernel.org/lkml/d97e86c7176f671405b4c15d75cb951349022a23.1747730982.git.mchehab+huawei@kernel.org/

> So, please, now that you are producing more warnings, make a missing
> file just a warning (or at least continue on).  I have enough to do
> every day without having to run "make htmldocs" more than once.

OK. As I said, the crash was unintended. The idea was to preserve
the old behavior as much as possible. Yet, it could probably be a
good idea to have a way in the future to control the docs behavior
for the cases where non-existing files are detected, allowing it to
stop quickly on such cases, instead of taking several minutes to
generate an html ouptut, as this could be useful for CI automation.

In order to simplify for Jon, I'll send those tree altogether(*).

(*) I'll add a 4th patch, adding scripts/lib/kdoc to linux-doc
content. Without that, Lore doesn't store kernel-doc changes as
part of docs. See: 
	https://lore.kernel.org/linux-doc/d97e86c7176f671405b4c15d75cb951349022a23.1747730982.git.mchehab+huawei@kernel.org/

Regards,
Mauro