Re: [TECH TOPIC] Kernel documentation - update and future directions

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Vegard Nossum]

(Added linux-doc and some more people to Cc)

On 30/08/2025 15:37, Jonathan Corbet wrote:
> Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:
> 
>> Hi Jon,
>>
>> On Fri, Aug 22, 2025 at 04:55:51PM -0600, Jonathan Corbet wrote:
>>> The last year has seen a massive amount of work in our documentation
>>> build infrastructure and continuous improvement in the docs themselves.
>>> This session will provide a brief update of what has happened recently,
>>> followed by discussion on what we want to do next.  How would we like
>>> our documentation to evolve in the next year or three?
>>
>> One area that I think could be improved is making documentation more
>> accessible, in particular to newcomers. We have a really impressive (and
>> ever increasing) amount of documentation that has mostly grown in an
>> organic fashion. As a consequence, many answers can be found when one
>> knows what they're searching for, but reading documentation is painful
>> for newcomers. It doesn't flow naturally, and lots of concepts are used
>> before being introduced much later (or in entirely different locations).
> 
> Trust me, I get it.  That's why I have pushed so hard to try to organize
> the docs with the intended reader in mind.  I think that has worked out
> well but, so far, the main effect has been to take a massive unorganized
> pile of stuff and arrange it into several pile of stuff, hopefully with
> slightly better organization.

+100 for continuing to organize the docs with the intended reader in mind.

If I may refer to my old email from the corresponding 2023 tech topic
thread, which also discusses the structure somewhat:

https://lore.kernel.org/all/e79d53e2-4a1a-4f7e-850c-7f412ba43d35@oracle.com/

"""
On the topic of the overall structure of the documentation: [4]
describes the idea that the kernel documentation is set of "books" --
user and admin guide, core API, drivers API, userspace API. I think this
needs to be emphasized more, as that _is_ the (philosophy of the)
current high-level organization of the documentation and it feels a bit
hidden where it currently is; maybe it should be placed prominently at
the top of that file and called "Organization and philosophy" or
something. At least I was very confused when I came across a passage
that read something like "This book covers ..." and I had no idea why a
kernel document was talking about books.

[4] 
https://docs.kernel.org/doc-guide/contributing.html#documentation-coherency
"""

> Occasionally I make an attempt to attack one of the top-level books and
> create a bit more order there.  But my teaspoon is going to take a while
> to drain that ocean.

I took a very small stab at organizing stuff in the right places last year:

https://lore.kernel.org/all/20240104160946.3450743-1-vegard.nossum@oracle.com/

I probably should have spun a v2 and pushed harder to get things done
but it might be worth taking a step back and try to analyze what
happened in this thread. As a submitter, it's hard to know how long to
wait for comments from others before sending a v2, it's not clear
whether people's comments are meant as improvement suggestions, if they
consider them blockers... should the maintainer be chiming in? Etc.

In general, it might be worth merging docs patches more aggressively and
requesting incremental fixups for non-critical things (it's not like
docs patches will ever result in unbootable kernels or corrupted
filesystems). This way we keep the flow going and don't get contributors
stuck on waiting, rebasing, resolving any conflicts that might appear in
the interim, and resubmitting... for what might be relatively minor issues.

At least I have a tendency to simply drop it and move on if my patches
meet resistance (and perhaps especially if the patches are relatively
inconsequential). I admit that this is largely my own fault, but I'm
guessing I'm not the only one either.

Another example that I don't think ever got merged (even with an ack
from Randy?), though admittedly that was RFC:

https://lore.kernel.org/all/e398ebb1-1d42-49ff-b355-b4bc3258fc10@oracle.com/#t

Looking at my local branches, I have a bunch of restructuring patches
that never even got sent out because the first parts were stuck in
limbo. Again, probably mostly my fault, but what do I do differently?

>> While some documents are clearly meant to be reference material, other
>> target developers who are not familiar with the topic being described.
>> They would benefit from being written in linear, story-telling way. I
>> don't know how to best achieve that though: developers writing any kind
>> of documentation in the first place is already an achievement, and
>> writing the documentation while putting yourself in the shoes of someone
>> not familiar with the topic is not an easy task.
> 
> It is common to divide technical documentation into four broad
> categories: tutorials (for learning), howtos (getting tasks done),
> explanation (understanding what's going on), and reference.  Each is
> aimed at a different audience.
> 
> Most of what we have is reference.  There's an occasional howto, and
> some explanation in spots.  We don't have much in the way of tutorials.
> 
> It would be nice to (1) recognize those categories in the organization
> of our documentation, and (2) fill them out somewhat.  But, as you note,
> getting people to do that is hard.  Doing it properly requires somebody
> whose job is to create that sort of material...and, as I've harped on
> for years:
> 
> 	Despite the fact that there are large number of people paid to
> 	work on the kernel, there is not a single person whose job is to
> 	work on kernel documentation.
> 
> Last year we tried an experiment with a bit of funding from the LF to
> create a bit of paid documentation; for a number of reasons, that
> experiment did not work out.  But it seems there should be a way to make
> some forward progress on this front.

I don't know if this has been suggested before, we seem to have a number
of people who are interested in documentation in its own right and I was
wondering if we could do more for community building around it: monthly
zoom call (which some other subsystems/interest groups do), IRC/Matrix/
Discord channel, a roadmap for docs (KernelNewbies project?).

(Speaking of which, why isn't linux-doc@ Cc'ed on this thread?)

I would personally be very happy to see a slightly less formal place
off-list to throw out some patches for quick feedback before submitting
them on the list so that I don't spend hours on something that's going
be met with either a wall of silence and zero enthusiasm or what I would
call mildly discouraging comments. Perhaps it sounds a bit hypocritical,
after all I have barely reviewed or acked any docs patches myself, but I
think it actually goes both ways: I was really happy to see the
kerneldoc perl-to-Python conversion (and the subsequent cleanups) but I
didn't have the time to look at it in detail at the time and chiming in
just to say "I like this" felt like I would just be adding noise.

Returning to the topic of getting people to do stuff, I think Jani (or
somebody, I forget who/where/when) suggested using the 'todo' Sphinx
plugin at some point, it basically lets you add todo:: nodes in .rst
(which can be rendered or hidden), which might be a good way to track
stuff that still needs to be done in docs land without having to do it
all at once -- and maybe draw in some contributions from others who come
across them?

Anyway, I don't mean to complain too much. Lots of great progress has
been made. Thanks Jon, Mauro, Randy, Bagas, and others -- good work!


Vegard

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Laurent Pinchart]

On Sat, Aug 30, 2025 at 06:00:42PM +0200, Vegard Nossum wrote:
> 
> (Added linux-doc and some more people to Cc)
> 
> On 30/08/2025 15:37, Jonathan Corbet wrote:
> > Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:
> >> On Fri, Aug 22, 2025 at 04:55:51PM -0600, Jonathan Corbet wrote:
> >>> The last year has seen a massive amount of work in our documentation
> >>> build infrastructure and continuous improvement in the docs themselves.
> >>> This session will provide a brief update of what has happened recently,
> >>> followed by discussion on what we want to do next.  How would we like
> >>> our documentation to evolve in the next year or three?
> >>
> >> One area that I think could be improved is making documentation more
> >> accessible, in particular to newcomers. We have a really impressive (and
> >> ever increasing) amount of documentation that has mostly grown in an
> >> organic fashion. As a consequence, many answers can be found when one
> >> knows what they're searching for, but reading documentation is painful
> >> for newcomers. It doesn't flow naturally, and lots of concepts are used
> >> before being introduced much later (or in entirely different locations).
> > 
> > Trust me, I get it.  That's why I have pushed so hard to try to organize
> > the docs with the intended reader in mind.  I think that has worked out
> > well but, so far, the main effect has been to take a massive unorganized
> > pile of stuff and arrange it into several pile of stuff, hopefully with
> > slightly better organization.

It's been a clear a noticeable improvement, even if there's still work
to do. I trust that you understand the issue.

> +100 for continuing to organize the docs with the intended reader in mind.
> 
> If I may refer to my old email from the corresponding 2023 tech topic
> thread, which also discusses the structure somewhat:
> 
> https://lore.kernel.org/all/e79d53e2-4a1a-4f7e-850c-7f412ba43d35@oracle.com/
> 
> """
> On the topic of the overall structure of the documentation: [4]
> describes the idea that the kernel documentation is set of "books" --
> user and admin guide, core API, drivers API, userspace API. I think this
> needs to be emphasized more, as that _is_ the (philosophy of the)
> current high-level organization of the documentation and it feels a bit
> hidden where it currently is; maybe it should be placed prominently at
> the top of that file and called "Organization and philosophy" or
> something. At least I was very confused when I came across a passage
> that read something like "This book covers ..." and I had no idea why a
> kernel document was talking about books.
> 
> [4] 
> https://docs.kernel.org/doc-guide/contributing.html#documentation-coherency
> """
> 
> > Occasionally I make an attempt to attack one of the top-level books and
> > create a bit more order there.  But my teaspoon is going to take a while
> > to drain that ocean.
> 
> I took a very small stab at organizing stuff in the right places last year:
> 
> https://lore.kernel.org/all/20240104160946.3450743-1-vegard.nossum@oracle.com/
> 
> I probably should have spun a v2 and pushed harder to get things done
> but it might be worth taking a step back and try to analyze what
> happened in this thread. As a submitter, it's hard to know how long to
> wait for comments from others before sending a v2, it's not clear
> whether people's comments are meant as improvement suggestions, if they
> consider them blockers... should the maintainer be chiming in? Etc.
> 
> In general, it might be worth merging docs patches more aggressively and
> requesting incremental fixups for non-critical things (it's not like
> docs patches will ever result in unbootable kernels or corrupted
> filesystems). This way we keep the flow going and don't get contributors
> stuck on waiting, rebasing, resolving any conflicts that might appear in
> the interim, and resubmitting... for what might be relatively minor issues.
> 
> At least I have a tendency to simply drop it and move on if my patches
> meet resistance (and perhaps especially if the patches are relatively
> inconsequential). I admit that this is largely my own fault, but I'm
> guessing I'm not the only one either.

I wouldn't use the word "fault" here, I don't think you should take any
blame for this. Looking at that particulat patch series, I think sending
a v2 would have helped moving forward.

> Another example that I don't think ever got merged (even with an ack
> from Randy?), though admittedly that was RFC:
> 
> https://lore.kernel.org/all/e398ebb1-1d42-49ff-b355-b4bc3258fc10@oracle.com/#t

For this one, Jon replied with a comment, and the discussion died out.
Replying to that could have helped.

> Looking at my local branches, I have a bunch of restructuring patches
> that never even got sent out because the first parts were stuck in
> limbo. Again, probably mostly my fault, but what do I do differently?

In general I'd say you shouldn't be afraid to ping.

> >> While some documents are clearly meant to be reference material, other
> >> target developers who are not familiar with the topic being described.
> >> They would benefit from being written in linear, story-telling way. I
> >> don't know how to best achieve that though: developers writing any kind
> >> of documentation in the first place is already an achievement, and
> >> writing the documentation while putting yourself in the shoes of someone
> >> not familiar with the topic is not an easy task.
> > 
> > It is common to divide technical documentation into four broad
> > categories: tutorials (for learning), howtos (getting tasks done),
> > explanation (understanding what's going on), and reference.  Each is
> > aimed at a different audience.
> > 
> > Most of what we have is reference.  There's an occasional howto, and
> > some explanation in spots.  We don't have much in the way of tutorials.
> > 
> > It would be nice to (1) recognize those categories in the organization
> > of our documentation, and (2) fill them out somewhat.  But, as you note,
> > getting people to do that is hard.  Doing it properly requires somebody
> > whose job is to create that sort of material...and, as I've harped on
> > for years:
> > 
> > 	Despite the fact that there are large number of people paid to
> > 	work on the kernel, there is not a single person whose job is to
> > 	work on kernel documentation.
> > 
> > Last year we tried an experiment with a bit of funding from the LF to
> > create a bit of paid documentation; for a number of reasons, that
> > experiment did not work out.  But it seems there should be a way to make
> > some forward progress on this front.

Is there anything we can learn from that failure and that number of
reasons to make the next attempt more successful ?

> I don't know if this has been suggested before, we seem to have a number
> of people who are interested in documentation in its own right and I was
> wondering if we could do more for community building around it: monthly
> zoom call (which some other subsystems/interest groups do), IRC/Matrix/
> Discord channel, a roadmap for docs (KernelNewbies project?).
> 
> (Speaking of which, why isn't linux-doc@ Cc'ed on this thread?)

Just an oversight I suppose.

> I would personally be very happy to see a slightly less formal place
> off-list to throw out some patches for quick feedback before submitting
> them on the list so that I don't spend hours on something that's going
> be met with either a wall of silence and zero enthusiasm or what I would
> call mildly discouraging comments.

When I'm really not sure if something is a good idea, or when I'm
convinced it's a good idea but I fear other may disagree, I often ask
relevant maintainers and developers on IRC. I don't know if there's an
IRC channel for the Linux kernel documentation, if there are enough
people interested in the topic, it could be useful.

> Perhaps it sounds a bit hypocritical,
> after all I have barely reviewed or acked any docs patches myself, but I
> think it actually goes both ways: I was really happy to see the
> kerneldoc perl-to-Python conversion (and the subsequent cleanups) but I
> didn't have the time to look at it in detail at the time and chiming in
> just to say "I like this" felt like I would just be adding noise.
> 
> Returning to the topic of getting people to do stuff, I think Jani (or
> somebody, I forget who/where/when) suggested using the 'todo' Sphinx
> plugin at some point, it basically lets you add todo:: nodes in .rst
> (which can be rendered or hidden), which might be a good way to track
> stuff that still needs to be done in docs land without having to do it
> all at once -- and maybe draw in some contributions from others who come
> across them?

I really don't know if we could count on newcomers interested in getting
into kernel development to start their journey in Documentation/. I can
imagine in theory that someone who tries to follow our documentation and
finds it suboptimal could send patches as a side product of whatever
work they're doing (without wanting to show off, that's how I wrote the
initial documentation for the KMS in-kernel API: I had to write a KMS
driver, had no idea how it worked, and there was no doc), but that will
be a rare occurence and most probably would come from an experienced
kernel contributor. We seem to scare lots of new contributors in
general, so I have a hard time imagining they would actively work on
documentation improvements. There could be exception, but I don't think
we should count on them.

> Anyway, I don't mean to complain too much. Lots of great progress has
> been made. Thanks Jon, Mauro, Randy, Bagas, and others -- good work!

-- 
Regards,

Laurent Pinchart

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jonathan Corbet]

Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:

>> > Last year we tried an experiment with a bit of funding from the LF to
>> > create a bit of paid documentation; for a number of reasons, that
>> > experiment did not work out.  But it seems there should be a way to make
>> > some forward progress on this front.
>
> Is there anything we can learn from that failure and that number of
> reasons to make the next attempt more successful ?

I think that the experiment didn't work for a couple of reasons:

- The topic area that we settled upon was a relatively advanced one, we
  really should have started with something simpler.

- The writer who was assigned was not really up to the task; I found
  myself repeatedly having to explain basic aspects of the C programming
  language, for example.  That made it almost impossible to get a
  satisfactory document out of the process, made worse by the first
  reason listed above.
  
What it comes down to, perhaps, is the same old problem: the people who
understand the problem domain well enough to document it can generally
make a more comfortable living creating more undocumented code instead.

jon

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Sat, 30 Aug 2025 17:08:53 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:
> 
> >> > Last year we tried an experiment with a bit of funding from the LF to
> >> > create a bit of paid documentation; for a number of reasons, that
> >> > experiment did not work out.  But it seems there should be a way to make
> >> > some forward progress on this front.  
> >
> > Is there anything we can learn from that failure and that number of
> > reasons to make the next attempt more successful ?  
> 
> I think that the experiment didn't work for a couple of reasons:
> 
> - The topic area that we settled upon was a relatively advanced one, we
>   really should have started with something simpler.
> 
> - The writer who was assigned was not really up to the task; I found
>   myself repeatedly having to explain basic aspects of the C programming
>   language, for example.  That made it almost impossible to get a
>   satisfactory document out of the process, made worse by the first
>   reason listed above.

The way I see, there are several orthogonal tasks.

1) Book set organization

The first one would almost certainly require LF or someone else sponsor
a person with lots of experience on organizing and reviewing technical
books that worked on similar projects.

Such person would be starting from top to down, organizing the books
on a way that the basic knowledge would be covered at the first chapters,
and more advanced items at the end, and missing chapters will contain
skeleton ReST files on it to be filled by someone else. After having
it, review the text, specially for the less technical chapters, to
ensure that it is accessible to kernel newbies.

2) uAPI documentation

This requires highly skilled people or advanced userspace developers,
and should be done subsystem by subsystem. 

We did that on media several years ago, and, although not perfect, 
I do think that we cover almost everything for uAPI, with examples, 
tables images etc.

From my experience, uAPI is easier to document than kAPI, as it doesn't
change that much (not counting sysfs/debugfs/configfs).

On media, we started enforcing documentation for all new media uAPI.
We then reviewed the gaps and filled in the blanks.

3) ABI documentation: sysfs, configfs, debugfs

Such ABIs are a different best: we have dozens of thousands of sysfs symbols
on a server. Last time I checked, most undocumented.

I wrote a function at get_abi tool to help finding the gaps: it can help
identifying missing gaps by converting ABI descriptions into regular
expressions and check against the real sysfs nodes found at the system(*). 

I'd say that someone without intermediate C knowledge can probably use it
to produce a lot of missing ABI documentation.

(*) The tool itself can easily be modified to also handle debugfs/configfs,
    but there are too many gaps at sysfs. I would start with them.

4) kernel-doc kAPI

It shouldn't be that hard to do the same for kernel-doc kAPI documentation:
kernel-doc now can parse the entire tree with:

	$ scripts/kernel-doc .

Someone can easily use it to discover the current gaps at the docs that
have already some kernel-doc markups and identify what of them aren't
yet placed under Documentation/ ".. kernel-doc::" markups.

So, I'd say the first step here would be to ensure that 100% of the
docs are there somewhere. Alternatively, we could place all the rest
of functions with kernel-doc markups outside Documentation inside an
"others/" book - or even "<subsystem>/others/", and then gradually move
them to the right places.

5) kAPI itself

Except for trivial cases, I don't think that only kernel-doc is
enough. On most of the cases, a document describing the main
concepts and the design behind the kAPI is needed. Almost certainly
only core developers and maintainers within each subsystem can
write those.

> What it comes down to, perhaps, is the same old problem: the people who
> understand the problem domain well enough to document it can generally
> make a more comfortable living creating more undocumented code instead.

True, but I guess this is valid mostly for (5).

Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> 4) kernel-doc kAPI
>
> It shouldn't be that hard to do the same for kernel-doc kAPI documentation:
> kernel-doc now can parse the entire tree with:
>
> 	$ scripts/kernel-doc .
>
> Someone can easily use it to discover the current gaps at the docs that
> have already some kernel-doc markups and identify what of them aren't
> yet placed under Documentation/ ".. kernel-doc::" markups.

...or one can use scripts/find-unused-docs.sh, which was written for
just this purpose :)

This sort of discoverability is part of why I want to move documentation
tooling into its own directory.

jon

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Randy Dunlap]

[-- Attachment #1: Type: text/plain, Size: 3172 bytes --]



On 8/31/25 1:16 PM, Jonathan Corbet wrote:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
>> 4) kernel-doc kAPI
>>
>> It shouldn't be that hard to do the same for kernel-doc kAPI documentation:
>> kernel-doc now can parse the entire tree with:
>>
>> 	$ scripts/kernel-doc .
>>
>> Someone can easily use it to discover the current gaps at the docs that
>> have already some kernel-doc markups and identify what of them aren't
>> yet placed under Documentation/ ".. kernel-doc::" markups.

Mauro, I tried that for one file: kernel/audit.c
but didn't see what I expected to see.
What options should I be using to find the gaps?

> ...or one can use scripts/find-unused-docs.sh, which was written for
> just this purpose :)

Yes, and I have used this script. It does what it was meant to do AFAIK.
It's reporting is at a gross file level.

I made a small subdirectory called "test" and copied kernel/audit.c to test/.

$ ./scripts/find-unused-docs.sh  test/
The following files contain kerneldoc comments for exported functions that are not used in the formatted documentation
test/audit.c

Sometime in the last 2-3 years Matthew Wilcox asked me about a tool (script, whatever)that would detect both EXPORTs without kernel-doc and kernel-doc without EXPORTs.
Either one of these can be noisy (with false positives) and they often don't lend
themselves to easy/beginner fixes.

Anyway, after some delay, I have such a script. It's written in Perl (I started
on it over a year ago!). It might have been desirable to add it to scripts/kernel-doc.pl
at the time, but it didn't seem to me like a good fit there, so it's independent.

Running (no options, just produce a summary)
$ kerndoc-export-search.pl test/audit.c
reports:
Missing kernel-doc for: audit_log_task_info
Missing kernel-doc for: audit_enabled
Missing kernel-doc for: audit_log_task_context
3 missing kernel-docs
Missing EXPORT for: audit_serial
Missing EXPORT for: audit_log_untrustedstring
Missing EXPORT for: audit_log_n_untrustedstring
Missing EXPORT for: audit_log_n_hex
Missing EXPORT for: audit_log_lost
Missing EXPORT for: audit_set_loginuid
Missing EXPORT for: auditd_test_task
Missing EXPORT for: audit_ctl_lock
Missing EXPORT for: audit_string_contains_control
Missing EXPORT for: audit_signal_info
Missing EXPORT for: audit_log_path_denied
Missing EXPORT for: audit_ctl_unlock
12 missing Exports

If that's not verbose enough (!), the -l (list) option lists each function's
location and short description. One example:
test/audit.c: 2006: audit_log_format:  * audit_log_format - format a message into the audit buffer.

But that generates lots of output.

And of course, for function, I mean function/struct/union/enum/typedef.

There is a "verbose" option but it currently does not print anything.

Here is a help summary:
$ kerndoc-export-search.pl -h
kerndoc-export-search.pl [--list|-l] [--verbose|-v] file(s)
  where --list    prints filename:line:funcname: short_description
  where --verbose prints more info.
  default: prints a doc/export summary + warnings.
  version: 0.9


Feel free to use in any way or to rewrite & merge it into the
kdoc python system.

FWIW.
-- 
~Randy

[-- Attachment #2: kerndoc-export-search.pl --]
[-- Type: application/x-perl, Size: 13692 bytes --]

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jani Nikula]

On Sun, 31 Aug 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> It shouldn't be that hard to do the same for kernel-doc kAPI documentation:
> kernel-doc now can parse the entire tree with:
>
> 	$ scripts/kernel-doc .
>
> Someone can easily use it to discover the current gaps at the docs that
> have already some kernel-doc markups and identify what of them aren't
> yet placed under Documentation/ ".. kernel-doc::" markups.
>
> So, I'd say the first step here would be to ensure that 100% of the
> docs are there somewhere. Alternatively, we could place all the rest
> of functions with kernel-doc markups outside Documentation inside an
> "others/" book - or even "<subsystem>/others/", and then gradually move
> them to the right places.

I don't agree that all the kernel-docs need to be in the html build in
the first place. Some of them would be better off with a simple
non-structured comment instead. For example, most static functions. Some
of the kernel-docs are useful for the structure the format provides, but
still having them in the html build is overkill. For example, many
complex but driver specific functions.

I think the API documentation in the Sphinx build is primarily useful
for kernel generic and subsystem APIs, or overviews of
functionality. But nobody's looking at the Sphinx build for highly
specific and isolated documentation for individual structures or
functions.

I'd say emphasize quality over quantity in the Sphinx build. An
overwhelming amount of (in the big picture) insignificant API
documentation does not make for good documentation.

That said, there *are* a lot of kernel-doc comments that absolutely
should be pulled into the Sphinx build. But don't be indiscriminate
about it.

---

I think a more interesting first step would be ensuring all the
kernel-docs we do have are free of kernel-doc and rst warnings. Because
they should be, and this would make them easier to pull into the Sphinx
build as needed.

Currently we only have the kernel-doc checks in W=1 builds for .c
files.

The i915 and xe drivers have local Makefile hacks to do it for more than
just W=1 builds and also headers. The attempts to expand the header
checks to the drm subsystem, however, failed infamously.

And still none of this checks for rst. But now that kernel-doc is
python, it shouldn't be too hard. Probably needs a dependency, but it
could only depend on it when passing some --lint-rst option.

Having this in place would also reduce the churn caused by merging
broken kernel-doc. It's fast enough to be done as part of the regular
build, while most people don't run the entire Sphinx build as part of
the development flow.


BR,
Jani.


-- 
Jani Nikula, Intel

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Randy Dunlap]

On 9/1/25 3:09 AM, Jani Nikula wrote:
> On Sun, 31 Aug 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>> It shouldn't be that hard to do the same for kernel-doc kAPI documentation:
>> kernel-doc now can parse the entire tree with:
>>
>> 	$ scripts/kernel-doc .
>>
>> Someone can easily use it to discover the current gaps at the docs that
>> have already some kernel-doc markups and identify what of them aren't
>> yet placed under Documentation/ ".. kernel-doc::" markups.
>>
>> So, I'd say the first step here would be to ensure that 100% of the
>> docs are there somewhere. Alternatively, we could place all the rest
>> of functions with kernel-doc markups outside Documentation inside an
>> "others/" book - or even "<subsystem>/others/", and then gradually move
>> them to the right places.
> 
> I don't agree that all the kernel-docs need to be in the html build in
> the first place. Some of them would be better off with a simple
> non-structured comment instead. For example, most static functions. Some
> of the kernel-docs are useful for the structure the format provides, but
> still having them in the html build is overkill. For example, many
> complex but driver specific functions.


IMO there are far too many static functions that use kernel-doc notation.
I certainly don't want to discourage function documentation, but I don't
think there was any ever intent to have those functions added to the
kernel docbooks.


> I think the API documentation in the Sphinx build is primarily useful
> for kernel generic and subsystem APIs, or overviews of
> functionality. But nobody's looking at the Sphinx build for highly
> specific and isolated documentation for individual structures or
> functions.
> 
> I'd say emphasize quality over quantity in the Sphinx build. An
> overwhelming amount of (in the big picture) insignificant API
> documentation does not make for good documentation.
> 
> That said, there *are* a lot of kernel-doc comments that absolutely
> should be pulled into the Sphinx build. But don't be indiscriminate
> about it.
> 
> ---
> 
> I think a more interesting first step would be ensuring all the
> kernel-docs we do have are free of kernel-doc and rst warnings. Because
> they should be, and this would make them easier to pull into the Sphinx
> build as needed.

ISTM that there are lots of non-docs developers who either just don't care
about that or never run 'make W=1 htmldocs' to see the problems in their
drivers or subsystems. OK, maybe it's just a very low priority for them.

Willy had a suggestion that we just make checking kernel-doc during
all .c builds a permanent feature instead of a W=1 option.
This helps, but still doesn't force 'make htmldocs' to be run.

And it causes around 450 build warnings in my testing of an x86_64 allmodconfig
build.

> Currently we only have the kernel-doc checks in W=1 builds for .c
> files.
> 
> The i915 and xe drivers have local Makefile hacks to do it for more than
> just W=1 builds and also headers. The attempts to expand the header
> checks to the drm subsystem, however, failed infamously.
> 
> And still none of this checks for rst. But now that kernel-doc is
> python, it shouldn't be too hard. Probably needs a dependency, but it
> could only depend on it when passing some --lint-rst option.
> 
> Having this in place would also reduce the churn caused by merging
> broken kernel-doc. It's fast enough to be done as part of the regular
> build, while most people don't run the entire Sphinx build as part of
> the development flow.

-- 
~Randy

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

On Mon, Sep 01, 2025 at 09:51:01AM -0700, Randy Dunlap wrote:

> Willy had a suggestion that we just make checking kernel-doc during
> all .c builds a permanent feature instead of a W=1 option.
> This helps, but still doesn't force 'make htmldocs' to be run.

make htmldocs is rather slow:

  $ time make -j56 htmldocs
  ...
  make -j56 htmldocs  2355.99s user 141.33s system 158% cpu 26:14.86 total

and produces a bunch of warnings with current mainline it seems.  That
compares unfavourably with allmodconfig on this system:

  $ time make -j56 allmodconfig
  ...
  make -j56 allmodconfig  5.31s user 1.93s system 146% cpu 4.931 total
  $ time make -j56
  ...
  make -j56  53468.11s user 4387.30s system 5084% cpu 18:57.77 total

and seems rather more likely to flag something for me.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Randy Dunlap]

Hi,

On 9/1/25 10:52 AM, Mark Brown wrote:
> On Mon, Sep 01, 2025 at 09:51:01AM -0700, Randy Dunlap wrote:
> 
>> Willy had a suggestion that we just make checking kernel-doc during
>> all .c builds a permanent feature instead of a W=1 option.
>> This helps, but still doesn't force 'make htmldocs' to be run.
> 
> make htmldocs is rather slow:
> 
>   $ time make -j56 htmldocs
>   ...
>   make -j56 htmldocs  2355.99s user 141.33s system 158% cpu 26:14.86 total

Wow. I just ran 'make -j8 htmldocs' on my 16 GB intel core-i7 laptop:

1205.85user 66.15system 6:05.82elapsed 347%CPU (0avgtext+0avgdata 1298044maxresident)k
697432inputs+1128016outputs (92major+16863601minor)pagefaults 0swaps

(using /usr/bin/time)

How much RAM does your build system have?

> 
> and produces a bunch of warnings with current mainline it seems.  That
> compares unfavourably with allmodconfig on this system:
> 
>   $ time make -j56 allmodconfig
>   ...
>   make -j56 allmodconfig  5.31s user 1.93s system 146% cpu 4.931 total
>   $ time make -j56
>   ...
>   make -j56  53468.11s user 4387.30s system 5084% cpu 18:57.77 total

whereas my x86_64 allmodconfig build takes over 1 hour.  :(

> and seems rather more likely to flag something for me.

OK, good to hear. Thanks.

-- 
~Randy

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

On Mon, Sep 01, 2025 at 11:15:50AM -0700, Randy Dunlap wrote:
> On 9/1/25 10:52 AM, Mark Brown wrote:

> > make htmldocs is rather slow:

> >   $ time make -j56 htmldocs
> >   ...
> >   make -j56 htmldocs  2355.99s user 141.33s system 158% cpu 26:14.86 total

> Wow. I just ran 'make -j8 htmldocs' on my 16 GB intel core-i7 laptop:

> 1205.85user 66.15system 6:05.82elapsed 347%CPU (0avgtext+0avgdata 1298044maxresident)k
> 697432inputs+1128016outputs (92major+16863601minor)pagefaults 0swaps

> How much RAM does your build system have?

64G, it's not swapping at all or anything.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jonathan Corbet]

Mark Brown <broonie@kernel.org> writes:

> On Mon, Sep 01, 2025 at 09:51:01AM -0700, Randy Dunlap wrote:
>
>> Willy had a suggestion that we just make checking kernel-doc during
>> all .c builds a permanent feature instead of a W=1 option.
>> This helps, but still doesn't force 'make htmldocs' to be run.
>
> make htmldocs is rather slow:
>
>   $ time make -j56 htmldocs
>   ...
>   make -j56 htmldocs  2355.99s user 141.33s system 158% cpu 26:14.86 total

That ... is weird... it takes me a little under 3 minutes to do an
htmldocs build, using a capable but not stellar desktop machine.

Which version of Sphinx are you using?  If you're not on Sphinx 8, you
really want to be; they finally fixed some really nasty performance
problems with that release.

Thanks,

jon

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Mon, 01 Sep 2025 13:09:15 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Sun, 31 Aug 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > It shouldn't be that hard to do the same for kernel-doc kAPI documentation:
> > kernel-doc now can parse the entire tree with:
> >
> > 	$ scripts/kernel-doc .
> >
> > Someone can easily use it to discover the current gaps at the docs that
> > have already some kernel-doc markups and identify what of them aren't
> > yet placed under Documentation/ ".. kernel-doc::" markups.
> >
> > So, I'd say the first step here would be to ensure that 100% of the
> > docs are there somewhere. Alternatively, we could place all the rest
> > of functions with kernel-doc markups outside Documentation inside an
> > "others/" book - or even "<subsystem>/others/", and then gradually move
> > them to the right places.  
> 
> I don't agree that all the kernel-docs need to be in the html build in
> the first place.

Not all, but those that are part of the kAPI requires good documentation.

> Some of them would be better off with a simple
> non-structured comment instead. For example, most static functions. Some
> of the kernel-docs are useful for the structure the format provides, but
> still having them in the html build is overkill. For example, many
> complex but driver specific functions.

Driver-specific functions could remain out of doc build - or be part
of the documentation. It should be a decision of the driver authors,
that may or may not be expecting contributions from the community.

> I think the API documentation in the Sphinx build is primarily useful
> for kernel generic and subsystem APIs, or overviews of
> functionality. But nobody's looking at the Sphinx build for highly
> specific and isolated documentation for individual structures or
> functions.
> 
> I'd say emphasize quality over quantity in the Sphinx build. An
> overwhelming amount of (in the big picture) insignificant API
> documentation does not make for good documentation.
> 
> That said, there *are* a lot of kernel-doc comments that absolutely
> should be pulled into the Sphinx build. But don't be indiscriminate
> about it.

Agreed. What I said is that this is a good start point, as it sounds
to me that we do have kAPI documentation inside headers but not
exported to the documentation.

> ---
> 
> I think a more interesting first step would be ensuring all the
> kernel-docs we do have are free of kernel-doc and rst warnings. 

Agreed. Things look better those days, but just because right now
there are several warnings disabled by default.

> Because they should be, and this would make them easier to pull into 
> the Sphinx build as needed.
> 
> Currently we only have the kernel-doc checks in W=1 builds for .c
> files.
> 
> The i915 and xe drivers have local Makefile hacks to do it for more than
> just W=1 builds and also headers. The attempts to expand the header
> checks to the drm subsystem, however, failed infamously.

On media, our CI builds with W=1, and aim to have no warnings.

> And still none of this checks for rst. But now that kernel-doc is
> python, it shouldn't be too hard. Probably needs a dependency, but it
> could only depend on it when passing some --lint-rst option.

Good idea. If you have some time, feel free to propose some patches.

> Having this in place would also reduce the churn caused by merging
> broken kernel-doc. It's fast enough to be done as part of the regular
> build, while most people don't run the entire Sphinx build as part of
> the development flow.

Checking the entire set of files inside the Kernel with kernel-doc
is fast. Using the new make mandocs, for instance, with reads the
entire tree takes about 45 seconds on my machine:

	$ time make mandocs
	...
	real    0m44,211s
	user    0m35,787s
	sys     0m3,274s

(and reports thousands of warnings)

Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Mon, 1 Sep 2025 09:51:01 -0700
Randy Dunlap <rdunlap@infradead.org> escreveu:

> On 9/1/25 3:09 AM, Jani Nikula wrote:
> > On Sun, 31 Aug 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:  
> >> It shouldn't be that hard to do the same for kernel-doc kAPI documentation:
> >> kernel-doc now can parse the entire tree with:
> >>
> >> 	$ scripts/kernel-doc .
> >>
> >> Someone can easily use it to discover the current gaps at the docs that
> >> have already some kernel-doc markups and identify what of them aren't
> >> yet placed under Documentation/ ".. kernel-doc::" markups.
> >>
> >> So, I'd say the first step here would be to ensure that 100% of the
> >> docs are there somewhere. Alternatively, we could place all the rest
> >> of functions with kernel-doc markups outside Documentation inside an
> >> "others/" book - or even "<subsystem>/others/", and then gradually move
> >> them to the right places.  
> > 
> > I don't agree that all the kernel-docs need to be in the html build in
> > the first place. Some of them would be better off with a simple
> > non-structured comment instead. For example, most static functions. Some
> > of the kernel-docs are useful for the structure the format provides, but
> > still having them in the html build is overkill. For example, many
> > complex but driver specific functions.  
> 
> 
> IMO there are far too many static functions that use kernel-doc notation.
> I certainly don't want to discourage function documentation, but I don't
> think there was any ever intent to have those functions added to the
> kernel docbooks.

Sure, but on the other hand, lots of those have warnings. Even if
not part of htmldocs, IMO the kernel-doc markups should reflect the
actual code.

> > I think the API documentation in the Sphinx build is primarily useful
> > for kernel generic and subsystem APIs, or overviews of
> > functionality. But nobody's looking at the Sphinx build for highly
> > specific and isolated documentation for individual structures or
> > functions.
> > 
> > I'd say emphasize quality over quantity in the Sphinx build. An
> > overwhelming amount of (in the big picture) insignificant API
> > documentation does not make for good documentation.
> > 
> > That said, there *are* a lot of kernel-doc comments that absolutely
> > should be pulled into the Sphinx build. But don't be indiscriminate
> > about it.
> > 
> > ---
> > 
> > I think a more interesting first step would be ensuring all the
> > kernel-docs we do have are free of kernel-doc and rst warnings. Because
> > they should be, and this would make them easier to pull into the Sphinx
> > build as needed.  
> 
> ISTM that there are lots of non-docs developers who either just don't care
> about that or never run 'make W=1 htmldocs' to see the problems in their
> drivers or subsystems. OK, maybe it's just a very low priority for them.
> 
> Willy had a suggestion that we just make checking kernel-doc during
> all .c builds a permanent feature instead of a W=1 option.
> This helps, but still doesn't force 'make htmldocs' to be run.

We can run it without actually building htmldocs. The only requirement
is to have Python 3.6 or above (this is enough to get error reports,
but 3.7 is needed to avoid having struct/function parameters out of
order).

The real problem is that, when we start doing it, Kernel build will 
have thousands of warnings. 

Perhaps one solution would be to have an image of our current
problems on a file, reporting only new stuff by default and using
WERROR policy, causing build to fail on new warnings.

This would at least avoid the problem to increase.

> And it causes around 450 build warnings in my testing of an x86_64 allmodconfig
> build.

It is a lot more if you check warnings for files with kernel-doc outside 
the htmldocs build and if you enable all kdoc warnings:

	$ time ./scripts/kernel-doc -Wall . --none 2>warnings

	real    0m33,063s
	user    0m31,233s
	sys     0m0,753s

	$ grep Warning warnings|wc -l
	36965

almost 37K warnings.


Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 878 bytes --]

On Mon, Sep 01, 2025 at 12:25:30PM -0600, Jonathan Corbet wrote:
> Mark Brown <broonie@kernel.org> writes:

> >   make -j56 htmldocs  2355.99s user 141.33s system 158% cpu 26:14.86 total

> That ... is weird... it takes me a little under 3 minutes to do an
> htmldocs build, using a capable but not stellar desktop machine.

> Which version of Sphinx are you using?  If you're not on Sphinx 8, you
> really want to be; they finally fixed some really nasty performance
> problems with that release.

I appear to be on version 5.3 which is what's in my distro.  I will get
8.1.3 when I upgrade, I don't really have any intention of manually
installing an unpackaged copy.  Three minutes would be more reasonable,
though without a clean build (and I guess something that errors on
warnings so I don't actually need to look at the output myself) I'm not
sure I'd notice any issues.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Mon, 1 Sep 2025 18:52:30 +0100
Mark Brown <broonie@kernel.org> escreveu:

> On Mon, Sep 01, 2025 at 09:51:01AM -0700, Randy Dunlap wrote:
> 
> > Willy had a suggestion that we just make checking kernel-doc during
> > all .c builds a permanent feature instead of a W=1 option.
> > This helps, but still doesn't force 'make htmldocs' to be run.  
> 
> make htmldocs is rather slow:
> 
>   $ time make -j56 htmldocs
>   ...
>   make -j56 htmldocs  2355.99s user 141.33s system 158% cpu 26:14.86 total

Sphinx itself is not too fast, but we don't need to generate
htmldocs to get kernel-doc results.

On a Razen 9 7900, kernel-doc takes about 32 seconds.

It should be noticed that kernel-doc doesn't run in parallel. Python
still suffers for a global big lock (called GIL). My attempts to run in
parallel actually made kernel-doc slower, but this is changing: the
next Python version is planning to get rid of GIL. So, maybe within
a year we can re-add the patches to run it in parallel.

> and produces a bunch of warnings with current mainline it seems. 

True.

> That
> compares unfavourably with allmodconfig on this system:
> 
>   $ time make -j56 allmodconfig
>   ...
>   make -j56 allmodconfig  5.31s user 1.93s system 146% cpu 4.931 total
>   $ time make -j56
>   ...
>   make -j56  53468.11s user 4387.30s system 5084% cpu 18:57.77 total
> 
> and seems rather more likely to flag something for me.

32 seconds more, on the top of 53468.11s doesn't sound that much.


Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1170 bytes --]

On Mon, Sep 01, 2025 at 08:46:35PM +0200, Mauro Carvalho Chehab wrote:

> It should be noticed that kernel-doc doesn't run in parallel. Python
> still suffers for a global big lock (called GIL). My attempts to run in
> parallel actually made kernel-doc slower, but this is changing: the
> next Python version is planning to get rid of GIL. So, maybe within
> a year we can re-add the patches to run it in parallel.

It'll take a lot longer for that to filter out to people's machines, for
example I'm running Debian stable on my desktop and I know a lot of
people have Ubuntu LTS.

> > That
> > compares unfavourably with allmodconfig on this system:
> > 
> >   $ time make -j56 allmodconfig
> >   ...
> >   make -j56 allmodconfig  5.31s user 1.93s system 146% cpu 4.931 total
> >   $ time make -j56
> >   ...
> >   make -j56  53468.11s user 4387.30s system 5084% cpu 18:57.77 total
> > 
> > and seems rather more likely to flag something for me.

> 32 seconds more, on the top of 53468.11s doesn't sound that much.

Yeah, if it was of that sort of order, ran clean with mainline and could
be checked automatically it'd be a lot more viable.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Andrew Lunn]

> We can run it without actually building htmldocs. The only requirement
> is to have Python 3.6 or above (this is enough to get error reports,
> but 3.7 is needed to avoid having struct/function parameters out of
> order).
> 
> The real problem is that, when we start doing it, Kernel build will 
> have thousands of warnings. 
> 
> Perhaps one solution would be to have an image of our current
> problems on a file, reporting only new stuff by default and using
> WERROR policy, causing build to fail on new warnings.
> 
> This would at least avoid the problem to increase.

netdev has a CI system which is used to try to evaluate every patch
for problems. It builds the HEAD of net-next, and counts the number of
compiler warnings, for the whole tree. If then applies the patches in
a patch series, one by one, and runs the build for each patch, and
counts the number of compiler Warnings. If the number of warnings goes
up, the test fails.

Does the kernel docs have any concept of incremental builds? Adding
one patch and rebuilding the kernel is generally fast, unless it
changes an important header. So the cost is reasonably small for two
builds. But if building the kernel documentation twice is going to
cost 6 minutes, this does not scale.

What the netdev CI also does is collect the names of the files a patch
will change. It runs ./scripts/kernel-doc -Wall -none $FILES, without
the patch, to get the number of warnings, applies the patch and does
./scripts/kernel-doc again and checks the number of warnings has not
gone up. That catches a number of undocumented new structure members
etc.

Could something similar be added to 0-day?

      Andrew

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

On Mon, Sep 01, 2025 at 09:05:19PM +0200, Andrew Lunn wrote:

> Does the kernel docs have any concept of incremental builds? Adding
> one patch and rebuilding the kernel is generally fast, unless it
> changes an important header. So the cost is reasonably small for two
> builds. But if building the kernel documentation twice is going to
> cost 6 minutes, this does not scale.

FWIW I do the same and would face similar issues, though for docs I
could probably cope with selectively building only tips of branches
given how infrequently they're touched.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Sun, 31 Aug 2025 23:17:22 -0700
Randy Dunlap <rdunlap@infradead.org> escreveu:

> On 8/31/25 1:16 PM, Jonathan Corbet wrote:
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> >   
> >> 4) kernel-doc kAPI
> >>
> >> It shouldn't be that hard to do the same for kernel-doc kAPI documentation:
> >> kernel-doc now can parse the entire tree with:
> >>
> >> 	$ scripts/kernel-doc .
> >>
> >> Someone can easily use it to discover the current gaps at the docs that
> >> have already some kernel-doc markups and identify what of them aren't
> >> yet placed under Documentation/ ".. kernel-doc::" markups.  
> 
> Mauro, I tried that for one file: kernel/audit.c
> but didn't see what I expected to see.
> What options should I be using to find the gaps?

If you want to check all warnings kernel-doc detect, use -Wall:

	$ ./scripts/kernel-doc -Wall kernel/audit.c --none
	Warning: kernel/audit.c:216 No description found for return value of 'auditd_test_task'
	Warning: kernel/audit.c:254 No description found for return value of 'audit_ctl_owner_current'
	Warning: kernel/audit.c:265 No description found for return value of 'auditd_pid_vnr'
	Warning: kernel/audit.c:289 No description found for return value of 'audit_get_sk'
	Warning: kernel/audit.c:498 No description found for return value of 'auditd_set'
	Warning: kernel/audit.c:687 No description found for return value of 'auditd_send_unicast_skb'
	Warning: kernel/audit.c:747 No description found for return value of 'kauditd_send_queue'
	Warning: kernel/audit.c:841 No description found for return value of 'kauditd_thread'
	Warning: kernel/audit.c:1828 No description found for return value of 'audit_serial'
	Warning: kernel/audit.c:1860 No description found for return value of 'audit_log_start'
	Warning: kernel/audit.c:1937 No description found for return value of 'audit_expand'
	Warning: kernel/audit.c:2093 No description found for return value of 'audit_string_contains_control'
	Warning: kernel/audit.c:2359 No description found for return value of 'audit_set_loginuid'
	Warning: kernel/audit.c:2394 No description found for return value of 'audit_signal_info'

To be bug-compatible with the Perl version, I ported -Wall to it,
as otherwise there would be tons of extra warnings after migration.

Ok, it doesn't tell that there are exports there, although kernel-doc
knows, as it has to process exports anyway.

> > ...or one can use scripts/find-unused-docs.sh, which was written for
> > just this purpose :)  

It should be very easy to add an option to kernel-doc to do the same.

IMO, reducing the number of scripts may help people to better use
the tools.

> 
> Yes, and I have used this script. It does what it was meant to do AFAIK.
> It's reporting is at a gross file level.
> 
> I made a small subdirectory called "test" and copied kernel/audit.c to test/.
> 
> $ ./scripts/find-unused-docs.sh  test/
> The following files contain kerneldoc comments for exported functions that are not used in the formatted documentation
> test/audit.c
> 
> Sometime in the last 2-3 years Matthew Wilcox asked me about a tool (script, whatever)that would detect both EXPORTs without kernel-doc and kernel-doc without EXPORTs.
> Either one of these can be noisy (with false positives) and they often don't lend
> themselves to easy/beginner fixes.

Heh, not knowing/remembering about find-unused-docs.sh, I actually
wrote a prototype for something similar sometime ago, in Perl.
Didn't upstream it though, nor used it much as I got sidetracked
by other things. My goal on that time were to find gaps on media kAPI.

> Anyway, after some delay, I have such a script. It's written in Perl (I started
> on it over a year ago!). It might have been desirable to add it to scripts/kernel-doc.pl
> at the time, but it didn't seem to me like a good fit there, so it's independent.
> 
> Running (no options, just produce a summary)
> $ kerndoc-export-search.pl test/audit.c
> reports:
> Missing kernel-doc for: audit_log_task_info
> Missing kernel-doc for: audit_enabled
> Missing kernel-doc for: audit_log_task_context
> 3 missing kernel-docs
> Missing EXPORT for: audit_serial
> Missing EXPORT for: audit_log_untrustedstring
> Missing EXPORT for: audit_log_n_untrustedstring
> Missing EXPORT for: audit_log_n_hex
> Missing EXPORT for: audit_log_lost
> Missing EXPORT for: audit_set_loginuid
> Missing EXPORT for: auditd_test_task
> Missing EXPORT for: audit_ctl_lock
> Missing EXPORT for: audit_string_contains_control
> Missing EXPORT for: audit_signal_info
> Missing EXPORT for: audit_log_path_denied
> Missing EXPORT for: audit_ctl_unlock
> 12 missing Exports
> 
> If that's not verbose enough (!), the -l (list) option lists each function's
> location and short description. One example:
> test/audit.c: 2006: audit_log_format:  * audit_log_format - format a message into the audit buffer.

Nice! Yet, I suggest trying to merge with kernel-doc, even if the
actual implementation would be on a separate class inside a
separate file.

Still, the way kernel-doc works allow one to just write a different
output class (or improve the output class used by --none) to use
the already parsed data on different ways.

> But that generates lots of output.
> 
> And of course, for function, I mean function/struct/union/enum/typedef.
> 
> There is a "verbose" option but it currently does not print anything.
> 
> Here is a help summary:
> $ kerndoc-export-search.pl -h
> kerndoc-export-search.pl [--list|-l] [--verbose|-v] file(s)
>   where --list    prints filename:line:funcname: short_description
>   where --verbose prints more info.
>   default: prints a doc/export summary + warnings.
>   version: 0.9
> 
> 
> Feel free to use in any way or to rewrite & merge it into the
> kdoc python system.

Good to know! I'll try to take a look on it later on.

Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jonathan Corbet]

Mark Brown <broonie@kernel.org> writes:

> On Mon, Sep 01, 2025 at 12:25:30PM -0600, Jonathan Corbet wrote:
>> Mark Brown <broonie@kernel.org> writes:
>
>> >   make -j56 htmldocs  2355.99s user 141.33s system 158% cpu 26:14.86 total
>
>> That ... is weird... it takes me a little under 3 minutes to do an
>> htmldocs build, using a capable but not stellar desktop machine.
>
>> Which version of Sphinx are you using?  If you're not on Sphinx 8, you
>> really want to be; they finally fixed some really nasty performance
>> problems with that release.
>
> I appear to be on version 5.3 which is what's in my distro.  I will get
> 8.1.3 when I upgrade, I don't really have any intention of manually
> installing an unpackaged copy.

That version will be far slower, certainly.

Just FWIW, installing a newer version in a venv is trivially easy to do.

jon

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> It should be noticed that kernel-doc doesn't run in parallel. Python
> still suffers for a global big lock (called GIL). My attempts to run in
> parallel actually made kernel-doc slower, but this is changing: the
> next Python version is planning to get rid of GIL. So, maybe within
> a year we can re-add the patches to run it in parallel.

I certainly wouldn't want to discourage work in this area, but I do
wonder if it would be worth the trouble; kernel-doc is nowhere near
being the bottleneck in this whole process.  Now, if you could
multi-thread the Sphinx HTML builder...

jon

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

On Mon, Sep 01, 2025 at 07:40:21PM +0100, Mark Brown wrote:
> On Mon, Sep 01, 2025 at 12:25:30PM -0600, Jonathan Corbet wrote:
> > Mark Brown <broonie@kernel.org> writes:
> 
> > >   make -j56 htmldocs  2355.99s user 141.33s system 158% cpu 26:14.86 total
> 
> > That ... is weird... it takes me a little under 3 minutes to do an
> > htmldocs build, using a capable but not stellar desktop machine.
> 
> > Which version of Sphinx are you using?  If you're not on Sphinx 8, you
> > really want to be; they finally fixed some really nasty performance
> > problems with that release.
> 
> I appear to be on version 5.3 which is what's in my distro.  I will get
> 8.1.3 when I upgrade, I don't really have any intention of manually
> installing an unpackaged copy.  Three minutes would be more reasonable,
> though without a clean build (and I guess something that errors on
> warnings so I don't actually need to look at the output myself) I'm not
> sure I'd notice any issues.

You can install it on a Python virtual env, using it only for doc builds.
Procedure is quite simple.

As Jon pointed out, from 3.x to 7.x, Sphinx had lots of performance
issues. On my 64GB RAM system, I even had to enable swap to avoid OOM
killer when testing builds with 7.x.

All of those were gone on 8.x. Build is about 3 minutes, on a big machine
and even on my 16GB i7core laptop.

Btw, on my experiences, building on Sphinx (even 8.x) with -j8 or -j24
is about the same time.

Btw, once this series is merged:

	https://lore.kernel.org/linux-doc/cover.1756740314.git.mchehab+huawei@kernel.org/T/#mdca010445a79da125b5113ca70da1b1d03a443e6

it is possible to build the docs directly at the virtual environment
without needing to enable/disable by calling directly the new 
sphinx-build-wrapper script:

	./tools/docs/sphinx-build-wrapper htmldocs -V

It will automatically search for the latest sphinx_<version> 
(or sphinx_latest) at the Kernel dir and use it for doc build. 
-V also accepts an optional argument if you want to force it to
use an specific virtual environment.


-- 
Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

On Mon, Sep 01, 2025 at 07:52:01PM +0100, Mark Brown wrote:
> On Mon, Sep 01, 2025 at 08:46:35PM +0200, Mauro Carvalho Chehab wrote:
> 
> > It should be noticed that kernel-doc doesn't run in parallel. Python
> > still suffers for a global big lock (called GIL). My attempts to run in
> > parallel actually made kernel-doc slower, but this is changing: the
> > next Python version is planning to get rid of GIL. So, maybe within
> > a year we can re-add the patches to run it in parallel.
> 
> It'll take a lot longer for that to filter out to people's machines, for
> example I'm running Debian stable on my desktop and I know a lot of
> people have Ubuntu LTS.
> 
> > > That
> > > compares unfavourably with allmodconfig on this system:
> > > 
> > >   $ time make -j56 allmodconfig
> > >   ...
> > >   make -j56 allmodconfig  5.31s user 1.93s system 146% cpu 4.931 total
> > >   $ time make -j56
> > >   ...
> > >   make -j56  53468.11s user 4387.30s system 5084% cpu 18:57.77 total
> > > 
> > > and seems rather more likely to flag something for me.
> 
> > 32 seconds more, on the top of 53468.11s doesn't sound that much.
> 
> Yeah, if it was of that sort of order, ran clean with mainline and could
> be checked automatically it'd be a lot more viable.

If you run:

	 kernel-doc . --none -Wall

You won't have troubles with Sphinx slowness. It would be worth timing
it on you machine and see how much time it takes to run. Probably
the run time depends a little bit on the Python version. Not sure how
much optimization it got(*).

(*) Here, I use Fedora 42, so it is a recent Python version, but I
    gues even on Debian and Ubuntu LTS, if python version affects,
    you can install a newer version from the official packages.

-- 
Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Mon, 01 Sep 2025 13:53:41 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > It should be noticed that kernel-doc doesn't run in parallel. Python
> > still suffers for a global big lock (called GIL). My attempts to run in
> > parallel actually made kernel-doc slower, but this is changing: the
> > next Python version is planning to get rid of GIL. So, maybe within
> > a year we can re-add the patches to run it in parallel.  
> 
> I certainly wouldn't want to discourage work in this area, but I do
> wonder if it would be worth the trouble; kernel-doc is nowhere near
> being the bottleneck in this whole process. 

I know kernel-doc is not a bottleneck. When I wrote it, I wrote to
run in parallel. The original patches supported even both 
multiprocess and multithread, and had command line parameters to
adjust, but I got disappointed with results, so I dropped it.
Re-adding shoudn't be hard, as I tried to ensure that this would
work since the beginning.

This is, btw, one of the main reasons why I created a separate
class to handle files and used an interactor for file names: the
entire parser logic is confined to one instance per file and
multiple instances can run in parallel. This way concurrent.futures
can distribute files to different processes and threads.

> Now, if you could
> multi-thread the Sphinx HTML builder...

Agreed. I suspect that, if Sphinx can run without GIL, in the future,
we'll have huge performance gains, specially on machines with multiple
CPU cores.

Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jani Nikula]

On Mon, 01 Sep 2025, Randy Dunlap <rdunlap@infradead.org> wrote:
> ISTM that there are lots of non-docs developers who either just don't care
> about that or never run 'make W=1 htmldocs' to see the problems in their
> drivers or subsystems. OK, maybe it's just a very low priority for them.
>
> Willy had a suggestion that we just make checking kernel-doc during
> all .c builds a permanent feature instead of a W=1 option.
> This helps, but still doesn't force 'make htmldocs' to be run.
>
> And it causes around 450 build warnings in my testing of an x86_64 allmodconfig
> build.

I think in general the build system lacks proper support for subsystems
or drivers being ahead of the curve in keeping them W=1 or kernel-doc
-Wall clean.

We hack around this in parts of drm Makefiles, and it really has helped
to keep the kernel-doc and W=1 breakage/fix churn down. But it's not
pretty.


BR,
Jani.

-- 
Jani Nikula, Intel

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jani Nikula]

On Mon, 01 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> On media, our CI builds with W=1, and aim to have no warnings.

How do you filter out non-media W=1 failures?

I'd love to be able to tell contributors to use a certain kernel config
or command-line for build, and tell them to fix *all* warnings, instead
of teaching them to ignore most of them.


BR,
Jani.

-- 
Jani Nikula, Intel

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 392 bytes --]

On Tue, Sep 02, 2025 at 12:56:57AM +0200, Mauro Carvalho Chehab wrote:

> If you run:

> 	 kernel-doc . --none -Wall

> You won't have troubles with Sphinx slowness. It would be worth timing
> it on you machine and see how much time it takes to run. Probably
> the run time depends a little bit on the Python version. Not sure how
> much optimization it got(*).

That takes about 90s for me.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Tue, 02 Sep 2025 13:42:45 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Mon, 01 Sep 2025, Randy Dunlap <rdunlap@infradead.org> wrote:
> > ISTM that there are lots of non-docs developers who either just don't care
> > about that or never run 'make W=1 htmldocs' to see the problems in their
> > drivers or subsystems. OK, maybe it's just a very low priority for them.
> >
> > Willy had a suggestion that we just make checking kernel-doc during
> > all .c builds a permanent feature instead of a W=1 option.
> > This helps, but still doesn't force 'make htmldocs' to be run.
> >
> > And it causes around 450 build warnings in my testing of an x86_64 allmodconfig
> > build.  
> 
> I think in general the build system lacks proper support for subsystems
> or drivers being ahead of the curve in keeping them W=1 or kernel-doc
> -Wall clean.

It is trivial to add a spinx/kerneldoc parameter to allow setting
-Wall per each .. kernel-doc markup. Yet, one would need to add it
for every markup within the subsystem.

It is probably easier to do something similar to that via a CI.

> 
> We hack around this in parts of drm Makefiles, and it really has helped
> to keep the kernel-doc and W=1 breakage/fix churn down. But it's not
> pretty.
> 
> 
> BR,
> Jani.
> 



Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Tue, 2 Sep 2025 12:15:51 +0100
Mark Brown <broonie@kernel.org> escreveu:

> On Tue, Sep 02, 2025 at 12:56:57AM +0200, Mauro Carvalho Chehab wrote:
> 
> > If you run:  
> 
> > 	 kernel-doc . --none -Wall  
> 
> > You won't have troubles with Sphinx slowness. It would be worth timing
> > it on you machine and see how much time it takes to run. Probably
> > the run time depends a little bit on the Python version. Not sure how
> > much optimization it got(*).  
> 
> That takes about 90s for me.


I wander why here is 3 times faster... disk cache? python version?
faster ssd?

What python version are you using?


Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Andrew Lunn]

On Tue, Sep 02, 2025 at 01:55:48PM +0300, Jani Nikula wrote:
> On Mon, 01 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > On media, our CI builds with W=1, and aim to have no warnings.
> 
> How do you filter out non-media W=1 failures?
> 
> I'd love to be able to tell contributors to use a certain kernel config
> or command-line for build, and tell them to fix *all* warnings, instead
> of teaching them to ignore most of them.

I cannot say anything for media, but for netdev we have the count
before and after the patch is applied. If the count goes up, the test
fails. We also generate a unified diff between the outputs, which is
enough to be able to tell the developer what they missed.

In theory, anybody can do the same on their own devel machine:

https://github.com/linux-netdev/nipa

I've no idea how many developers actually do.

	Andrew

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jani Nikula]

On Tue, 02 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Em Tue, 02 Sep 2025 13:42:45 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
>
>> On Mon, 01 Sep 2025, Randy Dunlap <rdunlap@infradead.org> wrote:
>> > ISTM that there are lots of non-docs developers who either just don't care
>> > about that or never run 'make W=1 htmldocs' to see the problems in their
>> > drivers or subsystems. OK, maybe it's just a very low priority for them.
>> >
>> > Willy had a suggestion that we just make checking kernel-doc during
>> > all .c builds a permanent feature instead of a W=1 option.
>> > This helps, but still doesn't force 'make htmldocs' to be run.
>> >
>> > And it causes around 450 build warnings in my testing of an x86_64 allmodconfig
>> > build.  
>> 
>> I think in general the build system lacks proper support for subsystems
>> or drivers being ahead of the curve in keeping them W=1 or kernel-doc
>> -Wall clean.
>
> It is trivial to add a spinx/kerneldoc parameter to allow setting
> -Wall per each .. kernel-doc markup. Yet, one would need to add it
> for every markup within the subsystem.

I'm not sure how that is relevant to what I'm saying.

>
> It is probably easier to do something similar to that via a CI.
>
>> 
>> We hack around this in parts of drm Makefiles, and it really has helped
>> to keep the kernel-doc and W=1 breakage/fix churn down. But it's not
>> pretty.
>> 
>> 
>> BR,
>> Jani.
>> 
>
>
>
> Thanks,
> Mauro

-- 
Jani Nikula, Intel

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Tue, 2 Sep 2025 13:59:38 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> Em Tue, 2 Sep 2025 12:15:51 +0100
> Mark Brown <broonie@kernel.org> escreveu:
> 
> > On Tue, Sep 02, 2025 at 12:56:57AM +0200, Mauro Carvalho Chehab wrote:
> >   
> > > If you run:    
> >   
> > > 	 kernel-doc . --none -Wall    
> >   
> > > You won't have troubles with Sphinx slowness. It would be worth timing
> > > it on you machine and see how much time it takes to run. Probably
> > > the run time depends a little bit on the Python version. Not sure how
> > > much optimization it got(*).    
> > 
> > That takes about 90s for me.  
> 
> 
> I wander why here is 3 times faster... disk cache? python version?
> faster ssd?
> 
> What python version are you using?

Heh, after running twice or three times to avoid cache issues, I tested
running it on my machine with two different python versions:

$ time python3.13 scripts/kernel-doc.py . --none

real    0m31,660s
user    0m30,911s
sys     0m0,588s

$ time python3.9 scripts/kernel-doc.py . --none

real    0m59,004s
user    0m58,014s
sys     0m0,730s

$ time python3.6 scripts/kernel-doc.py . --none

real    1m16,494s
user    1m15,400s
sys     0m0,765s

(after a fix I'm about to send to prevent lots of output about
 python version)

So, yeah, Python version seems to be one of the reasons why it is
taking so long on your machine.

Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

On Tue, Sep 02, 2025 at 02:14:34PM +0200, Mauro Carvalho Chehab wrote:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> > > That takes about 90s for me.  

> > I wander why here is 3 times faster... disk cache? python version?
> > faster ssd?

> > What python version are you using?

> Heh, after running twice or three times to avoid cache issues, I tested
> running it on my machine with two different python versions:

> $ time python3.13 scripts/kernel-doc.py . --none

> real    0m31,660s
> user    0m30,911s
> sys     0m0,588s

For me it's fairly consistent with python 3.11.2, I get some variation
depending on what else is going on with the system but nothing hugely
surprising.  It should mostly be running from cache, the underlying disk
is a reasonable SSD.  The single core performance on this machine isn't
amazing, it's more getting it's speed from having a bunch of cores.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

On Tue, Sep 02, 2025 at 02:00:43PM +0100, Mark Brown wrote:
> On Tue, Sep 02, 2025 at 02:14:34PM +0200, Mauro Carvalho Chehab wrote:
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:
> 
> > > > That takes about 90s for me.  
> 
> > > I wander why here is 3 times faster... disk cache? python version?
> > > faster ssd?
> 
> > > What python version are you using?
> 
> > Heh, after running twice or three times to avoid cache issues, I tested
> > running it on my machine with two different python versions:
> 
> > $ time python3.13 scripts/kernel-doc.py . --none
> 
> > real    0m31,660s
> > user    0m30,911s
> > sys     0m0,588s
> 
> For me it's fairly consistent with python 3.11.2, I get some variation
> depending on what else is going on with the system but nothing hugely
> surprising.  It should mostly be running from cache, the underlying disk
> is a reasonable SSD. 

Yeah, it sounds that the huge performance increment was indeed on 3.11:

$ time python3.6 ./scripts/kernel-doc --none .
real	1m17,854s
user	1m16,651s
sys	0m0,707s

$ time python3.9 ./scripts/kernel-doc --none .
real	1m0,193s
user	0m58,942s
sys	0m0,614s

$ time python3.10 ./scripts/kernel-doc --none .
real	0m55,376s
user	0m54,168s
sys	0m0,636s

$ time python3.11 ./scripts/kernel-doc --none .
real	0m34,878s
user	0m33,665s
sys	0m0,661s

$ time python3.12 ./scripts/kernel-doc --none .
real	0m34,590s
user	0m33,844s
sys	0m0,613s

$ time python3.13 ./scripts/kernel-doc --none .
real	0m31,751s
user	0m30,951s
sys	0m0,640s

==============  =============   =============   ================================
Python Version	Real Time (s)	User Time (s)	Performance Increase (Real Time)
==============  =============   =============   ================================
3.6		77.854 s	76.651 s	(baseline)
3.9		60.193 s	58.942 s	22.7% faster
3.10		55.376 s	54.168 s	28.9% faster
3.11		34.878 s	33.665 s	55.2% faster
3.12		34.590 s	33.844 s	55.6% faster
3.13		31.751 s	30.951 s	59.2% faster
==============  =============   =============   ================================

> The single core performance on this machine isn't
> amazing, it's more getting it's speed from having a bunch of cores.

As kernel-doc is currently single threaded, performance for a single
core is what matters most. I suspect that most of kernel-doc time is
spent handling regular expressions, specially when I/O is fast. 

-

To sum-up those discussions, I can propose a patchset for the next
merge window that would:

1. change kernel-doc exec to re-run using the latest available python
   version if version < 3.11, on a similar same way to what
   scripts-pre-install and scripts-build-wrapper does(*);

2. add a command line parameter for kernel-doc to make it pick only
   files that have a .. kernel-doc markup;

3. add a build logic to run it with make all, perhaps inside a Kconfig
   symbol like "config DOC_WARNINGS", disabled by default, but enabled
   with allyesconfig/allmodconfig.

4. with time, add more validations there, like checking for
   EXPORT_SYMBOL without documentation and other neat features.

This needs to be coordinated with some efforts to cleanup warnings,
to avoid having hundreds of new warnings at build time.

This way, even on LTS, we'll have fast kernel-doc check, and will likely
take lot less than 32 seconds, as it will only validate a small set of
files that are pointed by a kernel-doc markup notation inside
Documentation, specially if Python >= 3.11 is installed.

It should be noticed that can generate out of blue lots of new
warnings that are currently there by currently hidden, specially
if we add "-Wall" to the build target.

Comments?

(*) The logic there checks if python version is below a minimal
    version. If it is, it seeks for python 3.x exec files, picking
    the latest one if found, and re-running it with such version.

-- 
Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

On Tue, Sep 02, 2025 at 03:07:53PM +0300, Jani Nikula wrote:
> On Tue, 02 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > Em Tue, 02 Sep 2025 13:42:45 +0300
> > Jani Nikula <jani.nikula@intel.com> escreveu:
> >
> >> On Mon, 01 Sep 2025, Randy Dunlap <rdunlap@infradead.org> wrote:
> >> > ISTM that there are lots of non-docs developers who either just don't care
> >> > about that or never run 'make W=1 htmldocs' to see the problems in their
> >> > drivers or subsystems. OK, maybe it's just a very low priority for them.
> >> >
> >> > Willy had a suggestion that we just make checking kernel-doc during
> >> > all .c builds a permanent feature instead of a W=1 option.
> >> > This helps, but still doesn't force 'make htmldocs' to be run.
> >> >
> >> > And it causes around 450 build warnings in my testing of an x86_64 allmodconfig
> >> > build.  
> >> 
> >> I think in general the build system lacks proper support for subsystems
> >> or drivers being ahead of the curve in keeping them W=1 or kernel-doc
> >> -Wall clean.
> >
> > It is trivial to add a spinx/kerneldoc parameter to allow setting
> > -Wall per each .. kernel-doc markup. Yet, one would need to add it
> > for every markup within the subsystem.
> 
> I'm not sure how that is relevant to what I'm saying.

You said that the building system lacks support of W=1/-Wall per
subsystem. What I said is that, provided that we add a:

	.. kernel-doc:: drivers/drm/...
	:wall: 

you can set it per file inside a subsystem. Granted: this is doesn't
cover the entire subsystem.

Heh, there is another option. For instance lets assume you want
-Wall for drm subsystem. you could have this on your CI:

	$ ./scripts/kernel-doc -Wall --none drivers/gpu/

On a similar way, the build system can also W=1 inside a subsystem:

	$ make W=1 drivers/gpu/

(This is what we do on media)

In the specific case of the drm subsystem, you could try to modify
dim to run both as a condition to accept a git push - or modify
CI to only actually do the merge after passing both.

-- 
Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> To sum-up those discussions, I can propose a patchset for the next
> merge window that would:
>
> 1. change kernel-doc exec to re-run using the latest available python
>    version if version < 3.11, on a similar same way to what
>    scripts-pre-install and scripts-build-wrapper does(*);

I have to confess that I still feel some discomfort with this sort of
"pick a better version" magic.  Should we really be overriding the
search path that the user has set up?

Thanks,

jon

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Tue, 02 Sep 2025 09:15:49 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > To sum-up those discussions, I can propose a patchset for the next
> > merge window that would:
> >
> > 1. change kernel-doc exec to re-run using the latest available python
> >    version if version < 3.11, on a similar same way to what
> >    scripts-pre-install and scripts-build-wrapper does(*);  
> 
> I have to confess that I still feel some discomfort with this sort of
> "pick a better version" magic.  Should we really be overriding the
> search path that the user has set up?

The idea is not to override the search path: instead, to use it to
check if the user installed other /usr/bin/python3.* files (or on
some other part of PATH). Most distributions nowadays come with 
multiple python versions. I can't see a downside (*) of not using 
a newer version that the user had installed on his system and
has it on PATH.

For make htmldocs, if version is < 3.7 (or maybe 3.6), this is
mandatory: creating docs without that will fail. So, this is
actually a fallback measure in an attempt to save the day.
This is specially important for OpenSuse Leap, were we recommend
python311-sphinx package, which obviously require python 3.11
to run. The same applies for RHEL8-based distros and likely
old RHEL9 ones.

Now, for kernel-doc command line, checking against 3.11 is arguable,
as it runs slow, but works just fine with 3.7 to 3.10. 

Yet, trying to re-run costs about nothing, and make kernel-doc to
run 55% to 60% faster. IMO, it is worth. We can first check for
a PYTHON env to see if are there any overrides.

(*) The only possible issue is if the user installed python 3.11, but
    forgot to install python3.11-libs package, but it sounds easy enough
    to check if this is the case via a try/except logic.

Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Laurent Pinchart]

On Tue, Sep 02, 2025 at 07:19:29PM +0200, Mauro Carvalho Chehab wrote:
> Em Tue, 02 Sep 2025 09:15:49 -0600 Jonathan Corbet escreveu:
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> > 
> > > To sum-up those discussions, I can propose a patchset for the next
> > > merge window that would:
> > >
> > > 1. change kernel-doc exec to re-run using the latest available python
> > >    version if version < 3.11, on a similar same way to what
> > >    scripts-pre-install and scripts-build-wrapper does(*);  
> > 
> > I have to confess that I still feel some discomfort with this sort of
> > "pick a better version" magic.  Should we really be overriding the
> > search path that the user has set up?
> 
> The idea is not to override the search path: instead, to use it to
> check if the user installed other /usr/bin/python3.* files (or on
> some other part of PATH). Most distributions nowadays come with 
> multiple python versions. I can't see a downside (*) of not using 
> a newer version that the user had installed on his system and
> has it on PATH.

I'm with Jon here, I wouldn't blindly override the Python interpreter
selected by the user. What we could however do is print a message if we
detect a version of Python that could improve performance, telling the
user they could switch.

> For make htmldocs, if version is < 3.7 (or maybe 3.6), this is
> mandatory: creating docs without that will fail. So, this is
> actually a fallback measure in an attempt to save the day.
> This is specially important for OpenSuse Leap, were we recommend
> python311-sphinx package, which obviously require python 3.11
> to run. The same applies for RHEL8-based distros and likely
> old RHEL9 ones.
> 
> Now, for kernel-doc command line, checking against 3.11 is arguable,
> as it runs slow, but works just fine with 3.7 to 3.10. 
> 
> Yet, trying to re-run costs about nothing, and make kernel-doc to
> run 55% to 60% faster. IMO, it is worth. We can first check for
> a PYTHON env to see if are there any overrides.
> 
> (*) The only possible issue is if the user installed python 3.11, but
>     forgot to install python3.11-libs package, but it sounds easy enough
>     to check if this is the case via a try/except logic.

-- 
Regards,

Laurent Pinchart

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Em Tue, 02 Sep 2025 09:15:49 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
>
>> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>> 
>> > To sum-up those discussions, I can propose a patchset for the next
>> > merge window that would:
>> >
>> > 1. change kernel-doc exec to re-run using the latest available python
>> >    version if version < 3.11, on a similar same way to what
>> >    scripts-pre-install and scripts-build-wrapper does(*);  
>> 
>> I have to confess that I still feel some discomfort with this sort of
>> "pick a better version" magic.  Should we really be overriding the
>> search path that the user has set up?
>
> The idea is not to override the search path: instead, to use it to
> check if the user installed other /usr/bin/python3.* files (or on
> some other part of PATH). Most distributions nowadays come with 
> multiple python versions. I can't see a downside (*) of not using 
> a newer version that the user had installed on his system and
> has it on PATH.

But overriding the path is exactly what this would be doing.  It doesn't
seem right to say "we know better than you do" and circumvent the
configured path; the user may well have reasons for setting things up
the way they did.  Do you know of any other tool that behaves this way?

If we're really going to do it, I'd feel better if it had to be enabled
with a --fastest-python flag or some such.

> For make htmldocs, if version is < 3.7 (or maybe 3.6), this is
> mandatory: creating docs without that will fail. So, this is
> actually a fallback measure in an attempt to save the day.

Honestly, I'm not entirely convinced even here.  It really feels like
the kind of magic that may bite us one of these days.

> This is specially important for OpenSuse Leap, were we recommend
> python311-sphinx package, which obviously require python 3.11
> to run. The same applies for RHEL8-based distros and likely
> old RHEL9 ones.

...in which case, it seems to me we need to recommend that the path be
configured appropriately as well.

jon

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Tue, 02 Sep 2025 12:58:51 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Em Tue, 02 Sep 2025 09:15:49 -0600
> > Jonathan Corbet <corbet@lwn.net> escreveu:
> >  
> >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> >>   
> >> > To sum-up those discussions, I can propose a patchset for the next
> >> > merge window that would:
> >> >
> >> > 1. change kernel-doc exec to re-run using the latest available python
> >> >    version if version < 3.11, on a similar same way to what
> >> >    scripts-pre-install and scripts-build-wrapper does(*);    
> >> 
> >> I have to confess that I still feel some discomfort with this sort of
> >> "pick a better version" magic.  Should we really be overriding the
> >> search path that the user has set up?  
> >
> > The idea is not to override the search path: instead, to use it to
> > check if the user installed other /usr/bin/python3.* files (or on
> > some other part of PATH). Most distributions nowadays come with 
> > multiple python versions. I can't see a downside (*) of not using 
> > a newer version that the user had installed on his system and
> > has it on PATH.  
> 
> But overriding the path is exactly what this would be doing.  It doesn't
> seem right to say "we know better than you do" and circumvent the
> configured path; 

Again, it is not the path: it is the "alternatives" - well, except that
Python doesn't use alternatives.

This is on Fedora:

	$ ls -lctra /usr/bin/python???? /usr/bin/python? /usr/bin/python
	-rwxr-xr-x 1 root root 11704 ago 21 17:09 /usr/bin/python3.13
	lrwxrwxrwx 1 root root    10 ago 21 17:09 /usr/bin/python3 -> python3.13
	lrwxrwxrwx 1 root root     9 ago 21 17:09 /usr/bin/python -> ./python3
	-rwxr-xr-x 2 root root 11728 ago 21 17:09 /usr/bin/python3.6m
	-rwxr-xr-x 1 root root 11704 set  2 16:07 /usr/bin/python3.11
	-rwxr-xr-x 1 root root 11704 set  2 16:09 /usr/bin/python3.12
	-rwxr-xr-x 1 root root 11712 set  2 16:18 /usr/bin/python3.10

This is on openSuse Leap, after using the procedures recommended
by sphinx-pre-install:

	# ls -l /usr/bin/python*
	lrwxrwxrwx 1 root root     9 Apr  9  2024 /usr/bin/python3 -> python3.6
	-rwxr-xr-x 1 root root  6280 May 18  2024 /usr/bin/python3.11
	-rwxr-xr-x 2 root root 10560 Apr  9  2024 /usr/bin/python3.6
	-rwxr-xr-x 2 root root 10560 Apr  9  2024 /usr/bin/python3.6m

Recommending the user to run:

	# ln -sf python3.11 /usb/bin/python
	# ln -sf python3.11 /usb/bin/python3

Doesn't seem to be right. See below.

Also, adding at .bashrc is not enough, as running "make htmldocs" after
installing the packages will fail except if the user manually runs it.

So, at least for doc build, I still think that getting a python
compatible version is still the better solution - at least for doc
building.

> the user may well have reasons for setting things up
> the way they did. 

There are very good reasons why not doing that: replacing it
will very likely break the system. If I'm not mistaken, even package
install can break if one does something like that on some distros that
tries to run python at install time and whose scripts may not be 
forward-compatible with newer python versions.

The same problem occurs even if a bash alias would be created and placed
under .bashrc (plus, such procedure assumes shell is bash):

	alias python3=/usr/bin/python3.11

So, if doing something like that won't work, the other alternatives
are:

1. don't use Makefile; instead run:

	python3.11 scripts/tools/sphinx-build-wrapper htmldocs

2. create a <kernel_tree>/bin directory and add there an alias.
   every time you want to build docs, set:

	PATH=<kernel_tree>/bin:$PATH

3. use venv (probably the simplest solution)

the first two are IMHO, a lot more hackish than picking
/usr/bin/python3.11 from the patch where the user already installed
via a distro package and use it.

The third option works, but some developers don't like installing
packages outside the distro-provided package manager.

> Do you know of any other tool that behaves this way?

Not exactly this way. What several other build toolchains do is
to directly download everything that is needed to build (gcc,
make, python, ld, ...) - or to build themselves from the source
before starting building stuff.

> If we're really going to do it, I'd feel better if it had to be enabled
> with a --fastest-python flag or some such.

Makes sense for kernel-doc.

> > For make htmldocs, if version is < 3.7 (or maybe 3.6), this is
> > mandatory: creating docs without that will fail. So, this is
> > actually a fallback measure in an attempt to save the day.  
> 
> Honestly, I'm not entirely convinced even here.  It really feels like
> the kind of magic that may bite us one of these days.

If it bites, we can deal with it, but see: the alternative is to 
refuse to build the docs and bail out. IMHO, a worse solution(*).

Besides that, we already have since ever things like that, when,
for instance, we check if latexmk is available, falling back
to xelatex if not. The only difference here is that, instead of
requiring a non-python script (or makefile rule), we're handling
with multiple versions of the toolchain directly inside the
python script.

(*) Ok, with the new wrapper tool, he can still build docs by 
prepending "python3.11".

> > This is specially important for OpenSuse Leap, were we recommend
> > python311-sphinx package, which obviously require python 3.11
> > to run. The same applies for RHEL8-based distros and likely
> > old RHEL9 ones.  
> 
> ...in which case, it seems to me we need to recommend that the path be
> configured appropriately as well.

How? I can't see a way to recommend a solution without either
propose a hack or risk breaking the system.

Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Johannes Berg]

On Tue, 2025-09-02 at 12:58 -0600, Jonathan Corbet wrote:
> > 
> > The idea is not to override the search path: instead, to use it to
> > check if the user installed other /usr/bin/python3.* files (or on
> > some other part of PATH). Most distributions nowadays come with 
> > multiple python versions. I can't see a downside (*) of not using 
> > a newer version that the user had installed on his system and
> > has it on PATH.
> 
> But overriding the path is exactly what this would be doing.  It doesn't
> seem right to say "we know better than you do" and circumvent the
> configured path; the user may well have reasons for setting things up
> the way they did.

Absolutely! Please don't ever do this.

For example, use case we have: using nix-shell to lock down the software
used to build, for reproducible builds and similar reasons. Without --
pure, PATH may still contain (last!) software from the system itself,
but it should basically never be used.

johannes

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jani Nikula]

On Tue, 02 Sep 2025, Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
> I'm with Jon here, I wouldn't blindly override the Python interpreter
> selected by the user. What we could however do is print a message if we
> detect a version of Python that could improve performance, telling the
> user they could switch.

Just piling on here, totally agreed.

Don't surprise the user. Let the user be in control. If they choose a
silly combo, let them. You know, they might be debugging the issues in
that silly combo to begin with, and don't want to jump through hoops to
work around tools that think they know better.

Letting the user know they're using a silly combo and suggesting better
alternatives is a whole different matter.


BR,
Jani.

-- 
Jani Nikula, Intel

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Wed, 03 Sep 2025 10:47:28 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Tue, 02 Sep 2025, Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
> > I'm with Jon here, I wouldn't blindly override the Python interpreter
> > selected by the user. What we could however do is print a message if we
> > detect a version of Python that could improve performance, telling the
> > user they could switch.  
> 
> Just piling on here, totally agreed.
> 
> Don't surprise the user. Let the user be in control. If they choose a
> silly combo, let them. You know, they might be debugging the issues in
> that silly combo to begin with, and don't want to jump through hoops to
> work around tools that think they know better.
> 
> Letting the user know they're using a silly combo and suggesting better
> alternatives is a whole different matter.

The code can be changed to suggest a better alternative, and having an
optional argument to load the newest one automatically, but the issue
still remains:

if one has native python3.6, for instance, and he installed a newer one
to build htmldocs with:

	dnf install -y python3.11 # or zypper/apt-get/urpmi/... install python3.11

python3 will still be a soft link to python3.5, and all attempts to
build docs will fail.

With the approach I'm taken, Makefile will call sphinx-pre-install
and sphinx-build-wrapper. They both use a function that checks for
incompatible versions, re-running the code with one which works,
warning the user about that. So, on openSUSE leap, for instance, 
we have two official packages with Sphinx:

	$ grep PRETTY_NAME /etc/os-release 
	PRETTY_NAME="openSUSE Leap 15.6"

	$ zypper search sphinx |grep -E "python\S+\-Sphinx "
	   | python3-Sphinx                          | Python documentation generator                                              | package
	i+ | python311-Sphinx                        | Python documentation generator                                              | package


There, python3-Sphinx is too old:

	# sphinx-build --version
	sphinx-build 2.3.1

As docs minimal requirement is 3.4.3. Also, it is based on python3.6,
which is not compatible with the kernel Sphinx extensions.

Which the approach I took, once one installs python311-Sphinx like I did,
the build happens cleanly:

	$ make htmldocs
	# sphinx-pre-install, which checks is needed packages are installed
	Python 3.6.15 not supported. Changing to /usr/bin/python3.11
	# sphinx-build-wrapper, which calls Sphinx to build the docs
	Python version: 3.11.9
	Python 3.6.15 not supported. Changing to /usr/bin/python3.11
	Python version: 3.11.9
	Using alabaster theme
	Using Python kernel-doc
	...

but if we remove the switch logic:

        -print(f"Python {python_ver} not supported. Changing to {new_python_cmd}")
        -
        -try:
        -    os.execv(new_python_cmd, args)
        -except OSError as e:
        -    sys.exit(f"Failed to restart with {new_python_cmd}: {e}")
        -
	+print(f"Python {python_ver} not supported.")

The build will then fail:

	# make htmldocs
	Python 3.6.15 not supported.
	Python 3.6.15 not supported.
	Traceback (most recent call last):
	  File "/usr/bin/sphinx-build", line 5, in <module>
	    from sphinx.cmd.build import main
	ModuleNotFoundError: No module named 'sphinx'

because:

- python3.6 is not supported;
- sphinx-build is installed with python311-Sphinx, which also installed
  python311 package.

See, in this case, the user wanted docs build to use python3.11, as
it installed python311-Sphinx package from a distro official package.

Without the version-check logic, the only option would be for the user
to do:

	ln -sf python3 python3.11  # or something equivalent

-

Btw, the same type of packaging multiple versions of python and 
Sphinx is quite common on several distributions. This is not
an issue specific to openSUSE.

Also, notice that we're talking here about building docs with
distros that are using a 9+ years old python3, e.g.:

	- openSUSE Leap;
	- RHEL8-based distros;
	- other old LTS and EOL distros that have a too old Python 
	  versions.

Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jani Nikula]

On Wed, 03 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Em Wed, 03 Sep 2025 10:47:28 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
>
>> On Tue, 02 Sep 2025, Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
>> > I'm with Jon here, I wouldn't blindly override the Python interpreter
>> > selected by the user. What we could however do is print a message if we
>> > detect a version of Python that could improve performance, telling the
>> > user they could switch.  
>> 
>> Just piling on here, totally agreed.
>> 
>> Don't surprise the user. Let the user be in control. If they choose a
>> silly combo, let them. You know, they might be debugging the issues in
>> that silly combo to begin with, and don't want to jump through hoops to
>> work around tools that think they know better.
>> 
>> Letting the user know they're using a silly combo and suggesting better
>> alternatives is a whole different matter.
>
> The code can be changed to suggest a better alternative, and having an
> optional argument to load the newest one automatically, but the issue
> still remains:
>
> if one has native python3.6, for instance, and he installed a newer one
> to build htmldocs with:
>
> 	dnf install -y python3.11 # or zypper/apt-get/urpmi/... install python3.11
>
> python3 will still be a soft link to python3.5, and all attempts to
> build docs will fail.

If the user wants to shoot themselves in the foot, let them.

> With the approach I'm taken, Makefile will call sphinx-pre-install
> and sphinx-build-wrapper. They both use a function that checks for
> incompatible versions, re-running the code with one which works,
> warning the user about that. So, on openSUSE leap, for instance, 
> we have two official packages with Sphinx:
>
> 	$ grep PRETTY_NAME /etc/os-release 
> 	PRETTY_NAME="openSUSE Leap 15.6"
>
> 	$ zypper search sphinx |grep -E "python\S+\-Sphinx "
> 	   | python3-Sphinx                          | Python documentation generator                                              | package
> 	i+ | python311-Sphinx                        | Python documentation generator                                              | package
>
>
> There, python3-Sphinx is too old:
>
> 	# sphinx-build --version
> 	sphinx-build 2.3.1
>
> As docs minimal requirement is 3.4.3. Also, it is based on python3.6,
> which is not compatible with the kernel Sphinx extensions.
>
> Which the approach I took, once one installs python311-Sphinx like I did,
> the build happens cleanly:
>
> 	$ make htmldocs
> 	# sphinx-pre-install, which checks is needed packages are installed
> 	Python 3.6.15 not supported. Changing to /usr/bin/python3.11
> 	# sphinx-build-wrapper, which calls Sphinx to build the docs
> 	Python version: 3.11.9
> 	Python 3.6.15 not supported. Changing to /usr/bin/python3.11
> 	Python version: 3.11.9
> 	Using alabaster theme
> 	Using Python kernel-doc
> 	...
>
> but if we remove the switch logic:
>
>         -print(f"Python {python_ver} not supported. Changing to {new_python_cmd}")
>         -
>         -try:
>         -    os.execv(new_python_cmd, args)
>         -except OSError as e:
>         -    sys.exit(f"Failed to restart with {new_python_cmd}: {e}")
>         -
> 	+print(f"Python {python_ver} not supported.")
>
> The build will then fail:
>
> 	# make htmldocs
> 	Python 3.6.15 not supported.
> 	Python 3.6.15 not supported.
> 	Traceback (most recent call last):
> 	  File "/usr/bin/sphinx-build", line 5, in <module>
> 	    from sphinx.cmd.build import main
> 	ModuleNotFoundError: No module named 'sphinx'
>
> because:
>
> - python3.6 is not supported;
> - sphinx-build is installed with python311-Sphinx, which also installed
>   python311 package.
>
> See, in this case, the user wanted docs build to use python3.11, as
> it installed python311-Sphinx package from a distro official package.
>
> Without the version-check logic, the only option would be for the user
> to do:
>
> 	ln -sf python3 python3.11  # or something equivalent
>
> -
>
> Btw, the same type of packaging multiple versions of python and 
> Sphinx is quite common on several distributions. This is not
> an issue specific to openSUSE.
>
> Also, notice that we're talking here about building docs with
> distros that are using a 9+ years old python3, e.g.:
>
> 	- openSUSE Leap;
> 	- RHEL8-based distros;
> 	- other old LTS and EOL distros that have a too old Python 
> 	  versions.
>
> Thanks,
> Mauro

-- 
Jani Nikula, Intel

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Wed, 03 Sep 2025 08:29:34 +0200
Johannes Berg <johannes@sipsolutions.net> escreveu:

> On Tue, 2025-09-02 at 12:58 -0600, Jonathan Corbet wrote:
> > > 
> > > The idea is not to override the search path: instead, to use it to
> > > check if the user installed other /usr/bin/python3.* files (or on
> > > some other part of PATH). Most distributions nowadays come with 
> > > multiple python versions. I can't see a downside (*) of not using 
> > > a newer version that the user had installed on his system and
> > > has it on PATH.  
> > 
> > But overriding the path is exactly what this would be doing.  It doesn't
> > seem right to say "we know better than you do" and circumvent the
> > configured path; the user may well have reasons for setting things up
> > the way they did.  
> 
> Absolutely! Please don't ever do this.
> 
> For example, use case we have: using nix-shell to lock down the software
> used to build, for reproducible builds and similar reasons. Without --
> pure, PATH may still contain (last!) software from the system itself,
> but it should basically never be used.

if the PATH is mangled, you'll have a lot more problems than just
building docs as it will pick wrong exec files anyway.

In the particular case of docs, if you have, for instance, two different
python versions, one with sphinx and another one without it, it may pick
the wrong one, causing the build to fail. There's nothing the build system
can do to solve it. The proper fix would be to wrap the calling logic
to save the env before running under nix-shell, restoring env afterwards.

Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Johannes Berg]

On Wed, 2025-09-03 at 12:42 +0200, Mauro Carvalho Chehab wrote:
> > For example, use case we have: using nix-shell to lock down the software
> > used to build, for reproducible builds and similar reasons. Without --
> > pure, PATH may still contain (last!) software from the system itself,
> > but it should basically never be used.
> 
> if the PATH is mangled, you'll have a lot more problems than just
> building docs as it will pick wrong exec files anyway.

Err, no? To search a binary, directories in $PATH are meant to be
searched in order of appearance. It's well-defined which one you pick
for which, and this setup takes advantage of that (with a rather long
$PATH) to control the binaries used for the build.

> In the particular case of docs, if you have, for instance, two different
> python versions, one with sphinx and another one without it, it may pick
> the wrong one, causing the build to fail. There's nothing the build system
> can do to solve it. The proper fix would be to wrap the calling logic
> to save the env before running under nix-shell, restoring env afterwards.

I don't follow. If this setup breaks the build then that's good, I'll
fix the env. If the build does magic inside and sort of ignores $PATH,
that's bad.

johannes

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Johannes Berg]

On Wed, 2025-09-03 at 12:45 +0200, Johannes Berg wrote:
> 
> I don't follow. If this setup breaks the build then that's good, I'll
> fix the env. If the build does magic inside and sort of ignores $PATH,
> that's bad.

Or maybe it's not ignoring $PATH, but rather picking the "best"
python3.xy binary from $PATH - still that's annoying because you'd have
to control which ones are there and/or know which ones it might pick.

Far more predictable and usable to just use "python3" and print a
message saying you might want to use a better version if you think it's
too slow.

johannes

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

Em Wed, 03 Sep 2025 12:45:25 +0200
Johannes Berg <johannes@sipsolutions.net> escreveu:

> On Wed, 2025-09-03 at 12:42 +0200, Mauro Carvalho Chehab wrote:
> > > For example, use case we have: using nix-shell to lock down the software
> > > used to build, for reproducible builds and similar reasons. Without --
> > > pure, PATH may still contain (last!) software from the system itself,
> > > but it should basically never be used.  
> > 
> > if the PATH is mangled, you'll have a lot more problems than just
> > building docs as it will pick wrong exec files anyway.  
> 
> Err, no? To search a binary, directories in $PATH are meant to be
> searched in order of appearance. It's well-defined which one you pick
> for which, and this setup takes advantage of that (with a rather long
> $PATH) to control the binaries used for the build.

Yes. So? the logic does that.

> > In the particular case of docs, if you have, for instance, two different
> > python versions, one with sphinx and another one without it, it may pick
> > the wrong one, causing the build to fail. There's nothing the build system
> > can do to solve it. The proper fix would be to wrap the calling logic
> > to save the env before running under nix-shell, restoring env afterwards.  
> 
> I don't follow. If this setup breaks the build then that's good, I'll
> fix the env. If the build does magic inside and sort of ignores $PATH,
> that's bad.

The build logic does follow PATH. If python --version < 3.7, it will
seek, within PATH, for python > 3.6.

Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Laurent Pinchart]

On Wed, Sep 03, 2025 at 03:39:05PM +0200, Mauro Carvalho Chehab wrote:
> Em Wed, 03 Sep 2025 12:45:25 +0200 Johannes Berg escreveu:
> > On Wed, 2025-09-03 at 12:42 +0200, Mauro Carvalho Chehab wrote:
> > > > For example, use case we have: using nix-shell to lock down the software
> > > > used to build, for reproducible builds and similar reasons. Without --
> > > > pure, PATH may still contain (last!) software from the system itself,
> > > > but it should basically never be used.  
> > > 
> > > if the PATH is mangled, you'll have a lot more problems than just
> > > building docs as it will pick wrong exec files anyway.  
> > 
> > Err, no? To search a binary, directories in $PATH are meant to be
> > searched in order of appearance. It's well-defined which one you pick
> > for which, and this setup takes advantage of that (with a rather long
> > $PATH) to control the binaries used for the build.
> 
> Yes. So? the logic does that.
> 
> > > In the particular case of docs, if you have, for instance, two different
> > > python versions, one with sphinx and another one without it, it may pick
> > > the wrong one, causing the build to fail. There's nothing the build system
> > > can do to solve it. The proper fix would be to wrap the calling logic
> > > to save the env before running under nix-shell, restoring env afterwards.  
> > 
> > I don't follow. If this setup breaks the build then that's good, I'll
> > fix the env. If the build does magic inside and sort of ignores $PATH,
> > that's bad.
> 
> The build logic does follow PATH. If python --version < 3.7, it will
> seek, within PATH, for python > 3.6.

Please, let's stop here. As pointed out by Jani, Johannes and Jon,
that's not a good idea. If the user wants to shoot themselves in the
foot, that's all fine. Printing a message to indicate they may want to
use a more recent Python version is totally fine. Picking a Python
interpreter manually to override the python3 symlink is not fine.

-- 
Regards,

Laurent Pinchart

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

On Wed, Sep 03, 2025 at 12:54:04PM +0200, Johannes Berg wrote:
> On Wed, 2025-09-03 at 12:45 +0200, Johannes Berg wrote:
> > 
> > I don't follow. If this setup breaks the build then that's good, I'll
> > fix the env. If the build does magic inside and sort of ignores $PATH,
> > that's bad.
> 
> Or maybe it's not ignoring $PATH, but rather picking the "best"
> python3.xy binary from $PATH - still that's annoying because you'd have
> to control which ones are there and/or know which ones it might pick.
> 
> Far more predictable and usable to just use "python3" and print a
> message saying you might want to use a better version if you think it's
> too slow.

There are actually 3 different issues that depend on python version:

1. sphinx-pre-install:

    This used to be a Perl script. The goal is to check if sphinx-build
    is installed and works, and identify missing dependencies.

    The problem is: if one installs python3xx-Sphinx, instead of
    python3-Sphinx, the script will fail, except if it first switches
    to python3.xx;

2. sphinx-build logic inside makefile, required for doc-specific targets:

   - If python < 3.7, doc builds fail;
   - If python3xx-Sphinx is installed, build only works if started using
     the right python3.xx exec

3. kernel-doc via command line. Python >=3.6 and <= 3.11 works. It is
   just 60% slower.

For (3), I agree with you: let it run at the slow way, printing a warning;
eventually suggesting a newer version and, as Jon suggested, added a
command line like --newest-python that could optionally pick the fastest
version.

Now, for (1) and (2), it should be possible to allow building docs even
if the distro requires Python < 3.7, providing extra packages for newer
Python, as almost all distros do. See, several distros require python
on their minimal install images, because it can be using during package
install. Removing the default python replacing by a new version may break
the system, as the newer version may not be backward-compatible.

So, what distros do, instead, to ensure backward-compatibility is to
provide multpile Python versions (together with python libraries).

The htmldocs/pdfdocs/... targets must support it somehow.

The best alternative seems to check if:

- python version is bellow the minimal supported version;
- there is a newer python binary at PATH;
- check if sphinx-build runs with the newest version.

If all those 3 conditions are met, build docs with a version that works,
printing a message to tell the user what Python binary was used.

Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Laurent Pinchart]

On Wed, Sep 03, 2025 at 04:57:15PM +0200, Mauro Carvalho Chehab wrote:
> On Wed, Sep 03, 2025 at 12:54:04PM +0200, Johannes Berg wrote:
> > On Wed, 2025-09-03 at 12:45 +0200, Johannes Berg wrote:
> > > 
> > > I don't follow. If this setup breaks the build then that's good, I'll
> > > fix the env. If the build does magic inside and sort of ignores $PATH,
> > > that's bad.
> > 
> > Or maybe it's not ignoring $PATH, but rather picking the "best"
> > python3.xy binary from $PATH - still that's annoying because you'd have
> > to control which ones are there and/or know which ones it might pick.
> > 
> > Far more predictable and usable to just use "python3" and print a
> > message saying you might want to use a better version if you think it's
> > too slow.
> 
> There are actually 3 different issues that depend on python version:
> 
> 1. sphinx-pre-install:
> 
>     This used to be a Perl script. The goal is to check if sphinx-build
>     is installed and works, and identify missing dependencies.
> 
>     The problem is: if one installs python3xx-Sphinx, instead of
>     python3-Sphinx, the script will fail, except if it first switches
>     to python3.xx;
> 
> 2. sphinx-build logic inside makefile, required for doc-specific targets:
> 
>    - If python < 3.7, doc builds fail;
>    - If python3xx-Sphinx is installed, build only works if started using
>      the right python3.xx exec
> 
> 3. kernel-doc via command line. Python >=3.6 and <= 3.11 works. It is
>    just 60% slower.
> 
> For (3), I agree with you: let it run at the slow way, printing a warning;
> eventually suggesting a newer version and, as Jon suggested, added a
> command line like --newest-python that could optionally pick the fastest
> version.
> 
> Now, for (1) and (2), it should be possible to allow building docs even
> if the distro requires Python < 3.7, providing extra packages for newer
> Python, as almost all distros do. See, several distros require python
> on their minimal install images, because it can be using during package
> install. Removing the default python replacing by a new version may break
> the system, as the newer version may not be backward-compatible.
> 
> So, what distros do, instead, to ensure backward-compatibility is to
> provide multpile Python versions (together with python libraries).
> 
> The htmldocs/pdfdocs/... targets must support it somehow.
> 
> The best alternative seems to check if:
> 
> - python version is bellow the minimal supported version;
> - there is a newer python binary at PATH;
> - check if sphinx-build runs with the newest version.
> 
> If all those 3 conditions are met, build docs with a version that works,
> printing a message to tell the user what Python binary was used.

This should fail. If you really insist you could print a message telling
the user there's another Python version on their system that may work,
but selecting it automatically isn't a good idea.

-- 
Regards,

Laurent Pinchart

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Johannes Berg]

On Wed, 2025-09-03 at 16:57 +0200, Mauro Carvalho Chehab wrote:
> There are actually 3 different issues that depend on python version:
> 
> 1. sphinx-pre-install:
> 
>     This used to be a Perl script. The goal is to check if sphinx-build
>     is installed and works, and identify missing dependencies.
> 
>     The problem is: if one installs python3xx-Sphinx, instead of
>     python3-Sphinx, the script will fail, except if it first switches
>     to python3.xx;

So let it fail. Fail is fine, at least it's a clear signal. The python3-
Spinx package will anyway be a sort of meta-package that's basically
empty and depends on a specific version.

> 2. sphinx-build logic inside makefile, required for doc-specific targets:
> 
>    - If python < 3.7, doc builds fail;

Fine, python 3.7 is really old by now, if you actually have python3==3.7
you probably have other problems anyway. I don't think we need to
support that.

>    - If python3xx-Sphinx is installed, build only works if started using
>      the right python3.xx exec

Yeah, but again, we can require python3-sphinx here.

> Now, for (1) and (2), it should be possible to allow building docs even
> if the distro requires Python < 3.7, providing extra packages for newer
> Python, as almost all distros do. See, several distros require python
> on their minimal install images, because it can be using during package
> install. Removing the default python replacing by a new version may break
> the system, as the newer version may not be backward-compatible.

Umm, no? I'm not sure there's a need to cater to truly ancient software
in today's kernel build environment. Even debian *oldoldstable* has
python3==3.9:
https://packages.debian.org/bullseye/python3

> The best alternative seems to check if:
> 
> - python version is bellow the minimal supported version;
> - there is a newer python binary at PATH;
> - check if sphinx-build runs with the newest version.
> 
> If all those 3 conditions are met, build docs with a version that works,
> printing a message to tell the user what Python binary was used.

I still disagree. The only predictable thing is to use "python3" and
associated python3-xyz tools, and let things fail if those versions are
too old.

Picking a random different version that will depend on the kernel
version etc. is just going to introduce more moving parts and will
eventually be painful.

johannes

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Konstantin Ryabitsev]

On Wed, Sep 03, 2025 at 05:07:29PM +0200, Laurent Pinchart wrote:
> This should fail. If you really insist you could print a message telling
> the user there's another Python version on their system that may work,
> but selecting it automatically isn't a good idea.

Just to underline this -- as a systems person, the thing I expect from scripts
is predictability. If I run `python3 --version`, before running a script, I
expect that any script invoking python3 will be using that version. If a
script uses a different version of python without being explicitly told to do
so, it would be confusing and, I expect, frustrating.

-K

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Miguel Ojeda]

On Wed, Sep 3, 2025 at 5:07 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> If you really insist you could print a message telling
> the user there's another Python version on their system that may work,

There is also precedent for this in the kernel already, e.g. for Rust,
we do the warning thing (or error) depending on what we versions we
find etc.

The user is the one expected to fix the call to `make` and/or fix
their environment if needed.

Cheers,
Miguel

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

On Wed, Sep 03, 2025 at 05:11:12PM +0200, Johannes Berg wrote:
> On Wed, 2025-09-03 at 16:57 +0200, Mauro Carvalho Chehab wrote:
> > There are actually 3 different issues that depend on python version:
> > 
> > 1. sphinx-pre-install:
> > 
> >     This used to be a Perl script. The goal is to check if sphinx-build
> >     is installed and works, and identify missing dependencies.
> > 
> >     The problem is: if one installs python3xx-Sphinx, instead of
> >     python3-Sphinx, the script will fail, except if it first switches
> >     to python3.xx;
> 
> So let it fail. Fail is fine, at least it's a clear signal. The python3-
> Spinx package will anyway be a sort of meta-package that's basically
> empty and depends on a specific version.

No, that's not the case. On Leap, python3-Sphinx uses python 3.6 and has
Sphinx version 2.3.x, which is too old.

> > 2. sphinx-build logic inside makefile, required for doc-specific targets:
> > 
> >    - If python < 3.7, doc builds fail;
> 
> Fine, python 3.7 is really old by now, if you actually have python3==3.7
> you probably have other problems anyway. I don't think we need to
> support that.

On our past discusions on linux-doc ML, we opted to support the latest
openSUSE LTS version (Leap).

> >    - If python3xx-Sphinx is installed, build only works if started using
> >      the right python3.xx exec
> 
> Yeah, but again, we can require python3-sphinx here.

See above. python311-Sphinx is OK, and it is a distro-installed package.

I can't see why we can't use to build the docs if the user explicitly
installed it.

> > Now, for (1) and (2), it should be possible to allow building docs even
> > if the distro requires Python < 3.7, providing extra packages for newer
> > Python, as almost all distros do. See, several distros require python
> > on their minimal install images, because it can be using during package
> > install. Removing the default python replacing by a new version may break
> > the system, as the newer version may not be backward-compatible.
> 
> Umm, no? I'm not sure there's a need to cater to truly ancient software
> in today's kernel build environment. Even debian *oldoldstable* has
> python3==3.9:
> https://packages.debian.org/bullseye/python3

True, but at least one of the major LTS distros don't have it(*).

We can review it after Leap is replaced for the next openSUSE release.

(*) also, RHEL8 (and its derivated releases) suffer the same issues
     and they aren't EOL yet.

For most of us, I doubt the fallback logic would ever be used.

> > The best alternative seems to check if:
> > 
> > - python version is bellow the minimal supported version;
> > - there is a newer python binary at PATH;
> > - check if sphinx-build runs with the newest version.
> > 
> > If all those 3 conditions are met, build docs with a version that works,
> > printing a message to tell the user what Python binary was used.
> 
> I still disagree. The only predictable thing is to use "python3" and
> associated python3-xyz tools, and let things fail if those versions are
> too old.
> 
> Picking a random different version that will depend on the kernel
> version etc. is just going to introduce more moving parts and will
> eventually be painful.

When it becomes painful, we can drop it.

Anyway, I'll let it for Jon to decide.

-- 
Thanks,
Mauro

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> On Wed, Sep 03, 2025 at 05:11:12PM +0200, Johannes Berg wrote:
>> On Wed, 2025-09-03 at 16:57 +0200, Mauro Carvalho Chehab wrote:
>> > There are actually 3 different issues that depend on python version:
>> > 
>> > 1. sphinx-pre-install:
>> > 
>> >     This used to be a Perl script. The goal is to check if sphinx-build
>> >     is installed and works, and identify missing dependencies.
>> > 
>> >     The problem is: if one installs python3xx-Sphinx, instead of
>> >     python3-Sphinx, the script will fail, except if it first switches
>> >     to python3.xx;
>> 
>> So let it fail. Fail is fine, at least it's a clear signal. The python3-
>> Spinx package will anyway be a sort of meta-package that's basically
>> empty and depends on a specific version.
>
> No, that's not the case. On Leap, python3-Sphinx uses python 3.6 and has
> Sphinx version 2.3.x, which is too old.

That's Leap 15, presumably?  Given that 16 is due Real Soon Now, perhaps
before any kernel with these changes is released, do we need to concern
ourselves with that?

> True, but at least one of the major LTS distros don't have it(*).
>
> We can review it after Leap is replaced for the next openSUSE release.
>
> (*) also, RHEL8 (and its derivated releases) suffer the same issues
>      and they aren't EOL yet.
>
> For most of us, I doubt the fallback logic would ever be used.

CentOS 8 stream went EOL over a year ago.  How many people have systems
stuck on RHEL 8 and are using them to do docs builds?

> When it becomes painful, we can drop it.
>
> Anyway, I'll let it for Jon to decide.

I still really don't think that adding that stuff is a good idea; our
scripts should behave the way people expect them to and not go rooting
around for alternative interpreters to feed themselves to.  I appreciate
that you want to make things Just Work for people, that is a great goal,
but this seems a step too far.

Thanks,

jon

Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab]

On Wed, Sep 03, 2025 at 09:37:36AM -0600, Jonathan Corbet wrote:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > On Wed, Sep 03, 2025 at 05:11:12PM +0200, Johannes Berg wrote:
> >> On Wed, 2025-09-03 at 16:57 +0200, Mauro Carvalho Chehab wrote:
> >> > There are actually 3 different issues that depend on python version:
> >> > 
> >> > 1. sphinx-pre-install:
> >> > 
> >> >     This used to be a Perl script. The goal is to check if sphinx-build
> >> >     is installed and works, and identify missing dependencies.
> >> > 
> >> >     The problem is: if one installs python3xx-Sphinx, instead of
> >> >     python3-Sphinx, the script will fail, except if it first switches
> >> >     to python3.xx;
> >> 
> >> So let it fail. Fail is fine, at least it's a clear signal. The python3-
> >> Spinx package will anyway be a sort of meta-package that's basically
> >> empty and depends on a specific version.
> >
> > No, that's not the case. On Leap, python3-Sphinx uses python 3.6 and has
> > Sphinx version 2.3.x, which is too old.
> 
> That's Leap 15, presumably?

Yes. Leap 15.6 (the latest one)

> Given that 16 is due Real Soon Now, perhaps
> before any kernel with these changes is released, do we need to concern
> ourselves with that?

Not sure how it works on openSUSE, but on other LTS distros, people
usually wait at least for x.1 version (16.1) before migrating their
systems.

> > True, but at least one of the major LTS distros don't have it(*).
> >
> > We can review it after Leap is replaced for the next openSUSE release.
> >
> > (*) also, RHEL8 (and its derivated releases) suffer the same issues
> >      and they aren't EOL yet.
> >
> > For most of us, I doubt the fallback logic would ever be used.
> 
> CentOS 8 stream went EOL over a year ago.  How many people have systems
> stuck on RHEL 8 and are using them to do docs builds?
> 
> > When it becomes painful, we can drop it.
> >
> > Anyway, I'll let it for Jon to decide.
> 
> I still really don't think that adding that stuff is a good idea; our
> scripts should behave the way people expect them to and not go rooting
> around for alternative interpreters to feed themselves to.  I appreciate
> that you want to make things Just Work for people, that is a great goal,
> but this seems a step too far.

Ok, as I said, it is up to you to decide. I sent already a patch
series with the last patch making the build break with python 3.6:

    https://lore.kernel.org/linux-doc/cover.1756913837.git.mchehab+huawei@kernel.org/

Patches 1 and 2 should be OK to be merged. Patch 3 is the one that
will break for Leap15/RHEL8 and other distros where python 3.6 is
required for the distro default (and typically mandatory) python3
package.

Feel free to apply it or not as you wish.

Thanks,
Mauro