Re: [PATCH 07/13] docs: move sphinx-pre-install to tools/doc

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

Em Fri, 15 Aug 2025 07:18:51 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> >> Between "tools/doc" and "tools/docs" I don't really have overly strong
> >> feelings; if you work has the latter we can just stick with that.  If
> >> you propose "tools/Documentation", though, expect resistance :)  
> >
> > <joke>
> > Heh, I'm tempted to propose:
> > 	/Documentation -> /docs
> > or
> > 	/Documentation -> /Docs
> > </joke>  
> 
> I proposed the former at a maintainers summit a few years ago ... the
> response was less than fully positive.  I guess the fact that docs have
> the longest and only capitalized top-level directory name shows their
> relative importance :)

:-)

> > Ok, so let's keep tools/docs then. We need to decide about python
> > lib. On my series, I'm placing at tools/docs/lib, but I guess we
> > can change it later.
> >
> > From my side, I would prefer to have a single directory for tools,
> > as we may place there things that aren't specific to docs.
> >
> > For instance, I have my own class that I use for command execution,
> > using asyncio. The rationale is that it allows output messages in
> > real time without needing to wait for the entire process to end(*).
> >
> > (*) I recently discovered a way to do that without needing asyncio,
> >     which makes the code a little bit simpler.  
> 
> This is just flushing the output buffer?  Asyncio seems like a heavy
> tool for that; I guess I'm missing something.

Yes, flushing output as they arrive while storing results from stdout and
stderr and handling  "stdin", eventually dynamically printing stdin when
debug is enabled. The problem sounds simple, and this is really easy to
implement in Perl. However, on Python, this ends to be a very complex
problem that one can only get all the caveats after implementing unit 
tests and then discovering, for instance, that, if stdin is bigger than 
32KB, Python read logic freezes forever (or until gather() timeout, and
the stdin was incomplete.

To fix it, one needs to implement a different stdin function that can't
wait anymore for the end of a read operation, but instead needs to pick
buffers in small chunks. Plus, the subprocess execution method requires
some unusual buffering parameters.
 
> > Either using asyncio or not, a class for such purpose is something
> > that multiple tools could use. So, a generic dir like tools/lib, 
> > lib/python, ... IMO makes sense.  
> 
> I agree with this, anyway.  "tools/python" might end up as the winning
> suggestion. 
> 
> >> As I said, my series was an RFC to see what it would look like; it did't
> >> take all that long the first time around, and will be even quicker to
> >> redo on top of a new base, whatever that turns out to be.  
> >
> > With regards to the RFC, IMO we still may need to discuss how we'll end 
> > placing libraries under a LIBDIR. IMO, your RFC should also propose
> > a directory structure. I mean, we could have something like:
> >
> > 	LIBDIR     # either tools/docs/lib, tools/lib, lib/python or whatever
> > 	|
> > 	+---> common
> > 	\---> docs
> > 		|
> > 	    	+---> kdoc
> > 	    	\---> abi
> >
> > We could instead do:
> > 	- flatten "common" to LIBDIR; or:
> > 	- flatten "docs" to LIBDIR; or:
> > 	- flatten both but keeping kdoc, abi, ... directories inside
> > 	  LIBDIR; or:
> > 	- have a completely flatten directory with everything
> > 	  under LIBDIR.  
> 
> I'm not sure we need the common/docs intermediate directory.
> 
> Meanwhile, I had a related, possibly unpopular idea...  Start with
> .../tools/python/kernel and put a basic __init__.py file there;
> everything else would go into that directory or before.  The imports
> would then read something like:
> 
>   from kernel import abi_parser

Not against something similar to it, but IMO "kernel" is a bad
name as it sounds something that runs in kernel stace or for Kernel
build. It could be, instead:

	from lib import abi_parser

Yet, I guess it may still need to add something at PATH, depending from
where current working dir the script was called (but tests required).

For instance Documentation/sphinx would need to pick lib from
"../../tools/python". So, I guess Sphinx extensions need first to
do:

	sys.path.insert(0, "../../tools/python")

and then:

	from lib import abi_parser         # or:
	from lib.abi import abi_parser     # or:
	import .abi_parser


Btw, nothing prevents moving extensions from Documentation/sphinx
into tools/sphinx_extensions. We just need to add the path insert
at conf.py.

> That would make it clear which modules are ours and which come from
> elsewhere.
> 
> I was planning to toss together another RFC with that scheme, again to
> see what it would look like in practice.

> Thoughts?

Fine from my side.

Regards,
Mauro