Re: [PATCH v3 00/33] Implement kernel-doc in Python

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

Em Tue, 15 Apr 2025 10:03:47 +0300
Andy Shevchenko <andriy.shevchenko@intel.com> escreveu:

> On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote:
> > On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote:  
> > > Andy Shevchenko <andriy.shevchenko@intel.com> writes:  
> > > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote:  
> > > >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> > > >>   
> > > >> > This changeset contains the kernel-doc.py script to replace the verable
> > > >> > kernel-doc originally written in Perl. It replaces the first version and the
> > > >> > second series I sent on the top of it.  
> > > >> 
> > > >> OK, I've applied it, looked at the (minimal) changes in output, and
> > > >> concluded that it's good - all this stuff is now in docs-next.  Many
> > > >> thanks for doing this!
> > > >> 
> > > >> I'm going to hold off on other documentation patches for a day or two
> > > >> just in case anything turns up.  But it looks awfully good.  
> > > >
> > > > This started well, until it becomes a scripts/lib/kdoc.
> > > > So, it makes the `make O=...` builds dirty *). Please make sure this doesn't leave
> > > > "disgusting turd" )as said by Linus) in the clean tree.
> > > >
> > > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a solution.  
> > > 
> > > If nothing else, "make cleandocs" should clean it up, certainly.

Not sure about that, as __pycache__ is completely managed by Python: it
will not only create it for scripts/lib, but also for all Python libraries,
including the Sphinx ones.

IMO, it makes more sense, instead, to ensure that __pycache__ won't be
created at the sourcedir if O= is used, but ignore it if this is created.

Btw, the same problem should already happen with get_abi.py, as it also
uses "import" from scripts/lib. So, we need a more generic solution. See
below.

> > > 
> > > We can also tell CPython to not create that directory at all.  I'll run
> > > some tests to see what the effect is on the documentation build times;
> > > I'm guessing it will not be huge...  

I doubt it would have much impact for kernel-doc, but it can have some impact
for Sphinx, as disabling Python JIT to store bytecode would affect it too.

-

Andy, 

Could you please remove __pycache__ and set this env:

	PYTHONDONTWRITEBYTECODE=1

before building the Kernel? If this works, one alternative would be to 
set it when O= is used.

> > I do not build documentation at all, it's just a regular code build that leaves
> > tree dirty.
> > 
> > $ python3 --version
> > Python 3.13.2
> > 
> > It's standard Debian testing distribution, no customisation in the code.
> > 
> > To reproduce.
> > 1) I have just done a new build to reduce the churn, so, running make again does nothing;
> > 2) The following snippet in shell shows the issue
> > 
> > $ git clean -xdf
> > $ git status --ignored
> > On branch ...
> > nothing to commit, working tree clean
> > 
> > $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64
> > make[1]: Entering directory '...'
> >   GEN     Makefile
> >   DESCEND objtool
> >   CALL    .../scripts/checksyscalls.sh
> >   INSTALL libsubcmd_headers
> > .pylintrc: warning: ignored by one of the .gitignore files
> > Kernel: arch/x86/boot/bzImage is ready  (#23)
> > make[1]: Leaving directory '...'
> > 
> > $ touch drivers/gpio/gpiolib-acpi.c
> > 
> > $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64
> > make[1]: Entering directory '...'
> >   GEN     Makefile
> >   DESCEND objtool
> >   CALL    .../scripts/checksyscalls.sh
> >   INSTALL libsubcmd_headers
> > ...
> >   OBJCOPY arch/x86/boot/setup.bin
> >   BUILD   arch/x86/boot/bzImage
> > Kernel: arch/x86/boot/bzImage is ready  (#24)
> > make[1]: Leaving directory '...'
> > 
> > $ git status --ignored
> > On branch ...
> > Untracked files:
> >   (use "git add <file>..." to include in what will be committed)
> > 	scripts/lib/kdoc/__pycache__/
> > 
> > nothing added to commit but untracked files present (use "git add" to track)  
> 
> FWIW, I repeated this with removing the O=.../out folder completely, so it's
> fully clean build. Still the same issue.
> 
> And it appears at the very beginning of the build. You don't need to wait to
> have the kernel to be built actually.

Depending on your .config, kernel-doc will be called even without building
documentation to check for some problems at kernel-doc tags.

> 
> > It's 100% reproducible on my side. I am happy to test any patches to fix this.
> > It's really annoying "feature" for `make O=...` builds. Also note that
> > theoretically the Git worktree may be located on read-only storage / media
> > and this can induce subtle issues.  

Python's JIT compiler automatically creates __pycache__ whenever it
encounters an "import" and the *.pyc is older than the script (or doesn't
exist). This happens with external libraries, and also with the internal
ones, like the ones we now have at the Kernel.

I dunno what happens if the FS is read-only. I would expect that the JIT
compiler would just work as if bytecode creation is disabled.

That's said, I never played myself with __pycache__.

Yet, I have some raw ideas about how to deal with that. This requires
more tests, though. I can see some possible solutions for that:

1. Assuming that PYTHONDONTWRITEBYTECODE=1 works, the build system could
   set it if O= is used. This would have some performance impact for both
   Kernel compilation (because kernel-doc is called to check doc issues),
   and for Kernel compilation itself. I dunno how much it would impact,
   but this is probably the quickest solution to implement;

2. when O=<targetdir> is used, copy scripts/lib/*/*.py to the target
   directory and change kernel-doc.py to use <targetdir> for library search
   on such case. This way, the __pycache__ would be created at the 
   <targetdir>. This might work as well with symlinks. The performance
   impact here would be minimal, but it will require an extra step for
   O= to copy data (or to create symlinks);

3. eventually there is a way to teach Python to place the __pycache__
   at <targetdir>, instead of <sourcedir>. If supported, this would
   be the cleanest solution.

Regards,
Mauro