docs: sphinx: avoid using the deprecated node.set_class()

docs: sphinx: avoid using the deprecated node.set_class()

[Jonathan Corbet]

Docutils emits a deprecation warning when the set_class() element method is
used; that warning disappears into the ether, but it also causes a crash
with docutils 0.19 when combined with certain versions of Sphinx.

Avoid the deprecated function and just append directly to the "classes"
attribute like the documentation says instead.

Reported-by: Akira Yokosawa <akiyks@gmail.com>
Fixes: d6d1df92c25f ("docs: automarkup: Mark up undocumented entities too")
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
TODO for the future: figure out where the warning is going

 Documentation/sphinx/automarkup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index e67eb8e19c22..563033f764bb 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -240,7 +240,7 @@ def add_and_resolve_xref(app, docname, domain, reftype, target, contnode=None):
     # mark it as a broken xref
     #
     if contnode:
-        contnode.set_class("broken_xref")
+        contnode['classes'].append("broken_xref")
     return contnode
 
 #
-- 
2.49.0

Re: docs: sphinx: avoid using the deprecated node.set_class()

[Akira Yokosawa]

Hi Jon,

On Thu, 19 Jun 2025 15:26:56 -0600, Jonathan Corbet wrote:
> Docutils emits a deprecation warning when the set_class() element method is
> used; that warning disappears into the ether, but it also causes a crash
> with docutils 0.19 when combined with certain versions of Sphinx.

To be accurate, I'd rather say:
                                                but it also causes a crash
  with docutils 0.19 when combined with any version of Sphinx whose
  requirement accepts it.

> 
> Avoid the deprecated function and just append directly to the "classes"
> attribute like the documentation says instead.

Nice!  This is the kind of fix I wish I could have come up with by myself.

Tested OK against debian:12's Sphinx 5.3.0, as well as Sphinx 3.4.3 of
debian:11 and almalinux:9, Sphinx 4.2.0 of Ubuntu 22.04 and other recent
distro Sphinx packages.

> 
> Reported-by: Akira Yokosawa <akiyks@gmail.com>

Closes: https://lore.kernel.org/de7bae91-3200-481f-9db2-c0dc382c91dd@gmail.com/

> Fixes: d6d1df92c25f ("docs: automarkup: Mark up undocumented entities too")
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

Tested-by: Akira Yokosawa <akiyks@gmail.com>

> ---
> TODO for the future: figure out where the warning is going
>
>  Documentation/sphinx/automarkup.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
> index e67eb8e19c22..563033f764bb 100644
> --- a/Documentation/sphinx/automarkup.py
> +++ b/Documentation/sphinx/automarkup.py
> @@ -240,7 +240,7 @@ def add_and_resolve_xref(app, docname, domain, reftype, target, contnode=None):
>      # mark it as a broken xref
>      #
>      if contnode:
> -        contnode.set_class("broken_xref")
> +        contnode['classes'].append("broken_xref")
>      return contnode
>  
>  #

Re: docs: sphinx: avoid using the deprecated node.set_class()

[Mauro Carvalho Chehab]

Em Fri, 20 Jun 2025 11:22:48 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> Hi Jon,
> 
> On Thu, 19 Jun 2025 15:26:56 -0600, Jonathan Corbet wrote:
> > Docutils emits a deprecation warning when the set_class() element method is
> > used; that warning disappears into the ether, but it also causes a crash
> > with docutils 0.19 when combined with certain versions of Sphinx.  
> 
> To be accurate, I'd rather say:
>                                                 but it also causes a crash
>   with docutils 0.19 when combined with any version of Sphinx whose
>   requirement accepts it.
> 
> > 
> > Avoid the deprecated function and just append directly to the "classes"
> > attribute like the documentation says instead.  
> 
> Nice!  This is the kind of fix I wish I could have come up with by myself.
> 
> Tested OK against debian:12's Sphinx 5.3.0, as well as Sphinx 3.4.3 of
> debian:11 and almalinux:9, Sphinx 4.2.0 of Ubuntu 22.04 and other recent
> distro Sphinx packages.
> 
> > 
> > Reported-by: Akira Yokosawa <akiyks@gmail.com>  
> 
> Closes: https://lore.kernel.org/de7bae91-3200-481f-9db2-c0dc382c91dd@gmail.com/
> 
> > Fixes: d6d1df92c25f ("docs: automarkup: Mark up undocumented entities too")
> > Signed-off-by: Jonathan Corbet <corbet@lwn.net>  
> 
> Tested-by: Akira Yokosawa <akiyks@gmail.com>

I didn't test it yet, but yesterday I wrote a script which allows us to test
for Sphinx version breakages on multiple versions in one go.

Using it (and again before this patch, but after my parser-yaml series), I 
noticed that 6.0.1 with "-jauto" with those packages:

alabaster==0.7.13
babel==2.17.0
certifi==2025.6.15
charset-normalizer==3.4.2
docutils==0.18.1
idna==3.10
imagesize==1.4.1
importlib_metadata==8.7.0
Jinja2==3.1.2
MarkupSafe==2.0.0
packaging==25.0
Pygments==2.19.1
PyYAML==5.3.1
requests==2.32.4
snowballstemmer==3.0.1
Sphinx==6.0.1
sphinxcontrib-applehelp==1.0.4
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
urllib3==2.4.0
zipp==3.23.0

is crashing. It sounds to me that the issue is internal to Sphinx, as it
runs well with -j1. One possible solution would be to modify:

	Documentation/sphinx/parallel-wrapper.sh

To force "-j1" if Sphinx version is 6.0.1 (and probably 6.0).

---

Jon,

If you prefer, 
Maybe you could run the test script before, to check if no regressions
happened with other versions.

I'll prepare a new version of my patch series today placing the check
script at the beginning.

Regards,
Mauro

> 
> > ---
> > TODO for the future: figure out where the warning is going
> >
> >  Documentation/sphinx/automarkup.py | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
> > index e67eb8e19c22..563033f764bb 100644
> > --- a/Documentation/sphinx/automarkup.py
> > +++ b/Documentation/sphinx/automarkup.py
> > @@ -240,7 +240,7 @@ def add_and_resolve_xref(app, docname, domain, reftype, target, contnode=None):
> >      # mark it as a broken xref
> >      #
> >      if contnode:
> > -        contnode.set_class("broken_xref")
> > +        contnode['classes'].append("broken_xref")

Just in case, I would change it to:

	if 'classes' in countnode:
		contnode['classes'].append("broken_xref")

just to avoid eventual surprises.

Regards,
Mauro

Re: docs: sphinx: avoid using the deprecated node.set_class()

[Akira Yokosawa]

Mauro!

On Fri, 20 Jun 2025 09:44:30 +0200, Mauro Carvalho Chehab wrote:
> Em Fri, 20 Jun 2025 11:22:48 +0900
> Akira Yokosawa <akiyks@gmail.com> escreveu:
> 
[...]

> 
> I didn't test it yet, but yesterday I wrote a script which allows us to test
> for Sphinx version breakages on multiple versions in one go.
> 
> Using it (and again before this patch, but after my parser-yaml series), I 
> noticed that 6.0.1 with "-jauto" with those packages:

Why did you pick 6.0.1, which was in the middle of successive releases in
early 6.x days. 
No distro Sphinx packagers have picked this version.

Just see the release history:

[2022-10-16]  5.3.0  ### stable ###
[2022-12-29]  6.0.0
[2023-01-05]  6.0.1
[2023-01-05]  6.1.0  6.1.1 
[2023-01-07]  6.1.2
[2023-01-10]  6.1.3  ### stable ###
[2023-04-23]  6.2.0

The crash you observed is hardly related to this fix.

I'd ignore this report as a random noise.
My Tested-by: still stands.

Regards,
Akira

Re: docs: sphinx: avoid using the deprecated node.set_class()

[Mauro Carvalho Chehab]

Em Fri, 20 Jun 2025 20:14:57 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> Mauro!
> 
> On Fri, 20 Jun 2025 09:44:30 +0200, Mauro Carvalho Chehab wrote:
> > Em Fri, 20 Jun 2025 11:22:48 +0900
> > Akira Yokosawa <akiyks@gmail.com> escreveu:
> >   
> [...]
> 
> > 
> > I didn't test it yet, but yesterday I wrote a script which allows us to test
> > for Sphinx version breakages on multiple versions in one go.
> > 
> > Using it (and again before this patch, but after my parser-yaml series), I 
> > noticed that 6.0.1 with "-jauto" with those packages:  
> 
> Why did you pick 6.0.1, which was in the middle of successive releases in
> early 6.x days. 

I added all major,minor,latest-patch version since 3.4.3 and added to
the script. I didn't check what of those are inside a distro or not.

> No distro Sphinx packagers have picked this version.

The hole idea is to have a script where we can automate build tests
with old versions. Perhaps it makes a sense to add a flag at the table
indicating what major distros have what sphinx version and a command
line parameter to either test all or just the ones shipped on major
distros.
> 
> Just see the release history:
> 
> [2022-10-16]  5.3.0  ### stable ###
> [2022-12-29]  6.0.0
> [2023-01-05]  6.0.1
> [2023-01-05]  6.1.0  6.1.1 
> [2023-01-07]  6.1.2
> [2023-01-10]  6.1.3  ### stable ###
> [2023-04-23]  6.2.0
> 
> The crash you observed is hardly related to this fix.

Almost certainly, the breakage with 6.0.1 is unrelated to this
change.

> I'd ignore this report as a random noise.
> My Tested-by: still stands.
> 
> Regards,
> Akira
> 

Re: docs: sphinx: avoid using the deprecated node.set_class()

[Mauro Carvalho Chehab]

Em Fri, 20 Jun 2025 15:05:39 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> Em Fri, 20 Jun 2025 20:14:57 +0900
> Akira Yokosawa <akiyks@gmail.com> escreveu:
> 
> > Mauro!
> > 
> > On Fri, 20 Jun 2025 09:44:30 +0200, Mauro Carvalho Chehab wrote:  
> > > Em Fri, 20 Jun 2025 11:22:48 +0900
> > > Akira Yokosawa <akiyks@gmail.com> escreveu:
> > >     
> > [...]
> >   
> > > 
> > > I didn't test it yet, but yesterday I wrote a script which allows us to test
> > > for Sphinx version breakages on multiple versions in one go.
> > > 
> > > Using it (and again before this patch, but after my parser-yaml series), I 
> > > noticed that 6.0.1 with "-jauto" with those packages:    
> > 
> > Why did you pick 6.0.1, which was in the middle of successive releases in
> > early 6.x days.   
> 
> I added all major,minor,latest-patch version since 3.4.3 and added to
> the script. I didn't check what of those are inside a distro or not.
> 
> > No distro Sphinx packagers have picked this version.  
> 
> The hole idea is to have a script where we can automate build tests
> with old versions. Perhaps it makes a sense to add a flag at the table
> indicating what major distros have what sphinx version and a command
> line parameter to either test all or just the ones shipped on major
> distros.
> > 
> > Just see the release history:
> > 
> > [2022-10-16]  5.3.0  ### stable ###
> > [2022-12-29]  6.0.0
> > [2023-01-05]  6.0.1
> > [2023-01-05]  6.1.0  6.1.1 
> > [2023-01-07]  6.1.2
> > [2023-01-10]  6.1.3  ### stable ###
> > [2023-04-23]  6.2.0
> > 
> > The crash you observed is hardly related to this fix.  
> 
> Almost certainly, the breakage with 6.0.1 is unrelated to this
> change.

Heh, I'm not even sure that the problem is with 6.0.1 or with
Fedora OOM killer setup...

Even with 64GB ram and 8GB swap(*), I'm getting lots of those:

jun 20 03:23:46 myhost kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
jun 20 03:23:46 myhost kernel: [   1762]   998  1762     4074      467       96      371         0    77824      144          -900 systemd-oomd
jun 20 03:23:46 myhost kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.kde.konsole-433443.scope,task=sphinx-build,pid=1043271,uid=1000
jun 20 03:23:46 myhost kernel: Out of memory: Killed process 1043271 (sphinx-build) total-vm:4222280kB, anon-rss:3934380kB, file-rss:688kB, shmem-rss:0kB, UID:1000 pgtables:7812kB oom_score_adj:200
jun 20 03:24:28 myhost kernel: sphinx-build invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=200
jun 20 03:24:28 myhost kernel:  oom_kill_process.cold+0xa/0xbe

Will do some extra texts here and try to adjust this.

(*) Granted, I need more swap... the FS was generated when 8GB
    were good enough ;-)
    Still 64GB RAM should be enough. Will try to change overcommit
    and see how it goes.

Thanks,
Mauro

Re: docs: sphinx: avoid using the deprecated node.set_class()

[Mauro Carvalho Chehab]

Em Fri, 20 Jun 2025 20:44:59 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> Em Fri, 20 Jun 2025 15:05:39 +0200
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:
> 
> > Em Fri, 20 Jun 2025 20:14:57 +0900
> > Akira Yokosawa <akiyks@gmail.com> escreveu:
> > 
> > > Mauro!
> > > 
> > > On Fri, 20 Jun 2025 09:44:30 +0200, Mauro Carvalho Chehab wrote:  
> > > > Em Fri, 20 Jun 2025 11:22:48 +0900
> > > > Akira Yokosawa <akiyks@gmail.com> escreveu:
> > > >     
> > > [...]
> > >   
> > > > 
> > > > I didn't test it yet, but yesterday I wrote a script which allows us to test
> > > > for Sphinx version breakages on multiple versions in one go.
> > > > 
> > > > Using it (and again before this patch, but after my parser-yaml series), I 
> > > > noticed that 6.0.1 with "-jauto" with those packages:    
> > > 
> > > Why did you pick 6.0.1, which was in the middle of successive releases in
> > > early 6.x days.   
> > 
> > I added all major,minor,latest-patch version since 3.4.3 and added to
> > the script. I didn't check what of those are inside a distro or not.
> > 
> > > No distro Sphinx packagers have picked this version.  
> > 
> > The hole idea is to have a script where we can automate build tests
> > with old versions. Perhaps it makes a sense to add a flag at the table
> > indicating what major distros have what sphinx version and a command
> > line parameter to either test all or just the ones shipped on major
> > distros.
> > > 
> > > Just see the release history:
> > > 
> > > [2022-10-16]  5.3.0  ### stable ###
> > > [2022-12-29]  6.0.0
> > > [2023-01-05]  6.0.1
> > > [2023-01-05]  6.1.0  6.1.1 
> > > [2023-01-07]  6.1.2
> > > [2023-01-10]  6.1.3  ### stable ###
> > > [2023-04-23]  6.2.0
> > > 
> > > The crash you observed is hardly related to this fix.  
> > 
> > Almost certainly, the breakage with 6.0.1 is unrelated to this
> > change.
> 
> Heh, I'm not even sure that the problem is with 6.0.1 or with
> Fedora OOM killer setup...
> 
> Even with 64GB ram and 8GB swap(*), I'm getting lots of those:
> 
> jun 20 03:23:46 myhost kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
> jun 20 03:23:46 myhost kernel: [   1762]   998  1762     4074      467       96      371         0    77824      144          -900 systemd-oomd
> jun 20 03:23:46 myhost kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.kde.konsole-433443.scope,task=sphinx-build,pid=1043271,uid=1000
> jun 20 03:23:46 myhost kernel: Out of memory: Killed process 1043271 (sphinx-build) total-vm:4222280kB, anon-rss:3934380kB, file-rss:688kB, shmem-rss:0kB, UID:1000 pgtables:7812kB oom_score_adj:200
> jun 20 03:24:28 myhost kernel: sphinx-build invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=200
> jun 20 03:24:28 myhost kernel:  oom_kill_process.cold+0xa/0xbe
> 
> Will do some extra texts here and try to adjust this.
> 
> (*) Granted, I need more swap... the FS was generated when 8GB
>     were good enough ;-)
>     Still 64GB RAM should be enough. Will try to change overcommit
>     and see how it goes.

Yeah, the problem with 6.0.1 was indeed with OOM killer. Once
I added more 64GB of swap, and wait for a long time, it finally
compleded the task without crashes:

	$ ./scripts/test_doc_build.py -m -V 6.0.1 -v
	...
	Finished doc build for Sphinx 6.0.1. Elapsed time: 00:31:02

	Summary:
        	Sphinx 6.0.1 elapsed time: 00:31:02

Looking at the past log I have handy, this is by far the worse one:

	Finished doc build for Sphinx 6.1.3. Elapsed time: 00:11:15
	Finished doc build for Sphinx 6.2.1. Elapsed time: 00:09:21
	Finished doc build for Sphinx 7.0.1. Elapsed time: 00:09:17
	Finished doc build for Sphinx 7.1.2. Elapsed time: 00:09:22
	Finished doc build for Sphinx 7.2.3. Elapsed time: 00:09:17
	Finished doc build for Sphinx 7.3.7. Elapsed time: 00:09:34
	Finished doc build for Sphinx 7.4.7. Elapsed time: 00:04:54
	Finished doc build for Sphinx 8.0.2. Elapsed time: 00:03:40
	Finished doc build for Sphinx 8.1.3. Elapsed time: 00:03:47
	Finished doc build for Sphinx 8.2.3. Elapsed time: 00:03:45

(3.4.3 was the previous "champion" with about 14 minutes)

All of them are using "-jauto" on a machine with 24 CPU threads.

The only one that didn't work with my past scenario was 6.0.1,
so OOM killer seems the one to blame: it is killing a sub-process,
but keeping the main one active, thus causing Sphinx to run for
a long time, only to notice at the end that something bad happened
and producing a completely bogus log. 

Heh, systemd-oomd, shame on you!

Thanks,
Mauro

Re: docs: sphinx: avoid using the deprecated node.set_class()

[Jonathan Corbet]

Akira Yokosawa <akiyks@gmail.com> writes:

> Hi Jon,
>
> On Thu, 19 Jun 2025 15:26:56 -0600, Jonathan Corbet wrote:
>> Docutils emits a deprecation warning when the set_class() element method is
>> used; that warning disappears into the ether, but it also causes a crash
>> with docutils 0.19 when combined with certain versions of Sphinx.
>
> To be accurate, I'd rather say:
>                                                 but it also causes a crash
>   with docutils 0.19 when combined with any version of Sphinx whose
>   requirement accepts it.

...or just "with docutils 0.19" and put the period there, perhaps?

>> Avoid the deprecated function and just append directly to the "classes"
>> attribute like the documentation says instead.
>
> Nice!  This is the kind of fix I wish I could have come up with by myself.

It helps to have broken it in the first place :)

> Tested OK against debian:12's Sphinx 5.3.0, as well as Sphinx 3.4.3 of
> debian:11 and almalinux:9, Sphinx 4.2.0 of Ubuntu 22.04 and other recent
> distro Sphinx packages.
>
>> 
>> Reported-by: Akira Yokosawa <akiyks@gmail.com>
>
> Closes: https://lore.kernel.org/de7bae91-3200-481f-9db2-c0dc382c91dd@gmail.com/
>
>> Fixes: d6d1df92c25f ("docs: automarkup: Mark up undocumented entities too")
>> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
>
> Tested-by: Akira Yokosawa <akiyks@gmail.com>

Thanks,

jon

Re: docs: sphinx: avoid using the deprecated node.set_class()

[Akira Yokosawa]

On Fri, 20 Jun 2025 07:54:53 -0600, Jonathan Corbet wrote:
> Akira Yokosawa <akiyks@gmail.com> writes:
> 
>> Hi Jon,
>>
>> On Thu, 19 Jun 2025 15:26:56 -0600, Jonathan Corbet wrote:
>>> Docutils emits a deprecation warning when the set_class() element method is
>>> used; that warning disappears into the ether, but it also causes a crash
>>> with docutils 0.19 when combined with certain versions of Sphinx.
>>
>> To be accurate, I'd rather say:
>>                                                 but it also causes a crash
>>   with docutils 0.19 when combined with any version of Sphinx whose
>>   requirement accepts it.
> 
> ...or just "with docutils 0.19" and put the period there, perhaps?

Ah, that should be good enough.

        Thanks, Akira