docs: "translations" extension breaks "make latexdocs" with 6.1.3<=Sphinx<7.2

docs: "translations" extension breaks "make latexdocs" with 6.1.3<=Sphinx<7.2

[Akira Yokosawa]

Hi Vegard,

While testing update of sphinx_pre_install WRT pyyaml, I noticed build
errors in "make latexdocs" against Fedora 39's distro Sphinx (v6.2.1).

I don't mean the translations extension is doing anything wrong.

Repro:

    make cleandocs
    make SPHINXDIRS=doc-guide htmldocs  # for yaml -> rst conversion
    make SPHINXDIRS=doc-guide latexdocs

At current docs-next, Fedora 39's Sphinx ends up in the error:

    Exception occurred:
      File "/usr/lib/python3.12/site-packages/sphinx/util/nodes.py", line 624, in _copy_except__document
        newnode = self.__class__(rawsource=self.rawsource, **self.attributes)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    TypeError: LanguagesNode.__init__() missing 1 required positional argument: 'current_language'
    The full traceback has been saved in /tmp/sphinx-err-7xmwytuu.log, if you want to report the issue to the developers.


/tmp/sphinx-err-7xmwytuu.log reads:
------------------------------------------
# Platform:         linux; (Linux-6.7.3-200.fc39.x86_64-x86_64-with-glibc2.38)
# Sphinx version:   6.2.1
# Python version:   3.12.1 (CPython)
# Docutils version: 0.19
# Jinja2 version:   3.1.3
# Pygments version: 2.15.1

# Last messages:
#   building [latex]: all documents
#   updating environment:
#   0 added, 0 changed, 0 removed
#   reading sources...
#   
#   looking for now-outdated files...
#   none found
#   processing doc-guide.tex...
#   index
#   failed

# Loaded extensions:
#   sphinx.ext.mathjax (6.2.1)
#   alabaster (0.7.12)
#   sphinxcontrib.applehelp (1.0.2)
#   sphinxcontrib.devhelp (1.0.2)
#   sphinxcontrib.htmlhelp (2.0.0)
#   sphinxcontrib.serializinghtml (1.1.5)
#   sphinxcontrib.qthelp (1.0.3)
#   kerneldoc (1.0)
#   rstFlatTable (1.0)
#   kernel_include (1.0)
#   kfigure (1.0.0)
#   sphinx.ext.ifconfig (6.2.1)
#   automarkup (unknown version)
#   maintainers_include (1.0)
#   sphinx.ext.autosectionlabel (6.2.1)
#   kernel_abi (1.0)
#   kernel_feat (1.0)
#   translations (unknown version)
#   sphinx.ext.imgmath (6.2.1)

# Traceback:
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/sphinx/cmd/build.py", line 285, in build_main
    app.build(args.force_all, args.filenames)
  File "/usr/lib/python3.12/site-packages/sphinx/application.py", line 353, in build
    self.builder.build_update()
  File "/usr/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 308, in build_update
    self.build(['__all__'], to_build)
  File "/usr/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 378, in build
    self.write(docnames, list(updated_docnames), method)
  File "/usr/lib/python3.12/site-packages/sphinx/builders/latex/__init__.py", line 287, in write
    doctree = self.assemble_doctree(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/sphinx/builders/latex/__init__.py", line 344, in assemble_doctree
    largetree = inline_all_toctrees(self, self.docnames, indexfile, tree,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/sphinx/util/nodes.py", line 403, in inline_all_toctrees
    tree = tree.deepcopy()
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/nodes.py", line 1064, in deepcopy
    copy.extend([child.deepcopy() for child in self.children])
                 ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/nodes.py", line 1063, in deepcopy
    copy = self.copy()
           ^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/sphinx/util/nodes.py", line 624, in _copy_except__document
    newnode = self.__class__(rawsource=self.rawsource, **self.attributes)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: LanguagesNode.__init__() missing 1 required positional argument: 'current_language'
------------------------------------------

I did quick tests on Ubuntu 22.04 + venv (whose python3 is v3.10.12),
and found that Sphinx versions 6.1.3 and 7.1.2 also suffer from the
same issue.

Luckily, Sphinx v7.2.6 is not affected.

Bisection pointed commit 7418ec5b151f ("docs: translations: add
translations links when they exist") as the first bad commit and
reverting it resolved the issue.

Sphinx's git history shows that the code at line 624 of nodes.py
was added just before 6.1.3.  It was heavily refactored during
the 7.2.x development cycle [1] for improving TOC tree processing.

So this issue looks like a regression in Sphinx which has been
resolved upstream since.

Vegard, please let me know if am I missing something.

[1]: https://github.com/sphinx-doc/sphinx/pull/11565

        Thanks, Akira

Re: docs: "translations" extension breaks "make latexdocs" with 6.1.3<=Sphinx<7.2

[Vegard Nossum]

On 12/02/2024 10:52, Akira Yokosawa wrote:
> Hi Vegard,
> 
> While testing update of sphinx_pre_install WRT pyyaml, I noticed build
> errors in "make latexdocs" against Fedora 39's distro Sphinx (v6.2.1).
> 
> I don't mean the translations extension is doing anything wrong.
> 
> Repro:
> 
>      make cleandocs
>      make SPHINXDIRS=doc-guide htmldocs  # for yaml -> rst conversion
>      make SPHINXDIRS=doc-guide latexdocs
> 
> At current docs-next, Fedora 39's Sphinx ends up in the error:
> 
>      Exception occurred:
>        File "/usr/lib/python3.12/site-packages/sphinx/util/nodes.py", line 624, in _copy_except__document
>          newnode = self.__class__(rawsource=self.rawsource, **self.attributes)
>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>      TypeError: LanguagesNode.__init__() missing 1 required positional argument: 'current_language'
>      The full traceback has been saved in /tmp/sphinx-err-7xmwytuu.log, if you want to report the issue to the developers.

Thanks, with all your info it was quick and easy to reproduce.

I think we can solve it by making 'current_language' a true element
attribute:

diff --git a/Documentation/sphinx/translations.py 
b/Documentation/sphinx/translations.py
index 47161e6eba99..f20c30599ceb 100644
--- a/Documentation/sphinx/translations.py
+++ b/Documentation/sphinx/translations.py
@@ -32,7 +32,7 @@ class LanguagesNode(nodes.Element):
      def __init__(self, current_language, *args, **kwargs):
          super().__init__(*args, **kwargs)

-        self.current_language = current_language
+        self['current_language'] = current_language

  class TranslationsTransform(Transform):
      default_priority = 900
@@ -84,7 +84,7 @@ def process_languages(app, doctree, docname):

          html_content = app.builder.templates.render('translations.html',
              context={
-                'current_language': node.current_language,
+                'current_language': node['current_language'],
                  'languages': languages,
              })

This is probably more correct anyway.

I'll test this more thoroughly with a full build as I think translations
won't show up when using SPHINXDIRS= (because the translation is outside
the source tree) as well as with older/newer Sphinx versions.

Thanks,


Vegard

Re: docs: "translations" extension breaks "make latexdocs" with 6.1.3<=Sphinx<7.2

[Vegard Nossum]

On 12/02/2024 11:28, Vegard Nossum wrote:
> On 12/02/2024 10:52, Akira Yokosawa wrote:
>> While testing update of sphinx_pre_install WRT pyyaml, I noticed build
>> errors in "make latexdocs" against Fedora 39's distro Sphinx (v6.2.1).

> I think we can solve it by making 'current_language' a true element
> attribute:
[...]
> This is probably more correct anyway.
> 
> I'll test this more thoroughly with a full build as I think translations
> won't show up when using SPHINXDIRS= (because the translation is outside
> the source tree) as well as with older/newer Sphinx versions.

In the end, that first patch caused errors on Sphinx 4.x
("TypeError: LanguagesNode.__init__() got multiple values for argument
'current_language'").

This seems to work across 2.x/3.x/6.x/7.x for me:

diff --git a/Documentation/sphinx/translations.py 
b/Documentation/sphinx/translations.py
index 47161e6eba99..32c2b32b2b5e 100644
--- a/Documentation/sphinx/translations.py
+++ b/Documentation/sphinx/translations.py
@@ -29,10 +29,7 @@ all_languages = {
  }

  class LanguagesNode(nodes.Element):
-    def __init__(self, current_language, *args, **kwargs):
-        super().__init__(*args, **kwargs)
-
-        self.current_language = current_language
+    pass

  class TranslationsTransform(Transform):
      default_priority = 900
@@ -49,7 +46,8 @@ class TranslationsTransform(Transform):
              # normalize docname to be the untranslated one
              docname = os.path.join(*components[2:])

-        new_nodes = LanguagesNode(all_languages[this_lang_code])
+        new_nodes = LanguagesNode()
+        new_nodes['current_language'] = all_languages[this_lang_code]

          for lang_code, lang_name in all_languages.items():
              if lang_code == this_lang_code:
@@ -84,7 +82,7 @@ def process_languages(app, doctree, docname):

          html_content = app.builder.templates.render('translations.html',
              context={
-                'current_language': node.current_language,
+                'current_language': node['current_language'],
                  'languages': languages,
              })


I'll test a bit more and submit a proper patch if this works.


Vegard

Re: docs: "translations" extension breaks "make latexdocs" with 6.1.3<=Sphinx<7.2

[Jonathan Corbet]

Vegard Nossum <vegard.nossum@oracle.com> writes:

> On 12/02/2024 11:28, Vegard Nossum wrote:
>> On 12/02/2024 10:52, Akira Yokosawa wrote:
>>> While testing update of sphinx_pre_install WRT pyyaml, I noticed build
>>> errors in "make latexdocs" against Fedora 39's distro Sphinx (v6.2.1).
>
>> I think we can solve it by making 'current_language' a true element
>> attribute:
> [...]
>> This is probably more correct anyway.
>> 
>> I'll test this more thoroughly with a full build as I think translations
>> won't show up when using SPHINXDIRS= (because the translation is outside
>> the source tree) as well as with older/newer Sphinx versions.
>
> In the end, that first patch caused errors on Sphinx 4.x
> ("TypeError: LanguagesNode.__init__() got multiple values for argument
> 'current_language'").
>
> This seems to work across 2.x/3.x/6.x/7.x for me:

It occurs to me that me we should just have conf.py just not load the
translations extension at all if the builder is not "html"?

Thanks,

jon

Re: docs: "translations" extension breaks "make latexdocs" with 6.1.3<=Sphinx<7.2

[Vegard Nossum]

On 12/02/2024 22:52, Jonathan Corbet wrote:
> Vegard Nossum <vegard.nossum@oracle.com> writes:
> 
>> On 12/02/2024 11:28, Vegard Nossum wrote:
>>> On 12/02/2024 10:52, Akira Yokosawa wrote:
>>>> While testing update of sphinx_pre_install WRT pyyaml, I noticed build
>>>> errors in "make latexdocs" against Fedora 39's distro Sphinx (v6.2.1).
>>
>>> I think we can solve it by making 'current_language' a true element
>>> attribute:
>> [...]
>>> This is probably more correct anyway.
>>>
>>> I'll test this more thoroughly with a full build as I think translations
>>> won't show up when using SPHINXDIRS= (because the translation is outside
>>> the source tree) as well as with older/newer Sphinx versions.
>>
>> In the end, that first patch caused errors on Sphinx 4.x
>> ("TypeError: LanguagesNode.__init__() got multiple values for argument
>> 'current_language'").
>>
>> This seems to work across 2.x/3.x/6.x/7.x for me:
> 
> It occurs to me that me we should just have conf.py just not load the
> translations extension at all if the builder is not "html"?

I think we need to, otherwise it will have one of those issues that were
discovered during the original submission: if you 'make latexdocs'
first, then 'make htmldocs', the Translations nodes just won't be in the
saved doctrees and won't be there for the HTML output.


Vegard

Re: docs: "translations" extension breaks "make latexdocs" with 6.1.3<=Sphinx<7.2

[Jani Nikula]

On Mon, 12 Feb 2024, Vegard Nossum <vegard.nossum@oracle.com> wrote:
> On 12/02/2024 22:52, Jonathan Corbet wrote:
>> Vegard Nossum <vegard.nossum@oracle.com> writes:
>> 
>>> On 12/02/2024 11:28, Vegard Nossum wrote:
>>>> On 12/02/2024 10:52, Akira Yokosawa wrote:
>>>>> While testing update of sphinx_pre_install WRT pyyaml, I noticed build
>>>>> errors in "make latexdocs" against Fedora 39's distro Sphinx (v6.2.1).
>>>
>>>> I think we can solve it by making 'current_language' a true element
>>>> attribute:
>>> [...]
>>>> This is probably more correct anyway.
>>>>
>>>> I'll test this more thoroughly with a full build as I think translations
>>>> won't show up when using SPHINXDIRS= (because the translation is outside
>>>> the source tree) as well as with older/newer Sphinx versions.
>>>
>>> In the end, that first patch caused errors on Sphinx 4.x
>>> ("TypeError: LanguagesNode.__init__() got multiple values for argument
>>> 'current_language'").
>>>
>>> This seems to work across 2.x/3.x/6.x/7.x for me:
>> 
>> It occurs to me that me we should just have conf.py just not load the
>> translations extension at all if the builder is not "html"?
>
> I think we need to, otherwise it will have one of those issues that were
> discovered during the original submission: if you 'make latexdocs'
> first, then 'make htmldocs', the Translations nodes just won't be in the
> saved doctrees and won't be there for the HTML output.

"Note that the current builder tag is not available in conf.py, as it is
created after the builder is initialized."

https://www.sphinx-doc.org/en/master/usage/configuration.html


BR,
Jani.


-- 
Jani Nikula, Intel

Re: docs: "translations" extension breaks "make latexdocs" with 6.1.3<=Sphinx<7.2

[Jonathan Corbet]

Jani Nikula <jani.nikula@linux.intel.com> writes:

> "Note that the current builder tag is not available in conf.py, as it is
> created after the builder is initialized."
>
> https://www.sphinx-doc.org/en/master/usage/configuration.html

The command line is available, though, which should be enough to get the
job done.

jon