Re: Python 2.7 support and automarkup.py - Was: Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

Re: Python 2.7 support and automarkup.py - Was: Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Nícolas F. R. A. Prado]

On Fri Oct 30, 2020 at 11:39 AM -03, Matthew Wilcox wrote:
>
> On Fri, Oct 30, 2020 at 08:14:40AM -0600, Jonathan Corbet wrote:
> > On Fri, 30 Oct 2020 15:10:26 +0100
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> >
> > > I see a few alternatives:
> > >
> > > 1) fix automarkup.py for it to work again with python 2.7;
> > >
> > > 2) conf.py could gain some logic to disable automarkup with
> > >    Python < 3;
> > >
> > > 3) scripts/sphinx-pre-install already detects Python version.
> > >    It should likely be easy to ask the user to use python 3.x,
> > >    if an older version is detected.
> > >
> > > Doing (1) or (2) will require an additional step when we raise
> > > the bar for Python version.
> >
> > We haven't dropped support for Python 2 yet, so this constitutes a
> > regression.  My own approach would be something like this at the top of
> > automarkup.py:
> >
> > 	if python2:
> > 	    ascii = 0
> > 	else:
> > 	    ascii = re.ASCII
> >
> > ...then s/re.ASCII/ascii/ throughout.  I can probably put together
> > something later this morning.
>
> Could we have a warning somewhere that python 2.7 is going to produce
> inferior docs?

But I don't expect the docs to have inferior quality using python 2.7.
The fix proposed by Jon should be enough to solve the issue.

>
> Alternatively, https://docs.python.org/2/library/re.html suggests
> using "The third-party regex module".

I still think Jon's solution would be better, as it doesn't add any additional
dependency.
In the end the issue is just python 2 defaults to ASCII while python 3 defaults
to unicode.

Thanks,
Nícolas

[PATCH v2 0/5] docs: automarkup.py: Make automarkup ready for Sphinx 3.1+

[Nícolas F. R. A. Prado]

Hi,

this patch series makes the automatic markup extension ready for Sphinx 3.1+.
It was based on Mauro's Sphinx patch series, and requires it for the namespaces
to work, but could also be merged through the docs tree without regressions
(other than the increased build time explained below).

The first three patches make automarkup compatible with Sphinx 3.1. The first
patch makes use of the new C roles in Sphinx3 instead of the generic type role
from Sphinx 2, while patches 2 and 3 solve the warnings caused by Sphinx3's
stricter C domain.

Patch 4 adds cross-referencing to C macros with parameters for Sphinx 3.

Patch 5 enables cross-referencing inside C namespaces, which are new to Sphinx
3.1.

On an importante note:
In order to be able to support automatic cross-referencing inside C namespaces,
I needed to disable parallel source reading for Sphinx in patch 5. On my
machine, this makes the build process take about 4 additional minutes. This is
very bad, since the documentation building process already takes too long, but I
couldn't think of a way to sidestep this issue. If anyone has any idea, it would
be greatly appreciated.

Also, for some reason, disabling the source read parallelization makes
Sphinx output 2 warnings saying so, which is another annoyance.

Thanks,
Nícolas

Changes in v2:
- Split the single patch into patches 1, 2 and 3
- Change sphinx version verification in patch 1
- Thanks Mauro for the clarifications in v1:
  - Add patches 4 and 5 for the missing functionalities

Nícolas F. R. A. Prado (5):
  docs: automarkup.py: Use new C roles in Sphinx 3
  docs: automarkup.py: Fix regexes to solve sphinx 3 warnings
  docs: automarkup.py: Skip C reserved words when cross-referencing
  docs: automarkup.py: Add cross-reference for parametrized C macros
  docs: automarkup.py: Allow automatic cross-reference inside C
    namespace

 Documentation/sphinx/automarkup.py | 188 +++++++++++++++++++++++------
 1 file changed, 154 insertions(+), 34 deletions(-)

-- 
2.28.0

[PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Nícolas F. R. A. Prado]

While Sphinx 2 used a single c:type role for struct, union, enum and
typedef, Sphinx 3 uses a specific role for each one.
To keep backward compatibility, detect the Sphinx version and use the
correct roles for that version.

Signed-off-by: Nícolas F. R. A. Prado <nfraprado@protonmail.com>
---
 Documentation/sphinx/automarkup.py | 55 ++++++++++++++++++++++++++----
 1 file changed, 49 insertions(+), 6 deletions(-)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index a1b0f554cd82..db13fb15cedc 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -23,7 +23,21 @@ from itertools import chain
 # bit tries to restrict matches to things that won't create trouble.
 #
 RE_function = re.compile(r'(([\w_][\w\d_]+)\(\))')
-RE_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
+
+#
+# Sphinx 2 uses the same :c:type role for struct, union, enum and typedef
+#
+RE_generic_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
+
+#
+# Sphinx 3 uses a different C role for each one of struct, union, enum and
+# typedef
+#
+RE_struct = re.compile(r'\b(struct)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
+RE_union = re.compile(r'\b(union)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
+RE_enum = re.compile(r'\b(enum)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
+RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
+
 #
 # Detects a reference to a documentation page of the form Documentation/... with
 # an optional extension
@@ -48,9 +62,22 @@ def markup_refs(docname, app, node):
     #
     # Associate each regex with the function that will markup its matches
     #
-    markup_func = {RE_type: markup_c_ref,
-                   RE_function: markup_c_ref,
-                   RE_doc: markup_doc_ref}
+    markup_func_sphinx2 = {RE_doc: markup_doc_ref,
+                           RE_function: markup_c_ref,
+                           RE_generic_type: markup_c_ref}
+
+    markup_func_sphinx3 = {RE_doc: markup_doc_ref,
+                           RE_function: markup_c_ref,
+                           RE_struct: markup_c_ref,
+                           RE_union: markup_c_ref,
+                           RE_enum: markup_c_ref,
+                           RE_typedef: markup_c_ref}
+
+    if sphinx.version_info[0] >= 3:
+        markup_func = markup_func_sphinx3
+    else:
+        markup_func = markup_func_sphinx2
+
     match_iterators = [regex.finditer(t) for regex in markup_func]
     #
     # Sort all references by the starting position in text
@@ -79,8 +106,24 @@ def markup_refs(docname, app, node):
 # type_name) with an appropriate cross reference.
 #
 def markup_c_ref(docname, app, match):
-    class_str = {RE_function: 'c-func', RE_type: 'c-type'}
-    reftype_str = {RE_function: 'function', RE_type: 'type'}
+    class_str = {RE_function: 'c-func',
+                 # Sphinx 2 only
+                 RE_generic_type: 'c-type',
+                 # Sphinx 3+ only
+                 RE_struct: 'c-struct',
+                 RE_union: 'c-union',
+                 RE_enum: 'c-enum',
+                 RE_typedef: 'c-type',
+                 }
+    reftype_str = {RE_function: 'function',
+                   # Sphinx 2 only
+                   RE_generic_type: 'type',
+                   # Sphinx 3+ only
+                   RE_struct: 'struct',
+                   RE_union: 'union',
+                   RE_enum: 'enum',
+                   RE_typedef: 'type',
+                   }
 
     cdom = app.env.domains['c']
     #
-- 
2.28.0

Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Dafna Hirschfeld]

Hi

Am 14.10.20 um 01:13 schrieb Nícolas F. R. A. Prado:
> While Sphinx 2 used a single c:type role for struct, union, enum and
> typedef, Sphinx 3 uses a specific role for each one.
> To keep backward compatibility, detect the Sphinx version and use the
> correct roles for that version.
> 
> Signed-off-by: Nícolas F. R. A. Prado <nfraprado@protonmail.com>
> ---
>   Documentation/sphinx/automarkup.py | 55 ++++++++++++++++++++++++++----
>   1 file changed, 49 insertions(+), 6 deletions(-)
> 
> diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
> index a1b0f554cd82..db13fb15cedc 100644
> --- a/Documentation/sphinx/automarkup.py
> +++ b/Documentation/sphinx/automarkup.py
> @@ -23,7 +23,21 @@ from itertools import chain
>   # bit tries to restrict matches to things that won't create trouble.
>   #
>   RE_function = re.compile(r'(([\w_][\w\d_]+)\(\))')
> -RE_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
> +
> +#
> +# Sphinx 2 uses the same :c:type role for struct, union, enum and typedef
> +#
> +RE_generic_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
> +
> +#
> +# Sphinx 3 uses a different C role for each one of struct, union, enum and
> +# typedef
> +#
> +RE_struct = re.compile(r'\b(struct)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> +RE_union = re.compile(r'\b(union)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> +RE_enum = re.compile(r'\b(enum)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> +RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=re.ASCII)

I use ubuntu 18.04, my default python is 2.7,
when running 'make htmldocs' with that fix I get:

AttributeError: 'module' object has no attribute 'ASCII'

Thanks,
Dafna

> +
>   #
>   # Detects a reference to a documentation page of the form Documentation/... with
>   # an optional extension
> @@ -48,9 +62,22 @@ def markup_refs(docname, app, node):
>       #
>       # Associate each regex with the function that will markup its matches
>       #
> -    markup_func = {RE_type: markup_c_ref,
> -                   RE_function: markup_c_ref,
> -                   RE_doc: markup_doc_ref}
> +    markup_func_sphinx2 = {RE_doc: markup_doc_ref,
> +                           RE_function: markup_c_ref,
> +                           RE_generic_type: markup_c_ref}
> +
> +    markup_func_sphinx3 = {RE_doc: markup_doc_ref,
> +                           RE_function: markup_c_ref,
> +                           RE_struct: markup_c_ref,
> +                           RE_union: markup_c_ref,
> +                           RE_enum: markup_c_ref,
> +                           RE_typedef: markup_c_ref}
> +
> +    if sphinx.version_info[0] >= 3:
> +        markup_func = markup_func_sphinx3
> +    else:
> +        markup_func = markup_func_sphinx2
> +
>       match_iterators = [regex.finditer(t) for regex in markup_func]
>       #
>       # Sort all references by the starting position in text
> @@ -79,8 +106,24 @@ def markup_refs(docname, app, node):
>   # type_name) with an appropriate cross reference.
>   #
>   def markup_c_ref(docname, app, match):
> -    class_str = {RE_function: 'c-func', RE_type: 'c-type'}
> -    reftype_str = {RE_function: 'function', RE_type: 'type'}
> +    class_str = {RE_function: 'c-func',
> +                 # Sphinx 2 only
> +                 RE_generic_type: 'c-type',
> +                 # Sphinx 3+ only
> +                 RE_struct: 'c-struct',
> +                 RE_union: 'c-union',
> +                 RE_enum: 'c-enum',
> +                 RE_typedef: 'c-type',
> +                 }
> +    reftype_str = {RE_function: 'function',
> +                   # Sphinx 2 only
> +                   RE_generic_type: 'type',
> +                   # Sphinx 3+ only
> +                   RE_struct: 'struct',
> +                   RE_union: 'union',
> +                   RE_enum: 'enum',
> +                   RE_typedef: 'type',
> +                   }
>   
>       cdom = app.env.domains['c']
>       #
> 

Python 2.7 support and automarkup.py - Was: Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Mauro Carvalho Chehab]

Hi Dafna,

Em Fri, 30 Oct 2020 14:33:52 +0100
Dafna Hirschfeld <dafna.hirschfeld@collabora.com> escreveu:

> Hi
> 
> Am 14.10.20 um 01:13 schrieb Nícolas F. R. A. Prado:
> > While Sphinx 2 used a single c:type role for struct, union, enum and
> > typedef, Sphinx 3 uses a specific role for each one.
> > To keep backward compatibility, detect the Sphinx version and use the
> > correct roles for that version.
> > 
> > Signed-off-by: Nícolas F. R. A. Prado <nfraprado@protonmail.com>
> > ---
> >   Documentation/sphinx/automarkup.py | 55 ++++++++++++++++++++++++++----
> >   1 file changed, 49 insertions(+), 6 deletions(-)
> > 
> > diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
> > index a1b0f554cd82..db13fb15cedc 100644
> > --- a/Documentation/sphinx/automarkup.py
> > +++ b/Documentation/sphinx/automarkup.py
> > @@ -23,7 +23,21 @@ from itertools import chain
> >   # bit tries to restrict matches to things that won't create trouble.
> >   #
> >   RE_function = re.compile(r'(([\w_][\w\d_]+)\(\))')
> > -RE_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
> > +
> > +#
> > +# Sphinx 2 uses the same :c:type role for struct, union, enum and typedef
> > +#
> > +RE_generic_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
> > +
> > +#
> > +# Sphinx 3 uses a different C role for each one of struct, union, enum and
> > +# typedef
> > +#
> > +RE_struct = re.compile(r'\b(struct)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> > +RE_union = re.compile(r'\b(union)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> > +RE_enum = re.compile(r'\b(enum)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> > +RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=re.ASCII)  
> 
> I use ubuntu 18.04, my default python is 2.7,
> when running 'make htmldocs' with that fix I get:
> 
> AttributeError: 'module' object has no attribute 'ASCII'

FYI, there's a discussion at kernel-doc ML about dropping support for
python 2.7 at Kernel 5.11. While not explicitly mentioned at the
discussion, this is the rationale:

	https://www.python.org/doc/sunset-python-2/

As this is currently broken with Python 2.7, then perhaps we can
do that for 5.10 :-)

Jon,

What do you think? 

I see a few alternatives:

1) fix automarkup.py for it to work again with python 2.7;

2) conf.py could gain some logic to disable automarkup with
   Python < 3;

3) scripts/sphinx-pre-install already detects Python version. 
   It should likely be easy to ask the user to use python 3.x,
   if an older version is detected.

Doing (1) or (2) will require an additional step when we raise
the bar for Python version.

Thanks,
Mauro

Re: Python 2.7 support and automarkup.py - Was: Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Jonathan Corbet]

On Fri, 30 Oct 2020 15:10:26 +0100
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> I see a few alternatives:
> 
> 1) fix automarkup.py for it to work again with python 2.7;
> 
> 2) conf.py could gain some logic to disable automarkup with
>    Python < 3;
> 
> 3) scripts/sphinx-pre-install already detects Python version. 
>    It should likely be easy to ask the user to use python 3.x,
>    if an older version is detected.
> 
> Doing (1) or (2) will require an additional step when we raise
> the bar for Python version.

We haven't dropped support for Python 2 yet, so this constitutes a
regression.  My own approach would be something like this at the top of
automarkup.py:

	if python2:
	    ascii = 0
	else:
	    ascii = re.ASCII

...then s/re.ASCII/ascii/ throughout.  I can probably put together
something later this morning.

jon

Re: Python 2.7 support and automarkup.py - Was: Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Mauro Carvalho Chehab]

Em Fri, 30 Oct 2020 08:14:40 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> On Fri, 30 Oct 2020 15:10:26 +0100
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> 
> > I see a few alternatives:
> > 
> > 1) fix automarkup.py for it to work again with python 2.7;
> > 
> > 2) conf.py could gain some logic to disable automarkup with
> >    Python < 3;
> > 
> > 3) scripts/sphinx-pre-install already detects Python version. 
> >    It should likely be easy to ask the user to use python 3.x,
> >    if an older version is detected.
> > 
> > Doing (1) or (2) will require an additional step when we raise
> > the bar for Python version.  
> 
> We haven't dropped support for Python 2 yet, so this constitutes a
> regression.  My own approach would be something like this at the top of
> automarkup.py:
> 
> 	if python2:
> 	    ascii = 0
> 	else:
> 	    ascii = re.ASCII
> 
> ...then s/re.ASCII/ascii/ throughout.  I can probably put together
> something later this morning.

Makes sense.

> 
> jon



Thanks,
Mauro

Re: Python 2.7 support and automarkup.py - Was: Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Matthew Wilcox]

On Fri, Oct 30, 2020 at 08:14:40AM -0600, Jonathan Corbet wrote:
> On Fri, 30 Oct 2020 15:10:26 +0100
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> 
> > I see a few alternatives:
> > 
> > 1) fix automarkup.py for it to work again with python 2.7;
> > 
> > 2) conf.py could gain some logic to disable automarkup with
> >    Python < 3;
> > 
> > 3) scripts/sphinx-pre-install already detects Python version. 
> >    It should likely be easy to ask the user to use python 3.x,
> >    if an older version is detected.
> > 
> > Doing (1) or (2) will require an additional step when we raise
> > the bar for Python version.
> 
> We haven't dropped support for Python 2 yet, so this constitutes a
> regression.  My own approach would be something like this at the top of
> automarkup.py:
> 
> 	if python2:
> 	    ascii = 0
> 	else:
> 	    ascii = re.ASCII
> 
> ...then s/re.ASCII/ascii/ throughout.  I can probably put together
> something later this morning.

Could we have a warning somewhere that python 2.7 is going to produce
inferior docs?

Alternatively, https://docs.python.org/2/library/re.html suggests
using "The third-party regex module".