[PATCH RFC] Make the docs build "work" with Sphinx 3.x

[PATCH RFC] Make the docs build "work" with Sphinx 3.x

[Jonathan Corbet]

The Sphinx 3.x upgrade broke a number of things in our special "cdomain"
module that are not easy to fix.  For now, just disable that module for the
3.x build and put out a warning that the build will not be perfect.  The
problems affect primarily the media documentation where :name: is used.

This is not a great solution, but it at least allows the docs build to
proceed on systems running 3.x until somebody has the time to put together
a proper fix.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
It seems that much of what we have in cdomain.cpy isn't needed with 3.x,
though that has to be verified.  But the :name: extension is problematic.
That needs a hard look, it's guaranteed to be a hard thing to maintain
going forward and it would be nice if we could find a way to do without it.

Meanwhile, building the docs with Sphinx 3.2.1 takes nearly twice as long
as with 2.2.2.  Given that the process was not exactly fast before, this
is not a welcome change.  Sigh.

 Documentation/conf.py | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/Documentation/conf.py b/Documentation/conf.py
index c503188880d9..0a102d57437d 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -36,10 +36,23 @@ needs_sphinx = '1.3'
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain',
+extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include',
               'kfigure', 'sphinx.ext.ifconfig', 'automarkup',
               'maintainers_include', 'sphinx.ext.autosectionlabel' ]
 
+#
+# cdomain is badly broken in Sphinx 3+.  Leaving it out generates *most*
+# of the docs correctly, but not all.  Scream bloody murder but allow
+# the process to proceed; hopefully somebody will fix this properly soon.
+#
+if major >= 3:
+    sys.stderr.write('''WARNING: The kernel documentation build process
+	does not work correctly with Sphinx v3.0 and above.  Expect errors
+	in the generated output.
+	''')
+else:
+    extensions.append('cdomain')
+
 # Ensure that autosectionlabel will produce unique names
 autosectionlabel_prefix_document = True
 autosectionlabel_maxdepth = 2
-- 
2.26.2

Re: [PATCH RFC] Make the docs build "work" with Sphinx 3.x

[Matthew Wilcox]

On Fri, Sep 04, 2020 at 10:29:25AM -0600, Jonathan Corbet wrote:
> +# cdomain is badly broken in Sphinx 3+.  Leaving it out generates *most*
> +# of the docs correctly, but not all.  Scream bloody murder but allow
> +# the process to proceed; hopefully somebody will fix this properly soon.
> +#
> +if major >= 3:
> +    sys.stderr.write('''WARNING: The kernel documentation build process
> +	does not work correctly with Sphinx v3.0 and above.  Expect errors
> +	in the generated output.

Should we be converting the kernel-doc script to output 3.0-style markup
for structs?

https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#c-roles

Re: [PATCH RFC] Make the docs build "work" with Sphinx 3.x

[Mauro Carvalho Chehab]

Em Mon, 21 Sep 2020 23:26:39 +0100
Matthew Wilcox <willy@infradead.org> escreveu:

> On Fri, Sep 04, 2020 at 10:29:25AM -0600, Jonathan Corbet wrote:
> > +# cdomain is badly broken in Sphinx 3+.  Leaving it out generates *most*
> > +# of the docs correctly, but not all.  Scream bloody murder but allow
> > +# the process to proceed; hopefully somebody will fix this properly soon.
> > +#
> > +if major >= 3:
> > +    sys.stderr.write('''WARNING: The kernel documentation build process
> > +	does not work correctly with Sphinx v3.0 and above.  Expect errors
> > +	in the generated output.  
> 
> Should we be converting the kernel-doc script to output 3.0-style markup
> for structs?
> 
> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#c-roles

I wouldn't mind converting the media books to use 3.0 style if
they work as expected.

Yet, if we convert, we need to change the minimal version requirement
to follow it. 
Sphinx 3.x is still too new. I bet distros don't bring it yet.

Ok, the default procedure is to pick it using pip or venv. So,
no bit issue, IMHO.

Also, as Jon mentioned, it takes about twice the time for the
doc build.

Not sure if it is worth.


Thanks,
Mauro

Re: [PATCH RFC] Make the docs build "work" with Sphinx 3.x

[Jani Nikula]

On Mon, 21 Sep 2020, Matthew Wilcox <willy@infradead.org> wrote:
> On Fri, Sep 04, 2020 at 10:29:25AM -0600, Jonathan Corbet wrote:
>> +# cdomain is badly broken in Sphinx 3+.  Leaving it out generates *most*
>> +# of the docs correctly, but not all.  Scream bloody murder but allow
>> +# the process to proceed; hopefully somebody will fix this properly soon.
>> +#
>> +if major >= 3:
>> +    sys.stderr.write('''WARNING: The kernel documentation build process
>> +	does not work correctly with Sphinx v3.0 and above.  Expect errors
>> +	in the generated output.
>
> Should we be converting the kernel-doc script to output 3.0-style markup
> for structs?

IIUC it's not backwards compatible with Sphinx pre-3.0, so you'd either
have to require Sphinx 3.0 or have kernel-doc emit both, depending on
the Sphinx version.


BR,
Jani.


>
> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#c-roles

-- 
Jani Nikula, Intel Open Source Graphics Center

Re: [PATCH RFC] Make the docs build "work" with Sphinx 3.x

[Jonathan Corbet]

On Tue, 22 Sep 2020 10:23:46 +0300
Jani Nikula <jani.nikula@linux.intel.com> wrote:

> > Should we be converting the kernel-doc script to output 3.0-style markup
> > for structs?  
> 
> IIUC it's not backwards compatible with Sphinx pre-3.0, so you'd either
> have to require Sphinx 3.0 or have kernel-doc emit both, depending on
> the Sphinx version.

Certainly before requiring 3.0 we should, like, actually make it work with
3.0...  People with time to do that appear to be rather scarce at the
moment, though.  I'll get to it eventually but I have a few other things
on my plate as well.

jon

Re: [PATCH RFC] Make the docs build "work" with Sphinx 3.x

[Mauro Carvalho Chehab]

Em Tue, 22 Sep 2020 10:56:56 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> On Tue, 22 Sep 2020 10:23:46 +0300
> Jani Nikula <jani.nikula@linux.intel.com> wrote:
> 
> > > Should we be converting the kernel-doc script to output 3.0-style markup
> > > for structs?  
> > 
> > IIUC it's not backwards compatible with Sphinx pre-3.0, so you'd either
> > have to require Sphinx 3.0 or have kernel-doc emit both, depending on
> > the Sphinx version.
> 
> Certainly before requiring 3.0 we should, like, actually make it work with
> 3.0...  People with time to do that appear to be rather scarce at the
> moment, though.  I'll get to it eventually but I have a few other things
> on my plate as well.

Tried to build now with Sphinx 3.2.1. There are thousands of warnings:

	build succeeded, 3423 warnings.

Like those:

	Documentation/driver-api/basics.rst: WARNING: Duplicate C declaration, also defined in 'core-api/workqueue'.
	Declaration is 'unsigned long delay'.

And those:

	./include/uapi/linux/firewire-cdev.h:38: WARNING: Type must be either just a name or a typedef-like declaration.
	If just a name:
	  Error in declarator or parameters
	  Invalid C declaration: Expected identifier in nested name, got keyword: struct [error at 6]
	    struct fw_cdev_event_iso_interrupt
	    ------^
	If typedef-like declaration:
	  Error in declarator or parameters
	  Invalid C declaration: Expected identifier in nested name. [error at 34]
	    struct fw_cdev_event_iso_interrupt
	    ----------------------------------^

If I'm not mistaken, one of the reasons why cdomain was added is
due to the duplicate C declaration.

-

I guess changing kernel-doc to produce Sphinx 3.x syntax and
raise the bar requiring 3.0 to be the minimal version would be
one alternative, but this won't solve the duplicated C declarations. 

Does Sphinx 3.x have some way to enable "local" C references?

Another one could be to make cdomain compatible with 3.x and
make it handle the differences. Not sure how easy/hard would be
to do that.

Thanks,
Mauro

Re: [PATCH RFC] Make the docs build "work" with Sphinx 3.x

[Mauro Carvalho Chehab]

Em Wed, 23 Sep 2020 11:01:06 +0200
Mauro Carvalho Chehab <mchehab@kernel.org> escreveu:

> Em Tue, 22 Sep 2020 10:56:56 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
> 
> > On Tue, 22 Sep 2020 10:23:46 +0300
> > Jani Nikula <jani.nikula@linux.intel.com> wrote:
> >   
> > > > Should we be converting the kernel-doc script to output 3.0-style markup
> > > > for structs?    
> > > 
> > > IIUC it's not backwards compatible with Sphinx pre-3.0, so you'd either
> > > have to require Sphinx 3.0 or have kernel-doc emit both, depending on
> > > the Sphinx version.  
> > 
> > Certainly before requiring 3.0 we should, like, actually make it work with
> > 3.0...  People with time to do that appear to be rather scarce at the
> > moment, though.  I'll get to it eventually but I have a few other things
> > on my plate as well.  
> 
> Tried to build now with Sphinx 3.2.1. There are thousands of warnings:
> 
> 	build succeeded, 3423 warnings.
> 
> Like those:
> 
> 	Documentation/driver-api/basics.rst: WARNING: Duplicate C declaration, also defined in 'core-api/workqueue'.
> 	Declaration is 'unsigned long delay'.
> 
> And those:
> 
> 	./include/uapi/linux/firewire-cdev.h:38: WARNING: Type must be either just a name or a typedef-like declaration.
> 	If just a name:
> 	  Error in declarator or parameters
> 	  Invalid C declaration: Expected identifier in nested name, got keyword: struct [error at 6]
> 	    struct fw_cdev_event_iso_interrupt
> 	    ------^
> 	If typedef-like declaration:
> 	  Error in declarator or parameters
> 	  Invalid C declaration: Expected identifier in nested name. [error at 34]
> 	    struct fw_cdev_event_iso_interrupt
> 	    ----------------------------------^
> 
> If I'm not mistaken, one of the reasons why cdomain was added is
> due to the duplicate C declaration.
> 
> -
> 
> I guess changing kernel-doc to produce Sphinx 3.x syntax and
> raise the bar requiring 3.0 to be the minimal version would be
> one alternative, but this won't solve the duplicated C declarations. 
> 
> Does Sphinx 3.x have some way to enable "local" C references?
> 
> Another one could be to make cdomain compatible with 3.x and
> make it handle the differences. Not sure how easy/hard would be
> to do that.

Ok, I checked the code of cdomain.py. It sounds that Markus
tried to make cdomain compatible with Sphinx 3.x[1]:

	https://github.com/return42/linuxdoc/commit/48f09de2aff80169b7b9be5b0f2431ad8954e5cc

[1] Yet, he ended disabling cdomain.py at his latest commit there:

	https://github.com/return42/linuxdoc/commit/70673dcf69e705e08d81f53794895dc15c4920b3#diff-40ccfb63661f88fd819794047febd54d

    (Basically, if Sphinx > 3.0, the cdomain.py is just an alias for c.py)

I was hoping that porting such change would be enough for docs to build
with Sphinx 3.2, but it isn't[2]. It seems that the entire Cdomain code
was re-written:

	https://github.com/sphinx-doc/sphinx/commit/0f49e30c51b5cc5055cda5b4b294c2dd9d1df573#r38750737

There are now some environment vars that are required. Some of them
have things like:

        lastSymbol = self.env.temp_data['c:last_symbol']  # type: Symbol
        assert lastSymbol

If those aren't found, Sphinx crashes.

So, making anything that would be compatible with both Sphinx 2.x
and Sphinx 3.x seems very painful.

It sounds to me that we need to make a choice here about how to
proceed. We have a few alternatives:

1) stick forever with Sphinx 2.x syntax. We may end needing to 
   support a fork of Sphinx, if we follow this direction, as
   nobody currently fix bugs on older versions of Sphinx;

2) switch to use Sphinx 3.x c domains and drop support for
   version < 3.x;

3) copy the code from Sphinx 2.x c.py file into our cdomain.py,
   making it to work with both versions. We may have troubles
   with newer versions of Sphinx if we follow this path though.

If someone has enough time, one alternative would be to do a
a mix of (2) and (3) by first porting to Sphinx 3.x and then 
re-writing the code at cdomain.py for it to work as a "backport"
of the Sphinx 3.x c domain.

-

Assuming that we would take (1), I suspect that changing kernel-doc to
produce Sphinx 3 domain shouldn't be too hard, but I'm a little bit
afraid of namespace collisions. There is a new namespace directive 
there, introduced on Sphinx 3.1:

	.. c:namespace:: scope specification

We'll likely need to use things like these in order to avoid
namespace collisions, meaning that the new minimal version
would be 3.1.

Btw, with regards to namespace, there are two alternatives:

- the RST files that would need it would explicitly have
  such directive;

- the kernel-doc script could have a new option
  "namespace". If found, kernel-doc would output the
  namespace directive. IMO, this is more elegant.

In any case, "namespace" will break automarkup.py for symbols
that won't use the "NULL" namespace. Not sure how to fix it.

Comments?

Thanks,
Mauro

[2] After Markus patch, I applied this change, to solve the issues
with the number of rows at the items array:

diff --git a/Documentation/sphinx/cdomain.py b/Documentation/sphinx/cdomain.py
index f262586cd094..6f1524bbede5 100644
--- a/Documentation/sphinx/cdomain.py
+++ b/Documentation/sphinx/cdomain.py
@@ -160,7 +160,10 @@ class CObject(Base_CObject):
                             'duplicate C object description of %s, ' % name +
                             'other instance in ' + self.env.doc2path(inv[name][0]),
                             line=self.lineno)
-            inv[name] = (self.env.docname, self.objtype)
+            if major <= 2:
+                inv[name] = (self.env.docname, self.objtype)
+            else:
+                inv[name] = (self.env.docname, targetname, self.objtype)
 
         indextext = self.get_index_text(name)
         if indextext:

Re: [PATCH RFC] Make the docs build "work" with Sphinx 3.x

[Mauro Carvalho Chehab]

Em Wed, 23 Sep 2020 13:53:18 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> Em Wed, 23 Sep 2020 11:01:06 +0200
> Mauro Carvalho Chehab <mchehab@kernel.org> escreveu:
> 
> > Em Tue, 22 Sep 2020 10:56:56 -0600
> > Jonathan Corbet <corbet@lwn.net> escreveu:
> >   
> > > On Tue, 22 Sep 2020 10:23:46 +0300
> > > Jani Nikula <jani.nikula@linux.intel.com> wrote:
> > >     
> > > > > Should we be converting the kernel-doc script to output 3.0-style markup
> > > > > for structs?      
> > > > 
> > > > IIUC it's not backwards compatible with Sphinx pre-3.0, so you'd either
> > > > have to require Sphinx 3.0 or have kernel-doc emit both, depending on
> > > > the Sphinx version.    
> > > 
> > > Certainly before requiring 3.0 we should, like, actually make it work with
> > > 3.0...  People with time to do that appear to be rather scarce at the
> > > moment, though.  I'll get to it eventually but I have a few other things
> > > on my plate as well.    
> > 
> > Tried to build now with Sphinx 3.2.1. There are thousands of warnings:
> > 
> > 	build succeeded, 3423 warnings.
> > 
> > Like those:
> > 
> > 	Documentation/driver-api/basics.rst: WARNING: Duplicate C declaration, also defined in 'core-api/workqueue'.
> > 	Declaration is 'unsigned long delay'.
> > 
> > And those:
> > 
> > 	./include/uapi/linux/firewire-cdev.h:38: WARNING: Type must be either just a name or a typedef-like declaration.
> > 	If just a name:
> > 	  Error in declarator or parameters
> > 	  Invalid C declaration: Expected identifier in nested name, got keyword: struct [error at 6]
> > 	    struct fw_cdev_event_iso_interrupt
> > 	    ------^
> > 	If typedef-like declaration:
> > 	  Error in declarator or parameters
> > 	  Invalid C declaration: Expected identifier in nested name. [error at 34]
> > 	    struct fw_cdev_event_iso_interrupt
> > 	    ----------------------------------^
> > 
> > If I'm not mistaken, one of the reasons why cdomain was added is
> > due to the duplicate C declaration.
> > 
> > -
> > 
> > I guess changing kernel-doc to produce Sphinx 3.x syntax and
> > raise the bar requiring 3.0 to be the minimal version would be
> > one alternative, but this won't solve the duplicated C declarations. 
> > 
> > Does Sphinx 3.x have some way to enable "local" C references?
> > 
> > Another one could be to make cdomain compatible with 3.x and
> > make it handle the differences. Not sure how easy/hard would be
> > to do that.  
> 
> Ok, I checked the code of cdomain.py. It sounds that Markus
> tried to make cdomain compatible with Sphinx 3.x[1]:
> 
> 	https://github.com/return42/linuxdoc/commit/48f09de2aff80169b7b9be5b0f2431ad8954e5cc
> 
> [1] Yet, he ended disabling cdomain.py at his latest commit there:
> 
> 	https://github.com/return42/linuxdoc/commit/70673dcf69e705e08d81f53794895dc15c4920b3#diff-40ccfb63661f88fd819794047febd54d
> 
>     (Basically, if Sphinx > 3.0, the cdomain.py is just an alias for c.py)
> 
> I was hoping that porting such change would be enough for docs to build
> with Sphinx 3.2, but it isn't[2]. It seems that the entire Cdomain code
> was re-written:
> 
> 	https://github.com/sphinx-doc/sphinx/commit/0f49e30c51b5cc5055cda5b4b294c2dd9d1df573#r38750737
> 
> There are now some environment vars that are required. Some of them
> have things like:
> 
>         lastSymbol = self.env.temp_data['c:last_symbol']  # type: Symbol
>         assert lastSymbol
> 
> If those aren't found, Sphinx crashes.
> 
> So, making anything that would be compatible with both Sphinx 2.x
> and Sphinx 3.x seems very painful.
> 
> It sounds to me that we need to make a choice here about how to
> proceed. We have a few alternatives:
> 
> 1) stick forever with Sphinx 2.x syntax. We may end needing to 
>    support a fork of Sphinx, if we follow this direction, as
>    nobody currently fix bugs on older versions of Sphinx;
> 
> 2) switch to use Sphinx 3.x c domains and drop support for
>    version < 3.x;

Based on what it was described here:

	https://github.com/sphinx-doc/sphinx/issues/7421

I worked on a RFC patch converting one of the media files
(the CEC one, mentioned at the issue) in order to use the
new Sphinx 3.1 C domain.

Yet, it sounds that the c:namespace is broken on Sphinx 3.2.1:

	Documentation/userspace-api/media/cec/cec-func-close.rst:23: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fclose'.
	Declaration is 'int close( int fd )'.
	Documentation/userspace-api/media/cec/cec-func-close.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fclose'.
	Declaration is 'int close(int fd)'.
	Documentation/userspace-api/media/cec/cec-func-close.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fclose'.
	Declaration is 'int fd'.
	Documentation/userspace-api/media/cec/cec-func-open.rst:22: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fopen'.
	Declaration is 'int open( const char *device_name, int flags )'.
	Documentation/userspace-api/media/cec/cec-func-open.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fopen'.
	Declaration is 'int open(const char *deviceName, int flags)'.
	Documentation/userspace-api/media/cec/cec-func-open.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fopen'.


Or maybe it has to be defined on each individual .rst file that
uses an specific domain?

Thanks,
Mauro

RFC: convert CEC userspace docs to use Sphinx 3.1 domains

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

diff --git a/Documentation/userspace-api/media/cec/cec-api.rst b/Documentation/userspace-api/media/cec/cec-api.rst
index 4d229ed8a1d9..3c33d7f3bcf6 100644
--- a/Documentation/userspace-api/media/cec/cec-api.rst
+++ b/Documentation/userspace-api/media/cec/cec-api.rst
@@ -1,6 +1,8 @@
 .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
 .. include:: <isonum.txt>
 
+.. c:namespace:: CEC
+
 .. _cec:
 
 #########################################
@@ -44,3 +46,5 @@ Revision History
 :revision: 1.0.0 / 2016-03-17 (*hv*)
 
 Initial revision
+
+.. c:namespace:: NULL
diff --git a/Documentation/userspace-api/media/cec/cec-func-close.rst b/Documentation/userspace-api/media/cec/cec-func-close.rst
index 33c563f414a8..4849b315739b 100644
--- a/Documentation/userspace-api/media/cec/cec-func-close.rst
+++ b/Documentation/userspace-api/media/cec/cec-func-close.rst
@@ -21,13 +21,12 @@ Synopsis
 
 
 .. c:function:: int close( int fd )
-    :name: cec-close
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 
 Description
@@ -40,7 +39,7 @@ freed. The device configuration remain unchanged.
 Return Value
 ============
 
-:c:func:`close() <cec-close>` returns 0 on success. On error, -1 is returned, and
+:c:func:close() returns 0 on success. On error, -1 is returned, and
 ``errno`` is set appropriately. Possible error codes are:
 
 ``EBADF``
diff --git a/Documentation/userspace-api/media/cec/cec-func-ioctl.rst b/Documentation/userspace-api/media/cec/cec-func-ioctl.rst
index 3b88230fad80..a584d21afd73 100644
--- a/Documentation/userspace-api/media/cec/cec-func-ioctl.rst
+++ b/Documentation/userspace-api/media/cec/cec-func-ioctl.rst
@@ -20,13 +20,12 @@ Synopsis
 
 
 .. c:function:: int ioctl( int fd, int request, void *argp )
-   :name: cec-ioctl
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``request``
     CEC ioctl request code as defined in the cec.h header file, for
@@ -39,7 +38,7 @@ Arguments
 Description
 ===========
 
-The :c:func:`ioctl() <cec-ioctl>` function manipulates cec device parameters. The
+The :c:func:ioctl() function manipulates cec device parameters. The
 argument ``fd`` must be an open file descriptor.
 
 The ioctl ``request`` code specifies the cec function to be called. It
diff --git a/Documentation/userspace-api/media/cec/cec-func-open.rst b/Documentation/userspace-api/media/cec/cec-func-open.rst
index 887bfd2a755e..ce108e699dd7 100644
--- a/Documentation/userspace-api/media/cec/cec-func-open.rst
+++ b/Documentation/userspace-api/media/cec/cec-func-open.rst
@@ -20,7 +20,6 @@ Synopsis
 
 
 .. c:function:: int open( const char *device_name, int flags )
-   :name: cec-open
 
 
 Arguments
@@ -46,7 +45,7 @@ Arguments
 Description
 ===========
 
-To open a cec device applications call :c:func:`open() <cec-open>` with the
+To open a cec device applications call :c:func:open() with the
 desired device name. The function has no side effects; the device
 configuration remain unchanged.
 
@@ -58,7 +57,7 @@ EBADF.
 Return Value
 ============
 
-:c:func:`open() <cec-open>` returns the new file descriptor on success. On error,
+:c:func:open() returns the new file descriptor on success. On error,
 -1 is returned, and ``errno`` is set appropriately. Possible error codes
 include:
 
diff --git a/Documentation/userspace-api/media/cec/cec-func-poll.rst b/Documentation/userspace-api/media/cec/cec-func-poll.rst
index 2d87136e9a3f..1e1d95303b78 100644
--- a/Documentation/userspace-api/media/cec/cec-func-poll.rst
+++ b/Documentation/userspace-api/media/cec/cec-func-poll.rst
@@ -21,7 +21,6 @@ Synopsis
 
 
 .. c:function:: int poll( struct pollfd *ufds, unsigned int nfds, int timeout )
-   :name: cec-poll
 
 Arguments
 =========
@@ -39,10 +38,10 @@ Arguments
 Description
 ===========
 
-With the :c:func:`poll() <cec-poll>` function applications can wait for CEC
+With the :c:func:poll() function applications can wait for CEC
 events.
 
-On success :c:func:`poll() <cec-poll>` returns the number of file descriptors
+On success :c:func:poll() returns the number of file descriptors
 that have been selected (that is, file descriptors for which the
 ``revents`` field of the respective struct :c:type:`pollfd`
 is non-zero). CEC devices set the ``POLLIN`` and ``POLLRDNORM`` flags in
@@ -53,13 +52,13 @@ then the ``POLLPRI`` flag is set. When the function times out it returns
 a value of zero, on failure it returns -1 and the ``errno`` variable is
 set appropriately.
 
-For more details see the :c:func:`poll() <cec-poll>` manual page.
+For more details see the :c:func:poll() manual page.
 
 
 Return Value
 ============
 
-On success, :c:func:`poll() <cec-poll>` returns the number structures which have
+On success, :c:func:poll() returns the number structures which have
 non-zero ``revents`` fields, or zero if the call timed out. On error -1
 is returned, and the ``errno`` variable is set appropriately:
 
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-caps.rst b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-caps.rst
index 7f25365ce0fb..d74b6d0d44c6 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-caps.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-caps.rst
@@ -14,14 +14,15 @@ CEC_ADAP_G_CAPS - Query device capabilities
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_G_CAPS, struct cec_caps *argp )
-    :name: CEC_ADAP_G_CAPS
+.. c:macro:: CEC_ADAP_G_CAPS
+
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_G_CAPS, struct cec_caps *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
 
@@ -36,32 +37,29 @@ returns the information to the application. The ioctl never fails.
 
 .. tabularcolumns:: |p{1.2cm}|p{2.5cm}|p{13.8cm}|
 
-.. c:type:: cec_caps
+.. c:struct:: cec_caps
 
-.. flat-table:: struct cec_caps
-    :header-rows:  0
-    :stub-columns: 0
-    :widths:       1 1 16
-
-    * - char
-      - ``driver[32]``
-      - The name of the cec adapter driver.
-    * - char
-      - ``name[32]``
-      - The name of this CEC adapter. The combination ``driver`` and
-	``name`` must be unique.
-    * - __u32
-      - ``available_log_addrs``
-      - The maximum number of logical addresses that can be configured.
-    * - __u32
-      - ``capabilities``
-      - The capabilities of the CEC adapter, see
-	:ref:`cec-capabilities`.
-    * - __u32
-      - ``version``
-      - CEC Framework API version, formatted with the ``KERNEL_VERSION()``
-	macro.
+    .. c:var:: char driver[32]
+
+         - The name of the cec adapter driver.
+
+    .. c:var:: char name[32]
+
+         - The name of this CEC adapter. The combination :c:var:`driver` and
+	   :c:var:`name` must be unique.
+
+    .. c:var:: __u32 available_log_addrs
+
+         - The maximum number of logical addresses that can be configured.
+
+    .. c:var:: __u32 capabilities
+
+	 - The capabilities of the CEC adapter, see :ref:`cec-capabilities`.
+
+    .. c:var:: __u32 version
 
+         - CEC Framework API version, formatted with the ``KERNEL_VERSION()``
+	   macro.
 
 .. tabularcolumns:: |p{4.4cm}|p{2.5cm}|p{10.6cm}|
 
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-conn-info.rst b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-conn-info.rst
index 6818ddf1495c..a40c81c0779a 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-conn-info.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-conn-info.rst
@@ -16,14 +16,15 @@ CEC_ADAP_G_CONNECTOR_INFO - Query HDMI connector information
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_G_CONNECTOR_INFO, struct cec_connector_info *argp )
-    :name: CEC_ADAP_G_CONNECTOR_INFO
+.. c:macro:: CEC_ADAP_G_CONNECTOR_INFO
+
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_G_CONNECTOR_INFO, struct cec_connector_info *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
 
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-log-addrs.rst b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-log-addrs.rst
index 1ca893270ae9..7f5a17817c0e 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-log-addrs.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-log-addrs.rst
@@ -17,17 +17,21 @@ CEC_ADAP_G_LOG_ADDRS, CEC_ADAP_S_LOG_ADDRS - Get or set the logical addresses
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_G_LOG_ADDRS, struct cec_log_addrs *argp )
+.. c:macro:: CEC_ADAP_G_LOG_ADDRS
+
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_G_LOG_ADDRS, struct cec_log_addrs *argp)`
    :name: CEC_ADAP_G_LOG_ADDRS
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_S_LOG_ADDRS, struct cec_log_addrs *argp )
+.. c:macro:: CEC_ADAP_S_LOG_ADDRS
+
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_S_LOG_ADDRS, struct cec_log_addrs *argp)`
    :name: CEC_ADAP_S_LOG_ADDRS
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
     Pointer to struct :c:type:`cec_log_addrs`.
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-phys-addr.rst b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-phys-addr.rst
index a10443be1b26..2d83be209dcb 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-phys-addr.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-phys-addr.rst
@@ -17,17 +17,19 @@ CEC_ADAP_G_PHYS_ADDR, CEC_ADAP_S_PHYS_ADDR - Get or set the physical address
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_G_PHYS_ADDR, __u16 *argp )
-    :name: CEC_ADAP_G_PHYS_ADDR
+.. c:macro:: CEC_ADAP_G_PHYS_ADDR
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_S_PHYS_ADDR, __u16 *argp )
-    :name: CEC_ADAP_S_PHYS_ADDR
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_G_PHYS_ADDR, __u16 *argp)`
+
+.. c:macro:: CEC_ADAP_S_PHYS_ADDR
+
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_S_PHYS_ADDR, __u16 *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
     Pointer to the CEC address.
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-dqevent.rst b/Documentation/userspace-api/media/cec/cec-ioc-dqevent.rst
index 3bc81fc5a73f..688f8e88dd86 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-dqevent.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-dqevent.rst
@@ -15,14 +15,15 @@ CEC_DQEVENT - Dequeue a CEC event
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_DQEVENT, struct cec_event *argp )
-    :name: CEC_DQEVENT
+.. c:macro:: CEC_DQEVENT
+
+``int`` :c:expr:`ioctl(int fd, CEC_DQEVENT, struct cec_event *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
 
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-g-mode.rst b/Documentation/userspace-api/media/cec/cec-ioc-g-mode.rst
index 2093e373c93c..482b5a455b9f 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-g-mode.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-g-mode.rst
@@ -13,17 +13,19 @@ CEC_G_MODE, CEC_S_MODE - Get or set exclusive use of the CEC adapter
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_G_MODE, __u32 *argp )
-   :name: CEC_G_MODE
+.. c:macro:: CEC_G_MODE
 
-.. c:function:: int ioctl( int fd, CEC_S_MODE, __u32 *argp )
-   :name: CEC_S_MODE
+``int`` :c:expr:`ioctl(int fd, CEC_G_MODE, __u32 *argp)`
+
+.. c:macro:: CEC_S_MODE
+
+``int`` :c:expr:`ioctl(int fd, CEC_S_MODE, __u32 *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
     Pointer to CEC mode.
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-receive.rst b/Documentation/userspace-api/media/cec/cec-ioc-receive.rst
index 9d629d46973c..d5d8f63eeaf7 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-receive.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-receive.rst
@@ -16,17 +16,20 @@ CEC_RECEIVE, CEC_TRANSMIT - Receive or transmit a CEC message
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_RECEIVE, struct cec_msg \*argp )
-    :name: CEC_RECEIVE
+.. c:macro:: CEC_RECEIVE
 
-.. c:function:: int ioctl( int fd, CEC_TRANSMIT, struct cec_msg \*argp )
+``int`` :c:expr:`ioctl(int fd, CEC_RECEIVE, struct cec_msg \*argp)`
+
+.. c:macro:: CEC_TRANSMIT
+
+``int`` :c:expr:`ioctl(int fd, CEC_TRANSMIT, struct cec_msg \*argp)`
     :name: CEC_TRANSMIT
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
     Pointer to struct cec_msg.

Re: [PATCH RFC] Make the docs build "work" with Sphinx 3.x

[Mauro Carvalho Chehab]

Em Wed, 23 Sep 2020 16:02:10 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> Based on what it was described here:
> 
> 	https://github.com/sphinx-doc/sphinx/issues/7421
> 
> I worked on a RFC patch converting one of the media files
> (the CEC one, mentioned at the issue) in order to use the
> new Sphinx 3.1 C domain.
> 
> Yet, it sounds that the c:namespace is broken on Sphinx 3.2.1:
> 
> 	Documentation/userspace-api/media/cec/cec-func-close.rst:23: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fclose'.
> 	Declaration is 'int close( int fd )'.
> 	Documentation/userspace-api/media/cec/cec-func-close.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fclose'.
> 	Declaration is 'int close(int fd)'.
> 	Documentation/userspace-api/media/cec/cec-func-close.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fclose'.
> 	Declaration is 'int fd'.
> 	Documentation/userspace-api/media/cec/cec-func-open.rst:22: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fopen'.
> 	Declaration is 'int open( const char *device_name, int flags )'.
> 	Documentation/userspace-api/media/cec/cec-func-open.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fopen'.
> 	Declaration is 'int open(const char *deviceName, int flags)'.
> 	Documentation/userspace-api/media/cec/cec-func-open.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fopen'.
> 
> Or maybe it has to be defined on each individual .rst file that
> uses an specific domain?

Ok, I was able to get rid of the above warnings, after some discussions at:

	https://github.com/sphinx-doc/sphinx/issues/7421

In summary, the ".. c:namespace::" tag should be used on _every_ file
that uses a different namespace. For CEC, this is not too much,
but on v4l we have 249 files. Lots of them will require the usage of
the namespace directive.

I'm enclosing a patch with the minimal set of changes required to
make the CEC documentation compatible with Sphinx 3.1 and above.

Btw, according with Jakob, there was a bug that cause a run time
regression on Sphinx 3.2, affecting its performance:

	https://github.com/sphinx-doc/sphinx/issues/7421#issuecomment-697710707

Upstream Sphinx developers are working on a fix. So, maybe the
performance for an upcoming 3.2.2 will be similar to 2.x.

Jon,

Should we proceed enforcing version 3.1 to be the minimal one
required?

If so, I can take care of the conversion patches for the
media documents. I guess it should be easy to write a script that
would convert all the tags there to use the new notation.

I can also take a look at the kernel-doc issues.

-

If something has enough time to work at cdomain.py, we may
still have backward compatibility with versions 1.x/2.x
(support for 3.0.x would be a lot harder, IMHO), if cdomain.py 
would get support for at least those two tags:

	:c:expr:
	.. c:namespace::

Those two are the ones that are required to fix Sphinx 3.1+
support for the media docs (although there are other new
c domain tags that could also be interesting).

No idea wow easy/hard would be to add support for them over
there, as cdomain.py currently relies on the Sphinx 1.x/2.x
c domain code. I suspect that implementing support for namespace 
shouldn't be hard, as it internally does that already, but
using a different notation. Support for other tags can be
trickier.

I guess support for :c:expr:`foo()` should also be easy,
as this doesn't produce cross references. So, the code
could just convert this into something equivalent to:

	**``foo()``**

(e. g. into a bold monospaced font).

In any case, if we go to this direction, we'll be limited
to use a subset of the Sphinx 3.1 c domain notation.


Thanks,
Mauro

[PATCH 0/2] Start supporting builds with Sphinx 3.1+

[Mauro Carvalho Chehab]

Hi Jon,

This is a small patch series with just two patches.

The first one adds support at cdomain.py for two notations found on
Sphinx 3.1:

	:c:expr:
	.. c:namespace::

With that, it should now be possible to use those two C domain
tags at the media documentation, which will make it produce a
decent result with both Sphinx 1.x/2.x and Sphinx 3.1+.

The second patch manually changes the CEC documentation in
order for it to use those macros, instead of relying at the original
cdomain extensions.

I tested building the docs with both Sphinx 2.4.4 and 3.2.1.
They are identical, except by a minor difference:  the output of
:c:expr: uses a bold monospaced font with 3.1+, while it uses a
non-bold monospaced font with older versions.

Yet, the output looks decent on both versions.

I'm planning to use the same approach on all the other documents
under userspace-api/media. So, I guess it would be easier if
I could merge both the cdomain.py and the media patches via
the media tree, if this is ok for you.

-

With regards to patch 1, I tried first to use a hook at 'doctree-resolved',
just like the automarkup.py, but that is too late for  changing the
namespace. So, I ended hooking the extra logic at 'source-read'.

I suspect that this could be implemented on some other ways, but
this can be optimized later on, if needed.

Mauro Carvalho Chehab (2):
  docs: cdomain.py: add support for two new Sphinx 3.1+ tags
  media: docs: make CEC documents compatible with Sphinx 3.1+

 Documentation/sphinx/cdomain.py               | 56 ++++++++++++++++++-
 .../media/cec/cec-func-close.rst              |  7 ++-
 .../media/cec/cec-func-ioctl.rst              |  7 ++-
 .../userspace-api/media/cec/cec-func-open.rst |  7 ++-
 .../userspace-api/media/cec/cec-func-poll.rst | 11 ++--
 .../media/cec/cec-ioc-adap-g-caps.rst         |  9 ++-
 .../media/cec/cec-ioc-adap-g-conn-info.rst    | 11 ++--
 .../media/cec/cec-ioc-adap-g-log-addrs.rst    | 14 +++--
 .../media/cec/cec-ioc-adap-g-phys-addr.rst    | 14 +++--
 .../media/cec/cec-ioc-dqevent.rst             |  9 ++-
 .../media/cec/cec-ioc-g-mode.rst              | 14 +++--
 .../media/cec/cec-ioc-receive.rst             | 14 +++--
 12 files changed, 128 insertions(+), 45 deletions(-)

-- 
2.26.2

[PATCH 1/2] docs: cdomain.py: add support for two new Sphinx 3.1+ tags

[Mauro Carvalho Chehab]

Since Sphinx 3.0, the C domain code was rewritten, but only
after version 3.1 it got support for setting namespaces on
C domains, with is something that it is required, in order to
document system calls, like ioctl() and others.

As part of changing the documentation subsystem to properly
build with Sphinx 3.1+, add support for two tags:

	- :c:expr:`foo`
	- .. c:namespace::"

The first one just replaces the expresion by ``foo``, with
produces a monotext expression.

The second one replaces the optional "name" tag for functions,
setting a domain for all C references found after its usage.

With that, it should be possible to convert the existing
documentation to be compatible with both Sphinx 1.x/2.x and
3.1+.

Unfortunately, building the documentation with Sphinx 3.0
will produce lots of warnings, because the namespace tag
doesn't exist there, with will cause both warnings for the
usage of a non-existing tag and warnings about multiple
definitions for system calls. There's not much we can
do to solve such issues.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/sphinx/cdomain.py | 56 ++++++++++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/Documentation/sphinx/cdomain.py b/Documentation/sphinx/cdomain.py
index cbac8e608dc4..3f6228787282 100644
--- a/Documentation/sphinx/cdomain.py
+++ b/Documentation/sphinx/cdomain.py
@@ -40,14 +40,61 @@ from sphinx import addnodes
 from sphinx.domains.c import c_funcptr_sig_re, c_sig_re
 from sphinx.domains.c import CObject as Base_CObject
 from sphinx.domains.c import CDomain as Base_CDomain
+from itertools import chain
+import re
 
-__version__  = '1.0'
+__version__  = '1.1'
 
 # Get Sphinx version
 major, minor, patch = sphinx.version_info[:3]
 
+# Namespace to be prepended to the full name
+namespace = None
+
+#
+# Handle trivial newer c domain tags that are part of Sphinx 3.1 c domain tags
+# - Convert :c:expr:`foo` into ``foo``
+# - Store the namespace if ".. c:namespace::" tag is found
+
+RE_namespace = re.compile(r'^\s*..\s*c:namespace::\s*(\S+)\s*$')
+RE_expr = re.compile(r':c:expr:`([^\`]+)`')
+
+def markup_namespace(match):
+    namespace = match.group(1)
+
+    return ""
+
+def markup_c_expr(match):
+
+    return '\ ``' + match.group(1) + '``\ '
+
+def c_markups(app, docname, source):
+    result = ""
+    markup_func = {
+        RE_namespace: markup_namespace,
+        RE_expr: markup_c_expr
+    }
+
+    lines = iter(source[0].splitlines(True))
+    for n in lines:
+        match_iterators = [regex.finditer(n) for regex in markup_func]
+        matches = sorted(chain(*match_iterators), key=lambda m: m.start())
+        for m in matches:
+            n = n[:m.start()] + markup_func[m.re](m) + n[m.end():]
+
+        result = result + n
+
+    source[0] = result
+
+#
+# Now implements support for the cdomain namespacing logic
+#
+
 def setup(app):
 
+    # Handle easy Sphinx 3.1+ simple new tags: :c:expr and .. c:namespace::
+    app.connect('source-read', c_markups)
+
     if (major == 1 and minor < 8):
         app.override_domain(CDomain)
     else:
@@ -107,6 +154,9 @@ class CObject(Base_CObject):
             param += nodes.emphasis(argname, argname)
             paramlist += param
 
+        if namespace:
+            fullname = namespace + "." + fullname
+
         return fullname
 
     def handle_signature(self, sig, signode):
@@ -122,6 +172,10 @@ class CObject(Base_CObject):
             else:
                 # FIXME: handle :name: value of other declaration types?
                 pass
+        else:
+            if namespace:
+                fullname = namespace + "." + fullname
+
         return fullname
 
     def add_target_and_index(self, name, sig, signode):
-- 
2.26.2

Re: [PATCH 1/2] docs: cdomain.py: add support for two new Sphinx 3.1+ tags

[Jonathan Corbet]

So I'm just getting into this and trying to understand what's really going
on, but one thing jumped at me:

On Thu, 24 Sep 2020 13:22:04 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> +# Namespace to be prepended to the full name
> +namespace = None
> +
> +#
> +# Handle trivial newer c domain tags that are part of Sphinx 3.1 c domain tags
> +# - Convert :c:expr:`foo` into ``foo``
> +# - Store the namespace if ".. c:namespace::" tag is found
> +
> +RE_namespace = re.compile(r'^\s*..\s*c:namespace::\s*(\S+)\s*$')
> +RE_expr = re.compile(r':c:expr:`([^\`]+)`')
> +
> +def markup_namespace(match):
> +    namespace = match.group(1)
> +
> +    return ""
> +

How can this possibly work without a "global namespace" declaration in
markup_namespace()?

jon

Re: [PATCH 1/2] docs: cdomain.py: add support for two new Sphinx 3.1+ tags

[Mauro Carvalho Chehab]

Em Thu, 24 Sep 2020 09:43:35 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> So I'm just getting into this and trying to understand what's really going
> on, but one thing jumped at me:
> 
> On Thu, 24 Sep 2020 13:22:04 +0200
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> 
> > +# Namespace to be prepended to the full name
> > +namespace = None

^^^ See here....

> > +
> > +#
> > +# Handle trivial newer c domain tags that are part of Sphinx 3.1 c domain tags
> > +# - Convert :c:expr:`foo` into ``foo``
> > +# - Store the namespace if ".. c:namespace::" tag is found
> > +
> > +RE_namespace = re.compile(r'^\s*..\s*c:namespace::\s*(\S+)\s*$')
> > +RE_expr = re.compile(r':c:expr:`([^\`]+)`')
> > +
> > +def markup_namespace(match):
> > +    namespace = match.group(1)
> > +
> > +    return ""
> > +  
> 
> How can this possibly work without a "global namespace" declaration in
> markup_namespace()?

... While I'm not a python expert, the namespace variable is global
    because it was defined outside the "markup_namespace" function.

-

On a quick check at the internet, this is supposed to work properly:

	https://www.programiz.com/python-programming/global-local-nonlocal-variables

-

In any cases, on my tests, this is working properly.

Anyway, I'm sending a version 2 of the series, addressing the
namespace for the remaining files under userspace-api/media.

You can try building the media books without and with the
patch series, in order to see the differences.

There are still ~200 warnings produced after that, but it
sounds that the remaining issues will require changes at
kernel-doc.


Thanks,
Mauro

Re: [PATCH 1/2] docs: cdomain.py: add support for two new Sphinx 3.1+ tags

[Jonathan Corbet]

On Thu, 24 Sep 2020 18:13:54 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> > How can this possibly work without a "global namespace" declaration in
> > markup_namespace()?  
> 
> ... While I'm not a python expert, the namespace variable is global
>     because it was defined outside the "markup_namespace" function.

Assignments within functions are *always* local unless declared global.

Try this:

	$ python3
	>>> x = 0
	>>> def y(v):
	>>>	x = v
	>>>
	>>> y(1)
	>>> x
	0
	>>>

So your assignment to "namespace" in markup_namespace() cannot change the
global, since it's not declared global.

jon

Re: [PATCH 1/2] docs: cdomain.py: add support for two new Sphinx 3.1+ tags

[Mauro Carvalho Chehab]

Em Thu, 24 Sep 2020 10:22:25 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> On Thu, 24 Sep 2020 18:13:54 +0200
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> 
> > > How can this possibly work without a "global namespace" declaration in
> > > markup_namespace()?    
> > 
> > ... While I'm not a python expert, the namespace variable is global
> >     because it was defined outside the "markup_namespace" function.  
> 
> Assignments within functions are *always* local unless declared global.
> 
> Try this:
> 
> 	$ python3
> 	>>> x = 0
> 	>>> def y(v):
> 	>>>	x = v
> 	>>>
> 	>>> y(1)
> 	>>> x  
> 	0
> 	>>>  
> 
> So your assignment to "namespace" in markup_namespace() cannot change the
> global, since it's not declared global.

Ok! Thanks for helping with this. I'll declare namespace as global for
the next version.

Thanks,
Mauro

[PATCH 2/2] media: docs: make CEC documents compatible with Sphinx 3.1+

[Mauro Carvalho Chehab]

Sphinx 3.x broke support for the cdomain.py extension, as the
c domain code was rewritten. Due to that, the c tags need to
be re-written, in order to use the new c domain notation.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 .../userspace-api/media/cec/cec-func-close.rst     |  7 ++++---
 .../userspace-api/media/cec/cec-func-ioctl.rst     |  7 ++++---
 .../userspace-api/media/cec/cec-func-open.rst      |  7 ++++---
 .../userspace-api/media/cec/cec-func-poll.rst      | 11 ++++++-----
 .../media/cec/cec-ioc-adap-g-caps.rst              |  9 ++++++---
 .../media/cec/cec-ioc-adap-g-conn-info.rst         | 11 +++++++----
 .../media/cec/cec-ioc-adap-g-log-addrs.rst         | 14 +++++++++-----
 .../media/cec/cec-ioc-adap-g-phys-addr.rst         | 14 +++++++++-----
 .../userspace-api/media/cec/cec-ioc-dqevent.rst    |  9 ++++++---
 .../userspace-api/media/cec/cec-ioc-g-mode.rst     | 14 +++++++++-----
 .../userspace-api/media/cec/cec-ioc-receive.rst    | 14 +++++++++-----
 11 files changed, 73 insertions(+), 44 deletions(-)

diff --git a/Documentation/userspace-api/media/cec/cec-func-close.rst b/Documentation/userspace-api/media/cec/cec-func-close.rst
index 33c563f414a8..7771e40aa6e8 100644
--- a/Documentation/userspace-api/media/cec/cec-func-close.rst
+++ b/Documentation/userspace-api/media/cec/cec-func-close.rst
@@ -1,5 +1,7 @@
 .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
 
+.. c:namespace:: CEC
+
 .. _cec-func-close:
 
 ***********
@@ -21,13 +23,12 @@ Synopsis
 
 
 .. c:function:: int close( int fd )
-    :name: cec-close
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 
 Description
@@ -40,7 +41,7 @@ freed. The device configuration remain unchanged.
 Return Value
 ============
 
-:c:func:`close() <cec-close>` returns 0 on success. On error, -1 is returned, and
+:c:func:close() returns 0 on success. On error, -1 is returned, and
 ``errno`` is set appropriately. Possible error codes are:
 
 ``EBADF``
diff --git a/Documentation/userspace-api/media/cec/cec-func-ioctl.rst b/Documentation/userspace-api/media/cec/cec-func-ioctl.rst
index 3b88230fad80..3d2fed0470a8 100644
--- a/Documentation/userspace-api/media/cec/cec-func-ioctl.rst
+++ b/Documentation/userspace-api/media/cec/cec-func-ioctl.rst
@@ -1,5 +1,7 @@
 .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
 
+.. c:namespace:: CEC
+
 .. _cec-func-ioctl:
 
 ***********
@@ -20,13 +22,12 @@ Synopsis
 
 
 .. c:function:: int ioctl( int fd, int request, void *argp )
-   :name: cec-ioctl
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``request``
     CEC ioctl request code as defined in the cec.h header file, for
@@ -39,7 +40,7 @@ Arguments
 Description
 ===========
 
-The :c:func:`ioctl() <cec-ioctl>` function manipulates cec device parameters. The
+The :c:func:ioctl() function manipulates cec device parameters. The
 argument ``fd`` must be an open file descriptor.
 
 The ioctl ``request`` code specifies the cec function to be called. It
diff --git a/Documentation/userspace-api/media/cec/cec-func-open.rst b/Documentation/userspace-api/media/cec/cec-func-open.rst
index 887bfd2a755e..de43707768ca 100644
--- a/Documentation/userspace-api/media/cec/cec-func-open.rst
+++ b/Documentation/userspace-api/media/cec/cec-func-open.rst
@@ -1,5 +1,7 @@
 .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
 
+.. c:namespace:: CEC
+
 .. _cec-func-open:
 
 **********
@@ -20,7 +22,6 @@ Synopsis
 
 
 .. c:function:: int open( const char *device_name, int flags )
-   :name: cec-open
 
 
 Arguments
@@ -46,7 +47,7 @@ Arguments
 Description
 ===========
 
-To open a cec device applications call :c:func:`open() <cec-open>` with the
+To open a cec device applications call :c:func:open() with the
 desired device name. The function has no side effects; the device
 configuration remain unchanged.
 
@@ -58,7 +59,7 @@ EBADF.
 Return Value
 ============
 
-:c:func:`open() <cec-open>` returns the new file descriptor on success. On error,
+:c:func:open() returns the new file descriptor on success. On error,
 -1 is returned, and ``errno`` is set appropriately. Possible error codes
 include:
 
diff --git a/Documentation/userspace-api/media/cec/cec-func-poll.rst b/Documentation/userspace-api/media/cec/cec-func-poll.rst
index 2d87136e9a3f..3da4a96fb921 100644
--- a/Documentation/userspace-api/media/cec/cec-func-poll.rst
+++ b/Documentation/userspace-api/media/cec/cec-func-poll.rst
@@ -1,5 +1,7 @@
 .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
 
+.. c:namespace:: CEC
+
 .. _cec-func-poll:
 
 **********
@@ -21,7 +23,6 @@ Synopsis
 
 
 .. c:function:: int poll( struct pollfd *ufds, unsigned int nfds, int timeout )
-   :name: cec-poll
 
 Arguments
 =========
@@ -39,10 +40,10 @@ Arguments
 Description
 ===========
 
-With the :c:func:`poll() <cec-poll>` function applications can wait for CEC
+With the :c:func:poll() function applications can wait for CEC
 events.
 
-On success :c:func:`poll() <cec-poll>` returns the number of file descriptors
+On success :c:func:poll() returns the number of file descriptors
 that have been selected (that is, file descriptors for which the
 ``revents`` field of the respective struct :c:type:`pollfd`
 is non-zero). CEC devices set the ``POLLIN`` and ``POLLRDNORM`` flags in
@@ -53,13 +54,13 @@ then the ``POLLPRI`` flag is set. When the function times out it returns
 a value of zero, on failure it returns -1 and the ``errno`` variable is
 set appropriately.
 
-For more details see the :c:func:`poll() <cec-poll>` manual page.
+For more details see the :c:func:poll() manual page.
 
 
 Return Value
 ============
 
-On success, :c:func:`poll() <cec-poll>` returns the number structures which have
+On success, :c:func:poll() returns the number structures which have
 non-zero ``revents`` fields, or zero if the call timed out. On error -1
 is returned, and the ``errno`` variable is set appropriately:
 
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-caps.rst b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-caps.rst
index 7f25365ce0fb..f19e2e4e3f3d 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-caps.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-caps.rst
@@ -1,5 +1,7 @@
 .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
 
+.. c:namespace:: CEC
+
 .. _CEC_ADAP_G_CAPS:
 
 *********************
@@ -14,14 +16,15 @@ CEC_ADAP_G_CAPS - Query device capabilities
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_G_CAPS, struct cec_caps *argp )
-    :name: CEC_ADAP_G_CAPS
+.. c:macro:: CEC_ADAP_G_CAPS
+
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_G_CAPS, struct cec_caps *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
 
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-conn-info.rst b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-conn-info.rst
index 6818ddf1495c..68a6dcfedfe0 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-conn-info.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-conn-info.rst
@@ -1,7 +1,9 @@
 .. SPDX-License-Identifier: GPL-2.0
 ..
 .. Copyright 2019 Google LLC
-..
+
+.. c:namespace:: CEC
+
 .. _CEC_ADAP_G_CONNECTOR_INFO:
 
 *******************************
@@ -16,14 +18,15 @@ CEC_ADAP_G_CONNECTOR_INFO - Query HDMI connector information
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_G_CONNECTOR_INFO, struct cec_connector_info *argp )
-    :name: CEC_ADAP_G_CONNECTOR_INFO
+.. c:macro:: CEC_ADAP_G_CONNECTOR_INFO
+
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_G_CONNECTOR_INFO, struct cec_connector_info *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
 
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-log-addrs.rst b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-log-addrs.rst
index 1ca893270ae9..fafee09cd156 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-log-addrs.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-log-addrs.rst
@@ -1,5 +1,7 @@
 .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
 
+.. c:namespace:: CEC
+
 .. _CEC_ADAP_LOG_ADDRS:
 .. _CEC_ADAP_G_LOG_ADDRS:
 .. _CEC_ADAP_S_LOG_ADDRS:
@@ -17,17 +19,19 @@ CEC_ADAP_G_LOG_ADDRS, CEC_ADAP_S_LOG_ADDRS - Get or set the logical addresses
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_G_LOG_ADDRS, struct cec_log_addrs *argp )
-   :name: CEC_ADAP_G_LOG_ADDRS
+.. c:macro:: CEC_ADAP_G_LOG_ADDRS
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_S_LOG_ADDRS, struct cec_log_addrs *argp )
-   :name: CEC_ADAP_S_LOG_ADDRS
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_G_LOG_ADDRS, struct cec_log_addrs *argp)`
+
+.. c:macro:: CEC_ADAP_S_LOG_ADDRS
+
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_S_LOG_ADDRS, struct cec_log_addrs *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
     Pointer to struct :c:type:`cec_log_addrs`.
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-phys-addr.rst b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-phys-addr.rst
index a10443be1b26..ba5bca26fbd8 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-adap-g-phys-addr.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-adap-g-phys-addr.rst
@@ -1,5 +1,7 @@
 .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
 
+.. c:namespace:: CEC
+
 .. _CEC_ADAP_PHYS_ADDR:
 .. _CEC_ADAP_G_PHYS_ADDR:
 .. _CEC_ADAP_S_PHYS_ADDR:
@@ -17,17 +19,19 @@ CEC_ADAP_G_PHYS_ADDR, CEC_ADAP_S_PHYS_ADDR - Get or set the physical address
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_G_PHYS_ADDR, __u16 *argp )
-    :name: CEC_ADAP_G_PHYS_ADDR
+.. c:macro:: CEC_ADAP_G_PHYS_ADDR
 
-.. c:function:: int ioctl( int fd, CEC_ADAP_S_PHYS_ADDR, __u16 *argp )
-    :name: CEC_ADAP_S_PHYS_ADDR
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_G_PHYS_ADDR, __u16 *argp)`
+
+.. c:macro:: CEC_ADAP_S_PHYS_ADDR
+
+``int`` :c:expr:`ioctl(int fd, CEC_ADAP_S_PHYS_ADDR, __u16 *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
     Pointer to the CEC address.
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-dqevent.rst b/Documentation/userspace-api/media/cec/cec-ioc-dqevent.rst
index 3bc81fc5a73f..394e3c3acf9d 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-dqevent.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-dqevent.rst
@@ -1,5 +1,7 @@
 .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
 
+.. c:namespace:: CEC
+
 .. _CEC_DQEVENT:
 
 *****************
@@ -15,14 +17,15 @@ CEC_DQEVENT - Dequeue a CEC event
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_DQEVENT, struct cec_event *argp )
-    :name: CEC_DQEVENT
+.. c:macro:: CEC_DQEVENT
+
+``int`` :c:expr:`ioctl(int fd, CEC_DQEVENT, struct cec_event *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
 
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-g-mode.rst b/Documentation/userspace-api/media/cec/cec-ioc-g-mode.rst
index 2093e373c93c..a2daf4017e91 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-g-mode.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-g-mode.rst
@@ -1,5 +1,7 @@
 .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
 
+.. c:namespace:: CEC
+
 .. _CEC_MODE:
 .. _CEC_G_MODE:
 .. _CEC_S_MODE:
@@ -13,17 +15,19 @@ CEC_G_MODE, CEC_S_MODE - Get or set exclusive use of the CEC adapter
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_G_MODE, __u32 *argp )
-   :name: CEC_G_MODE
+.. c:macro:: CEC_G_MODE
 
-.. c:function:: int ioctl( int fd, CEC_S_MODE, __u32 *argp )
-   :name: CEC_S_MODE
+``int`` :c:expr:`ioctl(int fd, CEC_G_MODE, __u32 *argp)`
+
+.. c:macro:: CEC_S_MODE
+
+``int`` :c:expr:`ioctl(int fd, CEC_S_MODE, __u32 *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
     Pointer to CEC mode.
diff --git a/Documentation/userspace-api/media/cec/cec-ioc-receive.rst b/Documentation/userspace-api/media/cec/cec-ioc-receive.rst
index 9d629d46973c..c0689c52e3f7 100644
--- a/Documentation/userspace-api/media/cec/cec-ioc-receive.rst
+++ b/Documentation/userspace-api/media/cec/cec-ioc-receive.rst
@@ -1,5 +1,7 @@
 .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
 
+.. c:namespace:: CEC
+
 .. _CEC_TRANSMIT:
 .. _CEC_RECEIVE:
 
@@ -16,17 +18,19 @@ CEC_RECEIVE, CEC_TRANSMIT - Receive or transmit a CEC message
 Synopsis
 ========
 
-.. c:function:: int ioctl( int fd, CEC_RECEIVE, struct cec_msg \*argp )
-    :name: CEC_RECEIVE
+.. c:macro:: CEC_RECEIVE
 
-.. c:function:: int ioctl( int fd, CEC_TRANSMIT, struct cec_msg \*argp )
-    :name: CEC_TRANSMIT
+``int`` :c:expr:`ioctl(int fd, CEC_RECEIVE, struct cec_msg *argp)`
+
+.. c:macro:: CEC_TRANSMIT
+
+``int`` :c:expr:`ioctl(int fd, CEC_TRANSMIT, struct cec_msg *argp)`
 
 Arguments
 =========
 
 ``fd``
-    File descriptor returned by :c:func:`open() <cec-open>`.
+    File descriptor returned by :c:func:open().
 
 ``argp``
     Pointer to struct cec_msg.
-- 
2.26.2

Re: [PATCH RFC] Make the docs build "work" with Sphinx 3.x

[Mauro Carvalho Chehab]

Em Wed, 23 Sep 2020 14:31:55 +0200
Mauro Carvalho Chehab <mchehab@infradead.org> escreveu:

> Em Tue, 22 Sep 2020 10:23:46 +0300
> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> 
> > On Mon, 21 Sep 2020, Matthew Wilcox <willy@infradead.org> wrote:  
> > > On Fri, Sep 04, 2020 at 10:29:25AM -0600, Jonathan Corbet wrote:    
> > >> +# cdomain is badly broken in Sphinx 3+.  Leaving it out generates *most*
> > >> +# of the docs correctly, but not all.  Scream bloody murder but allow
> > >> +# the process to proceed; hopefully somebody will fix this properly soon.
> > >> +#
> > >> +if major >= 3:
> > >> +    sys.stderr.write('''WARNING: The kernel documentation build process
> > >> +	does not work correctly with Sphinx v3.0 and above.  Expect errors
> > >> +	in the generated output.    
> > >
> > > Should we be converting the kernel-doc script to output 3.0-style markup
> > > for structs?    
> > 
> > IIUC it's not backwards compatible with Sphinx pre-3.0, so you'd either
> > have to require Sphinx 3.0 or have kernel-doc emit both, depending on
> > the Sphinx version.  
> 
> Well, making kernel-doc to emit both may be not too hard, as it
> could be doing something like:

Hi Jani,

> 
> <snp
> diff --git a/Documentation/conf.py b/Documentation/conf.py
> index 0a102d57437d..0605949db383 100644
> --- a/Documentation/conf.py
> +++ b/Documentation/conf.py
> @@ -50,6 +50,7 @@ if major >= 3:
>  	does not work correctly with Sphinx v3.0 and above.  Expect errors
>  	in the generated output.
>  	''')
> +    tags.add('sphinx3')
>  else:
>      extensions.append('cdomain')
>  
> diff --git a/Documentation/userspace-api/media/v4l/v4l2.rst b/Documentation/userspace-api/media/v4l/v4l2.rst
> index ad7a2bf0cf26..5007f33a7920 100644
> --- a/Documentation/userspace-api/media/v4l/v4l2.rst
> +++ b/Documentation/userspace-api/media/v4l/v4l2.rst
> @@ -3,6 +3,10 @@
>  
>  .. _v4l2spec:
>  
> +.. only: sphinx3
> +
> +   .. c:namespace:: v4l_uAPI
> +
>  ############################
>  Part I - Video for Linux API
>  ############################
> @@ -416,3 +420,7 @@ Second draft, with corrections pointed out by Gerd Knorr.
>  
>  First draft, based on documentation by Bill Dirks and discussions on the
>  V4L mailing list.
> +
> +.. only: sphinx3
> +
> +   .. c:namespace:: NULL

I actually tried that, but this didn't work.

Btw, it is not that simple. It needs to convert also several other tags,
and some are in the middle of the tests.

So, even if this would work, a change like that will introduce lots
of "noise" at the docs.

The approach of adding support for c:namespace: at cdomain
(see the patchset I sent today) is a lot more cleaner.

Thanks,
Mauro