Path troubles when using SPHINXDIRS option

Path troubles when using SPHINXDIRS option

[Mauro Carvalho Chehab]

Hi,

While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option
stopped working.

What happens is that, if we don't pass this option, Sphinx assumes that
the "absolute" path is "Documentation/". So, include options like this:

	.. include:: /ABI/stable/firewire-cdev
	    :literal:


would be evaluated as Documentation/ABI/stable/firewire-cdev if built
without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then
the "absolute" patch would be "Documentation/driver-api", causing this
error:

	Sphinx parallel build error:
	docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:

This is specially bad for build jobs (like one we have on jenkins that
is meant to test media patches), as, due to some Sphinx bug, prevents
sphinx-build to stop, making the toolset to run forever.

I suspect that some change at conf.py could address the path issue.

I'll try to investigate further.

Just to be 100% sure that this is not some version-specific bad
behavior, I tested (using the latest patch version) Sphinx from
version 1.7.9 up to 2.4.4. The same two errors happen on all
versions.

Markus,

Maybe you may have some idea about how to fix those issues.

The parallel build error would likely require fixing something
inside Sphinx code, making it abort if it gets a "SEVERE" error.

Regards,
Mauro

Re: Path troubles when using SPHINXDIRS option

[Mauro Carvalho Chehab]

Em Tue, 21 Apr 2020 10:38:14 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> Hi,
> 
> While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option
> stopped working.
> 
> What happens is that, if we don't pass this option, Sphinx assumes that
> the "absolute" path is "Documentation/". So, include options like this:
> 
> 	.. include:: /ABI/stable/firewire-cdev
> 	    :literal:
> 
> 
> would be evaluated as Documentation/ABI/stable/firewire-cdev if built
> without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then
> the "absolute" patch would be "Documentation/driver-api", causing this
> error:
> 
> 	Sphinx parallel build error:
> 	docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:
> 
> This is specially bad for build jobs (like one we have on jenkins that
> is meant to test media patches), as, due to some Sphinx bug, prevents
> sphinx-build to stop, making the toolset to run forever.
> 
> I suspect that some change at conf.py could address the path issue.
> 
> I'll try to investigate further.

More updates on it:

It seems that the main issue is that Sphinx passes Documentation/foo
to docutils, when SPHINXDIRS=foo is used, and docutils prevent
access anything outside Documentation/foo on its include and toctree
directives.

So, even trying to use "../ABI/some_doc" would fail.

-

The other issue is that SphinxParallelError doesn't stop the
build multithreading tasks. As Documentation/docutils.conf is set to 
abort on severe errors, this causes Sphinx to keep running forever,
doing nothing.

I did already a workaround to prevent media Jenkins job to hang
due to those issues. So, I'm moving on. Hopefully, someone with
more experience with Sphinx, docutils and python multithreading
can work on some fixes for it.

Thanks,
Mauro

Re: Path troubles when using SPHINXDIRS option

[Markus Heiser]

Hi Mauro,

Am 21.04.20 um 10:38 schrieb Mauro Carvalho Chehab:
> 
> Hi,
> 
> While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option
> stopped working.
> 
> What happens is that, if we don't pass this option, Sphinx assumes that
> the "absolute" path is "Documentation/". So, include options like this:
> 
> 	.. include:: /ABI/stable/firewire-cdev
> 	    :literal:
> 
> 
> would be evaluated as Documentation/ABI/stable/firewire-cdev if built
> without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then
> the "absolute" patch would be "Documentation/driver-api", causing this
> error:
> 
> 	Sphinx parallel build error:
> 	docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:
> 
> This is specially bad for build jobs (like one we have on jenkins that
> is meant to test media patches), as, due to some Sphinx bug, prevents
> sphinx-build to stop, making the toolset to run forever.
> 
> I suspect that some change at conf.py could address the path issue.
> 
> I'll try to investigate further.
> 
> Just to be 100% sure that this is not some version-specific bad
> behavior, I tested (using the latest patch version) Sphinx from
> version 1.7.9 up to 2.4.4. The same two errors happen on all
> versions.
> 
> Markus,
> 
> Maybe you may have some idea about how to fix those issues.
> 
> The parallel build error would likely require fixing something
> inside Sphinx code, making it abort if it gets a "SEVERE" error.
> 
> Regards,
> Mauro
> 

I fixed it with the patch shown below.  Some words about: The use
of the :doc: rule should be replaced by a :ref:.  The path names
of include directives should always relative to the .rst file.

To go further ... with patch below:

    $ make SPHINXDIRS="driver-api" htmldocs

shows some " WARNING: undefined label: ..." messages.  To close
such links (addressing objects outside of the SPHINXDIR) we need
to activate intersphinx [1].

If we activate intersphinx we have to wait one release cycle to
get a objects.inv file at:

    https://www.kernel.org/doc/html/latest/

When the objects.inv file is available, the warnings are
disappear.  

If the process of intersphinx is unclear, please ask.


-- Markus --

[1] https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html

----

diff --git a/Documentation/driver-api/firewire.rst b/Documentation/driver-api/firewire.rst
index 94a2d7f01d99..67741aeae5e8 100644
--- a/Documentation/driver-api/firewire.rst
+++ b/Documentation/driver-api/firewire.rst
@@ -19,16 +19,16 @@ of kernel interfaces is available via exported symbols in `firewire-core` module
 Firewire char device data structures
 ====================================
 
-.. include:: /ABI/stable/firewire-cdev
+.. include:: ../ABI/stable/firewire-cdev
     :literal:
 
-.. kernel-doc:: include/uapi/linux/firewire-cdev.h
+.. kernel-doc:: /include/uapi/linux/firewire-cdev.h
     :internal:
 
 Firewire device probing and sysfs interfaces
 ============================================
 
-.. include:: /ABI/stable/sysfs-bus-firewire
+.. include:: ../ABI/stable/sysfs-bus-firewire
     :literal:
 
 .. kernel-doc:: drivers/firewire/core-device.c
diff --git a/Documentation/driver-api/gpio/using-gpio.rst b/Documentation/driver-api/gpio/using-gpio.rst
index dda069444032..f2bd1e72af4e 100644
--- a/Documentation/driver-api/gpio/using-gpio.rst
+++ b/Documentation/driver-api/gpio/using-gpio.rst
@@ -15,7 +15,7 @@ For any kind of mass produced system you want to support, such as servers,
 laptops, phones, tablets, routers, and any consumer or office or business goods
 using appropriate kernel drivers is paramount. Submit your code for inclusion
 in the upstream Linux kernel when you feel it is mature enough and you will get
-help to refine it, see :doc:`../../process/submitting-patches`.
+help to refine it, see :ref:`submittingpatches`.
 
 In Linux GPIO lines also have a userspace ABI.
 
diff --git a/Documentation/driver-api/ioctl.rst b/Documentation/driver-api/ioctl.rst
index c455db0e1627..ad91db68ba89 100644
--- a/Documentation/driver-api/ioctl.rst
+++ b/Documentation/driver-api/ioctl.rst
@@ -34,7 +34,7 @@ _IO/_IOR/_IOW/_IOWR
 
 type
    An 8-bit number, often a character literal, specific to a subsystem
-   or driver, and listed in :doc:`../userspace-api/ioctl/ioctl-number`
+   or driver, and listed in :ref:`ioctl numbers`
 
 nr
   An 8-bit number identifying the specific command, unique for a give
diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
index 52bf58417653..09647516ca52 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -1,3 +1,5 @@
+.. _ioctl numbers:
+
 =============
 Ioctl Numbers
 =============

Re: Path troubles when using SPHINXDIRS option

[Markus Heiser]

Hi Mauro,

one more addition: in patch

  9fc3a18a94 docs: remove extra conf.py files

the conf.py from SPHINXDIRS has been removed since some latex
stuff was no longer needed.  IMO this is not a really good idea,
this patch also remove the individual project titles like:

   project = "Core-API Documentation"

and it also removes the:

  tags.add("subproject")

which is widly used, e.g.:

  .. only::  subproject and html

     Indices
     =======

     * :ref:`genindex`

Anyway: if we do so we can also remove such directives and we can
also remove the 'loadConfig' which was added in

  606b9ac81a doc-rst: generic way to build only sphinx sub-folders

Sorry, that I do not have time to prepare a well formed patch.
But I maintain my opinion that we'd better not do that.

  -- Markus --


Am 21.04.20 um 12:52 schrieb Markus Heiser:
> Hi Mauro,
> 
> Am 21.04.20 um 10:38 schrieb Mauro Carvalho Chehab:
>>
>> Hi,
>>
>> While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option
>> stopped working.
>>
>> What happens is that, if we don't pass this option, Sphinx assumes that
>> the "absolute" path is "Documentation/". So, include options like this:
>>
>> 	.. include:: /ABI/stable/firewire-cdev
>> 	    :literal:
>>
>>
>> would be evaluated as Documentation/ABI/stable/firewire-cdev if built
>> without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then
>> the "absolute" patch would be "Documentation/driver-api", causing this
>> error:
>>
>> 	Sphinx parallel build error:
>> 	docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:
>>
>> This is specially bad for build jobs (like one we have on jenkins that
>> is meant to test media patches), as, due to some Sphinx bug, prevents
>> sphinx-build to stop, making the toolset to run forever.
>>
>> I suspect that some change at conf.py could address the path issue.
>>
>> I'll try to investigate further.
>>
>> Just to be 100% sure that this is not some version-specific bad
>> behavior, I tested (using the latest patch version) Sphinx from
>> version 1.7.9 up to 2.4.4. The same two errors happen on all
>> versions.
>>
>> Markus,
>>
>> Maybe you may have some idea about how to fix those issues.
>>
>> The parallel build error would likely require fixing something
>> inside Sphinx code, making it abort if it gets a "SEVERE" error.
>>
>> Regards,
>> Mauro
>>
> 
> I fixed it with the patch shown below.  Some words about: The use
> of the :doc: rule should be replaced by a :ref:.  The path names
> of include directives should always relative to the .rst file.
> 
> To go further ... with patch below:
> 
>      $ make SPHINXDIRS="driver-api" htmldocs
> 
> shows some " WARNING: undefined label: ..." messages.  To close
> such links (addressing objects outside of the SPHINXDIR) we need
> to activate intersphinx [1].
> 
> If we activate intersphinx we have to wait one release cycle to
> get a objects.inv file at:
> 
>      https://www.kernel.org/doc/html/latest/
> 
> When the objects.inv file is available, the warnings are
> disappear.
> 
> If the process of intersphinx is unclear, please ask.
> 
> 
> -- Markus --
> 
> [1] https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html
> 
> ----
> 
> diff --git a/Documentation/driver-api/firewire.rst b/Documentation/driver-api/firewire.rst
> index 94a2d7f01d99..67741aeae5e8 100644
> --- a/Documentation/driver-api/firewire.rst
> +++ b/Documentation/driver-api/firewire.rst
> @@ -19,16 +19,16 @@ of kernel interfaces is available via exported symbols in `firewire-core` module
>   Firewire char device data structures
>   ====================================
>   
> -.. include:: /ABI/stable/firewire-cdev
> +.. include:: ../ABI/stable/firewire-cdev
>       :literal:
>   
> -.. kernel-doc:: include/uapi/linux/firewire-cdev.h
> +.. kernel-doc:: /include/uapi/linux/firewire-cdev.h
>       :internal:
>   
>   Firewire device probing and sysfs interfaces
>   ============================================
>   
> -.. include:: /ABI/stable/sysfs-bus-firewire
> +.. include:: ../ABI/stable/sysfs-bus-firewire
>       :literal:
>   
>   .. kernel-doc:: drivers/firewire/core-device.c
> diff --git a/Documentation/driver-api/gpio/using-gpio.rst b/Documentation/driver-api/gpio/using-gpio.rst
> index dda069444032..f2bd1e72af4e 100644
> --- a/Documentation/driver-api/gpio/using-gpio.rst
> +++ b/Documentation/driver-api/gpio/using-gpio.rst
> @@ -15,7 +15,7 @@ For any kind of mass produced system you want to support, such as servers,
>   laptops, phones, tablets, routers, and any consumer or office or business goods
>   using appropriate kernel drivers is paramount. Submit your code for inclusion
>   in the upstream Linux kernel when you feel it is mature enough and you will get
> -help to refine it, see :doc:`../../process/submitting-patches`.
> +help to refine it, see :ref:`submittingpatches`.
>   
>   In Linux GPIO lines also have a userspace ABI.
>   
> diff --git a/Documentation/driver-api/ioctl.rst b/Documentation/driver-api/ioctl.rst
> index c455db0e1627..ad91db68ba89 100644
> --- a/Documentation/driver-api/ioctl.rst
> +++ b/Documentation/driver-api/ioctl.rst
> @@ -34,7 +34,7 @@ _IO/_IOR/_IOW/_IOWR
>   
>   type
>      An 8-bit number, often a character literal, specific to a subsystem
> -   or driver, and listed in :doc:`../userspace-api/ioctl/ioctl-number`
> +   or driver, and listed in :ref:`ioctl numbers`
>   
>   nr
>     An 8-bit number identifying the specific command, unique for a give
> diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
> index 52bf58417653..09647516ca52 100644
> --- a/Documentation/userspace-api/ioctl/ioctl-number.rst
> +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
> @@ -1,3 +1,5 @@
> +.. _ioctl numbers:
> +
>   =============
>   Ioctl Numbers
>   =============
> 

Re: Path troubles when using SPHINXDIRS option

[Mauro Carvalho Chehab]

Em Tue, 21 Apr 2020 12:52:42 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Hi Mauro,
> 
> Am 21.04.20 um 10:38 schrieb Mauro Carvalho Chehab:
> > 
> > Hi,
> > 
> > While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option
> > stopped working.
> > 
> > What happens is that, if we don't pass this option, Sphinx assumes that
> > the "absolute" path is "Documentation/". So, include options like this:
> > 
> > 	.. include:: /ABI/stable/firewire-cdev
> > 	    :literal:
> > 
> > 
> > would be evaluated as Documentation/ABI/stable/firewire-cdev if built
> > without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then
> > the "absolute" patch would be "Documentation/driver-api", causing this
> > error:
> > 
> > 	Sphinx parallel build error:
> > 	docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:
> > 
> > This is specially bad for build jobs (like one we have on jenkins that
> > is meant to test media patches), as, due to some Sphinx bug, prevents
> > sphinx-build to stop, making the toolset to run forever.
> > 
> > I suspect that some change at conf.py could address the path issue.
> > 
> > I'll try to investigate further.
> > 
> > Just to be 100% sure that this is not some version-specific bad
> > behavior, I tested (using the latest patch version) Sphinx from
> > version 1.7.9 up to 2.4.4. The same two errors happen on all
> > versions.
> > 
> > Markus,
> > 
> > Maybe you may have some idea about how to fix those issues.
> > 
> > The parallel build error would likely require fixing something
> > inside Sphinx code, making it abort if it gets a "SEVERE" error.
> > 
> > Regards,
> > Mauro
> >   
> 
> I fixed it with the patch shown below.  Some words about: The use
> of the :doc: rule should be replaced by a :ref:.  The path names
> of include directives should always relative to the .rst file.

There are lots of place already using :doc: directive, on both
relative and absolute ways:

	$ git grep ':doc:'|wc -l
	382

Replacing all of them to :ref: will be painful. Also, IMHO, :doc: fits
better on most needs, as it makes easier for people working with
just plain texts to find to what file a ref points.

Btw, not sure why, but, when doing some conversions I got broken
references when using :ref:. So, I ended using :doc: instead.

> 
> To go further ... with patch below:
> 
>     $ make SPHINXDIRS="driver-api" htmldocs
> 
> shows some " WARNING: undefined label: ..." messages.  To close
> such links (addressing objects outside of the SPHINXDIR) we need
> to activate intersphinx [1].
> 
> If we activate intersphinx we have to wait one release cycle to
> get a objects.inv file at:
> 
>     https://www.kernel.org/doc/html/latest/
> 
> When the objects.inv file is available, the warnings are
> disappear.  

Yeah, interphinx would help a lot to solve broken references.

Not sure, however, if it is worth using it (at least for html).

I mean, IMHO, the big reason why someone would use SPHINXDIRS is to
speedup the process of testing if a documentation patch did the right
thing.

Anyway, it is up to Jon to decide about enabling intersphinx or not.

> 
> If the process of intersphinx is unclear, please ask.
> 
> 
> -- Markus --
> 
> [1] https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html
> 
> ----
> 
> diff --git a/Documentation/driver-api/firewire.rst b/Documentation/driver-api/firewire.rst
> index 94a2d7f01d99..67741aeae5e8 100644
> --- a/Documentation/driver-api/firewire.rst
> +++ b/Documentation/driver-api/firewire.rst
> @@ -19,16 +19,16 @@ of kernel interfaces is available via exported symbols in `firewire-core` module
>  Firewire char device data structures
>  ====================================
>  
> -.. include:: /ABI/stable/firewire-cdev
> +.. include:: ../ABI/stable/firewire-cdev
>      :literal:

Hmm... I remember I tried that. It worked now on my local machine.

Yet, the big issue here is that we can't ensure that developers
will always use relative paths. We need somehow convince Sphinx to
always use a fixed base path (/Documentation), even when building
part of a big document.

>  -.. kernel-doc:: include/uapi/linux/firewire-cdev.h
> +.. kernel-doc:: /include/uapi/linux/firewire-cdev.h
>      :internal:

This is not needed. Probably because both conf.py and Makefile
passes a kerneldoc source directory:

	$ grep kerneldoc_ Documentation/conf.py
	kerneldoc_bin = '../scripts/kernel-doc'
	kerneldoc_srctree = '..'
	$ grep kerneldoc_ Documentation/Makefile 
	KERNELDOC_CONF  = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)

>  
>  Firewire device probing and sysfs interfaces
>  ============================================
>  
> -.. include:: /ABI/stable/sysfs-bus-firewire
> +.. include:: ../ABI/stable/sysfs-bus-firewire
>      :literal:
>  
>  .. kernel-doc:: drivers/firewire/core-device.c
> diff --git a/Documentation/driver-api/gpio/using-gpio.rst b/Documentation/driver-api/gpio/using-gpio.rst
> index dda069444032..f2bd1e72af4e 100644
> --- a/Documentation/driver-api/gpio/using-gpio.rst
> +++ b/Documentation/driver-api/gpio/using-gpio.rst
> @@ -15,7 +15,7 @@ For any kind of mass produced system you want to support, such as servers,
>  laptops, phones, tablets, routers, and any consumer or office or business goods
>  using appropriate kernel drivers is paramount. Submit your code for inclusion
>  in the upstream Linux kernel when you feel it is mature enough and you will get
> -help to refine it, see :doc:`../../process/submitting-patches`.
> +help to refine it, see :ref:`submittingpatches`.
>  
>  In Linux GPIO lines also have a userspace ABI.
>  
> diff --git a/Documentation/driver-api/ioctl.rst b/Documentation/driver-api/ioctl.rst
> index c455db0e1627..ad91db68ba89 100644
> --- a/Documentation/driver-api/ioctl.rst
> +++ b/Documentation/driver-api/ioctl.rst
> @@ -34,7 +34,7 @@ _IO/_IOR/_IOW/_IOWR
>  
>  type
>     An 8-bit number, often a character literal, specific to a subsystem
> -   or driver, and listed in :doc:`../userspace-api/ioctl/ioctl-number`
> +   or driver, and listed in :ref:`ioctl numbers`
>  
>  nr
>    An 8-bit number identifying the specific command, unique for a give
> diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
> index 52bf58417653..09647516ca52 100644
> --- a/Documentation/userspace-api/ioctl/ioctl-number.rst
> +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
> @@ -1,3 +1,5 @@
> +.. _ioctl numbers:
> +
>  =============
>  Ioctl Numbers
>  =============
> 



Thanks,
Mauro

Re: Path troubles when using SPHINXDIRS option

[Markus Heiser]

Hi Mauro,

about wording: I will use the term "book" for a SPHINXDIR and "bookshelf"
for the whole kernel documentation .. 

Am 22.04.20 um 10:57 schrieb Mauro Carvalho Chehab:
> Em Tue, 21 Apr 2020 12:52:42 +0200
> Markus Heiser <markus.heiser@darmarit.de> escreveu:
> 
>> Hi Mauro,
>>
>> Am 21.04.20 um 10:38 schrieb Mauro Carvalho Chehab:
>>>
>>> Hi,
>>>
>>> While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option
>>> stopped working.
>>>
>>> What happens is that, if we don't pass this option, Sphinx assumes that
>>> the "absolute" path is "Documentation/". So, include options like this:
>>>
>>> 	.. include:: /ABI/stable/firewire-cdev
>>> 	    :literal:
>>>
>>>
>>> would be evaluated as Documentation/ABI/stable/firewire-cdev if built
>>> without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then
>>> the "absolute" patch would be "Documentation/driver-api", causing this
>>> error:
>>>
>>> 	Sphinx parallel build error:
>>> 	docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:
>>>
>>> This is specially bad for build jobs (like one we have on jenkins that
>>> is meant to test media patches), as, due to some Sphinx bug, prevents
>>> sphinx-build to stop, making the toolset to run forever.
>>>
>>> I suspect that some change at conf.py could address the path issue.
>>>
>>> I'll try to investigate further.
>>>
>>> Just to be 100% sure that this is not some version-specific bad
>>> behavior, I tested (using the latest patch version) Sphinx from
>>> version 1.7.9 up to 2.4.4. The same two errors happen on all
>>> versions.
>>>
>>> Markus,
>>>
>>> Maybe you may have some idea about how to fix those issues.
>>>
>>> The parallel build error would likely require fixing something
>>> inside Sphinx code, making it abort if it gets a "SEVERE" error.
>>>
>>> Regards,
>>> Mauro
>>>    
>>
>> I fixed it with the patch shown below.  Some words about: The use
>> of the :doc: rule should be replaced by a :ref:.  The path names
>> of include directives should always relative to the .rst file.
> 
> There are lots of place already using :doc: directive, on both
> relative and absolute ways:
> 
> 	$ git grep ':doc:'|wc -l
> 	382

Your grep is wrong (it also matches the :doc: from kernel-include), use:

    $ git grep ':doc:`'|wc -l
    83

> 
> Replacing all of them to :ref: will be painful. Also, IMHO, :doc: fits
> better on most needs, as it makes easier for people working with
> just plain texts to find to what file a ref points.

The opposite of pointing to a file is, that we can only point the
file not a content in the file and we lost the flexibility of
dynamical referencing by target's name: If the target is moved,
the refs has to be changed also.  Not used right now, but we also
lost the ability to link from one "book" to another "book" in
the "bookshelf" or even into other books available in the WEB
(this is what intersphinx can do).

  Its a decission what counts more.

> Btw, not sure why, but, when doing some conversions I got broken
> references when using :ref:. So, I ended using :doc: instead.

Can't believe, there must have been other reasons.

>>
>> To go further ... with patch below:
>>
>>      $ make SPHINXDIRS="driver-api" htmldocs
>>
>> shows some " WARNING: undefined label: ..." messages.  To close
>> such links (addressing objects outside of the SPHINXDIR) we need
>> to activate intersphinx [1].
>>
>> If we activate intersphinx we have to wait one release cycle to
>> get a objects.inv file at:
>>
>>      https://www.kernel.org/doc/html/latest/
>>
>> When the objects.inv file is available, the warnings are
>> disappear.
> 
> Yeah, interphinx would help a lot to solve broken references.
> 
> Not sure, however, if it is worth using it (at least for html).

Why should a http link does not work in a PDF file? .. I mean, its
just inserting HTTP links.

> I mean, IMHO, the big reason why someone would use SPHINXDIRS is to
> speedup the process of testing if a documentation patch did the right
> thing.

I can't speak for other, but I often prefer a book with links in over
a complete bookshelf holding in my hands ;)

> Anyway, it is up to Jon to decide about enabling intersphinx or not.


-- Markus --

Re: Path troubles when using SPHINXDIRS option

[Mauro Carvalho Chehab]

Em Wed, 22 Apr 2020 11:27:50 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Hi Mauro,
> 
> about wording: I will use the term "book" for a SPHINXDIR and "bookshelf"
> for the whole kernel documentation .. 
> 
> Am 22.04.20 um 10:57 schrieb Mauro Carvalho Chehab:
> > Em Tue, 21 Apr 2020 12:52:42 +0200
> > Markus Heiser <markus.heiser@darmarit.de> escreveu:
> >   
> >> Hi Mauro,
> >>
> >> Am 21.04.20 um 10:38 schrieb Mauro Carvalho Chehab:  
> >>>
> >>> Hi,
> >>>
> >>> While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option
> >>> stopped working.
> >>>
> >>> What happens is that, if we don't pass this option, Sphinx assumes that
> >>> the "absolute" path is "Documentation/". So, include options like this:
> >>>
> >>> 	.. include:: /ABI/stable/firewire-cdev
> >>> 	    :literal:
> >>>
> >>>
> >>> would be evaluated as Documentation/ABI/stable/firewire-cdev if built
> >>> without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then
> >>> the "absolute" patch would be "Documentation/driver-api", causing this
> >>> error:
> >>>
> >>> 	Sphinx parallel build error:
> >>> 	docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:
> >>>
> >>> This is specially bad for build jobs (like one we have on jenkins that
> >>> is meant to test media patches), as, due to some Sphinx bug, prevents
> >>> sphinx-build to stop, making the toolset to run forever.
> >>>
> >>> I suspect that some change at conf.py could address the path issue.
> >>>
> >>> I'll try to investigate further.
> >>>
> >>> Just to be 100% sure that this is not some version-specific bad
> >>> behavior, I tested (using the latest patch version) Sphinx from
> >>> version 1.7.9 up to 2.4.4. The same two errors happen on all
> >>> versions.
> >>>
> >>> Markus,
> >>>
> >>> Maybe you may have some idea about how to fix those issues.
> >>>
> >>> The parallel build error would likely require fixing something
> >>> inside Sphinx code, making it abort if it gets a "SEVERE" error.
> >>>
> >>> Regards,
> >>> Mauro
> >>>      
> >>
> >> I fixed it with the patch shown below.  Some words about: The use
> >> of the :doc: rule should be replaced by a :ref:.  The path names
> >> of include directives should always relative to the .rst file.  
> > 
> > There are lots of place already using :doc: directive, on both
> > relative and absolute ways:
> > 
> > 	$ git grep ':doc:'|wc -l
> > 	382  
> 
> Your grep is wrong (it also matches the :doc: from kernel-include), use:
> 
>     $ git grep ':doc:`'|wc -l
>     83

Ah, true.

> > Replacing all of them to :ref: will be painful. Also, IMHO, :doc: fits
> > better on most needs, as it makes easier for people working with
> > just plain texts to find to what file a ref points.  
> 
> The opposite of pointing to a file is, that we can only point the
> file not a content in the file and we lost the flexibility of
> dynamical referencing by target's name: If the target is moved,
> the refs has to be changed also. 

True, but we have a script that tracks those problems and even
auto-correct most of such broken references.

Right now, there are *lots* of references to a /Documentation.*
file, without any sphinx tags:


	$ git grep Documentation/ Documentation/|grep -v ':doc:`'|grep -v ABI|grep -v binding|wc -l
	1515

An extension like Documentation/sphinx/automarkup.py could
easily replace them by :doc:`Documentation/foo`, creating
links on all those places at no cost.

Adding explicit :ref: would require manual work, and I bet some
maintainers will reject adding extra markups on some files.

> Not used right now, but we also
> lost the ability to link from one "book" to another "book" in
> the "bookshelf" or even into other books available in the WEB
> (this is what intersphinx can do).

As currently we don't use intersphinx, I'm not too much concerned
about what it could provide in the future.

In any case, maybe one day intersphinx will have support for :doc: 
types added there as well. 

Without looking into the dirty details, it doesn't sound too hard to add 
support for it: intersphinx could simply create a :ref: type of reference 
for all files (like, __doc_driver_api__media__drivers__tuners). Then,
any place with a :doc:`/driver-api/media/drivers/tuner` would be
dynamically replaced by a :ref:`__doc_driver_api__media__drivers__tuners`
at runtime.

>   Its a decission what counts more.
> 
> > Btw, not sure why, but, when doing some conversions I got broken
> > references when using :ref:. So, I ended using :doc: instead.  
> 
> Can't believe, there must have been other reasons.

Maybe. I tried several times on some documents, but it caused sphinx
warnings about unknown cross references. Not sure why, and didn't
investigated it further.

> >> To go further ... with patch below:
> >>
> >>      $ make SPHINXDIRS="driver-api" htmldocs
> >>
> >> shows some " WARNING: undefined label: ..." messages.  To close
> >> such links (addressing objects outside of the SPHINXDIR) we need
> >> to activate intersphinx [1].
> >>
> >> If we activate intersphinx we have to wait one release cycle to
> >> get a objects.inv file at:
> >>
> >>      https://www.kernel.org/doc/html/latest/
> >>
> >> When the objects.inv file is available, the warnings are
> >> disappear.  
> > 
> > Yeah, interphinx would help a lot to solve broken references.
> > 
> > Not sure, however, if it is worth using it (at least for html).  
> 
> Why should a http link does not work in a PDF file? .. I mean, its
> just inserting HTTP links.

The thing with PDF is that not many developers (I know) are using it.

Maybe now that we have a separate directory with just the PDF books,
without the other temporary files people may start using it more.

> > I mean, IMHO, the big reason why someone would use SPHINXDIRS is to
> > speedup the process of testing if a documentation patch did the right
> > thing.  
> 
> I can't speak for other, but I often prefer a book with links in over
> a complete bookshelf holding in my hands ;)

Well, I also often prefer reading a PDF book, when it provides a good
comprehensive documentation. On most places, we're not there yet: there 
are too many gaps inside the documentation for it to be used as a real
book. I hope someday this will change.

Thanks,
Mauro

Re: Path troubles when using SPHINXDIRS option

[Markus Heiser]

Am 22.04.20 um 12:05 schrieb Mauro Carvalho Chehab:
[...]
>> The opposite of pointing to a file is, that we can only point the
>> file not a content in the file and we lost the flexibility of
>> dynamical referencing by target's name: If the target is moved,
>> the refs has to be changed also.
> 
> True, but we have a script that tracks those problems and even
> auto-correct most of such broken references.

[...]

> An extension like Documentation/sphinx/automarkup.py could
> easily replace them by :doc:`Documentation/foo`, creating
> links on all those places at no cost.

[...]

> Adding explicit :ref: would require manual work, and I bet some
> maintainers will reject adding extra markups on some files.

[...]

> In any case, maybe one day intersphinx will have support for :doc:
> types added there as well.
> 
> Without looking into the dirty details, it doesn't sound too hard to add
> support for it: intersphinx could simply create a :ref: type of reference
> for all files (like, __doc_driver_api__media__drivers__tuners). Then,
> any place with a :doc:`/driver-api/media/drivers/tuner` would be
> dynamically replaced by a :ref:`__doc_driver_api__media__drivers__tuners`
> at runtime.

Yes, hack over hack .. sometimes I feel back in the "DocBook century".

We now have a very confusing build process and I have to admit that
I am also to blame.

I also believe that everyone has good reasons.  For one solution or
for the other solution.  _Hack_ or _standard_, it is always a compromise
that the Kernel community has to find. 

  I cannot rate that.

-- Markus --