Re: Path troubles when using SPHINXDIRS option

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

Em Wed, 22 Apr 2020 11:27:50 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Hi Mauro,
> 
> about wording: I will use the term "book" for a SPHINXDIR and "bookshelf"
> for the whole kernel documentation .. 
> 
> Am 22.04.20 um 10:57 schrieb Mauro Carvalho Chehab:
> > Em Tue, 21 Apr 2020 12:52:42 +0200
> > Markus Heiser <markus.heiser@darmarit.de> escreveu:
> >   
> >> Hi Mauro,
> >>
> >> Am 21.04.20 um 10:38 schrieb Mauro Carvalho Chehab:  
> >>>
> >>> Hi,
> >>>
> >>> While building from v5.7-rc2 + media, I noticed that SPHINXDIRS option
> >>> stopped working.
> >>>
> >>> What happens is that, if we don't pass this option, Sphinx assumes that
> >>> the "absolute" path is "Documentation/". So, include options like this:
> >>>
> >>> 	.. include:: /ABI/stable/firewire-cdev
> >>> 	    :literal:
> >>>
> >>>
> >>> would be evaluated as Documentation/ABI/stable/firewire-cdev if built
> >>> without SPHINXDIRS. However, if we do SPHINXDIRS=driver-api, then
> >>> the "absolute" patch would be "Documentation/driver-api", causing this
> >>> error:
> >>>
> >>> 	Sphinx parallel build error:
> >>> 	docutils.utils.SystemMessage: /devel/v4l/patchwork/Documentation/driver-api/firewire.rst:22: (SEVERE/4) Problems with "include" directive path:
> >>>
> >>> This is specially bad for build jobs (like one we have on jenkins that
> >>> is meant to test media patches), as, due to some Sphinx bug, prevents
> >>> sphinx-build to stop, making the toolset to run forever.
> >>>
> >>> I suspect that some change at conf.py could address the path issue.
> >>>
> >>> I'll try to investigate further.
> >>>
> >>> Just to be 100% sure that this is not some version-specific bad
> >>> behavior, I tested (using the latest patch version) Sphinx from
> >>> version 1.7.9 up to 2.4.4. The same two errors happen on all
> >>> versions.
> >>>
> >>> Markus,
> >>>
> >>> Maybe you may have some idea about how to fix those issues.
> >>>
> >>> The parallel build error would likely require fixing something
> >>> inside Sphinx code, making it abort if it gets a "SEVERE" error.
> >>>
> >>> Regards,
> >>> Mauro
> >>>      
> >>
> >> I fixed it with the patch shown below.  Some words about: The use
> >> of the :doc: rule should be replaced by a :ref:.  The path names
> >> of include directives should always relative to the .rst file.  
> > 
> > There are lots of place already using :doc: directive, on both
> > relative and absolute ways:
> > 
> > 	$ git grep ':doc:'|wc -l
> > 	382  
> 
> Your grep is wrong (it also matches the :doc: from kernel-include), use:
> 
>     $ git grep ':doc:`'|wc -l
>     83

Ah, true.

> > Replacing all of them to :ref: will be painful. Also, IMHO, :doc: fits
> > better on most needs, as it makes easier for people working with
> > just plain texts to find to what file a ref points.  
> 
> The opposite of pointing to a file is, that we can only point the
> file not a content in the file and we lost the flexibility of
> dynamical referencing by target's name: If the target is moved,
> the refs has to be changed also. 

True, but we have a script that tracks those problems and even
auto-correct most of such broken references.

Right now, there are *lots* of references to a /Documentation.*
file, without any sphinx tags:


	$ git grep Documentation/ Documentation/|grep -v ':doc:`'|grep -v ABI|grep -v binding|wc -l
	1515

An extension like Documentation/sphinx/automarkup.py could
easily replace them by :doc:`Documentation/foo`, creating
links on all those places at no cost.

Adding explicit :ref: would require manual work, and I bet some
maintainers will reject adding extra markups on some files.

> Not used right now, but we also
> lost the ability to link from one "book" to another "book" in
> the "bookshelf" or even into other books available in the WEB
> (this is what intersphinx can do).

As currently we don't use intersphinx, I'm not too much concerned
about what it could provide in the future.

In any case, maybe one day intersphinx will have support for :doc: 
types added there as well. 

Without looking into the dirty details, it doesn't sound too hard to add 
support for it: intersphinx could simply create a :ref: type of reference 
for all files (like, __doc_driver_api__media__drivers__tuners). Then,
any place with a :doc:`/driver-api/media/drivers/tuner` would be
dynamically replaced by a :ref:`__doc_driver_api__media__drivers__tuners`
at runtime.

>   Its a decission what counts more.
> 
> > Btw, not sure why, but, when doing some conversions I got broken
> > references when using :ref:. So, I ended using :doc: instead.  
> 
> Can't believe, there must have been other reasons.

Maybe. I tried several times on some documents, but it caused sphinx
warnings about unknown cross references. Not sure why, and didn't
investigated it further.

> >> To go further ... with patch below:
> >>
> >>      $ make SPHINXDIRS="driver-api" htmldocs
> >>
> >> shows some " WARNING: undefined label: ..." messages.  To close
> >> such links (addressing objects outside of the SPHINXDIR) we need
> >> to activate intersphinx [1].
> >>
> >> If we activate intersphinx we have to wait one release cycle to
> >> get a objects.inv file at:
> >>
> >>      https://www.kernel.org/doc/html/latest/
> >>
> >> When the objects.inv file is available, the warnings are
> >> disappear.  
> > 
> > Yeah, interphinx would help a lot to solve broken references.
> > 
> > Not sure, however, if it is worth using it (at least for html).  
> 
> Why should a http link does not work in a PDF file? .. I mean, its
> just inserting HTTP links.

The thing with PDF is that not many developers (I know) are using it.

Maybe now that we have a separate directory with just the PDF books,
without the other temporary files people may start using it more.

> > I mean, IMHO, the big reason why someone would use SPHINXDIRS is to
> > speedup the process of testing if a documentation patch did the right
> > thing.  
> 
> I can't speak for other, but I often prefer a book with links in over
> a complete bookshelf holding in my hands ;)

Well, I also often prefer reading a PDF book, when it provides a good
comprehensive documentation. On most places, we're not there yet: there 
are too many gaps inside the documentation for it to be used as a real
book. I hope someday this will change.

Thanks,
Mauro