Building directly with sphinx

Building directly with sphinx

[Konstantin Ryabitsev]

Hi, all:

I was playing around with readthedocs.org recently and wanted to see if 
I could build kernel docs there. I cannot directly run "make htmldocs" 
there, and it proved to be quite tricky to make sphinx do the right 
thing without all the things that are being defined in the Makefile.

Is it possible at all, or am I wasting my time?

-K

Re: Building directly with sphinx

[Markus Heiser]

Am 20.05.20 um 23:53 schrieb Konstantin Ryabitsev:
> Hi, all:
> 
> I was playing around with readthedocs.org recently and wanted to see if
> I could build kernel docs there. I cannot directly run "make htmldocs"
> there, and it proved to be quite tricky to make sphinx do the right
> thing without all the things that are being defined in the Makefile.
> 
> Is it possible at all, or am I wasting my time?

It is wasting time ;) .. The Makefile targets do build intermedaiate
files using perl and other scripts, this will never work on RTD.

FWIW: in other projects I worked some time with RTD but at the end
I gave up: If you have e.g. auto generated content in your build
process which is not generated by the python developer-mainstream
tools, RTD gives you too little freedom to implement your more
or less complex build procedures.  And .. often I get the RTD-Oops
links from search engines ..  RTD is (my experience is a  while
ago; "was") not very comfortable to to rebind obsolete URLs to
new content.

Overall I think kernel.org does a good job .. since years, no need
for additional RTD confusions;

    https://www.kernel.org/doc/html/latest/

-- Markus --

> 
> -K
> 

Re: Building directly with sphinx

[Konstantin Ryabitsev]

On Thu, May 21, 2020 at 10:35:51AM +0200, Markus Heiser wrote:
> > 
> > I was playing around with readthedocs.org recently and wanted to see if
> > I could build kernel docs there. I cannot directly run "make htmldocs"
> > there, and it proved to be quite tricky to make sphinx do the right
> > thing without all the things that are being defined in the Makefile.
> > 
> > Is it possible at all, or am I wasting my time?
> 
> It is wasting time ;) .. The Makefile targets do build intermedaiate
> files using perl and other scripts, this will never work on RTD.

Okay, I'd suspected as much after poking around things a bit. Another 
possibility would be to have a separate repository where we commit the 
doc tree (after it's massaged into the "ready to be built with sphinx" 
condition). We could run this automatically on our end on each mainline 
release, but it's certainly not a necessity.

> FWIW: in other projects I worked some time with RTD but at the end
> I gave up: If you have e.g. auto generated content in your build
> process which is not generated by the python developer-mainstream
> tools, RTD gives you too little freedom to implement your more
> or less complex build procedures.  And .. often I get the RTD-Oops
> links from search engines ..  RTD is (my experience is a  while
> ago; "was") not very comfortable to to rebind obsolete URLs to
> new content.
> 
> Overall I think kernel.org does a good job .. since years, no need
> for additional RTD confusions;
> 
>    https://www.kernel.org/doc/html/latest/

Glad to hear it. My primary interest in getting it hooked up with RTD 
was just from the perspective of creating an external mirror for folks 
who are already using RTD. For now, I'm going to shelve this effort and 
just continue building docs straight from the tree on our end.

Thanks for your help!

-K

Re: Building directly with sphinx

[Jani Nikula]

On Thu, 21 May 2020, Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Thu, May 21, 2020 at 10:35:51AM +0200, Markus Heiser wrote:
>> > 
>> > I was playing around with readthedocs.org recently and wanted to see if
>> > I could build kernel docs there. I cannot directly run "make htmldocs"
>> > there, and it proved to be quite tricky to make sphinx do the right
>> > thing without all the things that are being defined in the Makefile.
>> > 
>> > Is it possible at all, or am I wasting my time?
>> 
>> It is wasting time ;) .. The Makefile targets do build intermedaiate
>> files using perl and other scripts, this will never work on RTD.
>
> Okay, I'd suspected as much after poking around things a bit. Another 
> possibility would be to have a separate repository where we commit the 
> doc tree (after it's massaged into the "ready to be built with sphinx" 
> condition). We could run this automatically on our end on each mainline 
> release, but it's certainly not a necessity.

FWIW, when I was developing the Sphinx documentation build for the
kernel, I carefully tried to ensure everything would build directly, and
consequently also at readthedocs.org. I kept testing it at RTD too.

For this reason I kept resisting the addition of any Makefile trickery,
and insisting we should do everything through Sphinx. This would have
meant writing Sphinx extensions for all the hacks people wanted to
add. But it would also have lead to a self-contained system with fewer
moving parts, which I thought was worth pursuing. And, obviously, would
have made it possible to host the documentation at RTD.

I failed. Stuff just keeps piling up in the Makefiles.

In fairness, the kernel documentation build was growing too big even for
RTD. It kept timing out and/or consuming too much resources. I believe
that would have been possible to sort out with the RTD maintainers,
given the possibility to host kernel documentation, but there was no
point in pursuing this since the docs wouldn't build directly anyway.

BR,
Jani.


>> FWIW: in other projects I worked some time with RTD but at the end
>> I gave up: If you have e.g. auto generated content in your build
>> process which is not generated by the python developer-mainstream
>> tools, RTD gives you too little freedom to implement your more
>> or less complex build procedures.  And .. often I get the RTD-Oops
>> links from search engines ..  RTD is (my experience is a  while
>> ago; "was") not very comfortable to to rebind obsolete URLs to
>> new content.
>> 
>> Overall I think kernel.org does a good job .. since years, no need
>> for additional RTD confusions;
>> 
>>    https://www.kernel.org/doc/html/latest/
>
> Glad to hear it. My primary interest in getting it hooked up with RTD 
> was just from the perspective of creating an external mirror for folks 
> who are already using RTD. For now, I'm going to shelve this effort and 
> just continue building docs straight from the tree on our end.
>
> Thanks for your help!
>
> -K

-- 
Jani Nikula, Intel Open Source Graphics Center

Re: Building directly with sphinx

[Konstantin Ryabitsev]

On Mon, May 25, 2020 at 03:08:40PM +0300, Jani Nikula wrote:
> >> It is wasting time ;) .. The Makefile targets do build 
> >> intermedaiate
> >> files using perl and other scripts, this will never work on RTD.
> >
> > Okay, I'd suspected as much after poking around things a bit. Another 
> > possibility would be to have a separate repository where we commit the 
> > doc tree (after it's massaged into the "ready to be built with sphinx" 
> > condition). We could run this automatically on our end on each mainline 
> > release, but it's certainly not a necessity.
> 
> FWIW, when I was developing the Sphinx documentation build for the
> kernel, I carefully tried to ensure everything would build directly, and
> consequently also at readthedocs.org. I kept testing it at RTD too.
> 
> For this reason I kept resisting the addition of any Makefile trickery,
> and insisting we should do everything through Sphinx. This would have
> meant writing Sphinx extensions for all the hacks people wanted to
> add. But it would also have lead to a self-contained system with fewer
> moving parts, which I thought was worth pursuing. And, obviously, would
> have made it possible to host the documentation at RTD.
> 
> I failed. Stuff just keeps piling up in the Makefiles.

Well, if someone gives me a way to create a sphinx-buildable tree out of 
the contents of Documentation/*, I would be happy to maintain it and 
host on RTD. I think there is benefit to have a repository of just docs, 
even if it is secondary to the main repo and is auto-committed there on 
each mainline release.

> In fairness, the kernel documentation build was growing too big even for
> RTD. It kept timing out and/or consuming too much resources. I believe
> that would have been possible to sort out with the RTD maintainers,
> given the possibility to host kernel documentation, but there was no
> point in pursuing this since the docs wouldn't build directly anyway.

I've already negotiated with RTD.org to significantly extend the build 
time, so that's not a concern.

-K