[PATCH RFC] docs: experimental: build PDF with rst2pdf

[PATCH RFC] docs: experimental: build PDF with rst2pdf

[Mauro Carvalho Chehab]

Add an experimental PDF builder using rst2pdf

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/Makefile                     |  5 +++++
 Documentation/conf.py                      | 21 +++++++++++++++------
 Documentation/userspace-api/media/Makefile |  1 +
 Makefile                                   |  4 ++--
 4 files changed, 23 insertions(+), 8 deletions(-)

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 61a7310b49e0..c3c8fb10f94e 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -115,6 +115,10 @@ pdfdocs: latexdocs
 
 endif # HAVE_PDFLATEX
 
+rst2pdf:
+	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,pdf,$(var),pdf,$(var)))
+
 epubdocs:
 	@$(srctree)/scripts/sphinx-pre-install --version-check
 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
@@ -140,6 +144,7 @@ dochelp:
 	@echo  '  htmldocs        - HTML'
 	@echo  '  latexdocs       - LaTeX'
 	@echo  '  pdfdocs         - PDF'
+	@echo  '  rst2pdf         - PDF, using experimental rst2pdf support'
 	@echo  '  epubdocs        - EPUB'
 	@echo  '  xmldocs         - XML'
 	@echo  '  linkcheckdocs   - check for broken external links'
diff --git a/Documentation/conf.py b/Documentation/conf.py
index 66e121df59cd..6f2788aac81e 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -123,6 +123,12 @@ if (major == 1 and minor > 3) or (major > 1):
 else:
     extensions.append("sphinx.ext.pngmath")
 
+# Enable experimental rst2pdf, if available
+try:
+    extensions.append("rst2pdf.pdfbuilder")
+except:
+    sys.stderr.write('rst2pdf extension not available.\n')
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
@@ -614,12 +620,15 @@ epub_exclude_files = ['search.html']
 #
 # See the Sphinx chapter of https://ralsina.me/static/manual.pdf
 #
-# FIXME: Do not add the index file here; the result will be too big. Adding
-# multiple PDF files here actually tries to get the cross-referencing right
-# *between* PDF files.
-pdf_documents = [
-    ('kernel-documentation', u'Kernel', u'Kernel', u'J. Random Bozo'),
-]
+
+# Add all LaTeX files to PDF documents as well
+pdf_documents = []
+for l in latex_documents:
+    doc = l[0]
+    fn = l[1].replace(".tex", "")
+    name = l[2]
+    authors = l[3]
+    pdf_documents.append((doc, fn, name, authors))
 
 # kernel-doc extension configuration for running Sphinx directly (e.g. by Read
 # the Docs). In a normal build, these are supplied from the Makefile via command
diff --git a/Documentation/userspace-api/media/Makefile b/Documentation/userspace-api/media/Makefile
index 81a4a1a53bce..8c6b3ac4ecb0 100644
--- a/Documentation/userspace-api/media/Makefile
+++ b/Documentation/userspace-api/media/Makefile
@@ -59,6 +59,7 @@ all: $(IMGDOT) $(BUILDDIR) ${TARGETS}
 html: all
 epub: all
 xml: all
+pdf: all
 latex: $(IMGPDF) all
 linkcheck:
 
diff --git a/Makefile b/Makefile
index 43ecedeb3f02..db4043578eec 100644
--- a/Makefile
+++ b/Makefile
@@ -264,7 +264,7 @@ no-dot-config-targets := $(clean-targets) \
 			 cscope gtags TAGS tags help% %docs check% coccicheck \
 			 $(version_h) headers headers_% archheaders archscripts \
 			 %asm-generic kernelversion %src-pkg dt_binding_check \
-			 outputmakefile
+			 outputmakefile rst2pdf
 no-sync-config-targets := $(no-dot-config-targets) %install kernelrelease
 single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.s %.symtypes %/
 
@@ -1654,7 +1654,7 @@ $(help-board-dirs): help-%:
 
 # Documentation targets
 # ---------------------------------------------------------------------------
-DOC_TARGETS := xmldocs latexdocs pdfdocs htmldocs epubdocs cleandocs \
+DOC_TARGETS := xmldocs latexdocs pdfdocs rst2pdf htmldocs epubdocs cleandocs \
 	       linkcheckdocs dochelp refcheckdocs
 PHONY += $(DOC_TARGETS)
 $(DOC_TARGETS):
-- 
2.29.2

Re: [PATCH RFC] docs: experimental: build PDF with rst2pdf

[Mauro Carvalho Chehab]

Em Thu, 10 Dec 2020 17:01:19 +0100
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> Add an experimental PDF builder using rst2pdf
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

I opened an issue at:

https://github.com/rst2pdf/rst2pdf/issues/958

Let's hope someone at rst2pdf could help fixing this ;-)

Regards,
Mauro


> ---
>  Documentation/Makefile                     |  5 +++++
>  Documentation/conf.py                      | 21 +++++++++++++++------
>  Documentation/userspace-api/media/Makefile |  1 +
>  Makefile                                   |  4 ++--
>  4 files changed, 23 insertions(+), 8 deletions(-)
> 
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index 61a7310b49e0..c3c8fb10f94e 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -115,6 +115,10 @@ pdfdocs: latexdocs
>  
>  endif # HAVE_PDFLATEX
>  
> +rst2pdf:
> +	@$(srctree)/scripts/sphinx-pre-install --version-check
> +	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,pdf,$(var),pdf,$(var)))
> +
>  epubdocs:
>  	@$(srctree)/scripts/sphinx-pre-install --version-check
>  	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
> @@ -140,6 +144,7 @@ dochelp:
>  	@echo  '  htmldocs        - HTML'
>  	@echo  '  latexdocs       - LaTeX'
>  	@echo  '  pdfdocs         - PDF'
> +	@echo  '  rst2pdf         - PDF, using experimental rst2pdf support'
>  	@echo  '  epubdocs        - EPUB'
>  	@echo  '  xmldocs         - XML'
>  	@echo  '  linkcheckdocs   - check for broken external links'
> diff --git a/Documentation/conf.py b/Documentation/conf.py
> index 66e121df59cd..6f2788aac81e 100644
> --- a/Documentation/conf.py
> +++ b/Documentation/conf.py
> @@ -123,6 +123,12 @@ if (major == 1 and minor > 3) or (major > 1):
>  else:
>      extensions.append("sphinx.ext.pngmath")
>  
> +# Enable experimental rst2pdf, if available
> +try:
> +    extensions.append("rst2pdf.pdfbuilder")
> +except:
> +    sys.stderr.write('rst2pdf extension not available.\n')
> +
>  # Add any paths that contain templates here, relative to this directory.
>  templates_path = ['_templates']
>  
> @@ -614,12 +620,15 @@ epub_exclude_files = ['search.html']
>  #
>  # See the Sphinx chapter of https://ralsina.me/static/manual.pdf
>  #
> -# FIXME: Do not add the index file here; the result will be too big. Adding
> -# multiple PDF files here actually tries to get the cross-referencing right
> -# *between* PDF files.
> -pdf_documents = [
> -    ('kernel-documentation', u'Kernel', u'Kernel', u'J. Random Bozo'),
> -]
> +
> +# Add all LaTeX files to PDF documents as well
> +pdf_documents = []
> +for l in latex_documents:
> +    doc = l[0]
> +    fn = l[1].replace(".tex", "")
> +    name = l[2]
> +    authors = l[3]
> +    pdf_documents.append((doc, fn, name, authors))
>  
>  # kernel-doc extension configuration for running Sphinx directly (e.g. by Read
>  # the Docs). In a normal build, these are supplied from the Makefile via command
> diff --git a/Documentation/userspace-api/media/Makefile b/Documentation/userspace-api/media/Makefile
> index 81a4a1a53bce..8c6b3ac4ecb0 100644
> --- a/Documentation/userspace-api/media/Makefile
> +++ b/Documentation/userspace-api/media/Makefile
> @@ -59,6 +59,7 @@ all: $(IMGDOT) $(BUILDDIR) ${TARGETS}
>  html: all
>  epub: all
>  xml: all
> +pdf: all
>  latex: $(IMGPDF) all
>  linkcheck:
>  
> diff --git a/Makefile b/Makefile
> index 43ecedeb3f02..db4043578eec 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -264,7 +264,7 @@ no-dot-config-targets := $(clean-targets) \
>  			 cscope gtags TAGS tags help% %docs check% coccicheck \
>  			 $(version_h) headers headers_% archheaders archscripts \
>  			 %asm-generic kernelversion %src-pkg dt_binding_check \
> -			 outputmakefile
> +			 outputmakefile rst2pdf
>  no-sync-config-targets := $(no-dot-config-targets) %install kernelrelease
>  single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.s %.symtypes %/
>  
> @@ -1654,7 +1654,7 @@ $(help-board-dirs): help-%:
>  
>  # Documentation targets
>  # ---------------------------------------------------------------------------
> -DOC_TARGETS := xmldocs latexdocs pdfdocs htmldocs epubdocs cleandocs \
> +DOC_TARGETS := xmldocs latexdocs pdfdocs rst2pdf htmldocs epubdocs cleandocs \
>  	       linkcheckdocs dochelp refcheckdocs
>  PHONY += $(DOC_TARGETS)
>  $(DOC_TARGETS):



Thanks,
Mauro

[PATCH RFC v2] docs: experimental: build PDF with rst2pdf

[Mauro Carvalho Chehab]

Add an experimental PDF builder using rst2pdf

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---

Please notice that 18 documents (of a total of 71) won't build with 
rst2pdf. There's an opened issue about that at:

    https://github.com/rst2pdf/rst2pdf/issues/958

v2: usage of SPHINXDIRS was fixed.


 Documentation/Makefile                     |  5 +++++
 Documentation/conf.py                      | 21 +++++++++++++++------
 Documentation/sphinx/load_config.py        | 12 ++++++++++++
 Documentation/userspace-api/media/Makefile |  1 +
 Makefile                                   |  4 ++--
 5 files changed, 35 insertions(+), 8 deletions(-)

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 61a7310b49e0..c3c8fb10f94e 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -115,6 +115,10 @@ pdfdocs: latexdocs
 
 endif # HAVE_PDFLATEX
 
+rst2pdf:
+	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,pdf,$(var),pdf,$(var)))
+
 epubdocs:
 	@$(srctree)/scripts/sphinx-pre-install --version-check
 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
@@ -140,6 +144,7 @@ dochelp:
 	@echo  '  htmldocs        - HTML'
 	@echo  '  latexdocs       - LaTeX'
 	@echo  '  pdfdocs         - PDF'
+	@echo  '  rst2pdf         - PDF, using experimental rst2pdf support'
 	@echo  '  epubdocs        - EPUB'
 	@echo  '  xmldocs         - XML'
 	@echo  '  linkcheckdocs   - check for broken external links'
diff --git a/Documentation/conf.py b/Documentation/conf.py
index 66e121df59cd..6f2788aac81e 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -123,6 +123,12 @@ if (major == 1 and minor > 3) or (major > 1):
 else:
     extensions.append("sphinx.ext.pngmath")
 
+# Enable experimental rst2pdf, if available
+try:
+    extensions.append("rst2pdf.pdfbuilder")
+except:
+    sys.stderr.write('rst2pdf extension not available.\n')
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
@@ -614,12 +620,15 @@ epub_exclude_files = ['search.html']
 #
 # See the Sphinx chapter of https://ralsina.me/static/manual.pdf
 #
-# FIXME: Do not add the index file here; the result will be too big. Adding
-# multiple PDF files here actually tries to get the cross-referencing right
-# *between* PDF files.
-pdf_documents = [
-    ('kernel-documentation', u'Kernel', u'Kernel', u'J. Random Bozo'),
-]
+
+# Add all LaTeX files to PDF documents as well
+pdf_documents = []
+for l in latex_documents:
+    doc = l[0]
+    fn = l[1].replace(".tex", "")
+    name = l[2]
+    authors = l[3]
+    pdf_documents.append((doc, fn, name, authors))
 
 # kernel-doc extension configuration for running Sphinx directly (e.g. by Read
 # the Docs). In a normal build, these are supplied from the Makefile via command
diff --git a/Documentation/sphinx/load_config.py b/Documentation/sphinx/load_config.py
index eeb394b39e2c..8266afd438aa 100644
--- a/Documentation/sphinx/load_config.py
+++ b/Documentation/sphinx/load_config.py
@@ -43,6 +43,18 @@ def loadConfig(namespace):
 
             namespace['latex_documents'] = new_latex_docs
 
+            new_pdf_docs = []
+            pdf_documents = namespace['pdf_documents']
+
+            for l in pdf_documents:
+                if l[0].find(dir + '/') == 0:
+                    has = True
+                    fn = l[0][len(dir) + 1:]
+                    new_pdf_docs.append((fn, l[1], l[2], l[3]))
+                    break
+
+            namespace['pdf_documents'] = new_pdf_docs
+
         # If there is an extra conf.py file, load it
         if os.path.isfile(config_file):
             sys.stdout.write("load additional sphinx-config: %s\n" % config_file)
diff --git a/Documentation/userspace-api/media/Makefile b/Documentation/userspace-api/media/Makefile
index 81a4a1a53bce..8c6b3ac4ecb0 100644
--- a/Documentation/userspace-api/media/Makefile
+++ b/Documentation/userspace-api/media/Makefile
@@ -59,6 +59,7 @@ all: $(IMGDOT) $(BUILDDIR) ${TARGETS}
 html: all
 epub: all
 xml: all
+pdf: all
 latex: $(IMGPDF) all
 linkcheck:
 
diff --git a/Makefile b/Makefile
index 43ecedeb3f02..db4043578eec 100644
--- a/Makefile
+++ b/Makefile
@@ -264,7 +264,7 @@ no-dot-config-targets := $(clean-targets) \
 			 cscope gtags TAGS tags help% %docs check% coccicheck \
 			 $(version_h) headers headers_% archheaders archscripts \
 			 %asm-generic kernelversion %src-pkg dt_binding_check \
-			 outputmakefile
+			 outputmakefile rst2pdf
 no-sync-config-targets := $(no-dot-config-targets) %install kernelrelease
 single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.s %.symtypes %/
 
@@ -1654,7 +1654,7 @@ $(help-board-dirs): help-%:
 
 # Documentation targets
 # ---------------------------------------------------------------------------
-DOC_TARGETS := xmldocs latexdocs pdfdocs htmldocs epubdocs cleandocs \
+DOC_TARGETS := xmldocs latexdocs pdfdocs rst2pdf htmldocs epubdocs cleandocs \
 	       linkcheckdocs dochelp refcheckdocs
 PHONY += $(DOC_TARGETS)
 $(DOC_TARGETS):
-- 
2.29.2

Re: [PATCH RFC v2] docs: experimental: build PDF with rst2pdf

[Jonathan Corbet]

On Fri, 11 Dec 2020 09:33:32 +0100
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> Add an experimental PDF builder using rst2pdf
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
> 
> Please notice that 18 documents (of a total of 71) won't build with 
> rst2pdf. There's an opened issue about that at:
> 
>     https://github.com/rst2pdf/rst2pdf/issues/958
> 
> v2: usage of SPHINXDIRS was fixed.
> 
> 
>  Documentation/Makefile                     |  5 +++++
>  Documentation/conf.py                      | 21 +++++++++++++++------
>  Documentation/sphinx/load_config.py        | 12 ++++++++++++
>  Documentation/userspace-api/media/Makefile |  1 +
>  Makefile                                   |  4 ++--
>  5 files changed, 35 insertions(+), 8 deletions(-)

So I would dearly love to have rst2pdf working.

I applied this, then tried to see what would happen if I ran a build
without having rst2pdf installed:

> 1108 meer kernel: make htmldocs
>   SPHINX  htmldocs --> file:///stuff/k/git/kernel/Documentation/output
> make[2]: Nothing to be done for 'html'.
> WARNING: The kernel documentation build process
>         support for Sphinx v3.0 and above is brand new. Be prepared for
>         possible issues in the generated output.
>         enabling CJK for LaTeX builder
> 
> Extension error:
> Could not import extension rst2pdf.pdfbuilder (exception: No module named 'rst2pdf')
> make[1]: *** [Documentation/Makefile:91: htmldocs] Error 2
> make: *** [Makefile:1663: htmldocs] Error 2

Methinks it's perhaps not quite ready for linux-next yet :)

With rst2pdf installed I get a bunch of zero-length files, as promised.
Pretty much none of the larger "books" make it through.  It's a start,
though.  I'll happily apply this as a step forward once it doesn't break
the docs build if rst2pdf is missing.

Thanks,

jon

Re: [PATCH RFC v2] docs: experimental: build PDF with rst2pdf

[Mauro Carvalho Chehab]

Em Fri, 11 Dec 2020 13:48:59 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> On Fri, 11 Dec 2020 09:33:32 +0100
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> 
> > Add an experimental PDF builder using rst2pdf
> > 
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> > 
> > Please notice that 18 documents (of a total of 71) won't build with 
> > rst2pdf. There's an opened issue about that at:
> > 
> >     https://github.com/rst2pdf/rst2pdf/issues/958
> > 
> > v2: usage of SPHINXDIRS was fixed.
> > 
> > 
> >  Documentation/Makefile                     |  5 +++++
> >  Documentation/conf.py                      | 21 +++++++++++++++------
> >  Documentation/sphinx/load_config.py        | 12 ++++++++++++
> >  Documentation/userspace-api/media/Makefile |  1 +
> >  Makefile                                   |  4 ++--
> >  5 files changed, 35 insertions(+), 8 deletions(-)  
> 
> So I would dearly love to have rst2pdf working.
> 
> I applied this, then tried to see what would happen if I ran a build
> without having rst2pdf installed:
> 
> > 1108 meer kernel: make htmldocs
> >   SPHINX  htmldocs --> file:///stuff/k/git/kernel/Documentation/output
> > make[2]: Nothing to be done for 'html'.
> > WARNING: The kernel documentation build process
> >         support for Sphinx v3.0 and above is brand new. Be prepared for
> >         possible issues in the generated output.
> >         enabling CJK for LaTeX builder
> > 
> > Extension error:
> > Could not import extension rst2pdf.pdfbuilder (exception: No module named 'rst2pdf')
> > make[1]: *** [Documentation/Makefile:91: htmldocs] Error 2
> > make: *** [Makefile:1663: htmldocs] Error 2  
> 
> Methinks it's perhaps not quite ready for linux-next yet :)

Well, I haven't test this.

I'm not an usual python programmer, so, don't know much about its 
specifics... Yet, I would be expecting that something like this:

	try:
	    extensions.append("rst2pdf.pdfbuilder")
	except:
	    sys.stderr.write('rst2pdf extension not available.\n')
	

Would avoid it to crash, if the extension is not available.
Silly me :-)

Still, I suspect that it should not be hard to modify the above to
avoid the crash. 

I shouldn't be doing much development those days, as I'm taking
some vacations, after sending media stuff for 5.11. 

So, if you have a better idea about how to optionally probe an
extension, feel free to modify my patch.


> With rst2pdf installed I get a bunch of zero-length files, as promised.
> Pretty much none of the larger "books" make it through.  

Yeah. I guess one of the issues is with tables that don't fit into
a single page.

Yet, devicetree book is empty. That sounds really weird, as there are
few files on it, and I didn't see anything uncommon on the rst files.

> It's a start,
> though.  I'll happily apply this as a step forward once it doesn't break
> the docs build if rst2pdf is missing.

Sounds like a plan.

Thanks,
Mauro

Re: [PATCH RFC v2] docs: experimental: build PDF with rst2pdf

[Jonathan Corbet]

On Sat, 12 Dec 2020 00:54:35 +0100
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> I'm not an usual python programmer, so, don't know much about its 
> specifics... Yet, I would be expecting that something like this:
> 
> 	try:
> 	    extensions.append("rst2pdf.pdfbuilder")
> 	except:
> 	    sys.stderr.write('rst2pdf extension not available.\n')
> 	
> 
> Would avoid it to crash, if the extension is not available.
> Silly me :-)

No, that's not going to do it, for a couple of reasons.  First being that
all it's doing is appending a string to a list, which pretty much always
succeeds.  The attempt to actually import the module happens later.

...and you won't catch that either because it isn't actually throwing an
exception, it's just noting the problem and giving up.

The right solution is probably something like this:

	try:
	    import rst2pdf
	    extensions.append('rst2pdf.pdfbuilder')
	except ModuleNotFoundError:
	    pass # no rst2pdf for you

This is totally untested, of course.

[Incidentally, a blank "except:" clause like the one you had is, in my
experience, a bad idea.  That will catch *anything*, leading to hiding all
kinds of bugs.  Not that I've ever committed such a faux pas and suffered
the consequences myself...no...never...honest...]

I'll mess with this a bit more later.

Thanks,

jon