Making pdfdocs with sphinx - only select rst targets

Making pdfdocs with sphinx - only select rst targets

[Luis R. Rodriguez]

I'm excited to see the new documentation format, so I'm changing my
documentation in the pending patches I have to use it. I however
cannot generate anything other than the main

Documentation/output/pdf/Kernel.pdf

How can I see in PDF the other documentation?

I'm using:

make DOCBOOKS="" pdfdocs

The Documentation/output/pdf/Kernel.pdf only has a bit of the
documentation on how to write docs, nothing else. Using htmldocs as a
target works but I am not a fan of that output.

  Luis

Re: Making pdfdocs with sphinx - only select rst targets

[Mauro Carvalho Chehab]

Em Tue, 9 Aug 2016 18:19:14 -0700
"Luis R. Rodriguez" <mcgrof@kernel.org> escreveu:

> I'm excited to see the new documentation format, so I'm changing my
> documentation in the pending patches I have to use it. I however
> cannot generate anything other than the main
> 
> Documentation/output/pdf/Kernel.pdf
> 
> How can I see in PDF the other documentation?
> 
> I'm using:
> 
> make DOCBOOKS="" pdfdocs
> 
> The Documentation/output/pdf/Kernel.pdf only has a bit of the
> documentation on how to write docs, nothing else. Using htmldocs as a
> target works but I am not a fan of that output.

Hi Luis,

Unfortunately, on my tests with Sphinx, I noticed that pdf output is
not actually working fine. It relies on a tool called rst2pdf. On
my tests, I noticed that:

- rst2pdf is completely broken with Sphinx 1.3.x;
- rst2pdf doesn't handle complex documents, even with Sphinx 1.4.5;
- rst2pdf always report a successful build, even when it fails;
- I also consulted a developer with years of experience with Sphinx: he
  recommended removing the media book from the pdf build, as he doesn't
  know any fixup or replacement for the tool that would work for the
  media docs.

If you want to play with PDF output, you should take a look on this patch:
	https://patchwork.kernel.org/patch/9231687/

(commit 520a247760f750307b53db905a10a17df1700f3b)

Reverting it or adding your document main rst file there would let you play
with pdfdocs, if you're using Sphinx version 1.4.x.

If you revert it, you'll also need the enclosed patch.

Regards,


Cheers,
Mauro

---

doc-rst: build the dynamic rst files for non-html doc targets

Right now, the media makefile is called only for html docs.
Call it also for the other documentation targets.

NOTICE: Should not be applied upstream!

The pdf target won't actually build a PDF, because rst2pdf can't
handle complex documents and will fail with the media one.

Signed-off-by: Mauro Carvalho Chehab <mchehab@osg.samsung.com>

diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index 857f1e273418..e90167ba474d 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -49,13 +49,16 @@ ifeq ($(HAVE_RST2PDF),0)
 	$(warning The Python 'rst2pdf' module was not found. Make sure you have the module installed to produce PDF output.)
 	@echo "  SKIP    Sphinx $@ target."
 else # HAVE_RST2PDF
+	$(MAKE) BUILDDIR=$(BUILDDIR) -f $(srctree)/Documentation/media/Makefile $@
 	$(call cmd,sphinx,pdf)
 endif # HAVE_RST2PDF
 
 epubdocs:
+	$(MAKE) BUILDDIR=$(BUILDDIR) -f $(srctree)/Documentation/media/Makefile $@
 	$(call cmd,sphinx,epub)
 
 xmldocs:
+	$(MAKE) BUILDDIR=$(BUILDDIR) -f $(srctree)/Documentation/media/Makefile $@
 	$(call cmd,sphinx,xml)
 
 # no-ops for the Sphinx toolchain
diff --git a/Documentation/media/Makefile b/Documentation/media/Makefile
index 39e2d766dbe3..79784e848fc0 100644
--- a/Documentation/media/Makefile
+++ b/Documentation/media/Makefile
@@ -11,6 +11,9 @@ FILES = audio.h.rst ca.h.rst dmx.h.rst frontend.h.rst net.h.rst video.h.rst \
 TARGETS := $(addprefix $(BUILDDIR)/, $(FILES))
 
 htmldocs: $(BUILDDIR) ${TARGETS}
+epubdocs: $(BUILDDIR) ${TARGETS}
+xmldocs: $(BUILDDIR) ${TARGETS}
+pdfdocs: $(BUILDDIR) ${TARGETS}
 
 $(BUILDDIR):
 	$(Q)mkdir -p $@

Re: Making pdfdocs with sphinx - only select rst targets

[Mauro Carvalho Chehab]

Em Wed, 10 Aug 2016 07:42:42 -0700
"Luis R. Rodriguez" <mcgrof@kernel.org> escreveu:

> On Aug 10, 2016 1:41 AM, "Mauro Carvalho Chehab" <mchehab@infradead.org>
> wrote:
> >
> > Em Tue, 9 Aug 2016 18:19:14 -0700
> > "Luis R. Rodriguez" <mcgrof@kernel.org> escreveu:
> >  
> > > I'm excited to see the new documentation format, so I'm changing my
> > > documentation in the pending patches I have to use it. I however
> > > cannot generate anything other than the main
> > >
> > > Documentation/output/pdf/Kernel.pdf
> > >
> > > How can I see in PDF the other documentation?
> > >
> > > I'm using:
> > >
> > > make DOCBOOKS="" pdfdocs
> > >
> > > The Documentation/output/pdf/Kernel.pdf only has a bit of the
> > > documentation on how to write docs, nothing else. Using htmldocs as a
> > > target works but I am not a fan of that output.  
> >
> > Hi Luis,
> >
> > Unfortunately, on my tests with Sphinx, I noticed that pdf output is
> > not actually working fine. It relies on a tool called rst2pdf. On
> > my tests, I noticed that:
> >
> > - rst2pdf is completely broken with Sphinx 1.3.x;
> > - rst2pdf doesn't handle complex documents, even with Sphinx 1.4.5;
> > - rst2pdf always report a successful build, even when it fails;
> > - I also consulted a developer with years of experience with Sphinx: he
> >   recommended removing the media book from the pdf build, as he doesn't
> >   know any fixup or replacement for the tool that would work for the
> >   media docs.  
> 
> Hmm I see. What was the limitting issue ? The fact that media had embedded
> PDFs?

I guess not... I tried to debug rst2pdf, but it is very hard to identify
the problem, as it doesn't point where the error is. I remember I changed
some things to fix rst2pdf parser, but it still complains that some things
were missing.

> >
> > If you want to play with PDF output, you should take a look on this patch:
> >         https://patchwork.kernel.org/patch/9231687/
> >
> > (commit 520a247760f750307b53db905a10a17df1700f3b)
> >
> > Reverting it or adding your document main rst file there would let you  
> play
> > with pdfdocs, if you're using Sphinx version 1.4.x.
> >
> > If you revert it, you'll also need the enclosed patch.  
> 
> Ah OK thanks. So if our documentation works with this and PDF output we can
> add just that single doc (in my case a new folder), is that right ?

Not sure. Better ask Jon how he wants to proceed with that.

Regards,
Mauro

[RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

[Jani Nikula]

Hi Luis, all -

With these you should be able to get started with pdf generation. It's a
quick transition to pdflatex, the patches are not very pretty, but the
pdf output is. Patch 3/3 works as an example where to add your stuff
(latex_documents in conf.py) and how.

Mauro, please look at patch 1/3 for a quick fix for the media build.

Markus, you can now unleash your "I told you so" on the rst2pdf. ;)

BR,
Jani.


Jani Nikula (3):
  Documentation/sphinx: build the media intermediate rst files for all
    outputs
  Documentation: switch to pdflatex for pdf generation
  Documentation: exclude media documentation from pdf generation

 Documentation/Makefile.sphinx | 18 +++++++++---------
 Documentation/conf.py         |  4 +++-
 Documentation/media/Makefile  |  3 ++-
 3 files changed, 14 insertions(+), 11 deletions(-)

-- 
2.1.4

[RFC PATCH 1/3] Documentation/sphinx: build the media intermediate rst files for all outputs

[Jani Nikula]

This is a stopgap measure to allow building outputs other than html.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/Makefile.sphinx | 3 +--
 Documentation/media/Makefile  | 3 ++-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index 857f1e273418..d24a9f12a280 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -38,10 +38,9 @@ ALLSPHINXOPTS   = -D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) -d $(B
 I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
 
 quiet_cmd_sphinx = SPHINX  $@
-      cmd_sphinx = BUILDDIR=$(BUILDDIR) $(SPHINXBUILD) -b $2 $(ALLSPHINXOPTS) $(BUILDDIR)/$2
+      cmd_sphinx = $(MAKE) BUILDDIR=$(BUILDDIR) $(build)=Documentation/media all; BUILDDIR=$(BUILDDIR) $(SPHINXBUILD) -b $2 $(ALLSPHINXOPTS) $(BUILDDIR)/$2
 
 htmldocs:
-	$(MAKE) BUILDDIR=$(BUILDDIR) -f $(srctree)/Documentation/media/Makefile $@
 	$(call cmd,sphinx,html)
 
 pdfdocs:
diff --git a/Documentation/media/Makefile b/Documentation/media/Makefile
index 39e2d766dbe3..a7fb35291f6c 100644
--- a/Documentation/media/Makefile
+++ b/Documentation/media/Makefile
@@ -10,7 +10,8 @@ FILES = audio.h.rst ca.h.rst dmx.h.rst frontend.h.rst net.h.rst video.h.rst \
 
 TARGETS := $(addprefix $(BUILDDIR)/, $(FILES))
 
-htmldocs: $(BUILDDIR) ${TARGETS}
+.PHONY: all
+all: $(BUILDDIR) ${TARGETS}
 
 $(BUILDDIR):
 	$(Q)mkdir -p $@
-- 
2.1.4

Re: [RFC PATCH 1/3] Documentation/sphinx: build the media intermediate rst files for all outputs

[Mauro Carvalho Chehab]

Em Wed, 10 Aug 2016 18:54:07 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> This is a stopgap measure to allow building outputs other than html.

Didn't test, but looking at the patch, it looks OK.

So,

Acked-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>

> 
> Signed-off-by: Jani Nikula <jani.nikula@intel.com>
> ---
>  Documentation/Makefile.sphinx | 3 +--
>  Documentation/media/Makefile  | 3 ++-
>  2 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
> index 857f1e273418..d24a9f12a280 100644
> --- a/Documentation/Makefile.sphinx
> +++ b/Documentation/Makefile.sphinx
> @@ -38,10 +38,9 @@ ALLSPHINXOPTS   = -D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) -d $(B
>  I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
>  
>  quiet_cmd_sphinx = SPHINX  $@
> -      cmd_sphinx = BUILDDIR=$(BUILDDIR) $(SPHINXBUILD) -b $2 $(ALLSPHINXOPTS) $(BUILDDIR)/$2
> +      cmd_sphinx = $(MAKE) BUILDDIR=$(BUILDDIR) $(build)=Documentation/media all; BUILDDIR=$(BUILDDIR) $(SPHINXBUILD) -b $2 $(ALLSPHINXOPTS) $(BUILDDIR)/$2
>  
>  htmldocs:
> -	$(MAKE) BUILDDIR=$(BUILDDIR) -f $(srctree)/Documentation/media/Makefile $@
>  	$(call cmd,sphinx,html)
>  
>  pdfdocs:
> diff --git a/Documentation/media/Makefile b/Documentation/media/Makefile
> index 39e2d766dbe3..a7fb35291f6c 100644
> --- a/Documentation/media/Makefile
> +++ b/Documentation/media/Makefile
> @@ -10,7 +10,8 @@ FILES = audio.h.rst ca.h.rst dmx.h.rst frontend.h.rst net.h.rst video.h.rst \
>  
>  TARGETS := $(addprefix $(BUILDDIR)/, $(FILES))
>  
> -htmldocs: $(BUILDDIR) ${TARGETS}
> +.PHONY: all
> +all: $(BUILDDIR) ${TARGETS}
>  
>  $(BUILDDIR):
>  	$(Q)mkdir -p $@




Cheers,
Mauro

[RFC PATCH 2/3] Documentation: switch to pdflatex for pdf generation

[Jani Nikula]

Looks like rst2pdf is not robust enough, especially for large documents.

Use recursive make on the Sphinx generated makefile to convert latex to
pdf. The ugly detail is that pdf is generated into
Documentation/output/latex.

Unfortunately, the pdflatex build generates huge amounts of build log
noise, and also fails in the end. We'll fix that next.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/Makefile.sphinx | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index d24a9f12a280..fc29e08085aa 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -25,8 +25,8 @@ else ifneq ($(DOCBOOKS),)
 
 else # HAVE_SPHINX
 
-# User-friendly check for rst2pdf
-HAVE_RST2PDF := $(shell if python -c "import rst2pdf" >/dev/null 2>&1; then echo 1; else echo 0; fi)
+# User-friendly check for pdflatex
+HAVE_PDFLATEX := $(shell if which pdflatex >/dev/null 2>&1; then echo 1; else echo 0; fi)
 
 # Internal variables.
 PAPEROPT_a4     = -D latex_paper_size=a4
@@ -44,12 +44,13 @@ htmldocs:
 	$(call cmd,sphinx,html)
 
 pdfdocs:
-ifeq ($(HAVE_RST2PDF),0)
-	$(warning The Python 'rst2pdf' module was not found. Make sure you have the module installed to produce PDF output.)
+ifeq ($(HAVE_PDFLATEX),0)
+	$(warning The 'pdflatex' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
 	@echo "  SKIP    Sphinx $@ target."
-else # HAVE_RST2PDF
-	$(call cmd,sphinx,pdf)
-endif # HAVE_RST2PDF
+else # HAVE_PDFLATEX
+	$(call cmd,sphinx,latex)
+	$(Q)$(MAKE) -C $(BUILDDIR)/latex
+endif # HAVE_PDFLATEX
 
 epubdocs:
 	$(call cmd,sphinx,epub)
-- 
2.1.4

Re: [RFC PATCH 2/3] Documentation: switch to pdflatex for pdf generation

[Markus Heiser]

Am 10.08.2016 um 17:54 schrieb Jani Nikula <jani.nikula@intel.com>:

> Looks like rst2pdf is not robust enough, especially for large documents.
> 
> Use recursive make on the Sphinx generated makefile to convert latex to
> pdf. The ugly detail is that pdf is generated into
> Documentation/output/latex.
> 
> Unfortunately, the pdflatex build generates huge amounts of build log
> noise, and also fails in the end. We'll fix that next.

A good starting point might be to ship our own tex-Makefile:

create a folder e.g. Documentation/sphinx-tex place your
Makefile (a copy of Sphinx's Makefile) in. In conf.py set
the 

  latex_additional_files = [
      "sphinx-text/Makefile"
  ]

In the sphinx-tex/Makefile set e.g.

  # Additional LaTeX options
  LATEXOPTS = -interaction=batchmode
  LATEX_ENV = max_print_line=120

  LATEX = $(LATEX_ENV) latex
  PDFLATEX = $(LATEX_ENV) pdflatex
  MAKEINDEX = $(LATEX_ENV) makeindex

this is what I do in my projects to separate my tex-Makefile settings
from the main sphinx-Makefile.

-- Markus --


> 
> Signed-off-by: Jani Nikula <jani.nikula@intel.com>
> ---
> Documentation/Makefile.sphinx | 15 ++++++++-------
> 1 file changed, 8 insertions(+), 7 deletions(-)
> 
> diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
> index d24a9f12a280..fc29e08085aa 100644
> --- a/Documentation/Makefile.sphinx
> +++ b/Documentation/Makefile.sphinx
> @@ -25,8 +25,8 @@ else ifneq ($(DOCBOOKS),)
> 
> else # HAVE_SPHINX
> 
> -# User-friendly check for rst2pdf
> -HAVE_RST2PDF := $(shell if python -c "import rst2pdf" >/dev/null 2>&1; then echo 1; else echo 0; fi)
> +# User-friendly check for pdflatex
> +HAVE_PDFLATEX := $(shell if which pdflatex >/dev/null 2>&1; then echo 1; else echo 0; fi)
> 
> # Internal variables.
> PAPEROPT_a4     = -D latex_paper_size=a4
> @@ -44,12 +44,13 @@ htmldocs:
> 	$(call cmd,sphinx,html)
> 
> pdfdocs:
> -ifeq ($(HAVE_RST2PDF),0)
> -	$(warning The Python 'rst2pdf' module was not found. Make sure you have the module installed to produce PDF output.)
> +ifeq ($(HAVE_PDFLATEX),0)
> +	$(warning The 'pdflatex' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
> 	@echo "  SKIP    Sphinx $@ target."
> -else # HAVE_RST2PDF
> -	$(call cmd,sphinx,pdf)
> -endif # HAVE_RST2PDF
> +else # HAVE_PDFLATEX
> +	$(call cmd,sphinx,latex)
> +	$(Q)$(MAKE) -C $(BUILDDIR)/latex
> +endif # HAVE_PDFLATEX
> 
> epubdocs:
> 	$(call cmd,sphinx,epub)
> -- 
> 2.1.4
> 

Re: [RFC PATCH 2/3] Documentation: switch to pdflatex for pdf generation

[Luis R. Rodriguez]

On Wed, Aug 10, 2016 at 06:54:08PM +0300, Jani Nikula wrote:
> Looks like rst2pdf is not robust enough, especially for large documents.
> 
> Use recursive make on the Sphinx generated makefile to convert latex to
> pdf. The ugly detail is that pdf is generated into
> Documentation/output/latex.
> 
> Unfortunately, the pdflatex build generates huge amounts of build log
> noise, and also fails in the end. We'll fix that next.
> 
> Signed-off-by: Jani Nikula <jani.nikula@intel.com>

The commit log really does not much to explain to what bad extend
rst2pdf is in, mentioning at least Mauro's feedback that he is aware
this will not be fixed seems sufficient.

  Luis

[RFC PATCH 3/3] Documentation: exclude media documentation from pdf generation

[Jani Nikula]

Although pdflatex is more robust than rst2pdf, building media
documentation pdf still fails. Exclude media documentation from pdf
generation for now.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/conf.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Documentation/conf.py b/Documentation/conf.py
index 96b7aa66c89c..827dafc515b1 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -268,7 +268,9 @@ latex_elements = {
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, 'TheLinuxKernel.tex', 'The Linux Kernel Documentation',
+    ('kernel-documentation', 'kernel-documentation.tex', 'The Linux Kernel Documentation',
+     'The kernel development community', 'manual'),
+    ('gpu/index', 'gpu.tex', 'Linux GPU Driver Developer\'s Guide',
      'The kernel development community', 'manual'),
 ]
 
-- 
2.1.4

Re: [RFC PATCH 3/3] Documentation: exclude media documentation from pdf generation

[Mauro Carvalho Chehab]

Hi Jani,

Em Wed, 10 Aug 2016 18:54:09 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> Although pdflatex is more robust than rst2pdf, building media
> documentation pdf still fails. Exclude media documentation from pdf
> generation for now.

Again, didn't test, but do you think the issues with pdflatex and
media are solvable?

> 
> Signed-off-by: Jani Nikula <jani.nikula@intel.com>
> ---
>  Documentation/conf.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/conf.py b/Documentation/conf.py
> index 96b7aa66c89c..827dafc515b1 100644
> --- a/Documentation/conf.py
> +++ b/Documentation/conf.py
> @@ -268,7 +268,9 @@ latex_elements = {
>  # (source start file, target name, title,
>  #  author, documentclass [howto, manual, or own class]).
>  latex_documents = [
> -    (master_doc, 'TheLinuxKernel.tex', 'The Linux Kernel Documentation',
> +    ('kernel-documentation', 'kernel-documentation.tex', 'The Linux Kernel Documentation',
> +     'The kernel development community', 'manual'),
> +    ('gpu/index', 'gpu.tex', 'Linux GPU Driver Developer\'s Guide',
>       'The kernel development community', 'manual'),
>  ]
>  




Cheers,
Mauro

Re: [RFC PATCH 3/3] Documentation: exclude media documentation from pdf generation

[Markus Heiser]

Am 10.08.2016 um 18:16 schrieb Mauro Carvalho Chehab <mchehab@infradead.org>:

> Hi Jani,
> 
> Em Wed, 10 Aug 2016 18:54:09 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
> 
>> Although pdflatex is more robust than rst2pdf, building media
>> documentation pdf still fails. Exclude media documentation from pdf
>> generation for now.
> 
> Again, didn't test, but do you think the issues with pdflatex and
> media are solvable?

Not all, we have to test book by book and see what fails and what 
the reasons. Large tables will never format propper on paper sizes.

But first lets ack and merge all this together.

-- Markus --

> 
>> 
>> Signed-off-by: Jani Nikula <jani.nikula@intel.com>
>> ---
>> Documentation/conf.py | 4 +++-
>> 1 file changed, 3 insertions(+), 1 deletion(-)
>> 
>> diff --git a/Documentation/conf.py b/Documentation/conf.py
>> index 96b7aa66c89c..827dafc515b1 100644
>> --- a/Documentation/conf.py
>> +++ b/Documentation/conf.py
>> @@ -268,7 +268,9 @@ latex_elements = {
>> # (source start file, target name, title,
>> #  author, documentclass [howto, manual, or own class]).
>> latex_documents = [
>> -    (master_doc, 'TheLinuxKernel.tex', 'The Linux Kernel Documentation',
>> +    ('kernel-documentation', 'kernel-documentation.tex', 'The Linux Kernel Documentation',
>> +     'The kernel development community', 'manual'),
>> +    ('gpu/index', 'gpu.tex', 'Linux GPU Driver Developer\'s Guide',
>>      'The kernel development community', 'manual'),
>> ]
>> 
> 
> 
> 
> 
> Cheers,
> Mauro

Re: [RFC PATCH 3/3] Documentation: exclude media documentation from pdf generation

[Luis R. Rodriguez]

On Wed, Aug 10, 2016 at 06:54:09PM +0300, Jani Nikula wrote:
> Although pdflatex is more robust than rst2pdf, building media
> documentation pdf still fails. Exclude media documentation from pdf
> generation for now.
> 
> Signed-off-by: Jani Nikula <jani.nikula@intel.com>
> ---
>  Documentation/conf.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/conf.py b/Documentation/conf.py
> index 96b7aa66c89c..827dafc515b1 100644
> --- a/Documentation/conf.py
> +++ b/Documentation/conf.py
> @@ -268,7 +268,9 @@ latex_elements = {
>  # (source start file, target name, title,
>  #  author, documentclass [howto, manual, or own class]).
>  latex_documents = [
> -    (master_doc, 'TheLinuxKernel.tex', 'The Linux Kernel Documentation',
> +    ('kernel-documentation', 'kernel-documentation.tex', 'The Linux Kernel Documentation',
> +     'The kernel development community', 'manual'),
> +    ('gpu/index', 'gpu.tex', 'Linux GPU Driver Developer\'s Guide',
>       'The kernel development community', 'manual'),

Ah so other than expanding Documentation/index.rst we also have to
edit this file. It would be good to expand the new documentation
to reflect this as this was not clear.

With these changes I confirm I get my documentation in PDF format
for the new sections I am adding!

Thanks!

  Luis

Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

[Markus Heiser]

Am 10.08.2016 um 17:54 schrieb Jani Nikula <jani.nikula@intel.com>:

> Hi Luis, all -
> 
> With these you should be able to get started with pdf generation. It's a
> quick transition to pdflatex, the patches are not very pretty, but the
> pdf output is. Patch 3/3 works as an example where to add your stuff
> (latex_documents in conf.py) and how.
> 
> Mauro, please look at patch 1/3 for a quick fix for the media build.
> 
> Markus, you can now unleash your "I told you so" on the rst2pdf. ;)

:)

same here: Didn't test, but looking at the patch, it looks OK.

Acked-by: Markus Heiser <markus.heiser@darmarit.de>

One suggestion: later when my "generic way to build only sphinx sub-folders" 
patch ...

http://marc.info/?l=linux-doc&m=147066213515280&w=2

... is acked and merge, it might be better to place the latex_document 
option into a gpu/conf.py ?

latex_documents = [
    ('gpu/index', 'gpu.tex', 'Linux GPU Driver Developer\'s Guide',
     'The kernel development community', 'manual') ]

I haven't thought this to the end, just an idea.

-- Markus --

Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

[Luis R. Rodriguez]

On Wed, Aug 10, 2016 at 06:54:06PM +0300, Jani Nikula wrote:
> Hi Luis, all -
> 
> With these you should be able to get started with pdf generation. It's a
> quick transition to pdflatex, the patches are not very pretty, but the
> pdf output is. Patch 3/3 works as an example where to add your stuff
> (latex_documents in conf.py) and how.
> 
> Mauro, please look at patch 1/3 for a quick fix for the media build.
> 
> Markus, you can now unleash your "I told you so" on the rst2pdf. ;)
> 

Thanks this works! It would be nice to have a target for tex so one can
then just worry about fixing the errors generated to generate the tex
file. Otherwise I have to hit CTRL C right beore pdflatex is called.

  Luis

Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

[Jonathan Corbet]

On Wed, 10 Aug 2016 18:54:06 +0300
Jani Nikula <jani.nikula@intel.com> wrote:

> With these you should be able to get started with pdf generation. It's a
> quick transition to pdflatex, the patches are not very pretty, but the
> pdf output is. Patch 3/3 works as an example where to add your stuff
> (latex_documents in conf.py) and how.

OK, now I have a bone to pick with you.

I applied this, then decided to install the needed toolchain on the
Tumbleweed system I've been playing with; it wanted to install 1,727
packages to get pdflatex.  Pandoc just doesn't seem so bad anymore.

So I switched to the Fedora system, and found myself in a twisty maze of
missing font files, missing style files, missing babel crap, etc., each
doled out to me one file per run.  But I did eventually get PDFs out of
it.

The output isn't great; among other things, it seems to be about 1/2 blank
pages.  But it's something.

I've applied this so we have something to play with, but it doesn't feel
like a great solution.  This is the sort of installation hell that we
wanted to get away from.  It makes me wonder how hard it can really be to
fix rst2pdf; I wish I could say I'll find some time to figure that out.
Sigh.

jon

Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

[Markus Heiser]

Am 13.08.2016 um 00:40 schrieb Jonathan Corbet <corbet@lwn.net>:

> On Wed, 10 Aug 2016 18:54:06 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
> 
>> With these you should be able to get started with pdf generation. It's a
>> quick transition to pdflatex, the patches are not very pretty, but the
>> pdf output is. Patch 3/3 works as an example where to add your stuff
>> (latex_documents in conf.py) and how.
> 
> OK, now I have a bone to pick with you.
> 
> I applied this, then decided to install the needed toolchain on the
> Tumbleweed system I've been playing with; it wanted to install 1,727
> packages to get pdflatex.  Pandoc just doesn't seem so bad anymore.

I'am complete disenchanted on this topic. My experience is:

1) You wan't get any reasonable typesetting engine which preserves
your disk space. I don't know how many files or packages are installed,
the only thing I know is, a TeX installation is always >1GB.

2) You wan't get a (pdf, ps,..) book with a perfect layout without
any handcraft or at least a *theming*. TeX has many options to influence
the layout and Sphinx provides it's own LaTeX-document class (sphinxmanual)
which is IMHO awful.

> So I switched to the Fedora system, and found myself in a twisty maze of
> missing font files, missing style files, missing babel crap, etc., each
> doled out to me one file per run.  But I did eventually get PDFs out of
> it.

On debian it should be enough to install *base* and *recommended*

    sudo apt-get install
          texlive-base texlive-latex-recommended

> The output isn't great; among other things, it seems to be about 1/2 blank
> pages.

1/2 ? .. I have only empty pages at the start of parts or chapters, which
is a typical layout setting.

>  But it's something.

This is the sphinxmanual document class.
 
> I've applied this so we have something to play with, but it doesn't feel
> like a great solution.  This is the sort of installation hell that we
> wanted to get away from.  

See above, on debian it should be enough to install the two meta packages.

> It makes me wonder how hard it can really be to
> fix rst2pdf; I wish I could say I'll find some time to figure that out.
> Sigh.

I gave it a try, but as I come closer to the sources I realized that
it is hair-raising. I looked at the issues, added a comment to a related
issue, a few days later the issue was closed without any comment or code
change.

https://github.com/rst2pdf/rst2pdf/issues/556#issuecomment-228779542

My advice, if you don't like to waste your time: forget it.

Some thoughts of mine, wrote in an earlier mail:

> The sphinx-doc build-in LaTeX builder
> 
> * http://www.sphinx-doc.org/en/stable/config.html#options-for-latex-output
> 
> has some drawbacks, e.g. it produce LaTeX for the pdfTeX engine.
> LaTeX is by default ASCII and it needs some "inputenc" to supporta wider
> range of characters. This is not very helpful if you have a toolchain
> in an international community.
> 
> The alternative to LaTeX is to use the XeTeX engine, which supports UTF-8
> encoded input by default and supports TrueType/OpenType fonts directly.
> Thats why I started to write a XeLaTeX builder ...
> 
> * https://github.com/return42/sphkerneldoc/blob/master/scripts/site-python/xelatex_ext/__init__.py#L15 
> 
> ... but I can't predict when this will be finished ...
> 
> However which tool is used, my experience is, that building
> PDF (books) with a minimum of quality is not simple.
> Layout width tables, split table content over pages, switch
> from landscape to portrait and versus, the flow of objects etc.
> .. all this will need some manually interventions.


-- Markus --

Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

[Jani Nikula]

On Sat, 13 Aug 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> On Wed, 10 Aug 2016 18:54:06 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
>
>> With these you should be able to get started with pdf generation. It's a
>> quick transition to pdflatex, the patches are not very pretty, but the
>> pdf output is. Patch 3/3 works as an example where to add your stuff
>> (latex_documents in conf.py) and how.
>
> OK, now I have a bone to pick with you.
>
> I applied this, then decided to install the needed toolchain on the
> Tumbleweed system I've been playing with; it wanted to install 1,727
> packages to get pdflatex.  Pandoc just doesn't seem so bad anymore.

Jon, I sent these to unblock Luis, and as a starting point for a
discussion about rst2pdf vs. pdflatex. I didn't mean I'd want these
merged as-is! I'm sorry if I didn't make myself clear.

I don't mind at all if you want to drop them.

As I played around, it worked for *me* better than rst2pdf. I had
pdflatex already installed, so I didn't pay attention to the deps, and
the numbers you quote come as a suprise.

> I've applied this so we have something to play with, but it doesn't feel
> like a great solution.  This is the sort of installation hell that we
> wanted to get away from.  It makes me wonder how hard it can really be to
> fix rst2pdf; I wish I could say I'll find some time to figure that out.
> Sigh.

I'm afraid pdf generation is not very high on my list of priorities
right now. Surely I'll get back to this eventually, but not any time
soon. :(

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

[Mauro Carvalho Chehab]

Em Mon, 15 Aug 2016 12:40:21 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Sat, 13 Aug 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> > On Wed, 10 Aug 2016 18:54:06 +0300
> > Jani Nikula <jani.nikula@intel.com> wrote:
> >  
> >> With these you should be able to get started with pdf generation. It's a
> >> quick transition to pdflatex, the patches are not very pretty, but the
> >> pdf output is. Patch 3/3 works as an example where to add your stuff
> >> (latex_documents in conf.py) and how.  
> >
> > OK, now I have a bone to pick with you.
> >
> > I applied this, then decided to install the needed toolchain on the
> > Tumbleweed system I've been playing with; it wanted to install 1,727
> > packages to get pdflatex.  Pandoc just doesn't seem so bad anymore.  
> 
> Jon, I sent these to unblock Luis, and as a starting point for a
> discussion about rst2pdf vs. pdflatex. I didn't mean I'd want these
> merged as-is! I'm sorry if I didn't make myself clear.
> 
> I don't mind at all if you want to drop them.
> 
> As I played around, it worked for *me* better than rst2pdf. I had
> pdflatex already installed, so I didn't pay attention to the deps, and
> the numbers you quote come as a suprise.
> 
> > I've applied this so we have something to play with, but it doesn't feel
> > like a great solution.  This is the sort of installation hell that we
> > wanted to get away from.  It makes me wonder how hard it can really be to
> > fix rst2pdf; I wish I could say I'll find some time to figure that out.
> > Sigh.  
> 
> I'm afraid pdf generation is not very high on my list of priorities
> right now. Surely I'll get back to this eventually, but not any time
> soon. :(

I'm afraid that, no matter if we'll be using rst2pdf or pdflatex,
we'll need LaTex anyway.

One of the things that we want on media is to better represent the
math formulas there. Just enabling the math extension is enough for
it to require not only LaTex, but also this package (on Fedora):
	texlive-anyfontsize.noarch

So, unfortunately, I'm afraid that we'll end by having a much worse
dependency chain than what we used to have with DocBook.

An (incomplete) patch to let you test it is enclosed (it is actually after
patches 1 and 2 of this series).

Ah, it seems that rst2pdf is not compatible with math extension...

If we change the conf.py to:
	extensions = ['sphinx.ext.imgmath', 'kernel-doc', 'rstFlatTable', 'kernel_include']

	# Gracefully handle missing rst2pdf.
	try:
	    import rst2pdf
	    extensions += ['rst2pdf.pdfbuilder']
	except ImportError:
	    pass

The logic that tries to import rst2pdf produces the following error:

	Extension error:
	Config value 'math_number_all' already present

And, if we don't add the imgmath extension, the math expression
appears broken at the generated html page:

	The transfer function defined for SMPTE 170M is the same as the one defined in Rec. 709.
	L' = -1.099(-L)^{0.45} + 0.099 \text{ for } L \le-0.018 L' = 4.5L \text{ for } -0.018 < L < 0.018 L' = 1.099L^{0.45} - 0.099 \text{ for } L \ge 0.018

Btw, changing to math is a requirement for pdflatex to work on media,
as the font used by Sphinx/pdflatex doesn't have UTF-8 symbols used
on math. So, trying to build the documentation with pdfdocs (after
patches 1 and 2, before patch 3) would give fatal errors like:

	! Package inputenc Error: Unicode char \u8:≥ not set up for use with LaTeX.

	See the inputenc package documentation for explanation.
	Type  H <return>  for immediate help.
	 ...                                              
                                                  
	l.7759 ....099L $^{\text{0.45}}$ - 0.099 for L ≥
							 0.018

The problem is that the "grater or equal than" UTF-8 Math symbol
is not supported on the LaTex font.


Thanks,
Mauro


diff --git a/Documentation/conf.py b/Documentation/conf.py
index 96b7aa66c89c..0b60a55cea89 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -28,14 +28,14 @@ sys.path.insert(0, os.path.abspath('sphinx'))
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['kernel-doc', 'rstFlatTable', 'kernel_include']
+extensions = ['sphinx.ext.imgmath', 'kernel-doc', 'rstFlatTable', 'kernel_include']
 
 # Gracefully handle missing rst2pdf.
-try:
-    import rst2pdf
-    extensions += ['rst2pdf.pdfbuilder']
-except ImportError:
-    pass
+#try:
+#    import rst2pdf
+#    extensions += ['rst2pdf.pdfbuilder']
+#except ImportError:
+#    pass
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
diff --git a/Documentation/media/uapi/v4l/pixfmt-007.rst b/Documentation/media/uapi/v4l/pixfmt-007.rst
index 8c946b0c63a0..1253c3283b10 100644
--- a/Documentation/media/uapi/v4l/pixfmt-007.rst
+++ b/Documentation/media/uapi/v4l/pixfmt-007.rst
@@ -72,11 +72,13 @@ SMPTE C set, so this colorspace is sometimes called SMPTE C as well.
 The transfer function defined for SMPTE 170M is the same as the one
 defined in Rec. 709.
 
-    L' = -1.099(-L) :sup:`0.45` + 0.099 for L ≤ -0.018
+.. math::
 
-    L' = 4.5L for -0.018 < L < 0.018
+    L' = -1.099(-L)^{0.45} + 0.099 \text{ for } L \le-0.018
 
-    L' = 1.099L :sup:`0.45` - 0.099 for L ≥ 0.018
+    L' = 4.5L \text{ for } -0.018 < L < 0.018
+
+    L' = 1.099L^{0.45} - 0.099 \text{ for } L \ge 0.018
 
 Inverse Transfer function:
 

Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

[Mauro Carvalho Chehab]

Hi Jon,

Em Mon, 15 Aug 2016 09:17:52 -0300
Mauro Carvalho Chehab <mchehab@infradead.org> escreveu:

> Em Mon, 15 Aug 2016 12:40:21 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
> 
> > On Sat, 13 Aug 2016, Jonathan Corbet <corbet@lwn.net> wrote:  
> > > On Wed, 10 Aug 2016 18:54:06 +0300
> > > Jani Nikula <jani.nikula@intel.com> wrote:
> > >    
> > >> With these you should be able to get started with pdf generation. It's a
> > >> quick transition to pdflatex, the patches are not very pretty, but the
> > >> pdf output is. Patch 3/3 works as an example where to add your stuff
> > >> (latex_documents in conf.py) and how.    
> > >
> > > OK, now I have a bone to pick with you.
> > >
> > > I applied this, then decided to install the needed toolchain on the
> > > Tumbleweed system I've been playing with; it wanted to install 1,727
> > > packages to get pdflatex.  Pandoc just doesn't seem so bad anymore.    
> > 
> > Jon, I sent these to unblock Luis, and as a starting point for a
> > discussion about rst2pdf vs. pdflatex. I didn't mean I'd want these
> > merged as-is! I'm sorry if I didn't make myself clear.
> > 
> > I don't mind at all if you want to drop them.

I played for a while with rst2pdf and LaTeX-based tools, plus
using Sphinx math extension to improve the media documentation.

Here's my findings:

1) I'm pretty sure there's no way for us to get rid of LaTex.

The problem is that the Sphinx math extension depends on LaTeX amsmath
extension. Ok, someone could write some extension in the future to add math
support without requiring LaTeX and fix rst2pdf (or rewrite it), but this
would take time and efforts.

So, for now, I think we should just assume that anyone wanting to generate
pdf docs will need LaTeX (or, more likely, TeTeX).

2) rst2pdf seems to be incompatible with Sphinx math extension,
at least on Sphinx 1.4.5;

3) pdflatex doesn't handle UTF-8 chars. This is a problem for media,
as we use two UTF-8 symbols that are incompatible with the fonts
used by pdflatex:

	- ≥ U+2265 (GREATER-THAN OR EQUAL TO) utf-8 character
	- ≤ u+2264 (LESS-THAN OR EQUAL TO) utf-8 character

4) pdflatex output is not nice: all black and white; ugly (IMHO) fonts;

5) At media docs, some tables will only print ok in landscape.

After making the media books build, I think that the best way is to
use xelatex instead of pdfdocs. Visually, xelatex output is, IMHO,
nice - and it has colors :-)

It seems that there's yet another option: lualatex. I didn't try to
build with it. So, not sure if its output is better or not, nor if
some extra config for it is needed at conf.py.

I sent a patch series addressing most of the issues above.

If you want to see the output:
	https://mchehab.fedorapeople.org/TheLinuxKernel.pdf

This was built on Fedora 24.

Issues:
-------

Even with xelatex, there are still some issues to be addressed:

- There are two hacks to build media: one removes the *.h.rst files
  and the other one comments out two C code blocks inside tables;

- There are lots of noise at PDF output. I didn't even try to look
  into them;

- Almost all tables at the media books are mangled. Rotating them to
  landscape can fix several of them. I added support for it, however,
  that requires to manually add LaTeX tags before and after the tables,
  like:

    .. raw:: latex
    
        \begin{landscape}
    
    <some table(s)>
    
    .. raw:: latex
    
        \end{landscape}

As the above tags are LaTeX specific, they should not interfere on 
html (or e-pub) output.

One such example is this table:
	"Table 2.18: Packed RGB Image Formats"
	(at page 92 of the PDF file)

I'm seeking for a solution to scale it and rotate (as just rotating it
is not enough).

-- 
Thanks,
Mauro

Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

[Jani Nikula]

On Tue, 16 Aug 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> 5) At media docs, some tables will only print ok in landscape.
>
> After making the media books build, I think that the best way is to
> use xelatex instead of pdfdocs. Visually, xelatex output is, IMHO,
> nice - and it has colors :-)
>
> It seems that there's yet another option: lualatex. I didn't try to
> build with it. So, not sure if its output is better or not, nor if
> some extra config for it is needed at conf.py.
>
> I sent a patch series addressing most of the issues above.

I didn't see the series, but I'm surprised we now have HAVE_PDFLATEX
that actually checks for 'xelatex'. The commit subject doing the change
doesn't even mention xelatex, it's just about UTF-8:

commit a682ec4ba10c88231cdbb8bb9823b2cc749d6364
Author: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Date:   Tue Aug 16 13:25:39 2016 -0300

    docs-rst: Don't mangle with UTF-8 chars on LaTeX/PDF output

</bikeshedding after the fact>

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

[Mauro Carvalho Chehab]

Em Fri, 19 Aug 2016 15:54:37 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Tue, 16 Aug 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> > 5) At media docs, some tables will only print ok in landscape.
> >
> > After making the media books build, I think that the best way is to
> > use xelatex instead of pdfdocs. Visually, xelatex output is, IMHO,
> > nice - and it has colors :-)
> >
> > It seems that there's yet another option: lualatex. I didn't try to
> > build with it. So, not sure if its output is better or not, nor if
> > some extra config for it is needed at conf.py.
> >
> > I sent a patch series addressing most of the issues above.  
> 
> I didn't see the series, but I'm surprised we now have HAVE_PDFLATEX
> that actually checks for 'xelatex'. The commit subject doing the change
> doesn't even mention xelatex, it's just about UTF-8:
> 
> commit a682ec4ba10c88231cdbb8bb9823b2cc749d6364
> Author: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> Date:   Tue Aug 16 13:25:39 2016 -0300
> 
>     docs-rst: Don't mangle with UTF-8 chars on LaTeX/PDF output
> 
> </bikeshedding after the fact>

It was mentioned at patch 0/9:
	https://www.mail-archive.com/linux-media@vger.kernel.org/msg101408.html

I opted to not change the HAVE_PDFLATEX, as the macro that sets the
latex variant on Sphinx latex Makefile is called "PDFLATEX". Also,
we might change it in the future to use some other tool or even add
some logic to allow supporting multiple tools, like we have with
DocBook.

The main goal of XeTeX is to add unicode font support to LaTex,
according to:
	https://en.wikipedia.org/wiki/XeTeX

We need that, as media uses a lot of math Unicode symbols inside
the documentation.

-- 
Thanks,
Mauro