[PATCH 0/3] Add a way to build only media docs

[PATCH 0/3] Add a way to build only media docs

[Mauro Carvalho Chehab]

Being able to build just the media docs is important for us due to several
reasons:

1) Media developers community hosts a copy of the media documentation at linuxtv.org
    with the very latest  under development documents;

2) Nitpicking to identify broken references is important to identify documentation gaps
    that need to be addressed on future releases;

3) As media maintainers check patch per patch if a documentation gap is introduced, building
    media documentation should be as fast as possible.

This patchset adds a media file adding nitpick support and an extra build target that will
compile only the media documentation. It also groups all media documentation into one
section on the main Kernel document, with is, IMHO, a good thing as we start adding more
stuff there.

Jon,

I'd love to see this patch merged early at the -rc cycle, in order to avoid merge
conflicts when people start converting other docbooks to Sphinx, as it touches
at the main Makefile and at the Sphinx common stuff. Also, as I'll need to patch my
build scripts to check for documentation issues with Sphinx, I need them on my
master branch, as otherwise my workflow will be broken until the next Kernel release.

So, If you're ok with this patch series, can you submit to Linus on early -rc? Or 
if you prefer, I can do it myself, with your ack.

Thanks!
Mauro

PS.: I would prefer to have a more generic way to add support to build documentation
for only one subsystem, but, as we also need to load an extra python module to be
able to enable nitpick mode, I opted, for now, on not doing it too generic. We can rework
on it later, as other subsystems would need a similar feature.


Markus Heiser (1):
  doc-rst: support additional Sphinx build config override

Mauro Carvalho Chehab (2):
  doc-rst: add an option to build media documentation in nitpick mode
  doc-rst: remove a bogus comment from Documentation/index.rst

 Documentation/Makefile.sphinx       | 10 ++++-
 Documentation/conf.py               |  9 ++++
 Documentation/index.rst             |  7 +--
 Documentation/media/conf_nitpick.py | 85 +++++++++++++++++++++++++++++++++++++
 Documentation/media/index.rst       | 12 ++++++
 Documentation/sphinx/load_config.py | 25 +++++++++++
 Makefile                            |  6 +++
 7 files changed, 146 insertions(+), 8 deletions(-)
 create mode 100644 Documentation/media/conf_nitpick.py
 create mode 100644 Documentation/media/index.rst
 create mode 100644 Documentation/sphinx/load_config.py

-- 
2.7.4

[PATCH 1/3] doc-rst: support additional Sphinx build config override

[Mauro Carvalho Chehab]

From: Markus Heiser <markus.heiser@darmarIT.de>

Load an additional configuration file into conf.py namespace.

The name of the configuration file is taken from the environment
SPHINX_CONF. The external configuration file extends (or overwrites) the
configuration values from the origin conf.py.  With this you are
able to maintain *build themes*.

E.g. to create your own nit-picking *build theme*, create a file
Documentation/conf_nitpick.py::

  nitpicky=True
  nitpick_ignore = [
      ("c:func", "clock_gettime"),
      ...
      ]

and run make with SPHINX_CONF environment::

  make SPHINX_CONF=conf_nitpick.py htmldocs

Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 Documentation/conf.py               |  9 +++++++++
 Documentation/sphinx/load_config.py | 25 +++++++++++++++++++++++++
 2 files changed, 34 insertions(+)
 create mode 100644 Documentation/sphinx/load_config.py

diff --git a/Documentation/conf.py b/Documentation/conf.py
index 96b7aa66c89c..d5027750503a 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -20,6 +20,8 @@ import os
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 sys.path.insert(0, os.path.abspath('sphinx'))
 
+from load_config import loadConfig
+
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
@@ -419,3 +421,10 @@ pdf_documents = [
 # line arguments.
 kerneldoc_bin = '../scripts/kernel-doc'
 kerneldoc_srctree = '..'
+
+
+# ------------------------------------------------------------------------------
+# Since loadConfig overwrites settings from the global namespace, it has to be
+# the last statement in the conf.py file
+# ------------------------------------------------------------------------------
+loadConfig(globals())
diff --git a/Documentation/sphinx/load_config.py b/Documentation/sphinx/load_config.py
new file mode 100644
index 000000000000..44bdd2271d13
--- /dev/null
+++ b/Documentation/sphinx/load_config.py
@@ -0,0 +1,25 @@
+# -*- coding: utf-8; mode: python -*-
+# pylint: disable=R0903, C0330, R0914, R0912, E0401
+
+import os
+from sphinx.util.pycompat import execfile_
+
+# ------------------------------------------------------------------------------
+def loadConfig(namespace):
+# ------------------------------------------------------------------------------
+
+    u"""Load an additional configuration file into *namespace*.
+
+    The name of the configuration file is taken from the environment
+    ``SPHINX_CONF``. The external configuration file extends (or overwrites) the
+    configuration values from the origin ``conf.py``.  With this you are able to
+    maintain *build themes*.  """
+
+    config_file = os.environ.get("SPHINX_CONF", None)
+    if config_file is not None and os.path.exists(config_file):
+        config_file = os.path.abspath(config_file)
+        config = namespace.copy()
+        config['__file__'] = config_file
+        execfile_(config_file, config)
+        del config['__file__']
+        namespace.update(config)
-- 
2.7.4

[PATCH 2/3] doc-rst: add an option to build media documentation in nitpick mode

[Mauro Carvalho Chehab]

While writing the media documentation, it is important to be able
to check if all symbols that are internal to the documentation were
cross-referenced, as this ensures that newer patches won't be
introducing documentation gaps.

So, add a way to build only the media documentation in nitpick
mode.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 Documentation/Makefile.sphinx       | 10 ++++-
 Documentation/index.rst             |  5 +--
 Documentation/media/conf_nitpick.py | 85 +++++++++++++++++++++++++++++++++++++
 Documentation/media/index.rst       | 12 ++++++
 Makefile                            |  6 +++
 5 files changed, 112 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/media/conf_nitpick.py
 create mode 100644 Documentation/media/index.rst

diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index b10b6c598ae2..bbd7cd46f4a9 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -33,12 +33,17 @@ PAPEROPT_a4     = -D latex_paper_size=a4
 PAPEROPT_letter = -D latex_paper_size=letter
 KERNELDOC       = $(srctree)/scripts/kernel-doc
 KERNELDOC_CONF  = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
-ALLSPHINXOPTS   = -D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) -d $(BUILDDIR)/.doctrees $(KERNELDOC_CONF) $(PAPEROPT_$(PAPER)) -c $(srctree)/$(src) $(SPHINXOPTS) $(srctree)/$(src)
+ALLSPHINXOPTS   = -D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) -d $(BUILDDIR)/.doctrees $(KERNELDOC_CONF) $(PAPEROPT_$(PAPER)) -c $(srctree)/$(src) $(SPHINXOPTS)
 # the i18n builder cannot share the environment and doctrees with the others
 I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
 
 quiet_cmd_sphinx = SPHINX  $@
-      cmd_sphinx = BUILDDIR=$(BUILDDIR) $(SPHINXBUILD) -b $2 $(ALLSPHINXOPTS) $(BUILDDIR)/$2
+      cmd_sphinx = BUILDDIR=$(BUILDDIR) $(SPHINXBUILD) -b $2 $(ALLSPHINXOPTS) $(srctree)/$(src)$3 $(BUILDDIR)/$2
+
+# Build only the media docs, in nitpick mode
+mediadocs:
+	$(MAKE) BUILDDIR=$(BUILDDIR) SPHINX_CONF=media/conf_nitpick.py -f $(srctree)/Documentation/media/Makefile htmldocs
+	$(call cmd,sphinx,html,/media)
 
 htmldocs:
 	$(MAKE) BUILDDIR=$(BUILDDIR) -f $(srctree)/Documentation/media/Makefile $@
@@ -70,6 +75,7 @@ cleandocs:
 dochelp:
 	@echo  ' Linux kernel internal documentation in different formats (Sphinx):'
 	@echo  '  htmldocs        - HTML'
+	@echo  '  mediadocs       - built only media books in HTML on nitpick mode'
 	@echo  '  pdfdocs         - PDF'
 	@echo  '  epubdocs        - EPUB'
 	@echo  '  xmldocs         - XML'
diff --git a/Documentation/index.rst b/Documentation/index.rst
index e0fc72963e87..02255c1806f6 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -14,10 +14,7 @@ Contents:
    :maxdepth: 2
 
    kernel-documentation
-   media/media_uapi
-   media/media_kapi
-   media/dvb-drivers/index
-   media/v4l-drivers/index
+   media/index
    gpu/index
 
 Indices and tables
diff --git a/Documentation/media/conf_nitpick.py b/Documentation/media/conf_nitpick.py
new file mode 100644
index 000000000000..9034d7f19753
--- /dev/null
+++ b/Documentation/media/conf_nitpick.py
@@ -0,0 +1,85 @@
+nitpicky=True
+
+# It is possible to run Sphinx in nickpick mode with:
+#	make SPHINXOPTS=-n htmldocs
+# In such case, it will complain about lots of missing references that
+#	1) are just typedefs like: bool, __u32, etc;
+#	2) It will complain for things like: enum, NULL;
+#	3) It will complain for symbols that should be on different
+#	   books (but currently aren't ported to ReST)
+# The list below has a list of such symbols to be ignored in nitpick mode
+#
+nitpick_ignore = [
+	("c:func", "clock_gettime"),
+	("c:func", "close"),
+	("c:func", "container_of"),
+	("c:func", "determine_valid_ioctls"),
+	("c:func", "ERR_PTR"),
+	("c:func", "ioctl"),
+	("c:func", "IS_ERR"),
+	("c:func", "mmap"),
+	("c:func", "open"),
+	("c:func", "pci_name"),
+	("c:func", "poll"),
+	("c:func", "PTR_ERR"),
+	("c:func", "read"),
+	("c:func", "release"),
+	("c:func", "set"),
+	("c:func", "struct fd_set"),
+	("c:func", "struct pollfd"),
+	("c:func", "usb_make_path"),
+	("c:func", "write"),
+	("c:type", "atomic_t"),
+	("c:type", "bool"),
+	("c:type", "buf_queue"),
+	("c:type", "device"),
+	("c:type", "device_driver"),
+	("c:type", "device_node"),
+	("c:type", "enum"),
+	("c:type", "file"),
+	("c:type", "i2c_adapter"),
+	("c:type", "i2c_board_info"),
+	("c:type", "i2c_client"),
+	("c:type", "ktime_t"),
+	("c:type", "led_classdev_flash"),
+	("c:type", "list_head"),
+	("c:type", "lock_class_key"),
+	("c:type", "module"),
+	("c:type", "mutex"),
+	("c:type", "pci_dev"),
+	("c:type", "pdvbdev"),
+	("c:type", "poll_table_struct"),
+	("c:type", "s32"),
+	("c:type", "s64"),
+	("c:type", "sd"),
+	("c:type", "spi_board_info"),
+	("c:type", "spi_device"),
+	("c:type", "spi_master"),
+	("c:type", "struct fb_fix_screeninfo"),
+	("c:type", "struct pollfd"),
+	("c:type", "struct timeval"),
+	("c:type", "struct video_capability"),
+	("c:type", "u16"),
+	("c:type", "u32"),
+	("c:type", "u64"),
+	("c:type", "u8"),
+	("c:type", "union"),
+	("c:type", "usb_device"),
+
+	("cpp:type", "boolean"),
+	("cpp:type", "fd"),
+	("cpp:type", "fd_set"),
+	("cpp:type", "int16_t"),
+	("cpp:type", "NULL"),
+	("cpp:type", "off_t"),
+	("cpp:type", "pollfd"),
+	("cpp:type", "size_t"),
+	("cpp:type", "ssize_t"),
+	("cpp:type", "timeval"),
+	("cpp:type", "__u16"),
+	("cpp:type", "__u32"),
+	("cpp:type", "__u64"),
+	("cpp:type", "uint16_t"),
+	("cpp:type", "uint32_t"),
+	("cpp:type", "video_system_t"),
+]
diff --git a/Documentation/media/index.rst b/Documentation/media/index.rst
new file mode 100644
index 000000000000..e85c557eeea3
--- /dev/null
+++ b/Documentation/media/index.rst
@@ -0,0 +1,12 @@
+Linux Media Subsystem Documentation
+===================================
+
+Contents:
+
+.. toctree::
+   :maxdepth: 2
+
+   media_uapi
+   media_kapi
+   dvb-drivers/index
+   v4l-drivers/index
diff --git a/Makefile b/Makefile
index 35603556023e..08ef6c1a807b 100644
--- a/Makefile
+++ b/Makefile
@@ -1439,6 +1439,12 @@ $(DOC_TARGETS): scripts_basic FORCE
 	$(Q)$(MAKE) $(build)=Documentation -f $(srctree)/Documentation/Makefile.sphinx $@
 	$(Q)$(MAKE) $(build)=Documentation/DocBook $@
 
+DOC_NITPIC_TARGETS := mediadocs
+PHONY += $(DOC_NITPIC_TARGETS)
+$(DOC_NITPIC_TARGETS): scripts_basic FORCE
+	$(Q)$(MAKE) $(build)=scripts build_docproc build_check-lc_ctype
+	$(Q)$(MAKE) $(build)=Documentation -f $(srctree)/Documentation/Makefile.sphinx $@
+
 else # KBUILD_EXTMOD
 
 ###
-- 
2.7.4

[PATCH 3/3] doc-rst: remove a bogus comment from Documentation/index.rst

[Mauro Carvalho Chehab]

There's a bogus document at the documentation index saying
that there was nothing there, likely a left-over from the
initial Sphinx patches. Get rid of it!

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 Documentation/index.rst | 2 --
 1 file changed, 2 deletions(-)

diff --git a/Documentation/index.rst b/Documentation/index.rst
index 02255c1806f6..bdd9525e05aa 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -6,8 +6,6 @@
 Welcome to The Linux Kernel's documentation!
 ============================================
 
-Nothing for you to see here *yet*. Please move along.
-
 Contents:
 
 .. toctree::
-- 
2.7.4

Re: [PATCH 0/3] Add a way to build only media docs

[Mauro Carvalho Chehab]

Em Sat,  6 Aug 2016 09:00:31 -0300
Mauro Carvalho Chehab <mchehab@s-opensource.com> escreveu:

> Being able to build just the media docs is important for us due to several
> reasons:
> 
> 1) Media developers community hosts a copy of the media documentation at linuxtv.org
>     with the very latest  under development documents;
> 
> 2) Nitpicking to identify broken references is important to identify documentation gaps
>     that need to be addressed on future releases;
> 
> 3) As media maintainers check patch per patch if a documentation gap is introduced, building
>     media documentation should be as fast as possible.
> 
> This patchset adds a media file adding nitpick support and an extra build target that will
> compile only the media documentation. It also groups all media documentation into one
> section on the main Kernel document, with is, IMHO, a good thing as we start adding more
> stuff there.
> 
> Jon,
> 
> I'd love to see this patch merged early at the -rc cycle, in order to avoid merge
> conflicts when people start converting other docbooks to Sphinx, as it touches
> at the main Makefile and at the Sphinx common stuff. Also, as I'll need to patch my
> build scripts to check for documentation issues with Sphinx, I need them on my
> master branch, as otherwise my workflow will be broken until the next Kernel release.
> 
> So, If you're ok with this patch series, can you submit to Linus on early -rc? Or 
> if you prefer, I can do it myself, with your ack.

Forgot to comment, but those patches are applied against ustream master
branch. I applied them against my development tree at:
	https://git.linuxtv.org//mchehab/experimental.git/log/?h=docs-next

Regards,
Mauro

Re: [PATCH 0/3] Add a way to build only media docs

[Markus Heiser]

Am 06.08.2016 um 14:00 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:

> Being able to build just the media docs is important for us due to several
> reasons:
> 
> 1) Media developers community hosts a copy of the media documentation at linuxtv.org
>    with the very latest  under development documents;
> 
> 2) Nitpicking to identify broken references is important to identify documentation gaps
>    that need to be addressed on future releases;
> 
> 3) As media maintainers check patch per patch if a documentation gap is introduced,


Take into account: the gap is detected by "open references" (from the reST-header
files), but Sphinx tries to bind a ":ref:" to any anchor which fits (no matter where
it comes from). I mean, if we add more and more content or e.g. interpshinx objects,
we did not know which targets are available and the risk to get a *false bind* grows
with the content.

I don't know, may be in some future you will come into trouble, when you want
to refer targets outside of the ./media folder and *detect gaps* in the docs.

This need not be a problem, but you should be aware of, that your "test" is
only a test on open/closed links.


> building
>    media documentation should be as fast as possible.
> 
> This patchset adds a media file adding nitpick support and an extra build target that will
> compile only the media documentation. It also groups all media documentation into one
> section on the main Kernel document, with is, IMHO, a good thing as we start adding more
> stuff there.
> 
> Jon,
> 
> I'd love to see this patch merged early at the -rc cycle, in order to avoid merge
> conflicts when people start converting other docbooks to Sphinx, as it touches
> at the main Makefile and at the Sphinx common stuff. Also, as I'll need to patch my
> build scripts to check for documentation issues with Sphinx, I need them on my
> master branch, as otherwise my workflow will be broken until the next Kernel release.
> 
> So, If you're ok with this patch series, can you submit to Linus on early -rc? Or 
> if you prefer, I can do it myself, with your ack.
> 
> Thanks!
> Mauro
> 
> PS.: I would prefer to have a more generic way to add support to build documentation
> for only one subsystem, but, as we also need to load an extra python module to be
> able to enable nitpick mode, I opted, for now, on not doing it too generic. We can rework
> on it later, as other subsystems would need a similar feature.

Within my POC I called it "books", may it is better to call them "sphinx-projects",
but for the first and the concept it should good enough to stay with "books" / see 
Makefile.reST [1] ::

# Sphinx projects, we call them *books* which is more common to authors.
BOOKS_FOLDER = $(obj)/Documentation
BOOKS=$(filter-out books/intro, \
    $(patsubst $(BOOKS_FOLDER)/%/conf.py,books/%,$(wildcard $(BOOKS_FOLDER)/*/conf.py)) \
    )

# fine grained targets
BOOKS_HTML = $(patsubst %,%.html, $(BOOKS))
BOOKS_CLEAN = $(patsubst %,%.clean, $(BOOKS))
BOOKS_MAN = $(patsubst %,%.man, $(BOOKS))
BOOKS_PDF = $(patsubst %,%.pdf, $(BOOKS))

[1] https://github.com/return42/sphkerneldoc/blob/master/doc/Makefile#L40

In words: A "book" (or sphinx-project) is a folder with a conf.py in
and the "folder-name" is the name of the related target ::

  Documentation/*/conf.py

And the selective targets are

  make books/[media|...].[html|man|pdf|clean|...]

By example, build only the html of media::

  make books/media.html

The integration into the main Makefile is generic by adding the
wildcard target "books%"

modified   Makefile
@@ -478,7 +478,7 @@ version_h := include/generated/uapi/linux/version.h
 old_version_h := include/linux/version.h
 
 no-dot-config-targets := clean mrproper distclean \
-			 cscope gtags TAGS tags help% %docs check% coccicheck \
+			 cscope gtags TAGS tags help% %docs books% check% coccicheck \
 			 $(version_h) headers_% archheaders archscripts \
 			 kernelversion %src-pkg
 
+# more fine grained documentation targets
+books/%:
+	$(Q)$(MAKE) $(build)=Documentation -f Documentation/Makefile.reST $@
+


If you are interested in, I prepare a patch series for the Makefiles and the
conf.py. Any suggestions about the prefix "books"? ... I preferred "books" over
"sphinx-project" or similar, because it is pregnant and short (less to type).

-- Markus --

> 
> 
> Markus Heiser (1):
>  doc-rst: support additional Sphinx build config override
> 
> Mauro Carvalho Chehab (2):
>  doc-rst: add an option to build media documentation in nitpick mode
>  doc-rst: remove a bogus comment from Documentation/index.rst
> 
> Documentation/Makefile.sphinx       | 10 ++++-
> Documentation/conf.py               |  9 ++++
> Documentation/index.rst             |  7 +--
> Documentation/media/conf_nitpick.py | 85 +++++++++++++++++++++++++++++++++++++
> Documentation/media/index.rst       | 12 ++++++
> Documentation/sphinx/load_config.py | 25 +++++++++++
> Makefile                            |  6 +++
> 7 files changed, 146 insertions(+), 8 deletions(-)
> create mode 100644 Documentation/media/conf_nitpick.py
> create mode 100644 Documentation/media/index.rst
> create mode 100644 Documentation/sphinx/load_config.py
> 
> -- 
> 2.7.4
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] Add a way to build only media docs

[Mauro Carvalho Chehab]

Em Sun, 7 Aug 2016 11:55:27 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> 
> Am 06.08.2016 um 14:00 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> 
> > Being able to build just the media docs is important for us due to several
> > reasons:
> > 
> > 1) Media developers community hosts a copy of the media documentation at linuxtv.org
> >    with the very latest  under development documents;
> > 
> > 2) Nitpicking to identify broken references is important to identify documentation gaps
> >    that need to be addressed on future releases;
> > 
> > 3) As media maintainers check patch per patch if a documentation gap is introduced,
> 
> 
> Take into account: the gap is detected by "open references" (from the reST-header
> files), but Sphinx tries to bind a ":ref:" to any anchor which fits (no matter where
> it comes from). I mean, if we add more and more content or e.g. interpshinx objects,
> we did not know which targets are available and the risk to get a *false bind* grows
> with the content.
> 
> I don't know, may be in some future you will come into trouble, when you want
> to refer targets outside of the ./media folder and *detect gaps* in the docs.
> 
> This need not be a problem, but you should be aware of, that your "test" is
> only a test on open/closed links.

Yeah, we'll still have some references outside the media system. There
are already a few of them at the kABI books, and to get mistakes there,
we'll need to use nitpick mode for the entire stuff, with can be a
nightmare, specially since right now we have 300+ missing references just
inside the media books. We need to address those first, before being able
to address the external ones.

> > building
> >    media documentation should be as fast as possible.
> > 
> > This patchset adds a media file adding nitpick support and an extra build target that will
> > compile only the media documentation. It also groups all media documentation into one
> > section on the main Kernel document, with is, IMHO, a good thing as we start adding more
> > stuff there.
> > 
> > Jon,
> > 
> > I'd love to see this patch merged early at the -rc cycle, in order to avoid merge
> > conflicts when people start converting other docbooks to Sphinx, as it touches
> > at the main Makefile and at the Sphinx common stuff. Also, as I'll need to patch my
> > build scripts to check for documentation issues with Sphinx, I need them on my
> > master branch, as otherwise my workflow will be broken until the next Kernel release.
> > 
> > So, If you're ok with this patch series, can you submit to Linus on early -rc? Or 
> > if you prefer, I can do it myself, with your ack.
> > 
> > Thanks!
> > Mauro
> > 
> > PS.: I would prefer to have a more generic way to add support to build documentation
> > for only one subsystem, but, as we also need to load an extra python module to be
> > able to enable nitpick mode, I opted, for now, on not doing it too generic. We can rework
> > on it later, as other subsystems would need a similar feature.
> 
> Within my POC I called it "books", may it is better to call them "sphinx-projects",
> but for the first and the concept it should good enough to stay with "books" / see 
> Makefile.reST [1] ::
> 
> # Sphinx projects, we call them *books* which is more common to authors.

The nomenclature can be very confusing, specially when we're grouping
different documents altogether.

The way I see, at the media subsystem, there are 4 books:
	- uAPI - with has internally 5 parts;
	- kAPI;
	- V4L drivers;
	- DVB drivers.

Yet, IMHO, the best is to put one master index.rst pointing to all of
them, so we actually have a media "book set", and just one "override"
conf.py for all of them. So, for now, I guess it is OK to group them 
altogether when checking for references, although in reality, at least 
in the first moment, I'm only interested on fixing bad references at
the uAPI and kAPI books. The other two books have lots of legacy stuff
that needs to be updated before even looking into any bad references there.

Yet, at the makefile, I don't mind calling the book set as "BOOK".


> BOOKS_FOLDER = $(obj)/Documentation
> BOOKS=$(filter-out books/intro, \
>     $(patsubst $(BOOKS_FOLDER)/%/conf.py,books/%,$(wildcard $(BOOKS_FOLDER)/*/conf.py)) \
>     )
> 
> # fine grained targets
> BOOKS_HTML = $(patsubst %,%.html, $(BOOKS))
> BOOKS_CLEAN = $(patsubst %,%.clean, $(BOOKS))
> BOOKS_MAN = $(patsubst %,%.man, $(BOOKS))
> BOOKS_PDF = $(patsubst %,%.pdf, $(BOOKS))
> 
> [1] https://github.com/return42/sphkerneldoc/blob/master/doc/Makefile#L40
> 
> In words: A "book" (or sphinx-project) is a folder with a conf.py in
> and the "folder-name" is the name of the related target ::
> 
>   Documentation/*/conf.py
> 
> And the selective targets are
> 
>   make books/[media|...].[html|man|pdf|clean|...]
> 
> By example, build only the html of media::
> 
>   make books/media.html
> 
> The integration into the main Makefile is generic by adding the
> wildcard target "books%"
> 
> modified   Makefile
> @@ -478,7 +478,7 @@ version_h := include/generated/uapi/linux/version.h
>  old_version_h := include/linux/version.h
>  
>  no-dot-config-targets := clean mrproper distclean \
> -			 cscope gtags TAGS tags help% %docs check% coccicheck \
> +			 cscope gtags TAGS tags help% %docs books% check% coccicheck \
>  			 $(version_h) headers_% archheaders archscripts \
>  			 kernelversion %src-pkg
>  
> +# more fine grained documentation targets
> +books/%:
> +	$(Q)$(MAKE) $(build)=Documentation -f Documentation/Makefile.reST $@
> +
> 
> 
> If you are interested in, I prepare a patch series for the Makefiles and the
> conf.py.

Interesting approach. That would work for me. Not sure what the others
think about that.

Yet, my understanding is that we should have just one conf.py
for the "all-in-one" book (e. g. Documentation/index.rst), plus 
additional ones to be used where we want to build standalone books 
with nitpick mode enabled.

So, we should keep having the
	make htmldocs

target that would build the all-in-one book, plus some other way to
build those books that would have a conf.py that would be adding the
extra nitpick options.

I really don't mind if we'll be compiling those with:
	make books/media.html

or with something like:
	make SPHINXDIRS="./media" htmldocs

Actually, the last syntax is better, IMHO, because people are already
familiar with something similar to that.

Provided that it would work and the patch would be simple enough to
be added on an early -rc Kernel (or alternatively, if it can be
sent via my master devel branch with Jon's ack).

> Any suggestions about the prefix "books"? ... I preferred "books" over
> "sphinx-project" or similar, because it is pregnant and short (less to type).
> 
> -- Markus --
> 
> > 
> > 
> > Markus Heiser (1):
> >  doc-rst: support additional Sphinx build config override
> > 
> > Mauro Carvalho Chehab (2):
> >  doc-rst: add an option to build media documentation in nitpick mode
> >  doc-rst: remove a bogus comment from Documentation/index.rst
> > 
> > Documentation/Makefile.sphinx       | 10 ++++-
> > Documentation/conf.py               |  9 ++++
> > Documentation/index.rst             |  7 +--
> > Documentation/media/conf_nitpick.py | 85 +++++++++++++++++++++++++++++++++++++
> > Documentation/media/index.rst       | 12 ++++++
> > Documentation/sphinx/load_config.py | 25 +++++++++++
> > Makefile                            |  6 +++
> > 7 files changed, 146 insertions(+), 8 deletions(-)
> > create mode 100644 Documentation/media/conf_nitpick.py
> > create mode 100644 Documentation/media/index.rst
> > create mode 100644 Documentation/sphinx/load_config.py
> > 
> > -- 
> > 2.7.4
> > 
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-media" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 



Thanks,
Mauro

Re: [PATCH 0/3] Add a way to build only media docs

[Markus Heiser]

Am 07.08.2016 um 14:38 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:

> Em Sun, 7 Aug 2016 11:55:27 +0200
> Markus Heiser <markus.heiser@darmarit.de> escreveu:
> 
>> 
>> Am 06.08.2016 um 14:00 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
>> 
>>> Being able to build just the media docs is important for us due to several
>>> reasons:
>>> 
>>> 1) Media developers community hosts a copy of the media documentation at linuxtv.org
>>>   with the very latest  under development documents;
>>> 
>>> 2) Nitpicking to identify broken references is important to identify documentation gaps
>>>   that need to be addressed on future releases;
>>> 
>>> 3) As media maintainers check patch per patch if a documentation gap is introduced,
>> 
>> 
>> Take into account: the gap is detected by "open references" (from the reST-header
>> files), but Sphinx tries to bind a ":ref:" to any anchor which fits (no matter where
>> it comes from). I mean, if we add more and more content or e.g. interpshinx objects,
>> we did not know which targets are available and the risk to get a *false bind* grows
>> with the content.
>> 
>> I don't know, may be in some future you will come into trouble, when you want
>> to refer targets outside of the ./media folder and *detect gaps* in the docs.
>> 
>> This need not be a problem, but you should be aware of, that your "test" is
>> only a test on open/closed links.
> 
> Yeah, we'll still have some references outside the media system. There
> are already a few of them at the kABI books, and to get mistakes there,
> we'll need to use nitpick mode for the entire stuff, with can be a
> nightmare, specially since right now we have 300+ missing references just
> inside the media books. We need to address those first, before being able
> to address the external ones.
> 
>>> building
>>>   media documentation should be as fast as possible.
>>> 
>>> This patchset adds a media file adding nitpick support and an extra build target that will
>>> compile only the media documentation. It also groups all media documentation into one
>>> section on the main Kernel document, with is, IMHO, a good thing as we start adding more
>>> stuff there.
>>> 
>>> Jon,
>>> 
>>> I'd love to see this patch merged early at the -rc cycle, in order to avoid merge
>>> conflicts when people start converting other docbooks to Sphinx, as it touches
>>> at the main Makefile and at the Sphinx common stuff. Also, as I'll need to patch my
>>> build scripts to check for documentation issues with Sphinx, I need them on my
>>> master branch, as otherwise my workflow will be broken until the next Kernel release.
>>> 
>>> So, If you're ok with this patch series, can you submit to Linus on early -rc? Or 
>>> if you prefer, I can do it myself, with your ack.
>>> 
>>> Thanks!
>>> Mauro
>>> 
>>> PS.: I would prefer to have a more generic way to add support to build documentation
>>> for only one subsystem, but, as we also need to load an extra python module to be
>>> able to enable nitpick mode, I opted, for now, on not doing it too generic. We can rework
>>> on it later, as other subsystems would need a similar feature.
>> 
>> Within my POC I called it "books", may it is better to call them "sphinx-projects",
>> but for the first and the concept it should good enough to stay with "books" / see 
>> Makefile.reST [1] ::
>> 
>> # Sphinx projects, we call them *books* which is more common to authors.
> 
> The nomenclature can be very confusing, specially when we're grouping
> different documents altogether.
> 
> The way I see, at the media subsystem, there are 4 books:
> 	- uAPI - with has internally 5 parts;
> 	- kAPI;
> 	- V4L drivers;
> 	- DVB drivers.
> 
> Yet, IMHO, the best is to put one master index.rst pointing to all of
> them, so we actually have a media "book set", and just one "override"
> conf.py for all of them. So, for now, I guess it is OK to group them 
> altogether when checking for references, although in reality, at least 
> in the first moment, I'm only interested on fixing bad references at
> the uAPI and kAPI books. The other two books have lots of legacy stuff
> that needs to be updated before even looking into any bad references there.
> 
> Yet, at the makefile, I don't mind calling the book set as "BOOK".
> 
> 
>> BOOKS_FOLDER = $(obj)/Documentation
>> BOOKS=$(filter-out books/intro, \
>>    $(patsubst $(BOOKS_FOLDER)/%/conf.py,books/%,$(wildcard $(BOOKS_FOLDER)/*/conf.py)) \
>>    )
>> 
>> # fine grained targets
>> BOOKS_HTML = $(patsubst %,%.html, $(BOOKS))
>> BOOKS_CLEAN = $(patsubst %,%.clean, $(BOOKS))
>> BOOKS_MAN = $(patsubst %,%.man, $(BOOKS))
>> BOOKS_PDF = $(patsubst %,%.pdf, $(BOOKS))
>> 
>> [1] https://github.com/return42/sphkerneldoc/blob/master/doc/Makefile#L40
>> 
>> In words: A "book" (or sphinx-project) is a folder with a conf.py in
>> and the "folder-name" is the name of the related target ::
>> 
>>  Documentation/*/conf.py
>> 
>> And the selective targets are
>> 
>>  make books/[media|...].[html|man|pdf|clean|...]
>> 
>> By example, build only the html of media::
>> 
>>  make books/media.html
>> 
>> The integration into the main Makefile is generic by adding the
>> wildcard target "books%"
>> 
>> modified   Makefile
>> @@ -478,7 +478,7 @@ version_h := include/generated/uapi/linux/version.h
>> old_version_h := include/linux/version.h
>> 
>> no-dot-config-targets := clean mrproper distclean \
>> -			 cscope gtags TAGS tags help% %docs check% coccicheck \
>> +			 cscope gtags TAGS tags help% %docs books% check% coccicheck \
>> 			 $(version_h) headers_% archheaders archscripts \
>> 			 kernelversion %src-pkg
>> 
>> +# more fine grained documentation targets
>> +books/%:
>> +	$(Q)$(MAKE) $(build)=Documentation -f Documentation/Makefile.reST $@
>> +
>> 
>> 
>> If you are interested in, I prepare a patch series for the Makefiles and the
>> conf.py.
> 
> Interesting approach. That would work for me. Not sure what the others
> think about that.
> 
> Yet, my understanding is that we should have just one conf.py
> for the "all-in-one" book (e. g. Documentation/index.rst), plus 
> additional ones to be used where we want to build standalone books 
> with nitpick mode enabled.
> 
> So, we should keep having the
> 	make htmldocs
> 
> target that would build the all-in-one book, plus some other way to
> build those books that would have a conf.py that would be adding the
> extra nitpick options.
> 
> I really don't mind if we'll be compiling those with:
> 	make books/media.html
> 
> or with something like:
> 	make SPHINXDIRS="./media" htmldocs
> 
> Actually, the last syntax is better, IMHO, because people are already
> familiar with something similar to that.
> 
> Provided that it would work and the patch would be simple enough to
> be added on an early -rc Kernel (or alternatively, if it can be
> sent via my master devel branch with Jon's ack).

Hi Mauro,

I agree with all, nevertheless I think, building a partial product should
be a make target. Anyway, I prepare a patch (with SPHINXDIRS="./media") 
on top of your patch series.

-- Markus --