[docs-next PATCH v2 00/11] Documentation/sphinx follow-up, again

[docs-next PATCH v2 00/11] Documentation/sphinx follow-up, again

[Jani Nikula]

The first 9 patches are just a resend of [1], there's two patches on top
here.

Pull details below.

BR,
Jani.

[1] http://mid.gmane.org/cover.1465567017.git.jani.nikula@intel.com


The following changes since commit 8569de68e79e94cce6709831edd94accb6942ade:

  docs: kernel-doc: Add "example" and "note" to the magic section types (2016-06-09 13:35:05 -0600)

are available in the git repository at:

  git://people.freedesktop.org/~jani/drm sphinx-for-docs-next

for you to fetch changes up to ebc88ef05c825024a5d95285459b8c842c095c0f:

  Documentation: add top level 'make help' output for Sphinx (2016-06-23 15:11:51 +0300)

----------------------------------------------------------------
Jani Nikula (11):
      kernel-doc: remove old debug cruft from dump_section()
      kernel-doc: do not warn about duplicate default section names
      kernel-doc: add missing semi-colons in option parsing
      kernel-doc: abstract filename mapping
      kernel-doc: add support for specifying extra files for EXPORT_SYMBOLs
      kernel-doc: unify all EXPORT_SYMBOL scanning to one place
      Documentation/sphinx: remove unnecessary temporary variable
      Documentation/sphinx: use a more sensible string split in kernel-doc extension
      Documentation/sphinx: add support for specifying extra export files
      Documentation/sphinx: drop modindex, we don't have python modules
      Documentation: add top level 'make help' output for Sphinx

 Documentation/DocBook/Makefile     |  4 +-
 Documentation/Makefile.sphinx      |  8 ++++
 Documentation/index.rst            |  2 -
 Documentation/sphinx/kernel-doc.py | 18 +++++---
 Makefile                           |  2 +
 scripts/kernel-doc                 | 88 +++++++++++++++++++++++++++-----------
 6 files changed, 87 insertions(+), 35 deletions(-)


-- 
2.1.4

[docs-next PATCH v2 01/11] kernel-doc: remove old debug cruft from dump_section()

[Jani Nikula]

No functional changes.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 27757c21551a..ac18eb5ed776 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -529,21 +529,18 @@ sub dump_section {
     my $contents = join "\n", @_;
 
     if ($name =~ m/$type_param/) {
-#	print STDERR "parameter def '$1' = '$contents'\n";
 	$name = $1;
 	$parameterdescs{$name} = $contents;
 	$sectcheck = $sectcheck . $name . " ";
         $parameterdesc_start_lines{$name} = $new_start_line;
         $new_start_line = 0;
     } elsif ($name eq "@\.\.\.") {
-#	print STDERR "parameter def '...' = '$contents'\n";
 	$name = "...";
 	$parameterdescs{$name} = $contents;
 	$sectcheck = $sectcheck . $name . " ";
         $parameterdesc_start_lines{$name} = $new_start_line;
         $new_start_line = 0;
     } else {
-#	print STDERR "other section '$name' = '$contents'\n";
 	if (defined($sections{$name}) && ($sections{$name} ne "")) {
 	    print STDERR "${file}:$.: warning: duplicate section name '$name'\n";
 	    ++$warnings;
-- 
2.1.4

[docs-next PATCH v2 02/11] kernel-doc: do not warn about duplicate default section names

[Jani Nikula]

Since

commit 32217761ee9db0215350dfe1ca4e66f312fb8c54
Author: Jani Nikula <jani.nikula@intel.com>
Date:   Sun May 29 09:40:44 2016 +0300

    kernel-doc: concatenate contents of colliding sections

we started getting (more) errors on duplicate section names, especially
on the default section name "Description":

include/net/mac80211.h:3174: warning: duplicate section name 'Description'

This is usually caused by a slightly unorthodox placement of parameter
descriptions, like in the above case, and kernel-doc resetting back to
the default section more than once within a kernel-doc comment.

Ignore warnings on the duplicate section name automatically assigned by
kernel-doc, and only consider explicitly user assigned duplicate section
names an issue.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index ac18eb5ed776..710615f3a4ff 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -542,8 +542,11 @@ sub dump_section {
         $new_start_line = 0;
     } else {
 	if (defined($sections{$name}) && ($sections{$name} ne "")) {
-	    print STDERR "${file}:$.: warning: duplicate section name '$name'\n";
-	    ++$warnings;
+	    # Only warn on user specified duplicate section names.
+	    if ($name ne $section_default) {
+		print STDERR "${file}:$.: warning: duplicate section name '$name'\n";
+		++$warnings;
+	    }
 	    $sections{$name} .= $contents;
 	} else {
 	    $sections{$name} = $contents;
-- 
2.1.4

[docs-next PATCH v2 03/11] kernel-doc: add missing semi-colons in option parsing

[Jani Nikula]

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 710615f3a4ff..a6f82c812c15 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -484,10 +484,10 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	$function_table{$function} = 1;
     } elsif ($cmd eq "-export") { # only exported symbols
 	$output_selection = OUTPUT_EXPORTED;
-	%function_table = ()
+	%function_table = ();
     } elsif ($cmd eq "-internal") { # only non-exported symbols
 	$output_selection = OUTPUT_INTERNAL;
-	%function_table = ()
+	%function_table = ();
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
-- 
2.1.4

[docs-next PATCH v2 04/11] kernel-doc: abstract filename mapping

[Jani Nikula]

Reduce duplication in follow-up work. No functional changes.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index a6f82c812c15..516d95fcefb7 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -2730,26 +2730,35 @@ sub local_unescape($) {
 	return $text;
 }
 
-sub process_file($) {
+sub map_filename($) {
     my $file;
-    my $identifier;
-    my $func;
-    my $descr;
-    my $in_purpose = 0;
-    my $initial_section_counter = $section_counter;
     my ($orig_file) = @_;
-    my $leading_space;
 
     if (defined($ENV{'SRCTREE'})) {
 	$file = "$ENV{'SRCTREE'}" . "/" . $orig_file;
-    }
-    else {
+    } else {
 	$file = $orig_file;
     }
+
     if (defined($source_map{$file})) {
 	$file = $source_map{$file};
     }
 
+    return $file;
+}
+
+sub process_file($) {
+    my $file;
+    my $identifier;
+    my $func;
+    my $descr;
+    my $in_purpose = 0;
+    my $initial_section_counter = $section_counter;
+    my ($orig_file) = @_;
+    my $leading_space;
+
+    $file = map_filename($orig_file);
+
     if (!open(IN,"<$file")) {
 	print STDERR "Error: Cannot open file $file\n";
 	++$errors;
-- 
2.1.4

[docs-next PATCH v2 05/11] kernel-doc: add support for specifying extra files for EXPORT_SYMBOLs

[Jani Nikula]

If the kernel-doc comments for functions are not in the same file as the
EXPORT_SYMBOL statements, the -export and -internal output selections do
not work as expected. This is typically the case when the kernel-doc
comments are in header files next to the function declarations and the
EXPORT_SYMBOL statements are next to the function definitions in the
source files.

Let the user specify additional source files in which to look for the
EXPORT_SYMBOLs using the new -export-file FILE option, which may be
given multiple times.

The pathological example for this is include/net/mac80211.h, which has
all the kernel-doc documentation for the exported functions defined in a
plethora of source files net/mac80211/*.c.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 39 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 37 insertions(+), 2 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 516d95fcefb7..9708a87c7069 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -61,10 +61,10 @@ Output format selection (mutually exclusive):
 Output selection (mutually exclusive):
   -export		Only output documentation for symbols that have been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-			in the same FILE.
+                        in the same FILE or any -export-file FILE.
   -internal		Only output documentation for symbols that have NOT been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-			in the same FILE.
+                        in the same FILE or any -export-file FILE.
   -function NAME	Only output documentation for the given function(s)
 			or DOC: section title(s). All other functions and DOC:
 			sections are ignored. May be specified multiple times.
@@ -76,6 +76,9 @@ Output selection modifiers:
   -no-doc-sections	Do not output DOC: sections.
   -enable-lineno        Enable output of #define LINENO lines. Only works with
                         reStructuredText format.
+  -export-file FILE     Specify an additional FILE in which to look for
+                        EXPORT_SYMBOL() and EXPORT_SYMBOL_GPL(). To be used with
+                        -export or -internal. May be specified multiple times.
 
 Other parameters:
   -v			Verbose output, more warnings and other information.
@@ -336,6 +339,8 @@ use constant {
 my $output_selection = OUTPUT_ALL;
 my $show_not_found = 0;
 
+my @export_file_list;
+
 my @build_time;
 if (defined($ENV{'KBUILD_BUILD_TIMESTAMP'}) &&
     (my $seconds = `date -d"${ENV{'KBUILD_BUILD_TIMESTAMP'}}" +%s`) ne '') {
@@ -488,6 +493,9 @@ while ($ARGV[0] =~ m/^-(.*)/) {
     } elsif ($cmd eq "-internal") { # only non-exported symbols
 	$output_selection = OUTPUT_INTERNAL;
 	%function_table = ();
+    } elsif ($cmd eq "-export-file") {
+	my $file = shift @ARGV;
+	push(@export_file_list, $file);
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
@@ -2747,6 +2755,25 @@ sub map_filename($) {
     return $file;
 }
 
+sub process_export_file($) {
+    my ($orig_file) = @_;
+    my $file = map_filename($orig_file);
+
+    if (!open(IN,"<$file")) {
+	print STDERR "Error: Cannot open file $file\n";
+	++$errors;
+	return;
+    }
+
+    while (<IN>) {
+	if (/$export_symbol/) {
+	    $function_table{$2} = 1;
+	}
+    }
+
+    close(IN);
+}
+
 sub process_file($) {
     my $file;
     my $identifier;
@@ -3081,6 +3108,14 @@ if (open(SOURCE_MAP, "<.tmp_filelist.txt")) {
 	close(SOURCE_MAP);
 }
 
+if ($output_selection == OUTPUT_EXPORTED ||
+    $output_selection == OUTPUT_INTERNAL) {
+    foreach (@export_file_list) {
+	chomp;
+	process_export_file($_);
+    }
+}
+
 foreach (@ARGV) {
     chomp;
     process_file($_);
-- 
2.1.4

[docs-next PATCH v2 06/11] kernel-doc: unify all EXPORT_SYMBOL scanning to one place

[Jani Nikula]

Scan all input files for EXPORT_SYMBOLs along with the explicitly
specified export files before actually parsing anything.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 18 +++++-------------
 1 file changed, 5 insertions(+), 13 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 9708a87c7069..932b3f34ff06 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -61,10 +61,10 @@ Output format selection (mutually exclusive):
 Output selection (mutually exclusive):
   -export		Only output documentation for symbols that have been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-                        in the same FILE or any -export-file FILE.
+                        in any input FILE or -export-file FILE.
   -internal		Only output documentation for symbols that have NOT been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-                        in the same FILE or any -export-file FILE.
+                        in any input FILE or -export-file FILE.
   -function NAME	Only output documentation for the given function(s)
 			or DOC: section title(s). All other functions and DOC:
 			sections are ignored. May be specified multiple times.
@@ -2792,17 +2792,6 @@ sub process_file($) {
 	return;
     }
 
-    # two passes for -export and -internal
-    if ($output_selection == OUTPUT_EXPORTED ||
-	$output_selection == OUTPUT_INTERNAL) {
-	while (<IN>) {
-	    if (/$export_symbol/o) {
-		$function_table{$2} = 1;
-	    }
-	}
-	seek(IN, 0, 0);
-    }
-
     $. = 1;
 
     $section_counter = 0;
@@ -3110,6 +3099,9 @@ if (open(SOURCE_MAP, "<.tmp_filelist.txt")) {
 
 if ($output_selection == OUTPUT_EXPORTED ||
     $output_selection == OUTPUT_INTERNAL) {
+
+    push(@export_file_list, @ARGV);
+
     foreach (@export_file_list) {
 	chomp;
 	process_export_file($_);
-- 
2.1.4

[docs-next PATCH v2 07/11] Documentation/sphinx: remove unnecessary temporary variable

[Jani Nikula]

Leftover cruft. No functional changes.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/sphinx/kernel-doc.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/Documentation/sphinx/kernel-doc.py b/Documentation/sphinx/kernel-doc.py
index 4adfb0e91ecc..2856376cc62c 100644
--- a/Documentation/sphinx/kernel-doc.py
+++ b/Documentation/sphinx/kernel-doc.py
@@ -59,7 +59,6 @@ class KernelDocDirective(Directive):
         env.note_dependency(os.path.abspath(filename))
 
         tab_width = self.options.get('tab-width', self.state.document.settings.tab_width)
-        source = filename
 
         # FIXME: make this nicer and more robust against errors
         if 'export' in self.options:
@@ -105,7 +104,7 @@ class KernelDocDirective(Directive):
                     lineoffset = int(match.group(1)) - 1
                     # we must eat our comments since the upset the markup
                 else:
-                    result.append(line, source, lineoffset)
+                    result.append(line, filename, lineoffset)
                     lineoffset += 1
 
             node = nodes.section()
-- 
2.1.4

[docs-next PATCH v2 08/11] Documentation/sphinx: use a more sensible string split in kernel-doc extension

[Jani Nikula]

Using the default str.split doesn't return empty strings like the
current version does.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/sphinx/kernel-doc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/sphinx/kernel-doc.py b/Documentation/sphinx/kernel-doc.py
index 2856376cc62c..d6a76f9a0ba5 100644
--- a/Documentation/sphinx/kernel-doc.py
+++ b/Documentation/sphinx/kernel-doc.py
@@ -68,7 +68,7 @@ class KernelDocDirective(Directive):
         elif 'doc' in self.options:
             cmd += ['-function', str(self.options.get('doc'))]
         elif 'functions' in self.options:
-            for f in str(self.options.get('functions')).split(' '):
+            for f in str(self.options.get('functions')).split():
                 cmd += ['-function', f]
 
         cmd += [filename]
-- 
2.1.4

[docs-next PATCH v2 09/11] Documentation/sphinx: add support for specifying extra export files

[Jani Nikula]

Let the user specify file patterns where to look for the EXPORT_SYMBOLs
in addition to the file with kernel-doc comments. This is directly based
on the -export-file FILE option added to kernel-doc in "kernel-doc: add
support for specifying extra files for EXPORT_SYMBOLs", but we extend
that with globbing patterns in the Sphinx extension.

The file patterns are added as options to the :export: and :internal:
arguments of the kernel-doc directive. For example, to extract the
documentation of exported functions from include/net/mac80211.h:

.. kernel-doc:: include/net/mac80211.h
   :export: net/mac80211/*.c

Without the file pattern, no exported functions would be found, as the
EXPORT_SYMBOLs are placed in the various source files under
net/mac80211.

The matched files are also added as dependencies on the document in
Sphinx, as they may affect the output. This is one of the reasons to do
the globbing in the Sphinx extension instead of in scripts/kernel-doc.

The file pattern remains optional, and is not needed if the kernel-doc
comments and EXPORT_SYMBOLs are placed in the source file passed in as
the main argument to the kernel-doc directive. This is the most common
case across the kernel source tree.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/sphinx/kernel-doc.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/Documentation/sphinx/kernel-doc.py b/Documentation/sphinx/kernel-doc.py
index d6a76f9a0ba5..dedb24e3bb7a 100644
--- a/Documentation/sphinx/kernel-doc.py
+++ b/Documentation/sphinx/kernel-doc.py
@@ -31,6 +31,7 @@ import os
 import subprocess
 import sys
 import re
+import glob
 
 from docutils import nodes, statemachine
 from docutils.statemachine import ViewList
@@ -44,8 +45,8 @@ class KernelDocDirective(Directive):
     option_spec = {
         'doc': directives.unchanged_required,
         'functions': directives.unchanged_required,
-        'export': directives.flag,
-        'internal': directives.flag,
+        'export': directives.unchanged,
+        'internal': directives.unchanged,
     }
     has_content = False
 
@@ -54,6 +55,7 @@ class KernelDocDirective(Directive):
         cmd = [env.config.kerneldoc_bin, '-rst', '-enable-lineno']
 
         filename = env.config.kerneldoc_srctree + '/' + self.arguments[0]
+        export_file_patterns = []
 
         # Tell sphinx of the dependency
         env.note_dependency(os.path.abspath(filename))
@@ -63,14 +65,21 @@ class KernelDocDirective(Directive):
         # FIXME: make this nicer and more robust against errors
         if 'export' in self.options:
             cmd += ['-export']
+            export_file_patterns = str(self.options.get('export')).split()
         elif 'internal' in self.options:
             cmd += ['-internal']
+            export_file_patterns = str(self.options.get('internal')).split()
         elif 'doc' in self.options:
             cmd += ['-function', str(self.options.get('doc'))]
         elif 'functions' in self.options:
             for f in str(self.options.get('functions')).split():
                 cmd += ['-function', f]
 
+        for pattern in export_file_patterns:
+            for f in glob.glob(env.config.kerneldoc_srctree + '/' + pattern):
+                env.note_dependency(os.path.abspath(f))
+                cmd += ['-export-file', f]
+
         cmd += [filename]
 
         try:
-- 
2.1.4

[docs-next PATCH v2 10/11] Documentation/sphinx: drop modindex, we don't have python modules

[Jani Nikula]

The modindex is for python modules.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/index.rst | 2 --
 1 file changed, 2 deletions(-)

diff --git a/Documentation/index.rst b/Documentation/index.rst
index 71a276f34c7f..1139a7ea39ec 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -18,6 +18,4 @@ Indices and tables
 ==================
 
 * :ref:`genindex`
-* :ref:`modindex`
 * :ref:`search`
-
-- 
2.1.4

[docs-next PATCH v2 11/11] Documentation: add top level 'make help' output for Sphinx

[Jani Nikula]

While there's slight overlap with the DocBook help now, this can stay
intact when the DocBook help goes away.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/DocBook/Makefile | 4 ++--
 Documentation/Makefile.sphinx  | 8 ++++++++
 Makefile                       | 2 ++
 3 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile
index e0c7e1e0590b..496d4295ec38 100644
--- a/Documentation/DocBook/Makefile
+++ b/Documentation/DocBook/Makefile
@@ -218,7 +218,7 @@ silent_gen_xml = :
 ###
 # Help targets as used by the top-level makefile
 dochelp:
-	@echo  ' Linux kernel internal documentation in different formats:'
+	@echo  ' Linux kernel internal documentation in different formats (DocBook):'
 	@echo  '  htmldocs        - HTML'
 	@echo  '  pdfdocs         - PDF'
 	@echo  '  psdocs          - Postscript'
@@ -227,7 +227,7 @@ dochelp:
 	@echo  '  installmandocs  - install man pages generated by mandocs'
 	@echo  '  cleandocs       - clean all generated DocBook files'
 	@echo
-	@echo  'make DOCBOOKS="s1.xml s2.xml" [target] Generate only docs s1.xml s2.xml'
+	@echo  '  make DOCBOOKS="s1.xml s2.xml" [target] Generate only docs s1.xml s2.xml'
 	@echo  '  valid values for DOCBOOKS are: $(DOCBOOKS)'
 
 
diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index addf32309bc3..810c11f0d37f 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -60,4 +60,12 @@ installmandocs:
 cleandocs:
 	$(Q)rm -rf $(BUILDDIR)
 
+dochelp:
+	@echo  ' Linux kernel internal documentation in different formats (Sphinx):'
+	@echo  '  htmldocs        - HTML'
+	@echo  '  pdfdocs         - PDF'
+	@echo  '  epubdocs        - EPUB'
+	@echo  '  xmldocs         - XML'
+	@echo  '  cleandocs       - clean all generated files'
+
 endif # HAVE_SPHINX
diff --git a/Makefile b/Makefile
index 8908a51b895a..f7f393c40886 100644
--- a/Makefile
+++ b/Makefile
@@ -1364,6 +1364,8 @@ help:
 	@$(MAKE) $(build)=$(package-dir) help
 	@echo  ''
 	@echo  'Documentation targets:'
+	@$(MAKE) -f $(srctree)/Documentation/Makefile.sphinx dochelp
+	@echo  ''
 	@$(MAKE) -f $(srctree)/Documentation/DocBook/Makefile dochelp
 	@echo  ''
 	@echo  'Architecture specific targets ($(SRCARCH)):'
-- 
2.1.4

Re: [docs-next PATCH v2 00/11] Documentation/sphinx follow-up, again

[Jonathan Corbet]

On Thu, 23 Jun 2016 15:28:00 +0300
Jani Nikula <jani.nikula@intel.com> wrote:

> The first 9 patches are just a resend of [1], there's two patches on top
> here.

OK, these all look good to me.  Applied to the docs tree, sorry for the
delay.

jon