[docs-next PATCH 0/9] Documentation/sphinx follow-up

[docs-next PATCH 0/9] Documentation/sphinx follow-up

[Jani Nikula]

Hi Jon -

Thanks for merging the main Sphinx series! I greet you with another set
of patches on top. ;)

The main things here are reducing the noise on duplicate sections and
adding better support for extracting exported symbols when
EXPORT_SYMBOLs and kernel-doc are in separate files. And then there are
some drive-by cleanups and fixes on top.

I'm still working on rewriting Documentation/kernel-doc-nano-HOWTO.txt
to reflect all the changes, but didn't want to let these wait for that.

BR,
Jani.


PS. One wrinkle I spotted is that in the kernel-doc directive,
:functions: and :internal: also match structs and other types. This is
just kernel-doc the script shining its awesomeness through to the upper
layer; I just totally missed this earlier. I added :doc: to make the
distinction for documents, even though internally that uses kernel-doc
-function parameter as well.

Should we rename :functions: to, mmh, maybe :symbols: (for want of a
better word) while we still can, or leave that in as a historical
curiosity? I'm not sure there's much value in adding a separate :types:
(or something) for non-functions either? And should :internal: keep
returning non-functions as well? Or make that return just functions and
have another argument to get the types? Thoughts?


Jani Nikula (9):
  kernel-doc: remove old debug cruft from dump_section()
  kernel-doc: do not warn about duplicate default section names
  kernel-doc: add missing semi-colons in option parsing
  kernel-doc: abstract filename mapping
  kernel-doc: add support for specifying extra files for EXPORT_SYMBOLs
  kernel-doc: unify all EXPORT_SYMBOL scanning to one place
  Documentation/sphinx: remove unnecessary temporary variable
  Documentation/sphinx: use a more sensible string split in kernel-doc
    extension
  Documentation/sphinx: add support for specifying extra export files

 Documentation/sphinx/kernel-doc.py | 18 +++++---
 scripts/kernel-doc                 | 88 +++++++++++++++++++++++++++-----------
 2 files changed, 75 insertions(+), 31 deletions(-)

-- 
2.1.4

[docs-next PATCH 1/9] kernel-doc: remove old debug cruft from dump_section()

[Jani Nikula]

No functional changes.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 27757c21551a..ac18eb5ed776 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -529,21 +529,18 @@ sub dump_section {
     my $contents = join "\n", @_;
 
     if ($name =~ m/$type_param/) {
-#	print STDERR "parameter def '$1' = '$contents'\n";
 	$name = $1;
 	$parameterdescs{$name} = $contents;
 	$sectcheck = $sectcheck . $name . " ";
         $parameterdesc_start_lines{$name} = $new_start_line;
         $new_start_line = 0;
     } elsif ($name eq "@\.\.\.") {
-#	print STDERR "parameter def '...' = '$contents'\n";
 	$name = "...";
 	$parameterdescs{$name} = $contents;
 	$sectcheck = $sectcheck . $name . " ";
         $parameterdesc_start_lines{$name} = $new_start_line;
         $new_start_line = 0;
     } else {
-#	print STDERR "other section '$name' = '$contents'\n";
 	if (defined($sections{$name}) && ($sections{$name} ne "")) {
 	    print STDERR "${file}:$.: warning: duplicate section name '$name'\n";
 	    ++$warnings;
-- 
2.1.4

[docs-next PATCH 2/9] kernel-doc: do not warn about duplicate default section names

[Jani Nikula]

Since

commit 32217761ee9db0215350dfe1ca4e66f312fb8c54
Author: Jani Nikula <jani.nikula@intel.com>
Date:   Sun May 29 09:40:44 2016 +0300

    kernel-doc: concatenate contents of colliding sections

we started getting (more) errors on duplicate section names, especially
on the default section name "Description":

include/net/mac80211.h:3174: warning: duplicate section name 'Description'

This is usually caused by a slightly unorthodox placement of parameter
descriptions, like in the above case, and kernel-doc resetting back to
the default section more than once within a kernel-doc comment.

Ignore warnings on the duplicate section name automatically assigned by
kernel-doc, and only consider explicitly user assigned duplicate section
names an issue.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index ac18eb5ed776..710615f3a4ff 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -542,8 +542,11 @@ sub dump_section {
         $new_start_line = 0;
     } else {
 	if (defined($sections{$name}) && ($sections{$name} ne "")) {
-	    print STDERR "${file}:$.: warning: duplicate section name '$name'\n";
-	    ++$warnings;
+	    # Only warn on user specified duplicate section names.
+	    if ($name ne $section_default) {
+		print STDERR "${file}:$.: warning: duplicate section name '$name'\n";
+		++$warnings;
+	    }
 	    $sections{$name} .= $contents;
 	} else {
 	    $sections{$name} = $contents;
-- 
2.1.4

[docs-next PATCH 3/9] kernel-doc: add missing semi-colons in option parsing

[Jani Nikula]

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 710615f3a4ff..a6f82c812c15 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -484,10 +484,10 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	$function_table{$function} = 1;
     } elsif ($cmd eq "-export") { # only exported symbols
 	$output_selection = OUTPUT_EXPORTED;
-	%function_table = ()
+	%function_table = ();
     } elsif ($cmd eq "-internal") { # only non-exported symbols
 	$output_selection = OUTPUT_INTERNAL;
-	%function_table = ()
+	%function_table = ();
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
-- 
2.1.4

[docs-next PATCH 4/9] kernel-doc: abstract filename mapping

[Jani Nikula]

Reduce duplication in follow-up work. No functional changes.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index a6f82c812c15..516d95fcefb7 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -2730,26 +2730,35 @@ sub local_unescape($) {
 	return $text;
 }
 
-sub process_file($) {
+sub map_filename($) {
     my $file;
-    my $identifier;
-    my $func;
-    my $descr;
-    my $in_purpose = 0;
-    my $initial_section_counter = $section_counter;
     my ($orig_file) = @_;
-    my $leading_space;
 
     if (defined($ENV{'SRCTREE'})) {
 	$file = "$ENV{'SRCTREE'}" . "/" . $orig_file;
-    }
-    else {
+    } else {
 	$file = $orig_file;
     }
+
     if (defined($source_map{$file})) {
 	$file = $source_map{$file};
     }
 
+    return $file;
+}
+
+sub process_file($) {
+    my $file;
+    my $identifier;
+    my $func;
+    my $descr;
+    my $in_purpose = 0;
+    my $initial_section_counter = $section_counter;
+    my ($orig_file) = @_;
+    my $leading_space;
+
+    $file = map_filename($orig_file);
+
     if (!open(IN,"<$file")) {
 	print STDERR "Error: Cannot open file $file\n";
 	++$errors;
-- 
2.1.4

[docs-next PATCH 5/9] kernel-doc: add support for specifying extra files for EXPORT_SYMBOLs

[Jani Nikula]

If the kernel-doc comments for functions are not in the same file as the
EXPORT_SYMBOL statements, the -export and -internal output selections do
not work as expected. This is typically the case when the kernel-doc
comments are in header files next to the function declarations and the
EXPORT_SYMBOL statements are next to the function definitions in the
source files.

Let the user specify additional source files in which to look for the
EXPORT_SYMBOLs using the new -export-file FILE option, which may be
given multiple times.

The pathological example for this is include/net/mac80211.h, which has
all the kernel-doc documentation for the exported functions defined in a
plethora of source files net/mac80211/*.c.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 39 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 37 insertions(+), 2 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 516d95fcefb7..9708a87c7069 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -61,10 +61,10 @@ Output format selection (mutually exclusive):
 Output selection (mutually exclusive):
   -export		Only output documentation for symbols that have been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-			in the same FILE.
+                        in the same FILE or any -export-file FILE.
   -internal		Only output documentation for symbols that have NOT been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-			in the same FILE.
+                        in the same FILE or any -export-file FILE.
   -function NAME	Only output documentation for the given function(s)
 			or DOC: section title(s). All other functions and DOC:
 			sections are ignored. May be specified multiple times.
@@ -76,6 +76,9 @@ Output selection modifiers:
   -no-doc-sections	Do not output DOC: sections.
   -enable-lineno        Enable output of #define LINENO lines. Only works with
                         reStructuredText format.
+  -export-file FILE     Specify an additional FILE in which to look for
+                        EXPORT_SYMBOL() and EXPORT_SYMBOL_GPL(). To be used with
+                        -export or -internal. May be specified multiple times.
 
 Other parameters:
   -v			Verbose output, more warnings and other information.
@@ -336,6 +339,8 @@ use constant {
 my $output_selection = OUTPUT_ALL;
 my $show_not_found = 0;
 
+my @export_file_list;
+
 my @build_time;
 if (defined($ENV{'KBUILD_BUILD_TIMESTAMP'}) &&
     (my $seconds = `date -d"${ENV{'KBUILD_BUILD_TIMESTAMP'}}" +%s`) ne '') {
@@ -488,6 +493,9 @@ while ($ARGV[0] =~ m/^-(.*)/) {
     } elsif ($cmd eq "-internal") { # only non-exported symbols
 	$output_selection = OUTPUT_INTERNAL;
 	%function_table = ();
+    } elsif ($cmd eq "-export-file") {
+	my $file = shift @ARGV;
+	push(@export_file_list, $file);
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
@@ -2747,6 +2755,25 @@ sub map_filename($) {
     return $file;
 }
 
+sub process_export_file($) {
+    my ($orig_file) = @_;
+    my $file = map_filename($orig_file);
+
+    if (!open(IN,"<$file")) {
+	print STDERR "Error: Cannot open file $file\n";
+	++$errors;
+	return;
+    }
+
+    while (<IN>) {
+	if (/$export_symbol/) {
+	    $function_table{$2} = 1;
+	}
+    }
+
+    close(IN);
+}
+
 sub process_file($) {
     my $file;
     my $identifier;
@@ -3081,6 +3108,14 @@ if (open(SOURCE_MAP, "<.tmp_filelist.txt")) {
 	close(SOURCE_MAP);
 }
 
+if ($output_selection == OUTPUT_EXPORTED ||
+    $output_selection == OUTPUT_INTERNAL) {
+    foreach (@export_file_list) {
+	chomp;
+	process_export_file($_);
+    }
+}
+
 foreach (@ARGV) {
     chomp;
     process_file($_);
-- 
2.1.4

[docs-next PATCH 6/9] kernel-doc: unify all EXPORT_SYMBOL scanning to one place

[Jani Nikula]

Scan all input files for EXPORT_SYMBOLs along with the explicitly
specified export files before actually parsing anything.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 18 +++++-------------
 1 file changed, 5 insertions(+), 13 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 9708a87c7069..932b3f34ff06 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -61,10 +61,10 @@ Output format selection (mutually exclusive):
 Output selection (mutually exclusive):
   -export		Only output documentation for symbols that have been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-                        in the same FILE or any -export-file FILE.
+                        in any input FILE or -export-file FILE.
   -internal		Only output documentation for symbols that have NOT been
 			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
-                        in the same FILE or any -export-file FILE.
+                        in any input FILE or -export-file FILE.
   -function NAME	Only output documentation for the given function(s)
 			or DOC: section title(s). All other functions and DOC:
 			sections are ignored. May be specified multiple times.
@@ -2792,17 +2792,6 @@ sub process_file($) {
 	return;
     }
 
-    # two passes for -export and -internal
-    if ($output_selection == OUTPUT_EXPORTED ||
-	$output_selection == OUTPUT_INTERNAL) {
-	while (<IN>) {
-	    if (/$export_symbol/o) {
-		$function_table{$2} = 1;
-	    }
-	}
-	seek(IN, 0, 0);
-    }
-
     $. = 1;
 
     $section_counter = 0;
@@ -3110,6 +3099,9 @@ if (open(SOURCE_MAP, "<.tmp_filelist.txt")) {
 
 if ($output_selection == OUTPUT_EXPORTED ||
     $output_selection == OUTPUT_INTERNAL) {
+
+    push(@export_file_list, @ARGV);
+
     foreach (@export_file_list) {
 	chomp;
 	process_export_file($_);
-- 
2.1.4

[docs-next PATCH 7/9] Documentation/sphinx: remove unnecessary temporary variable

[Jani Nikula]

Leftover cruft. No functional changes.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/sphinx/kernel-doc.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/Documentation/sphinx/kernel-doc.py b/Documentation/sphinx/kernel-doc.py
index 4adfb0e91ecc..2856376cc62c 100644
--- a/Documentation/sphinx/kernel-doc.py
+++ b/Documentation/sphinx/kernel-doc.py
@@ -59,7 +59,6 @@ class KernelDocDirective(Directive):
         env.note_dependency(os.path.abspath(filename))
 
         tab_width = self.options.get('tab-width', self.state.document.settings.tab_width)
-        source = filename
 
         # FIXME: make this nicer and more robust against errors
         if 'export' in self.options:
@@ -105,7 +104,7 @@ class KernelDocDirective(Directive):
                     lineoffset = int(match.group(1)) - 1
                     # we must eat our comments since the upset the markup
                 else:
-                    result.append(line, source, lineoffset)
+                    result.append(line, filename, lineoffset)
                     lineoffset += 1
 
             node = nodes.section()
-- 
2.1.4

[docs-next PATCH 8/9] Documentation/sphinx: use a more sensible string split in kernel-doc extension

[Jani Nikula]

Using the default str.split doesn't return empty strings like the
current version does.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/sphinx/kernel-doc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/sphinx/kernel-doc.py b/Documentation/sphinx/kernel-doc.py
index 2856376cc62c..d6a76f9a0ba5 100644
--- a/Documentation/sphinx/kernel-doc.py
+++ b/Documentation/sphinx/kernel-doc.py
@@ -68,7 +68,7 @@ class KernelDocDirective(Directive):
         elif 'doc' in self.options:
             cmd += ['-function', str(self.options.get('doc'))]
         elif 'functions' in self.options:
-            for f in str(self.options.get('functions')).split(' '):
+            for f in str(self.options.get('functions')).split():
                 cmd += ['-function', f]
 
         cmd += [filename]
-- 
2.1.4

[docs-next PATCH 9/9] Documentation/sphinx: add support for specifying extra export files

[Jani Nikula]

Let the user specify file patterns where to look for the EXPORT_SYMBOLs
in addition to the file with kernel-doc comments. This is directly based
on the -export-file FILE option added to kernel-doc in "kernel-doc: add
support for specifying extra files for EXPORT_SYMBOLs", but we extend
that with globbing patterns in the Sphinx extension.

The file patterns are added as options to the :export: and :internal:
arguments of the kernel-doc directive. For example, to extract the
documentation of exported functions from include/net/mac80211.h:

.. kernel-doc:: include/net/mac80211.h
   :export: net/mac80211/*.c

Without the file pattern, no exported functions would be found, as the
EXPORT_SYMBOLs are placed in the various source files under
net/mac80211.

The matched files are also added as dependencies on the document in
Sphinx, as they may affect the output. This is one of the reasons to do
the globbing in the Sphinx extension instead of in scripts/kernel-doc.

The file pattern remains optional, and is not needed if the kernel-doc
comments and EXPORT_SYMBOLs are placed in the source file passed in as
the main argument to the kernel-doc directive. This is the most common
case across the kernel source tree.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/sphinx/kernel-doc.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/Documentation/sphinx/kernel-doc.py b/Documentation/sphinx/kernel-doc.py
index d6a76f9a0ba5..dedb24e3bb7a 100644
--- a/Documentation/sphinx/kernel-doc.py
+++ b/Documentation/sphinx/kernel-doc.py
@@ -31,6 +31,7 @@ import os
 import subprocess
 import sys
 import re
+import glob
 
 from docutils import nodes, statemachine
 from docutils.statemachine import ViewList
@@ -44,8 +45,8 @@ class KernelDocDirective(Directive):
     option_spec = {
         'doc': directives.unchanged_required,
         'functions': directives.unchanged_required,
-        'export': directives.flag,
-        'internal': directives.flag,
+        'export': directives.unchanged,
+        'internal': directives.unchanged,
     }
     has_content = False
 
@@ -54,6 +55,7 @@ class KernelDocDirective(Directive):
         cmd = [env.config.kerneldoc_bin, '-rst', '-enable-lineno']
 
         filename = env.config.kerneldoc_srctree + '/' + self.arguments[0]
+        export_file_patterns = []
 
         # Tell sphinx of the dependency
         env.note_dependency(os.path.abspath(filename))
@@ -63,14 +65,21 @@ class KernelDocDirective(Directive):
         # FIXME: make this nicer and more robust against errors
         if 'export' in self.options:
             cmd += ['-export']
+            export_file_patterns = str(self.options.get('export')).split()
         elif 'internal' in self.options:
             cmd += ['-internal']
+            export_file_patterns = str(self.options.get('internal')).split()
         elif 'doc' in self.options:
             cmd += ['-function', str(self.options.get('doc'))]
         elif 'functions' in self.options:
             for f in str(self.options.get('functions')).split():
                 cmd += ['-function', f]
 
+        for pattern in export_file_patterns:
+            for f in glob.glob(env.config.kerneldoc_srctree + '/' + pattern):
+                env.note_dependency(os.path.abspath(f))
+                cmd += ['-export-file', f]
+
         cmd += [filename]
 
         try:
-- 
2.1.4