[PATCH 0/6] kernel-doc: add DOC: sections support

[PATCH 0/6] kernel-doc: add DOC: sections support

[Randy Dunlap]

This patch series from Johannes Berg adds support for DOC: sections
that are embedded in source files.


-- 

[PATCH 6/6] kernel-doc: new P directive for DOC: sections

[Randy Dunlap]

From: Johannes Berg <johannes@sipsolutions.net>

The !P directive includes the contents of a DOC: section
given by title, e.g.
!Pfilename Title of the section

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>

---
 scripts/basic/docproc.c |   40 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 39 insertions(+), 1 deletion(-)

--- linux-2.6.23-git19.orig/scripts/basic/docproc.c
+++ linux-2.6.23-git19/scripts/basic/docproc.c
@@ -30,6 +30,7 @@
  *		!Ifilename
  *		!Dfilename
  *		!Ffilename
+ *		!Pfilename
  *
  */
 
@@ -57,6 +58,7 @@ FILEONLY *symbolsonly;
 typedef void FILELINE(char * file, char * line);
 FILELINE * singlefunctions;
 FILELINE * entity_system;
+FILELINE * docsection;
 
 #define MAXLINESZ     2048
 #define MAXFILES      250
@@ -289,12 +291,36 @@ void singfunc(char * filename, char * li
 }
 
 /*
+ * Insert specific documentation section from a file.
+ * Call kernel-doc with the following parameters:
+ * kernel-doc -docbook -function "doc section" filename
+ */
+void docsect(char *filename, char *line)
+{
+	char *vec[6]; /* kerneldoc -docbook -function "section" file NULL */
+	char *s;
+
+	for (s = line; *s; s++)
+		if (*s == '\n')
+			*s = '\0';
+
+	vec[0] = KERNELDOC;
+	vec[1] = DOCBOOK;
+	vec[2] = FUNCTION;
+	vec[3] = line;
+	vec[4] = filename;
+	vec[5] = NULL;
+	exec_kernel_doc(vec);
+}
+
+/*
  * Parse file, calling action specific functions for:
  * 1) Lines containing !E
  * 2) Lines containing !I
  * 3) Lines containing !D
  * 4) Lines containing !F
- * 5) Default lines - lines not matching the above
+ * 5) Lines containing !P
+ * 6) Default lines - lines not matching the above
  */
 void parse_file(FILE *infile)
 {
@@ -328,6 +354,15 @@ void parse_file(FILE *infile)
 						s++;
 					singlefunctions(line +2, s);
 					break;
+				case 'P':
+					/* filename */
+					while (*s && !isspace(*s)) s++;
+					*s++ = '\0';
+					/* DOC: section name */
+					while (isspace(*s))
+						s++;
+					docsection(line +2, s);
+					break;
 				default:
 					defaultline(line);
 			}
@@ -374,6 +409,7 @@ int main(int argc, char *argv[])
 		externalfunctions = find_export_symbols;
 		symbolsonly       = find_export_symbols;
 		singlefunctions   = noaction2;
+		docsection        = noaction2;
 		parse_file(infile);
 
 		/* Rewind to start from beginning of file again */
@@ -383,6 +419,7 @@ int main(int argc, char *argv[])
 		externalfunctions = extfunc;
 		symbolsonly       = printline;
 		singlefunctions   = singfunc;
+		docsection        = docsect;
 
 		parse_file(infile);
 	}
@@ -396,6 +433,7 @@ int main(int argc, char *argv[])
 		externalfunctions = adddep;
 		symbolsonly       = adddep;
 		singlefunctions   = adddep2;
+		docsection        = adddep2;
 		parse_file(infile);
 		printf("\n");
 	}


-- 

[PATCH 2/6] kernel-doc: init kernel version

[Randy Dunlap]

From: Johannes Berg <johannes@sipsolutions.net>

The kernel-doc script triggers a perl warning when invoked
without KERNELVERSION in the environment, rather make it use
the string "unknown kernel version" instead.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>

---
 scripts/kernel-doc |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- linux-2.6.23-git19.orig/scripts/kernel-doc
+++ linux-2.6.23-git19/scripts/kernel-doc
@@ -334,7 +334,7 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 
 # get kernel version from env
 sub get_kernel_version() {
-    my $version;
+    my $version = 'unknown kernel version';
 
     if (defined($ENV{'KERNELVERSION'})) {
 	$version = $ENV{'KERNELVERSION'};


-- 

[PATCH 4/6] kernel-doc: process functions, not DOC:

[Randy Dunlap]

From: Johannes Berg <johannes@sipsolutions.net>

This flag is necessary for the next patch for docproc to output
only the functions and not DOC: sections when a function list
is requested.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>

---
 scripts/kernel-doc |   14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

--- linux-2.6.23-git19.orig/scripts/kernel-doc
+++ linux-2.6.23-git19/scripts/kernel-doc
@@ -46,13 +46,16 @@ use strict;
 # Note: This only supports 'c'.
 
 # usage:
-# kernel-doc [ -docbook | -html | -text | -man ]
+# kernel-doc [ -docbook | -html | -text | -man ] [ -no-doc-sections ]
 #           [ -function funcname [ -function funcname ...] ] c file(s)s > outputfile
 # or
 #           [ -nofunction funcname [ -function funcname ...] ] c file(s)s > outputfile
 #
 #  Set output format using one of -docbook -html -text or -man.  Default is man.
 #
+#  -no-doc-sections
+#	Do not output DOC: sections
+#
 #  -function funcname
 #	If set, then only generate documentation for the given function(s) or
 #	DOC: section titles.  All other functions and DOC: sections are ignored.
@@ -211,7 +214,7 @@ my $blankline_text = "";
 
 
 sub usage {
-    print "Usage: $0 [ -v ] [ -docbook | -html | -text | -man ]\n";
+    print "Usage: $0 [ -v ] [ -docbook | -html | -text | -man ] [ -no-doc-sections ]\n";
     print "         [ -function funcname [ -function funcname ...] ]\n";
     print "         [ -nofunction funcname [ -nofunction funcname ...] ]\n";
     print "         c source file(s) > outputfile\n";
@@ -225,6 +228,7 @@ if ($#ARGV==-1) {
 
 my $verbose = 0;
 my $output_mode = "man";
+my $no_doc_sections = 0;
 my %highlights = %highlights_man;
 my $blankline = $blankline_man;
 my $modulename = "Kernel API";
@@ -329,6 +333,8 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	usage();
     } elsif ($cmd eq '-filelist') {
 	    $filelist = shift @ARGV;
+    } elsif ($cmd eq '-no-doc-sections') {
+	    $no_doc_sections = 1;
     }
 }
 
@@ -380,6 +386,10 @@ sub dump_doc_section {
     my $name = shift;
     my $contents = join "\n", @_;
 
+    if ($no_doc_sections) {
+	return;
+    }
+
     if (($function_only==0) ||
 	( $function_only == 1 && defined($function_table{$name})) ||
 	( $function_only == 2 && !defined($function_table{$name})))


-- 

[PATCH 1/6] kernel-doc: fix xml output mode

[Randy Dunlap]

From: Johannes Berg <johannes@sipsolutions.net>

After Randy's patch fixing the HTML output in DOC: sections
(6b5b55f6c404fa730a09a8254eb19f5a038afcc2) the same bug remained in XML
mode, this fixes it.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>

---
 scripts/kernel-doc |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

--- linux-2.6.23-git19.orig/scripts/kernel-doc
+++ linux-2.6.23-git19/scripts/kernel-doc
@@ -182,10 +182,10 @@ my $blankline_html = $local_lt . "p" . $
 my %highlights_xml = ( "([^=])\\\"([^\\\"<]+)\\\"", "\$1<quote>\$2</quote>",
 			$type_constant, "<constant>\$1</constant>",
 			$type_func, "<function>\$1</function>",
-			$type_struct, "<structname>\$1</structname>",
+			$type_struct_xml, "<structname>\$1</structname>",
 			$type_env, "<envar>\$1</envar>",
 			$type_param, "<parameter>\$1</parameter>" );
-my $blankline_xml = "</para><para>\n";
+my $blankline_xml = $local_lt . "/para" . $local_gt . $local_lt . "para" . $local_gt . "\n";
 
 # gnome, docbook format
 my %highlights_gnome = ( $type_constant, "<replaceable class=\"option\">\$1</replaceable>",
@@ -394,7 +394,7 @@ sub output_highlight {
 #	confess "output_highlight got called with no args?\n";
 #   }
 
-    if ($output_mode eq "html") {
+    if ($output_mode eq "html" || $output_mode eq "xml") {
 	$contents = local_unescape($contents);
 	# convert data read & converted thru xml_escape() into &xyz; format:
 	$contents =~ s/\\\\\\/&/g;


-- 

[PATCH 5/6] kernel-doc: use no-doc option

[Randy Dunlap]

From: Johannes Berg <johannes@sipsolutions.net>

When asked by a template to include all functions from a file,
it will also include DOC: sections wreaking havoc in the generated
docbook file. This patch makes it use the new -no-doc-sections
flag for kernel-doc to avoid this.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>

---
 scripts/basic/docproc.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

--- linux-2.6.23-git19.orig/scripts/basic/docproc.c
+++ linux-2.6.23-git19/scripts/basic/docproc.c
@@ -65,6 +65,7 @@ FILELINE * entity_system;
 #define DOCBOOK       "-docbook"
 #define FUNCTION      "-function"
 #define NOFUNCTION    "-nofunction"
+#define NODOCSECTIONS "-no-doc-sections"
 
 char *srctree;
 
@@ -231,13 +232,14 @@ void docfunctions(char * filename, char 
 
 	for (i=0; i <= symfilecnt; i++)
 		symcnt += symfilelist[i].symbolcnt;
-	vec = malloc((2 + 2 * symcnt + 2) * sizeof(char*));
+	vec = malloc((2 + 2 * symcnt + 3) * sizeof(char*));
 	if (vec == NULL) {
 		perror("docproc: ");
 		exit(1);
 	}
 	vec[idx++] = KERNELDOC;
 	vec[idx++] = DOCBOOK;
+	vec[idx++] = NODOCSECTIONS;
 	for (i=0; i < symfilecnt; i++) {
 		struct symfile * sym = &symfilelist[i];
 		for (j=0; j < sym->symbolcnt; j++) {


-- 

[PATCH 3/6] kernel-doc: single DOC: selection

[Randy Dunlap]

From: Johannes Berg <johannes@sipsolutions.net>

Currently, DOC: sections are always output even if only a single
function is requested, fix this and also make it possible to just
output a single DOC: section by giving its title as the function
name to output.

Also fixes docbook XML well-formedness for sections with examples.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Signed-off-by: Randy Dunlap <randy.dunlap@oracle.com>

---
 scripts/kernel-doc |   63 ++++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 43 insertions(+), 20 deletions(-)

--- linux-2.6.23-git19.orig/scripts/kernel-doc
+++ linux-2.6.23-git19/scripts/kernel-doc
@@ -54,13 +54,13 @@ use strict;
 #  Set output format using one of -docbook -html -text or -man.  Default is man.
 #
 #  -function funcname
-#	If set, then only generate documentation for the given function(s).  All
-#	other functions are ignored.
+#	If set, then only generate documentation for the given function(s) or
+#	DOC: section titles.  All other functions and DOC: sections are ignored.
 #
 #  -nofunction funcname
-#	If set, then only generate documentation for the other function(s).
-#	Cannot be used together with -function
-#	(yes, that's a bug -- perl hackers can fix it 8))
+#	If set, then only generate documentation for the other function(s)/DOC:
+#	sections. Cannot be used together with -function (yes, that's a bug --
+#	perl hackers can fix it 8))
 #
 #  c files - list of 'c' files to process
 #
@@ -374,6 +374,25 @@ sub dump_section {
 }
 
 ##
+# dump DOC: section after checking that it should go out
+#
+sub dump_doc_section {
+    my $name = shift;
+    my $contents = join "\n", @_;
+
+    if (($function_only == 0) ||
+	( $function_only == 1 && defined($function_table{$name})) ||
+	( $function_only == 2 && !defined($function_table{$name})))
+    {
+	dump_section $name, $contents;
+	output_blockhead({'sectionlist' => \@sectionlist,
+			  'sections' => \%sections,
+			  'module' => $modulename,
+			  'content-only' => ($function_only != 0), });
+    }
+}
+
+##
 # output function
 #
 # parameterdescs, a hash.
@@ -564,8 +583,8 @@ sub output_function_html(%) {
     print "<hr>\n";
 }
 
-# output intro in html
-sub output_intro_html(%) {
+# output DOC: block header in html
+sub output_blockhead_html(%) {
     my %args = %{$_[0]};
     my ($parameter, $section);
     my $count;
@@ -871,7 +890,7 @@ sub output_typedef_xml(%) {
 }
 
 # output in XML DocBook
-sub output_intro_xml(%) {
+sub output_blockhead_xml(%) {
     my %args = %{$_[0]};
     my ($parameter, $section);
     my $count;
@@ -882,15 +901,23 @@ sub output_intro_xml(%) {
     # print out each section
     $lineprefix="   ";
     foreach $section (@{$args{'sectionlist'}}) {
-	print "<refsect1>\n <title>$section</title>\n <para>\n";
+    	if (!$args{'content-only'}) {
+		print "<refsect1>\n <title>$section</title>\n";
+	}
 	if ($section =~ m/EXAMPLE/i) {
 	    print "<example><para>\n";
+	} else {
+	    print "<para>\n";
 	}
 	output_highlight($args{'sections'}{$section});
 	if ($section =~ m/EXAMPLE/i) {
 	    print "</para></example>\n";
+	} else {
+	    print "</para>";
+	}
+	if (!$args{'content-only'}) {
+		print "\n</refsect1>\n";
 	}
-	print " </para>\n</refsect1>\n";
     }
 
     print "\n\n";
@@ -1137,7 +1164,7 @@ sub output_typedef_man(%) {
     }
 }
 
-sub output_intro_man(%) {
+sub output_blockhead_man(%) {
     my %args = %{$_[0]};
     my ($parameter, $section);
     my $count;
@@ -1294,7 +1321,7 @@ sub output_struct_text(%) {
     output_section_text(@_);
 }
 
-sub output_intro_text(%) {
+sub output_blockhead_text(%) {
     my %args = %{$_[0]};
     my ($parameter, $section);
 
@@ -1325,9 +1352,9 @@ sub output_declaration {
 
 ##
 # generic output function - calls the right one based on current output mode.
-sub output_intro {
+sub output_blockhead {
     no strict 'refs';
-    my $func = "output_intro_".$output_mode;
+    my $func = "output_blockhead_".$output_mode;
     &$func(@_);
     $section_counter++;
 }
@@ -1926,9 +1953,7 @@ sub process_file($) {
 	} elsif ($state == 4) {
 		# Documentation block
 		if (/$doc_block/) {
-			dump_section($section, xml_escape($contents));
-			output_intro({'sectionlist' => \@sectionlist,
-				      'sections' => \%sections });
+			dump_doc_section($section, xml_escape($contents));
 			$contents = "";
 			$function = "";
 			%constants = ();
@@ -1946,9 +1971,7 @@ sub process_file($) {
 		}
 		elsif (/$doc_end/)
 		{
-			dump_section($section, xml_escape($contents));
-			output_intro({'sectionlist' => \@sectionlist,
-				      'sections' => \%sections });
+			dump_doc_section($section, xml_escape($contents));
 			$contents = "";
 			$function = "";
 			%constants = ();


-- 

Re: [PATCH 0/6] kernel-doc: add DOC: sections support

[Matt Mackall]

On Wed, Oct 24, 2007 at 03:08:48PM -0700, Randy Dunlap wrote:
> 
> This patch series from Johannes Berg adds support for DOC: sections
> that are embedded in source files.

What do these look like?

Kerneldoc's documentation is sadly lacking..

-- 
Mathematics is the supreme nostalgia of our time.

Re: [PATCH 0/6] kernel-doc: add DOC: sections support

[Randy Dunlap]

On Thu, 25 Oct 2007 12:29:32 -0500 Matt Mackall wrote:

> On Wed, Oct 24, 2007 at 03:08:48PM -0700, Randy Dunlap wrote:
> > 
> > This patch series from Johannes Berg adds support for DOC: sections
> > that are embedded in source files.
> 
> What do these look like?

There are currently 2 files in the kernel tree that contain DOC:
sections.  These are not used in the docbooks, but you can see what
they look like here:

./drivers/net/3c501.c: * DOC: 3c501 Card Notes
./drivers/net/3c501.c: * DOC: Problems
./drivers/net/3c527.c: * DOC: Traps for the unwary
./drivers/net/3c527.c: * DOC: Theory Of Operation
./drivers/net/3c527.c: * DOC: Notes

Also, Johannes has patches to use DOC: in mac80211 files.


> Kerneldoc's documentation is sadly lacking..

point taken.

---
~Randy

Re: [PATCH 0/6] kernel-doc: add DOC: sections support

[Matt Mackall]

On Thu, Oct 25, 2007 at 10:36:11AM -0700, Randy Dunlap wrote:
> On Thu, 25 Oct 2007 12:29:32 -0500 Matt Mackall wrote:
> 
> > On Wed, Oct 24, 2007 at 03:08:48PM -0700, Randy Dunlap wrote:
> > > 
> > > This patch series from Johannes Berg adds support for DOC: sections
> > > that are embedded in source files.
> > 
> > What do these look like?
> 
> There are currently 2 files in the kernel tree that contain DOC:
> sections.  These are not used in the docbooks, but you can see what
> they look like here:
> 
> ./drivers/net/3c501.c: * DOC: 3c501 Card Notes
> ./drivers/net/3c501.c: * DOC: Problems
> ./drivers/net/3c527.c: * DOC: Traps for the unwary
> ./drivers/net/3c527.c: * DOC: Theory Of Operation
> ./drivers/net/3c527.c: * DOC: Notes

Aside from those being about the least interesting drivers in the
tree, that's pretty neat.

The docbook infrastructure probably needs to figure out a way to pick
up this stuff automatically if it's not otherwise tied in.

With Python, I can do pydoc foo.py or pydoc foo/ and immediately get
browsable extracted docs in $PAGER for the associated file or
directory. Would be nice to get a little closer to that here.

-- 
Mathematics is the supreme nostalgia of our time.

Re: [PATCH 0/6] kernel-doc: add DOC: sections support

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 637 bytes --]


> > ./drivers/net/3c527.c: * DOC: Notes
> 
> Aside from those being about the least interesting drivers in the
> tree, that's pretty neat.
> 
> The docbook infrastructure probably needs to figure out a way to pick
> up this stuff automatically if it's not otherwise tied in.

The 5th and 6th patch in this series do that by allowing you to write

!Pfilename section title

in a docbook template file to insert the contents of such a section.
It's what I'm doing for mac80211's docs so that browsing the files gives
you the same documentation as reading the mac80211 docbook book just
organised differently.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 0/6] kernel-doc: add DOC: sections support

[Matt Mackall]

On Fri, Oct 26, 2007 at 12:48:17PM +0200, Johannes Berg wrote:
> 
> > > ./drivers/net/3c527.c: * DOC: Notes
> > 
> > Aside from those being about the least interesting drivers in the
> > tree, that's pretty neat.
> > 
> > The docbook infrastructure probably needs to figure out a way to pick
> > up this stuff automatically if it's not otherwise tied in.
> 
> The 5th and 6th patch in this series do that by allowing you to write
> 
> !Pfilename section title
> 
> in a docbook template file to insert the contents of such a section.
> It's what I'm doing for mac80211's docs so that browsing the files gives
> you the same documentation as reading the mac80211 docbook book just
> organised differently.

There are lots and lots of files that currently contain kerneldoc
comments that are unreferenced by docbook (and have been for years).
We should somehow sweep them all up into some sort of bucket
automatically.

-- 
Mathematics is the supreme nostalgia of our time.

Re: [PATCH 0/6] kernel-doc: add DOC: sections support

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 344 bytes --]


> There are lots and lots of files that currently contain kerneldoc
> comments that are unreferenced by docbook (and have been for years).
> We should somehow sweep them all up into some sort of bucket
> automatically.

That'd be another thing, but yes, it'd be nice as well. Not something
I'm aiming for right now though.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 0/6] kernel-doc: add DOC: sections support

[Sam Ravnborg]

On Wed, Oct 24, 2007 at 03:08:48PM -0700, Randy Dunlap wrote:
> 
> This patch series from Johannes Berg adds support for DOC: sections
> that are embedded in source files.

Thanks - added to kbuild.git after a few small fix-ups.
[Whitespace + the last bit did not apply].

	Sam