[PATCH 0/6] scripts/kernel-doc: Kernel-doc improvements

[PATCH 0/6] scripts/kernel-doc: Kernel-doc improvements

[Danilo Cesar Lemes de Paula]

The following series contains:
 * kernel-doc: markdown support and improvements.
 * Fixing kernel-doc highlights.
 * Improve doc support for functions and structs with same name.
 * misc small fixes for drm docbook.

Signed-off-by: Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Johannes Berg <johannes.berg@intel.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: intel-gfx <intel-gfx@lists.freedesktop.org>
Cc: dri-devel <dri-devel@lists.freedesktop.org>

Danilo Cesar Lemes de Paula (6):
  scripts/kernel-doc: Replacing highlights hash by an array
  scripts/kernel-doc: Adding infrastructure for markdown support
  drm/doc: Convert to markdown
  drm/doc: Fixing xml documentation warning
  scripts/kernel-doc: Improve Markdown results
  scripts/kernel-doc: Processing -nofunc for functions only

 Documentation/DocBook/Makefile     |  25 +++--
 Documentation/DocBook/drm.tmpl     |  86 ---------------
 drivers/gpu/drm/drm_modes.c        |  12 +--
 drivers/gpu/drm/drm_modeset_lock.c |  14 ++-
 drivers/gpu/drm/drm_prime.c        |  16 ++-
 drivers/gpu/drm/i915/i915_reg.h    |  48 ++++-----
 include/drm/drm_modeset_lock.h     |  10 +-
 include/drm/drm_vma_manager.h      |  10 +-
 scripts/docproc.c                  |  54 +++++++---
 scripts/kernel-doc                 | 216 ++++++++++++++++++++++++++-----------
 10 files changed, 264 insertions(+), 227 deletions(-)

-- 
2.4.3

[PATCH 1/6] scripts/kernel-doc: Replacing highlights hash by an array

[Danilo Cesar Lemes de Paula]

The "highlight" code is very sensible to the order of the hash keys,
but the order of the keys cannot be predicted. It generates
faulty DocBook entries like:
	- @<function>device_for_each_child</function>

Sorting the result is not enough some times (as it's deterministic but
we can't control it).
We should use an array for that job, so we can guarantee that the order
of the regex execution on dohighlight is correct.

Signed-off-by: Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Stephan Mueller <smueller@chronox.de>
Cc: Michal Marek <mmarek@suse.cz>
Cc: linux-kernel@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: intel-gfx <intel-gfx@lists.freedesktop.org>
Cc: dri-devel <dri-devel@lists.freedesktop.org>
---
 scripts/kernel-doc | 104 ++++++++++++++++++++++++++++++-----------------------
 1 file changed, 60 insertions(+), 44 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 9a08fb5..0affe4f 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -206,59 +206,73 @@ my $type_env = '(\$\w+)';
 #  One for each output format
 
 # these work fairly well
-my %highlights_html = ( $type_constant, "<i>\$1</i>",
-			$type_func, "<b>\$1</b>",
-			$type_struct_xml, "<i>\$1</i>",
-			$type_env, "<b><i>\$1</i></b>",
-			$type_param, "<tt><b>\$1</b></tt>" );
+my @highlights_html = (
+                       [$type_constant, "<i>\$1</i>"],
+                       [$type_func, "<b>\$1</b>"],
+                       [$type_struct_xml, "<i>\$1</i>"],
+                       [$type_env, "<b><i>\$1</i></b>"],
+                       [$type_param, "<tt><b>\$1</b></tt>"]
+                      );
 my $local_lt = "\\\\\\\\lt:";
 my $local_gt = "\\\\\\\\gt:";
 my $blankline_html = $local_lt . "p" . $local_gt;	# was "<p>"
 
 # html version 5
-my %highlights_html5 = ( $type_constant, "<span class=\"const\">\$1</span>",
-			$type_func, "<span class=\"func\">\$1</span>",
-			$type_struct_xml, "<span class=\"struct\">\$1</span>",
-			$type_env, "<span class=\"env\">\$1</span>",
-			$type_param, "<span class=\"param\">\$1</span>" );
+my @highlights_html5 = (
+                        [$type_constant, "<span class=\"const\">\$1</span>"],
+                        [$type_func, "<span class=\"func\">\$1</span>"],
+                        [$type_struct_xml, "<span class=\"struct\">\$1</span>"],
+                        [$type_env, "<span class=\"env\">\$1</span>"],
+                        [$type_param, "<span class=\"param\">\$1</span>]"]
+		       );
 my $blankline_html5 = $local_lt . "br /" . $local_gt;
 
 # XML, docbook format
-my %highlights_xml = ( "([^=])\\\"([^\\\"<]+)\\\"", "\$1<quote>\$2</quote>",
-			$type_constant, "<constant>\$1</constant>",
-			$type_func, "<function>\$1</function>",
-			$type_struct_xml, "<structname>\$1</structname>",
-			$type_env, "<envar>\$1</envar>",
-			$type_param, "<parameter>\$1</parameter>" );
+my @highlights_xml = (
+                      ["([^=])\\\"([^\\\"<]+)\\\"", "\$1<quote>\$2</quote>"],
+                      [$type_constant, "<constant>\$1</constant>"],
+                      [$type_struct_xml, "<structname>\$1</structname>"],
+                      [$type_param, "<parameter>\$1</parameter>"],
+                      [$type_func, "<function>\$1</function>"],
+                      [$type_env, "<envar>\$1</envar>"]
+		     );
 my $blankline_xml = $local_lt . "/para" . $local_gt . $local_lt . "para" . $local_gt . "\n";
 
 # gnome, docbook format
-my %highlights_gnome = ( $type_constant, "<replaceable class=\"option\">\$1</replaceable>",
-			 $type_func, "<function>\$1</function>",
-			 $type_struct, "<structname>\$1</structname>",
-			 $type_env, "<envar>\$1</envar>",
-			 $type_param, "<parameter>\$1</parameter>" );
+my @highlights_gnome = (
+                        [$type_constant, "<replaceable class=\"option\">\$1</replaceable>"],
+                        [$type_func, "<function>\$1</function>"],
+                        [$type_struct, "<structname>\$1</structname>"],
+                        [$type_env, "<envar>\$1</envar>"],
+                        [$type_param, "<parameter>\$1</parameter>" ]
+		       );
 my $blankline_gnome = "</para><para>\n";
 
 # these are pretty rough
-my %highlights_man = ( $type_constant, "\$1",
-		       $type_func, "\\\\fB\$1\\\\fP",
-		       $type_struct, "\\\\fI\$1\\\\fP",
-		       $type_param, "\\\\fI\$1\\\\fP" );
+my @highlights_man = (
+                      [$type_constant, "\$1"],
+                      [$type_func, "\\\\fB\$1\\\\fP"],
+                      [$type_struct, "\\\\fI\$1\\\\fP"],
+                      [$type_param, "\\\\fI\$1\\\\fP"]
+		     );
 my $blankline_man = "";
 
 # text-mode
-my %highlights_text = ( $type_constant, "\$1",
-			$type_func, "\$1",
-			$type_struct, "\$1",
-			$type_param, "\$1" );
+my @highlights_text = (
+                       [$type_constant, "\$1"],
+                       [$type_func, "\$1"],
+                       [$type_struct, "\$1"],
+                       [$type_param, "\$1"]
+		      );
 my $blankline_text = "";
 
 # list mode
-my %highlights_list = ( $type_constant, "\$1",
-			$type_func, "\$1",
-			$type_struct, "\$1",
-			$type_param, "\$1" );
+my @highlights_list = (
+                       [$type_constant, "\$1"],
+                       [$type_func, "\$1"],
+                       [$type_struct, "\$1"],
+                       [$type_param, "\$1"]
+		      );
 my $blankline_list = "";
 
 # read arguments
@@ -273,7 +287,7 @@ my $verbose = 0;
 my $output_mode = "man";
 my $output_preformatted = 0;
 my $no_doc_sections = 0;
-my %highlights = %highlights_man;
+my @highlights = @highlights_man;
 my $blankline = $blankline_man;
 my $modulename = "Kernel API";
 my $function_only = 0;
@@ -374,31 +388,31 @@ while ($ARGV[0] =~ m/^-(.*)/) {
     my $cmd = shift @ARGV;
     if ($cmd eq "-html") {
 	$output_mode = "html";
-	%highlights = %highlights_html;
+	@highlights = @highlights_html;
 	$blankline = $blankline_html;
     } elsif ($cmd eq "-html5") {
 	$output_mode = "html5";
-	%highlights = %highlights_html5;
+	@highlights = @highlights_html5;
 	$blankline = $blankline_html5;
     } elsif ($cmd eq "-man") {
 	$output_mode = "man";
-	%highlights = %highlights_man;
+	@highlights = @highlights_man;
 	$blankline = $blankline_man;
     } elsif ($cmd eq "-text") {
 	$output_mode = "text";
-	%highlights = %highlights_text;
+	@highlights = @highlights_text;
 	$blankline = $blankline_text;
     } elsif ($cmd eq "-docbook") {
 	$output_mode = "xml";
-	%highlights = %highlights_xml;
+	@highlights = @highlights_xml;
 	$blankline = $blankline_xml;
     } elsif ($cmd eq "-list") {
 	$output_mode = "list";
-	%highlights = %highlights_list;
+	@highlights = @highlights_list;
 	$blankline = $blankline_list;
     } elsif ($cmd eq "-gnome") {
 	$output_mode = "gnome";
-	%highlights = %highlights_gnome;
+	@highlights = @highlights_gnome;
 	$blankline = $blankline_gnome;
     } elsif ($cmd eq "-module") { # not needed for XML, inherits from calling document
 	$modulename = shift @ARGV;
@@ -2671,9 +2685,11 @@ $kernelversion = get_kernel_version();
 
 # generate a sequence of code that will splice in highlighting information
 # using the s// operator.
-foreach my $pattern (sort keys %highlights) {
-#   print STDERR "scanning pattern:$pattern, highlight:($highlights{$pattern})\n";
-    $dohighlight .=  "\$contents =~ s:$pattern:$highlights{$pattern}:gs;\n";
+foreach my $k (keys @highlights) {
+    my $pattern = $highlights[$k][0];
+    my $result = $highlights[$k][1];
+#   print STDERR "scanning pattern:$pattern, highlight:($result)\n";
+    $dohighlight .=  "\$contents =~ s:$pattern:$result:gs;\n";
 }
 
 # Read the file that maps relative names to absolute names for
-- 
2.4.3

Re: [PATCH 1/6] scripts/kernel-doc: Replacing highlights hash by an array

[Jonathan Corbet]

On Mon,  7 Sep 2015 17:01:59 -0300
Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk> wrote:

> The "highlight" code is very sensible to the order of the hash keys,
> but the order of the keys cannot be predicted. It generates
> faulty DocBook entries like:
> 	- @<function>device_for_each_child</function>
> 
> Sorting the result is not enough some times (as it's deterministic but
> we can't control it).
> We should use an array for that job, so we can guarantee that the order
> of the regex execution on dohighlight is correct.

OK, I've spent a bunch of time with this, comparing the results before
and after.  The output you mention is clearly wrong, but there might be
room to differ over what the root cause is.

That output is caused by @device_for_each_child() in the comments.  This
happens for a few other functions as well, and I think it's wrong.  @ is
used to indicate parameters (or structure fields); I'm not sure why
people are using it for functions that are *not* one of the above.
Formatting the function names as a parameter doesn't seem right either.

There is the occasional case where the parameter *is* a function and the
text uses the () notation (threadfn(), for example).  Having the
"parameter" style win out over the "function" style in such cases is OK,
I guess, but it would be good to format the parentheses along with the
name.  The function patterns there now drop the parentheses entirely,
which seems a bit strange to me.

I guess I'll apply the patch; determinism is good, and it doesn't make
anything screwy that wasn't already that way.  But I think we should fix
the misapplied @s and sort out function formatting in general.  One of
these days...

Thanks,

jon

Re: [PATCH 1/6] scripts/kernel-doc: Replacing highlights hash by an array

[Lukas Wunner]

Hi,

On Sun, Sep 13, 2015 at 02:36:47PM -0600, Jonathan Corbet wrote:
> On Mon,  7 Sep 2015 17:01:59 -0300
> Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk> wrote:
> > The "highlight" code is very sensible to the order of the hash keys,
> > but the order of the keys cannot be predicted. It generates
> > faulty DocBook entries like:
> > 	- @<function>device_for_each_child</function>
> > 
> > Sorting the result is not enough some times (as it's deterministic but
> > we can't control it).
> > We should use an array for that job, so we can guarantee that the order
> > of the regex execution on dohighlight is correct.
> OK, I've spent a bunch of time with this, comparing the results before
> and after.  The output you mention is clearly wrong, but there might be
> room to differ over what the root cause is.
> 
> That output is caused by @device_for_each_child() in the comments.  This
> happens for a few other functions as well, and I think it's wrong.  @ is
> used to indicate parameters (or structure fields); I'm not sure why
> people are using it for functions that are *not* one of the above.
> Formatting the function names as a parameter doesn't seem right either.

Shouldn't kernel-doc print a warning for syntactic mistakes like this
rather than silently accomodating to it in whatever way?

As to the usage of markdown in general, there's documentation coming up
for vga_switcheroo which makes use of that so I'd love to see it merged:
https://github.com/l1k/linux/commit/d1476d748b5f1adf5bffe8e0a8bafad1e879d22f
https://github.com/l1k/linux/commit/11c55ae65788162970d8fa23cd1fd2518af55d34

The large set of dependencies pulled in on Fedora is likely to be blamed
on the RedHat packaging being notoriously coarse-grained. By comparison,
Debian is extremely fine-grained, kind of the opposite extreme, and
therefore has comparatively few prerequisites:
https://packages.debian.org/jessie/pandoc

I do agree however that alternative tools with fewer dependencies should
be supported and it would be great if the markdown patches were amended
to that end.

Best regards,

Lukas

[PATCH 2/6] scripts/kernel-doc: Adding infrastructure for markdown support

[Danilo Cesar Lemes de Paula]

Markdown support is given by calling an external tool, pandoc, for all
highlighted text on kernel-doc.

Pandoc converts Markdown text to proper Docbook tags, which will be
later translated to pdf, html or other targets.

This adds the capability of adding human-readle text highlight (bold,
underline, etc), bullet and numbered lists, simple tables, fixed-width
text (including asciiart), requiring minimal changes to current
documentation.

At this moment, pandoc is totally optional. Docbooks ready for markdown
should be added to the MARKDOWNREADY variable inside the Makefile. In
case the developer doesn't have pandoc installed, Make will throw a
warning and the documentation build will continue, generating
simple Documentation without the features brought by pandoc.

Signed-off-by: Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Stephan Mueller <smueller@chronox.de>
Cc: Michal Marek <mmarek@suse.cz>
Cc: linux-kernel@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: intel-gfx <intel-gfx@lists.freedesktop.org>
Cc: dri-devel <dri-devel@lists.freedesktop.org>
---
 Documentation/DocBook/Makefile | 25 +++++++++++-----
 scripts/docproc.c              | 54 ++++++++++++++++++++++++----------
 scripts/kernel-doc             | 66 ++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 119 insertions(+), 26 deletions(-)

diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile
index 93eff64..c1ff834 100644
--- a/Documentation/DocBook/Makefile
+++ b/Documentation/DocBook/Makefile
@@ -17,6 +17,8 @@ DOCBOOKS := z8530book.xml device-drivers.xml \
 	    tracepoint.xml drm.xml media_api.xml w1.xml \
 	    writing_musb_glue_layer.xml crypto-API.xml iio.xml
 
+MARKDOWNREADY := 
+
 include Documentation/DocBook/media/Makefile
 
 ###
@@ -81,18 +83,23 @@ XMLTOFLAGS += --skip-validation
 # The following rules are used to generate the .xml documentation
 # required to generate the final targets. (ps, pdf, html).
 quiet_cmd_docproc = DOCPROC $@
-      cmd_docproc = SRCTREE=$(srctree)/ $(DOCPROC) doc $< >$@
+      cmd_docproc = SRCTREE=$(srctree)/ $(DOCPROC) doc $<
 define rule_docproc
-	set -e;								\
-        $(if $($(quiet)cmd_$(1)),echo '  $($(quiet)cmd_$(1))';) 	\
-        $(cmd_$(1)); 							\
-        ( 								\
-          echo 'cmd_$@ := $(cmd_$(1))'; 				\
-          echo $@: `SRCTREE=$(srctree) $(DOCPROC) depend $<`; 		\
+	set -e;									\
+	USEMARKDOWN="";								\
+	FILE=`basename $@`;							\
+	[[ "$(MARKDOWNREADY)" =~ "$${FILE}" ]] && USEMARKDOWN="-use-markdown";	\
+        $(if $($(quiet)cmd_$(1)),echo '  $($(quiet)cmd_$(1))';) 		\
+        $(cmd_$(1)) $$USEMARKDOWN >$@;						\
+        ( 									\
+          echo 'cmd_$@ := $(cmd_$(1))';				 		\
+          echo $@: `SRCTREE=$(srctree) $(DOCPROC) depend $<`;			\
         ) > $(dir $@).$(notdir $@).cmd
 endef
 
 %.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE
+	@(which pandoc > /dev/null 2>&1) || \
+	(echo "*** To get propper documentation you need to install pandoc ***";)
 	$(call if_changed_rule,docproc)
 
 # Tell kbuild to always build the programs
@@ -103,6 +110,10 @@ notfoundtemplate = echo "*** You have to install docbook-utils or xmlto ***"; \
 db2xtemplate = db2TYPE -o $(dir $@) $<
 xmltotemplate = xmlto TYPE $(XMLTOFLAGS) -o $(dir $@) $<
 
+ifneq ($(shell which pandoc >/dev/null 2>&1 && echo found),found)
+	MARKDOWNREADY := "";
+endif
+
 # determine which methods are available
 ifeq ($(shell which db2ps >/dev/null 2>&1 && echo found),found)
 	use-db2x = db2x
diff --git a/scripts/docproc.c b/scripts/docproc.c
index e267e621..7c6b225 100644
--- a/scripts/docproc.c
+++ b/scripts/docproc.c
@@ -73,12 +73,15 @@ FILELINE * docsection;
 #define NOFUNCTION    "-nofunction"
 #define NODOCSECTIONS "-no-doc-sections"
 #define SHOWNOTFOUND  "-show-not-found"
+#define USEMARKDOWN   "-use-markdown"
 
 static char *srctree, *kernsrctree;
 
 static char **all_list = NULL;
 static int all_list_len = 0;
 
+static int use_markdown = 0;
+
 static void consume_symbol(const char *sym)
 {
 	int i;
@@ -95,10 +98,11 @@ static void consume_symbol(const char *sym)
 
 static void usage (void)
 {
-	fprintf(stderr, "Usage: docproc {doc|depend} file\n");
+	fprintf(stderr, "Usage: docproc {doc|depend} [-use-markdown] file\n");
 	fprintf(stderr, "Input is read from file.tmpl. Output is sent to stdout\n");
 	fprintf(stderr, "doc: frontend when generating kernel documentation\n");
 	fprintf(stderr, "depend: generate list of files referenced within file\n");
+	fprintf(stderr, "-use-markdown: pass -use-markdown to kernel-doc\n");
 	fprintf(stderr, "Environment variable SRCTREE: absolute path to sources.\n");
 	fprintf(stderr, "                     KBUILD_SRC: absolute path to kernel source tree.\n");
 }
@@ -257,12 +261,15 @@ static void docfunctions(char * filename, char * type)
 
 	for (i=0; i <= symfilecnt; i++)
 		symcnt += symfilelist[i].symbolcnt;
-	vec = malloc((2 + 2 * symcnt + 3) * sizeof(char *));
+	vec = malloc((2 + 2 * symcnt + 4) * sizeof(char *));
 	if (vec == NULL) {
 		perror("docproc: ");
 		exit(1);
 	}
 	vec[idx++] = KERNELDOC;
+	if (use_markdown)
+		vec[idx++] = USEMARKDOWN;
+
 	vec[idx++] = DOCBOOK;
 	vec[idx++] = NODOCSECTIONS;
 	for (i=0; i < symfilecnt; i++) {
@@ -294,6 +301,9 @@ static void singfunc(char * filename, char * line)
 	int i, idx = 0;
 	int startofsym = 1;
 	vec[idx++] = KERNELDOC;
+	if (use_markdown)
+		vec[idx++] = USEMARKDOWN;
+
 	vec[idx++] = DOCBOOK;
 	vec[idx++] = SHOWNOTFOUND;
 
@@ -328,8 +338,9 @@ static void singfunc(char * filename, char * line)
 static void docsect(char *filename, char *line)
 {
 	/* kerneldoc -docbook -show-not-found -function "section" file NULL */
-	char *vec[7];
+	char *vec[8];
 	char *s;
+	int idx = 0;
 
 	for (s = line; *s; s++)
 		if (*s == '\n')
@@ -342,30 +353,37 @@ static void docsect(char *filename, char *line)
 	consume_symbol(s);
 	free(s);
 
-	vec[0] = KERNELDOC;
-	vec[1] = DOCBOOK;
-	vec[2] = SHOWNOTFOUND;
-	vec[3] = FUNCTION;
-	vec[4] = line;
-	vec[5] = filename;
-	vec[6] = NULL;
+	vec[idx++] = KERNELDOC;
+	if (use_markdown)
+		vec[idx++] = USEMARKDOWN;
+
+	vec[idx++] = DOCBOOK;
+	vec[idx++] = SHOWNOTFOUND;
+	vec[idx++] = FUNCTION;
+	vec[idx++] = line;
+	vec[idx++] = filename;
+	vec[idx] = NULL;
 	exec_kernel_doc(vec);
 }
 
 static void find_all_symbols(char *filename)
 {
-	char *vec[4]; /* kerneldoc -list file NULL */
+	char *vec[5]; /* kerneldoc -list file NULL */
 	pid_t pid;
 	int ret, i, count, start;
 	char real_filename[PATH_MAX + 1];
 	int pipefd[2];
 	char *data, *str;
 	size_t data_len = 0;
+	int idx = 0;
+
+	vec[idx++] = KERNELDOC;
+	if (use_markdown)
+		vec[idx++] = USEMARKDOWN;
 
-	vec[0] = KERNELDOC;
-	vec[1] = LIST;
-	vec[2] = filename;
-	vec[3] = NULL;
+	vec[idx++] = LIST;
+	vec[idx++] = filename;
+	vec[idx] = NULL;
 
 	if (pipe(pipefd)) {
 		perror("pipe");
@@ -509,7 +527,7 @@ int main(int argc, char *argv[])
 	kernsrctree = getenv("KBUILD_SRC");
 	if (!kernsrctree || !*kernsrctree)
 		kernsrctree = srctree;
-	if (argc != 3) {
+	if (argc < 3 || argc > 4) {
 		usage();
 		exit(1);
 	}
@@ -521,6 +539,10 @@ int main(int argc, char *argv[])
 		exit(2);
 	}
 
+	if (argc == 4 && strcmp("-use-markdown", argv[3]) == 0) {
+		use_markdown = 1;
+	}
+
 	if (strcmp("doc", argv[1]) == 0) {
 		/* Need to do this in two passes.
 		 * First pass is used to collect all symbols exported
diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 0affe4f..5fe572a 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1,6 +1,7 @@
 #!/usr/bin/perl -w
 
 use strict;
+use IPC::Open3;
 
 ## Copyright (c) 1998 Michael Zucchi, All Rights Reserved        ##
 ## Copyright (C) 2000, 1  Tim Waugh <twaugh@redhat.com>          ##
@@ -282,6 +283,7 @@ if ($#ARGV == -1) {
 
 my $kernelversion;
 my $dohighlight = "";
+my $use_markdown = 0;
 
 my $verbose = 0;
 my $output_mode = "man";
@@ -424,6 +426,8 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	$function_only = 2;
 	$function = shift @ARGV;
 	$function_table{$function} = 1;
+    } elsif ($cmd eq "-use-markdown") {
+	$use_markdown = 1;
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
@@ -442,6 +446,7 @@ sub usage {
     print "         [ -no-doc-sections ]\n";
     print "         [ -function funcname [ -function funcname ...] ]\n";
     print "         [ -nofunction funcname [ -nofunction funcname ...] ]\n";
+    print "         [ -use-markdown ]\n";
     print "         [ -v ]\n";
     print "         c source file(s) > outputfile\n";
     print "         -v : verbose output, more warnings & other info listed\n";
@@ -515,6 +520,49 @@ sub dump_doc_section {
     }
 }
 
+sub markdown_to_docbook {
+	my $orig_content = $_[0];
+
+	my $pid = open3( \*CHLD_IN, \*CHLD_OUT, \*CHLD_ERR, "pandoc  --columns=80 -f markdown -t docbook" );
+
+	print CHLD_IN "$orig_content";
+	close(CHLD_IN);
+
+	waitpid($pid, 0);
+
+	my $content = "";
+	chomp(my @lines = <CHLD_OUT>);
+	foreach my $line (@lines) {
+		$content .= $line . "\n";
+	}
+	close(CHLD_OUT);
+	close(CHLD_ERR);
+
+	# pandoc insists in adding Main <para></para>, we should remove them.
+	$content =~ s:\A\s*<para>\s*\n(.*)\n</para>\Z$:$1:egsm;
+
+	return $content;
+}
+
+# Markdown->Docbook conversion by pandoc requires unescaped text
+# Kernel-doc converts every & to "&amp;", we need to convert it back.
+sub markdown_unescape
+{
+	my $text = shift;
+	my @lines = split /\n/, $text;
+
+	my @result;
+	foreach my $line (@lines) {
+		if ( $line =~ m /^    /s ) {
+			$line =~ s:\&amp;:\&:gs
+		}
+		push @result, $line;
+	}
+
+	return join "\n",@result;
+
+}
+
 ##
 # output function
 #
@@ -541,11 +589,19 @@ sub output_highlight {
 	$contents = local_unescape($contents);
 	# convert data read & converted thru xml_escape() into &xyz; format:
 	$contents =~ s/\\\\\\/\&/g;
+
+	if ($use_markdown) {
+		$contents = markdown_unescape($contents);
+	}
     }
+
 #   print STDERR "contents b4:$contents\n";
     eval $dohighlight;
     die $@ if $@;
 #   print STDERR "contents af:$contents\n";
+    if ($use_markdown) {
+        $contents = markdown_to_docbook($contents);
+    }
 
 #   strip whitespaces when generating html5
     if ($output_mode eq "html5") {
@@ -553,7 +609,8 @@ sub output_highlight {
 	$contents =~ s/\s+$//;
     }
     foreach $line (split "\n", $contents) {
-	if (! $output_preformatted) {
+	if (! $output_preformatted &&
+	    !($use_markdown && $line =~ m /^    /s)) {
 	    $line =~ s/^\s*//;
 	}
 	if ($line eq ""){
@@ -974,7 +1031,9 @@ sub output_section_xml(%) {
 	print "<refsect1>\n";
 	print "<title>$section</title>\n";
 	if ($section =~ m/EXAMPLE/i) {
-	    print "<informalexample><programlisting>\n";
+	    print "<informalexample>\n";
+	    # programlisting is already included by pandoc
+	    print "<programlisting>\n" unless $use_markdown;
 	    $output_preformatted = 1;
 	} else {
 	    print "<para>\n";
@@ -982,7 +1041,8 @@ sub output_section_xml(%) {
 	output_highlight($args{'sections'}{$section});
 	$output_preformatted = 0;
 	if ($section =~ m/EXAMPLE/i) {
-	    print "</programlisting></informalexample>\n";
+	    print "</programlisting>\n" unless $use_markdown;
+	    print "</informalexample>\n";
 	} else {
 	    print "</para>\n";
 	}
-- 
2.4.3

Re: [PATCH 2/6] scripts/kernel-doc: Adding infrastructure for markdown support

[Jani Nikula]

On Mon, 07 Sep 2015, Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk> wrote:
>  %.xml: %.tmpl $(KERNELDOC) $(DOCPROC) $(KERNELDOCXMLREF) FORCE
> +	@(which pandoc > /dev/null 2>&1) || \
> +	(echo "*** To get propper documentation you need to install pandoc ***";)

s/propper/proper/

This only seems to be applied to our (drm-intel) topical branch,
otherwise would've sent a patch.

BR,
Jani.



-- 
Jani Nikula, Intel Open Source Technology Center

[PATCH 3/6] drm/doc: Convert to markdown

[Danilo Cesar Lemes de Paula]

DRM Docbook is now Markdown ready. This means its doc is able to
use markdown text on it.

* Documentation/DocBook/drm.tmpl: Contains a table duplicated from
  drivers/gpu/drm/i915/i915_reg.h. This is not needed anymore

* drivers/gpu/drm/drm_modeset_lock.c: had a code example that used
  to look pretty bad on html. Fixed by using proper code markup.

* drivers/gpu/drm/drm_prime.c: Remove spaces between lines to make
  a proper markup list.

* drivers/gpu/drm/i915/i915_reg.h: Altought pandoc supports tables,
  it doesn't support table cell spanning. But we can use fixed-width
  for those special cases.

* include/drm/drm_vma_manager.h: Another code example that should be
  proper indented with four spaces.

Signed-off-by: Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Stephan Mueller <smueller@chronox.de>
Cc: Michal Marek <mmarek@suse.cz>
Cc: linux-kernel@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: intel-gfx <intel-gfx@lists.freedesktop.org>
Cc: dri-devel <dri-devel@lists.freedesktop.org>
---
 Documentation/DocBook/Makefile     |  2 +-
 Documentation/DocBook/drm.tmpl     | 86 --------------------------------------
 drivers/gpu/drm/drm_modes.c        | 12 +++---
 drivers/gpu/drm/drm_modeset_lock.c | 14 +++----
 drivers/gpu/drm/drm_prime.c        | 16 +++----
 drivers/gpu/drm/i915/i915_reg.h    | 48 ++++++++++-----------
 include/drm/drm_vma_manager.h      | 10 ++---
 7 files changed, 48 insertions(+), 140 deletions(-)

diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile
index c1ff834..75b7d9b 100644
--- a/Documentation/DocBook/Makefile
+++ b/Documentation/DocBook/Makefile
@@ -17,7 +17,7 @@ DOCBOOKS := z8530book.xml device-drivers.xml \
 	    tracepoint.xml drm.xml media_api.xml w1.xml \
 	    writing_musb_glue_layer.xml crypto-API.xml iio.xml
 
-MARKDOWNREADY := 
+MARKDOWNREADY := drm.xml
 
 include Documentation/DocBook/media/Makefile
 
diff --git a/Documentation/DocBook/drm.tmpl b/Documentation/DocBook/drm.tmpl
index 9ddf8c6..4111902 100644
--- a/Documentation/DocBook/drm.tmpl
+++ b/Documentation/DocBook/drm.tmpl
@@ -4076,92 +4076,6 @@ int num_ioctls;</synopsis>
       <sect2>
         <title>DPIO</title>
 !Pdrivers/gpu/drm/i915/i915_reg.h DPIO
-	<table id="dpiox2">
-	  <title>Dual channel PHY (VLV/CHV/BXT)</title>
-	  <tgroup cols="8">
-	    <colspec colname="c0" />
-	    <colspec colname="c1" />
-	    <colspec colname="c2" />
-	    <colspec colname="c3" />
-	    <colspec colname="c4" />
-	    <colspec colname="c5" />
-	    <colspec colname="c6" />
-	    <colspec colname="c7" />
-	    <spanspec spanname="ch0" namest="c0" nameend="c3" />
-	    <spanspec spanname="ch1" namest="c4" nameend="c7" />
-	    <spanspec spanname="ch0pcs01" namest="c0" nameend="c1" />
-	    <spanspec spanname="ch0pcs23" namest="c2" nameend="c3" />
-	    <spanspec spanname="ch1pcs01" namest="c4" nameend="c5" />
-	    <spanspec spanname="ch1pcs23" namest="c6" nameend="c7" />
-	    <thead>
-	      <row>
-		<entry spanname="ch0">CH0</entry>
-		<entry spanname="ch1">CH1</entry>
-	      </row>
-	    </thead>
-	    <tbody valign="top" align="center">
-	      <row>
-		<entry spanname="ch0">CMN/PLL/REF</entry>
-		<entry spanname="ch1">CMN/PLL/REF</entry>
-	      </row>
-	      <row>
-		<entry spanname="ch0pcs01">PCS01</entry>
-		<entry spanname="ch0pcs23">PCS23</entry>
-		<entry spanname="ch1pcs01">PCS01</entry>
-		<entry spanname="ch1pcs23">PCS23</entry>
-	      </row>
-	      <row>
-		<entry>TX0</entry>
-		<entry>TX1</entry>
-		<entry>TX2</entry>
-		<entry>TX3</entry>
-		<entry>TX0</entry>
-		<entry>TX1</entry>
-		<entry>TX2</entry>
-		<entry>TX3</entry>
-	      </row>
-	      <row>
-		<entry spanname="ch0">DDI0</entry>
-		<entry spanname="ch1">DDI1</entry>
-	      </row>
-	    </tbody>
-	  </tgroup>
-	</table>
-	<table id="dpiox1">
-	  <title>Single channel PHY (CHV/BXT)</title>
-	  <tgroup cols="4">
-	    <colspec colname="c0" />
-	    <colspec colname="c1" />
-	    <colspec colname="c2" />
-	    <colspec colname="c3" />
-	    <spanspec spanname="ch0" namest="c0" nameend="c3" />
-	    <spanspec spanname="ch0pcs01" namest="c0" nameend="c1" />
-	    <spanspec spanname="ch0pcs23" namest="c2" nameend="c3" />
-	    <thead>
-	      <row>
-		<entry spanname="ch0">CH0</entry>
-	      </row>
-	    </thead>
-	    <tbody valign="top" align="center">
-	      <row>
-		<entry spanname="ch0">CMN/PLL/REF</entry>
-	      </row>
-	      <row>
-		<entry spanname="ch0pcs01">PCS01</entry>
-		<entry spanname="ch0pcs23">PCS23</entry>
-	      </row>
-	      <row>
-		<entry>TX0</entry>
-		<entry>TX1</entry>
-		<entry>TX2</entry>
-		<entry>TX3</entry>
-	      </row>
-	      <row>
-		<entry spanname="ch0">DDI2</entry>
-	      </row>
-	    </tbody>
-	  </tgroup>
-	</table>
       </sect2>
 
       <sect2>
diff --git a/drivers/gpu/drm/drm_modes.c b/drivers/gpu/drm/drm_modes.c
index cd74a09..9480464 100644
--- a/drivers/gpu/drm/drm_modes.c
+++ b/drivers/gpu/drm/drm_modes.c
@@ -553,10 +553,10 @@ EXPORT_SYMBOL(drm_gtf_mode_complex);
  * drivers/video/fbmon.c
  *
  * Standard GTF parameters:
- * M = 600
- * C = 40
- * K = 128
- * J = 20
+ *     M = 600
+ *     C = 40
+ *     K = 128
+ *     J = 20
  *
  * Returns:
  * The modeline based on the GTF algorithm stored in a drm_display_mode object.
@@ -1212,7 +1212,7 @@ EXPORT_SYMBOL(drm_mode_connector_list_update);
  * This uses the same parameters as the fb modedb.c, except for an extra
  * force-enable, force-enable-digital and force-disable bit at the end:
  *
- *	<xres>x<yres>[M][R][-<bpp>][@<refresh>][i][m][eDd]
+ * <xres>x<yres>[M][R][-<bpp>][@<refresh>][i][m][eDd]
  *
  * The intermediate drm_cmdline_mode structure is required to store additional
  * options from the command line modline like the force-enable/disable flag.
@@ -1491,4 +1491,4 @@ int drm_mode_convert_umode(struct drm_display_mode *out,
 
 out:
 	return ret;
-}
\ No newline at end of file
+}
diff --git a/drivers/gpu/drm/drm_modeset_lock.c b/drivers/gpu/drm/drm_modeset_lock.c
index fba321c..9abee87 100644
--- a/drivers/gpu/drm/drm_modeset_lock.c
+++ b/drivers/gpu/drm/drm_modeset_lock.c
@@ -40,17 +40,15 @@
  * The basic usage pattern is to:
  *
  *     drm_modeset_acquire_init(&ctx)
- *   retry:
+ *     retry:
  *     foreach (lock in random_ordered_set_of_locks) {
- *       ret = drm_modeset_lock(lock, &ctx)
- *       if (ret == -EDEADLK) {
- *          drm_modeset_backoff(&ctx);
- *          goto retry;
- *       }
+ *         ret = drm_modeset_lock(lock, &ctx)
+ *         if (ret == -EDEADLK) {
+ *             drm_modeset_backoff(&ctx);
+ *             goto retry;
+ *         }
  *     }
- *
  *     ... do stuff ...
- *
  *     drm_modeset_drop_locks(&ctx);
  *     drm_modeset_acquire_fini(&ctx);
  */
diff --git a/drivers/gpu/drm/drm_prime.c b/drivers/gpu/drm/drm_prime.c
index 9f935f5..27aa718 100644
--- a/drivers/gpu/drm/drm_prime.c
+++ b/drivers/gpu/drm/drm_prime.c
@@ -313,19 +313,15 @@ static const struct dma_buf_ops drm_gem_prime_dmabuf_ops =  {
  *
  * Export callbacks:
  *
- *  - @gem_prime_pin (optional): prepare a GEM object for exporting
- *
- *  - @gem_prime_get_sg_table: provide a scatter/gather table of pinned pages
- *
- *  - @gem_prime_vmap: vmap a buffer exported by your driver
- *
- *  - @gem_prime_vunmap: vunmap a buffer exported by your driver
- *
- *  - @gem_prime_mmap (optional): mmap a buffer exported by your driver
+ *  * @gem_prime_pin (optional): prepare a GEM object for exporting
+ *  * @gem_prime_get_sg_table: provide a scatter/gather table of pinned pages
+ *  * @gem_prime_vmap: vmap a buffer exported by your driver
+ *  * @gem_prime_vunmap: vunmap a buffer exported by your driver
+ *  * @gem_prime_mmap (optional): mmap a buffer exported by your driver
  *
  * Import callback:
  *
- *  - @gem_prime_import_sg_table (import): produce a GEM object from another
+ *  * @gem_prime_import_sg_table (import): produce a GEM object from another
  *    driver's scatter/gather table
  */
 
diff --git a/drivers/gpu/drm/i915/i915_reg.h b/drivers/gpu/drm/i915/i915_reg.h
index 83a0888..db5e248 100644
--- a/drivers/gpu/drm/i915/i915_reg.h
+++ b/drivers/gpu/drm/i915/i915_reg.h
@@ -799,31 +799,31 @@ enum skl_disp_power_wells {
  *
  * Note: DDI0 is digital port B, DD1 is digital port C, and DDI2 is
  * digital port D (CHV) or port A (BXT).
- */
-/*
- * Dual channel PHY (VLV/CHV/BXT)
- * ---------------------------------
- * |      CH0      |      CH1      |
- * |  CMN/PLL/REF  |  CMN/PLL/REF  |
- * |---------------|---------------| Display PHY
- * | PCS01 | PCS23 | PCS01 | PCS23 |
- * |-------|-------|-------|-------|
- * |TX0|TX1|TX2|TX3|TX0|TX1|TX2|TX3|
- * ---------------------------------
- * |     DDI0      |     DDI1      | DP/HDMI ports
- * ---------------------------------
  *
- * Single channel PHY (CHV/BXT)
- * -----------------
- * |      CH0      |
- * |  CMN/PLL/REF  |
- * |---------------| Display PHY
- * | PCS01 | PCS23 |
- * |-------|-------|
- * |TX0|TX1|TX2|TX3|
- * -----------------
- * |     DDI2      | DP/HDMI port
- * -----------------
+ *
+ *     Dual channel PHY (VLV/CHV/BXT)
+ *     ---------------------------------
+ *     |      CH0      |      CH1      |
+ *     |  CMN/PLL/REF  |  CMN/PLL/REF  |
+ *     |---------------|---------------| Display PHY
+ *     | PCS01 | PCS23 | PCS01 | PCS23 |
+ *     |-------|-------|-------|-------|
+ *     |TX0|TX1|TX2|TX3|TX0|TX1|TX2|TX3|
+ *     ---------------------------------
+ *     |     DDI0      |     DDI1      | DP/HDMI ports
+ *     ---------------------------------
+ *
+ *     Single channel PHY (CHV/BXT)
+ *     -----------------
+ *     |      CH0      |
+ *     |  CMN/PLL/REF  |
+ *     |---------------| Display PHY
+ *     | PCS01 | PCS23 |
+ *     |-------|-------|
+ *     |TX0|TX1|TX2|TX3|
+ *     -----------------
+ *     |     DDI2      | DP/HDMI port
+ *     -----------------
  */
 #define DPIO_DEVFN			0
 
diff --git a/include/drm/drm_vma_manager.h b/include/drm/drm_vma_manager.h
index 8cd402c..2ca44db 100644
--- a/include/drm/drm_vma_manager.h
+++ b/include/drm/drm_vma_manager.h
@@ -110,11 +110,11 @@ drm_vma_offset_exact_lookup(struct drm_vma_offset_manager *mgr,
  * Note: You're in atomic-context while holding this lock!
  *
  * Example:
- *   drm_vma_offset_lock_lookup(mgr);
- *   node = drm_vma_offset_lookup_locked(mgr);
- *   if (node)
- *       kref_get_unless_zero(container_of(node, sth, entr));
- *   drm_vma_offset_unlock_lookup(mgr);
+ *     drm_vma_offset_lock_lookup(mgr);
+ *     node = drm_vma_offset_lookup_locked(mgr);
+ *     if (node)
+ *         kref_get_unless_zero(container_of(node, sth, entr));
+ *     drm_vma_offset_unlock_lookup(mgr);
  */
 static inline void drm_vma_offset_lock_lookup(struct drm_vma_offset_manager *mgr)
 {
-- 
2.4.3

[PATCH 4/6] drm/doc: Fixing xml documentation warning

[Danilo Cesar Lemes de Paula]

"/**" should be used for kernel-doc documentation only.
It causes a warning with the new "in struct body" format.

Signed-off-by: Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Stephan Mueller <smueller@chronox.de>
Cc: Michal Marek <mmarek@suse.cz>
Cc: linux-kernel@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: intel-gfx <intel-gfx@lists.freedesktop.org>
Cc: dri-devel <dri-devel@lists.freedesktop.org>
Cc: Graham Whaley <graham.whaley@linux.intel.com>
---
 include/drm/drm_modeset_lock.h | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/include/drm/drm_modeset_lock.h b/include/drm/drm_modeset_lock.h
index 5dd18bf..94938d8 100644
--- a/include/drm/drm_modeset_lock.h
+++ b/include/drm/drm_modeset_lock.h
@@ -43,19 +43,19 @@ struct drm_modeset_acquire_ctx {
 
 	struct ww_acquire_ctx ww_ctx;
 
-	/**
+	/*
 	 * Contended lock: if a lock is contended you should only call
 	 * drm_modeset_backoff() which drops locks and slow-locks the
 	 * contended lock.
 	 */
 	struct drm_modeset_lock *contended;
 
-	/**
+	/*
 	 * list of held locks (drm_modeset_lock)
 	 */
 	struct list_head locked;
 
-	/**
+	/*
 	 * Trylock mode, use only for panic handlers!
 	 */
 	bool trylock_only;
@@ -70,12 +70,12 @@ struct drm_modeset_acquire_ctx {
  * Used for locking CRTCs and other modeset resources.
  */
 struct drm_modeset_lock {
-	/**
+	/*
 	 * modeset lock
 	 */
 	struct ww_mutex mutex;
 
-	/**
+	/*
 	 * Resources that are locked as part of an atomic update are added
 	 * to a list (so we know what to unlock at the end).
 	 */
-- 
2.4.3

[PATCH 5/6] scripts/kernel-doc: Improve Markdown results

[Danilo Cesar Lemes de Paula]

Using pandoc as the Markdown engine cause some minor side effects as
pandoc includes  main <para> tags for almost everything.
Original Markdown support approach removes those main tags, but it caused
some inconsistencies when that tag is not the main one, like:
<something>..</something>
<para>...</para>

As kernel-doc was already including a <para> tag, it causes the presence
of double <para> tags (<para><para>), which is not supported by DocBook
spec.

Html target gets away with it, so it causes no harm, although other
targets might not be so lucky (pdf as example).

We're now delegating the inclusion of the main <para> tag to pandoc
only, as it knows when it's necessary or not.

That behavior causes a corner case, the only situation where we're
certainly that <para> is not needed, which is the <refpurpose> content.
For those cases, we're using a $output_markdown_nopara = 1 control var.

Signed-off-by: Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Stephan Mueller <smueller@chronox.de>
Cc: Michal Marek <mmarek@suse.cz>
Cc: linux-kernel@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: intel-gfx <intel-gfx@lists.freedesktop.org>
Cc: dri-devel <dri-devel@lists.freedesktop.org>
Cc: Graham Whaley <graham.whaley@linux.intel.com>
---
 scripts/kernel-doc | 48 ++++++++++++++++++++++++++++++++++--------------
 1 file changed, 34 insertions(+), 14 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 5fe572a..d6129e7 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -288,6 +288,7 @@ my $use_markdown = 0;
 my $verbose = 0;
 my $output_mode = "man";
 my $output_preformatted = 0;
+my $output_markdown_nopara = 0;
 my $no_doc_sections = 0;
 my @highlights = @highlights_man;
 my $blankline = $blankline_man;
@@ -538,8 +539,11 @@ sub markdown_to_docbook {
 	close(CHLD_OUT);
 	close(CHLD_ERR);
 
-	# pandoc insists in adding Main <para></para>, we should remove them.
-	$content =~ s:\A\s*<para>\s*\n(.*)\n</para>\Z$:$1:egsm;
+	if ($output_markdown_nopara) {
+		# pandoc insists in adding Main <para></para>, sometimes we
+		# want to remove them.
+		$content =~ s:\A\s*<para>\s*\n(.*)\n</para>\Z$:$1:egsm;
+	}
 
 	return $content;
 }
@@ -614,7 +618,7 @@ sub output_highlight {
 	    $line =~ s/^\s*//;
 	}
 	if ($line eq ""){
-	    if (! $output_preformatted) {
+	    if (! $output_preformatted && ! $use_markdown) {
 		print $lineprefix, local_unescape($blankline);
 	    }
 	} else {
@@ -1035,7 +1039,7 @@ sub output_section_xml(%) {
 	    # programlisting is already included by pandoc
 	    print "<programlisting>\n" unless $use_markdown;
 	    $output_preformatted = 1;
-	} else {
+	} elsif (! $use_markdown) {
 	    print "<para>\n";
 	}
 	output_highlight($args{'sections'}{$section});
@@ -1043,7 +1047,7 @@ sub output_section_xml(%) {
 	if ($section =~ m/EXAMPLE/i) {
 	    print "</programlisting>\n" unless $use_markdown;
 	    print "</informalexample>\n";
-	} else {
+	} elsif (! $use_markdown) {
 	    print "</para>\n";
 	}
 	print "</refsect1>\n";
@@ -1075,7 +1079,9 @@ sub output_function_xml(%) {
     print " <refname>" . $args{'function'} . "</refname>\n";
     print " <refpurpose>\n";
     print "  ";
+    $output_markdown_nopara = 1;
     output_highlight ($args{'purpose'});
+    $output_markdown_nopara = 0;
     print " </refpurpose>\n";
     print "</refnamediv>\n";
 
@@ -1113,10 +1119,12 @@ sub output_function_xml(%) {
 	    $parameter_name =~ s/\[.*//;
 
 	    print "  <varlistentry>\n   <term><parameter>$parameter</parameter></term>\n";
-	    print "   <listitem>\n    <para>\n";
+	    print "   <listitem>\n";
+	    print "    <para>\n" unless $use_markdown;
 	    $lineprefix="     ";
 	    output_highlight($args{'parameterdescs'}{$parameter_name});
-	    print "    </para>\n   </listitem>\n  </varlistentry>\n";
+	    print "    </para>\n" unless $use_markdown;
+	    print "   </listitem>\n  </varlistentry>\n";
 	}
 	print " </variablelist>\n";
     } else {
@@ -1152,7 +1160,9 @@ sub output_struct_xml(%) {
     print " <refname>" . $args{'type'} . " " . $args{'struct'} . "</refname>\n";
     print " <refpurpose>\n";
     print "  ";
+    $output_markdown_nopara = 1;
     output_highlight ($args{'purpose'});
+    $output_markdown_nopara = 0;
     print " </refpurpose>\n";
     print "</refnamediv>\n";
 
@@ -1205,9 +1215,11 @@ sub output_struct_xml(%) {
       ($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
       print "    <varlistentry>";
       print "      <term>$parameter</term>\n";
-      print "      <listitem><para>\n";
+      print "      <listitem>\n";
+      print "         <para>\n" unless $use_markdown;
       output_highlight($args{'parameterdescs'}{$parameter_name});
-      print "      </para></listitem>\n";
+      print "         </para>\n" unless $use_markdown;
+      print "      </listitem>\n";
       print "    </varlistentry>\n";
     }
     print "  </variablelist>\n";
@@ -1246,7 +1258,9 @@ sub output_enum_xml(%) {
     print " <refname>enum " . $args{'enum'} . "</refname>\n";
     print " <refpurpose>\n";
     print "  ";
+    $output_markdown_nopara = 1;
     output_highlight ($args{'purpose'});
+    $output_markdown_nopara = 0;
     print " </refpurpose>\n";
     print "</refnamediv>\n";
 
@@ -1276,9 +1290,11 @@ sub output_enum_xml(%) {
 
       print "    <varlistentry>";
       print "      <term>$parameter</term>\n";
-      print "      <listitem><para>\n";
+      print "      <listitem>\n";
+      print "         <para>\n" unless $use_markdown;
       output_highlight($args{'parameterdescs'}{$parameter_name});
-      print "      </para></listitem>\n";
+      print "         </para>\n" unless $use_markdown;
+      print "      </listitem>\n";
       print "    </varlistentry>\n";
     }
     print "  </variablelist>\n";
@@ -1344,14 +1360,14 @@ sub output_blockhead_xml(%) {
 	if ($section =~ m/EXAMPLE/i) {
 	    print "<example><para>\n";
 	    $output_preformatted = 1;
-	} else {
+	} elsif (! $use_markdown) {
 	    print "<para>\n";
 	}
 	output_highlight($args{'sections'}{$section});
 	$output_preformatted = 0;
 	if ($section =~ m/EXAMPLE/i) {
 	    print "</para></example>\n";
-	} else {
+	} elsif (! $use_markdown) {
 	    print "</para>";
 	}
 	if (!$args{'content-only'}) {
@@ -2695,7 +2711,11 @@ sub process_file($) {
 		{
 			if ( $1 eq "" )
 			{
-				$contents .= $blankline;
+				if ($use_markdown) {
+					$contents .= "\n";
+				} else {
+					$contents .= $blankline;
+				}
 			}
 			else
 			{
-- 
2.4.3

[PATCH 6/6] scripts/kernel-doc: Processing -nofunc for functions only

[Danilo Cesar Lemes de Paula]

Docproc process EXPORT_SYMBOL(f1) macro and uses -nofunc f1 to
avoid duplicated documentation in the next call.
It works for most of the cases, but there are some specific situations
where a struct has the same name of an already-exported function.

Current kernel-doc behavior ignores those structs and do not add them
to the final documentation. This patch fixes it.

This is non-usual and the only case I've found is the drm_modeset_lock
(function and struct) defined in drm_modeset_lock.h and
drm_modeset_lock.c. Considering this, it should only affect the DRM
documentation by including struct drm_modeset_lock to the final Docbook.

Signed-off-by: Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Johannes Berg <johannes.berg@intel.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: intel-gfx <intel-gfx@lists.freedesktop.org>
Cc: dri-devel <dri-devel@lists.freedesktop.org>
---
 scripts/kernel-doc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index d6129e7..a01b20ea 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1836,7 +1836,7 @@ sub output_declaration {
     my $func = "output_${functype}_$output_mode";
     if (($function_only==0) ||
 	( $function_only == 1 && defined($function_table{$name})) ||
-	( $function_only == 2 && !defined($function_table{$name})))
+	( $function_only == 2 && !($functype eq "function" && defined($function_table{$name}))))
     {
 	&$func(@_);
 	$section_counter++;
-- 
2.4.3

Re: [PATCH 0/6] scripts/kernel-doc: Kernel-doc improvements

[Jonathan Corbet]

On Mon,  7 Sep 2015 17:01:58 -0300
Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk> wrote:

> The following series contains:
>  * kernel-doc: markdown support and improvements.

OK, I've spent a while looking this stuff over.  I like the general idea,
but I do have a couple of concerns.

1 Installing pandoc on a Fedora system wants to drag in 70(!) packages
  for 100MB of total stuff.  Installing it on Arch is ... well ... enough
  to make you want to switch to Fedora.  If we add a dependency on a tool
  this massive, people are going to complain, loudly.

  Have you looked at using something like cmark instead?  I don't know
  the tool well, but it seems like it can do the job simply enough.  It's
  focused, written in C, and doesn't drag in a diskful of Haskell
  stuff.  There's lot of other converters out there too, I'm not tied to
  this one (though I think CommonMark deserves support), but I do think
  this question needs to be considered.

2 We're constructing an increasingly complicated document-processing
  mechanism with a lot of independently moving parts.  What if we
  converted the whole document to markdown and dispensed with the XML
  part altogether?  Making the source files simpler and dispensing with
  the xmlto requirement would be a big win, IMO.

I will not make #2 be a precondition to getting some form of this work
merged, but I would like to have a good answer for #1.  Adding such a
heavyweight dependency (even as an optional one) needs to have a pretty
good story behind it.

Thanks,

jon

Re: [PATCH 0/6] scripts/kernel-doc: Kernel-doc improvements

[Daniel Vetter]

On Sat, Sep 12, 2015 at 03:24:49PM -0600, Jonathan Corbet wrote:
> On Mon,  7 Sep 2015 17:01:58 -0300
> Danilo Cesar Lemes de Paula <danilo.cesar@collabora.co.uk> wrote:
> 
> > The following series contains:
> >  * kernel-doc: markdown support and improvements.
> 
> OK, I've spent a while looking this stuff over.  I like the general idea,
> but I do have a couple of concerns.
> 
> 1 Installing pandoc on a Fedora system wants to drag in 70(!) packages
>   for 100MB of total stuff.  Installing it on Arch is ... well ... enough
>   to make you want to switch to Fedora.  If we add a dependency on a tool
>   this massive, people are going to complain, loudly.
> 
>   Have you looked at using something like cmark instead?  I don't know
>   the tool well, but it seems like it can do the job simply enough.  It's
>   focused, written in C, and doesn't drag in a diskful of Haskell
>   stuff.  There's lot of other converters out there too, I'm not tied to
>   this one (though I think CommonMark deserves support), but I do think
>   this question needs to be considered.

Personally I don't care which kind of text markup we pick and wich
converter, as long as the project looks reasonable far away from
immeninent death (way too many one-person projects on github in this
area).

But if we have this discussion I'd like to decouple it from the other
kerneldoc improvemnts in this series (patches 1, 5 and 6). If we cant
agree on the text markup then drm docs will simply look a bit funny for
everyone else. But if the inline struct stuff won't happen 0-day will
scream around (and there's already patches which use the new layout).

> 2 We're constructing an increasingly complicated document-processing
>   mechanism with a lot of independently moving parts.  What if we
>   converted the whole document to markdown and dispensed with the XML
>   part altogether?  Making the source files simpler and dispensing with
>   the xmlto requirement would be a big win, IMO.

Who's going to convert the almost 30kloc of xml template (which often have
large amounts of texts) and the over 60k kerneldoc comments that we have
already?

> I will not make #2 be a precondition to getting some form of this work
> merged, but I would like to have a good answer for #1.  Adding such a
> heavyweight dependency (even as an optional one) needs to have a pretty
> good story behind it.

Should we discuss #1 at ks? Imo as long as no one pipes up to do the
massive conversion away from the current toolchain (and subsystem
maintainers won't just kill that effort because it causes too much churn)
moving forward with the kerneldoc toolchain we have makes sense. Hence I'd
like to see those patches landed before we resolve #1 if possible.

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [PATCH 0/6] scripts/kernel-doc: Kernel-doc improvements

[Jonathan Corbet]

On Sun, 13 Sep 2015 12:36:07 +0200
Daniel Vetter <daniel@ffwll.ch> wrote:

> Personally I don't care which kind of text markup we pick and wich
> converter, as long as the project looks reasonable far away from
> immeninent death (way too many one-person projects on github in this
> area).
> 
> But if we have this discussion I'd like to decouple it from the other
> kerneldoc improvemnts in this series (patches 1, 5 and 6). If we cant
> agree on the text markup then drm docs will simply look a bit funny for
> everyone else. But if the inline struct stuff won't happen 0-day will
> scream around (and there's already patches which use the new layout).

Those patches are:

1)  scripts/kernel-doc: Replacing highlights hash by an array
5)  scripts/kernel-doc: Improve Markdown results
6)  scripts/kernel-doc: Processing -nofunc for functions only

#1 is fine, I'll merge that today.  #6 is already merged.  #5 is a markdown
patch, though, and doesn't make sense without the others?  Are you thinking
about #3 (drm/doc: Convert to markdown)?  That one would almost work (it
depends on #2 currently) and it nicely shows *why* I'd like to get away from
XML... 

> > 2 We're constructing an increasingly complicated document-processing
> >   mechanism with a lot of independently moving parts.  What if we
> >   converted the whole document to markdown and dispensed with the XML
> >   part altogether?  Making the source files simpler and dispensing with
> >   the xmlto requirement would be a big win, IMO.
> 
> Who's going to convert the almost 30kloc of xml template (which often have
> large amounts of texts) and the over 60k kerneldoc comments that we have
> already?

I thought you'd do that :)

Seriously, though, a change would be a big job.  There's a reason I've said
several times that it would make no sense to delay the work at hand in the
hopes of somebody doing this whole job instead.  But we can certainly
ponder what might be better.

Getting rid of the XML stuff may well simplify the whole process and make
the documentation much more accessible for those who would change it; that
could be a goal worth going for.  Oh, and anything requiring changes to the
kerneldoc comments is going nowhere, that was never something I'd
contemplated.  Those comments are fine.

But any such change needs a lot of thought and a reasonable proof of
concept.  Meanwhile we have work that can make the docs better now, and I
want to merge it.  But we should choose the tools we use carefully, and I
anticipate a lot of opposition to one that has to drag in 70 Haskell
packages with it.

> > I will not make #2 be a precondition to getting some form of this work
> > merged, but I would like to have a good answer for #1.  Adding such a
> > heavyweight dependency (even as an optional one) needs to have a pretty
> > good story behind it.
> 
> Should we discuss #1 at ks? Imo as long as no one pipes up to do the
> massive conversion away from the current toolchain (and subsystem
> maintainers won't just kill that effort because it causes too much churn)
> moving forward with the kerneldoc toolchain we have makes sense. Hence I'd
> like to see those patches landed before we resolve #1 if possible.

I've tossed the topic out there; the kernel summit agenda hasn't been made
yet, though.  In any case, I'd be happy to resolve this particular issue
before then if it were possible.

jon

Re: [PATCH 0/6] scripts/kernel-doc: Kernel-doc improvements

[Daniel Vetter]

On Sun, Sep 13, 2015 at 9:13 PM, Jonathan Corbet <corbet@lwn.net> wrote:
> On Sun, 13 Sep 2015 12:36:07 +0200
> Daniel Vetter <daniel@ffwll.ch> wrote:
>
>> Personally I don't care which kind of text markup we pick and wich
>> converter, as long as the project looks reasonable far away from
>> immeninent death (way too many one-person projects on github in this
>> area).
>>
>> But if we have this discussion I'd like to decouple it from the other
>> kerneldoc improvemnts in this series (patches 1, 5 and 6). If we cant
>> agree on the text markup then drm docs will simply look a bit funny for
>> everyone else. But if the inline struct stuff won't happen 0-day will
>> scream around (and there's already patches which use the new layout).
>
> Those patches are:
>
> 1)  scripts/kernel-doc: Replacing highlights hash by an array
> 5)  scripts/kernel-doc: Improve Markdown results
> 6)  scripts/kernel-doc: Processing -nofunc for functions only
>
> #1 is fine, I'll merge that today.  #6 is already merged.  #5 is a markdown
> patch, though, and doesn't make sense without the others?  Are you thinking
> about #3 (drm/doc: Convert to markdown)?  That one would almost work (it
> depends on #2 currently) and it nicely shows *why* I'd like to get away from
> XML...

Sorry I mixed things up, #5 is ok to leave out. I thought about
"scripts/kernel-doc Allow struct arguments documentation in struct
body" but you already merged that one for 4.2. Which I missed, but
which is great since that's the one that would cause the big
conflicts.

>> > 2 We're constructing an increasingly complicated document-processing
>> >   mechanism with a lot of independently moving parts.  What if we
>> >   converted the whole document to markdown and dispensed with the XML
>> >   part altogether?  Making the source files simpler and dispensing with
>> >   the xmlto requirement would be a big win, IMO.
>>
>> Who's going to convert the almost 30kloc of xml template (which often have
>> large amounts of texts) and the over 60k kerneldoc comments that we have
>> already?
>
> I thought you'd do that :)
>
> Seriously, though, a change would be a big job.  There's a reason I've said
> several times that it would make no sense to delay the work at hand in the
> hopes of somebody doing this whole job instead.  But we can certainly
> ponder what might be better.
>
> Getting rid of the XML stuff may well simplify the whole process and make
> the documentation much more accessible for those who would change it; that
> could be a goal worth going for.  Oh, and anything requiring changes to the
> kerneldoc comments is going nowhere, that was never something I'd
> contemplated.  Those comments are fine.

Well my goal is to be able to have all the programmer docs in the
code, including any high-level overview sections to tie everything
together (hence hyperlinks and better formatting). But then you still
need some skeleton to make a coherent whole out of all the parts, and
I think the docbook xml templates we have serve rather fine for that
role. Writing text in xml is indeed horrid, but if we can create
in-line DOC: sections with decent formatting there's really no need
for that. From that angle this work here already has the goal to get
rid of the xml - I plane to move all the existing text in the drm.tmpl
xml into inline DOC: kerneldoc commets.

And at least gtkdoc (which we use extensively for the userpace
test/tools repo) relies on some xml thing to tie different topics
together too. So that seems pretty standard.

> But any such change needs a lot of thought and a reasonable proof of
> concept.  Meanwhile we have work that can make the docs better now, and I
> want to merge it.  But we should choose the tools we use carefully, and I
> anticipate a lot of opposition to one that has to drag in 70 Haskell
> packages with it.

Well personally I don't care about the exact tooling, as long as:
- it's a reasonable alive project
- packages available on distros (works for me on both debian and fedora)
- we can muck around with how things are integrated into overall docs
(which is the case since pandoc is only used as a paragraph formatter
here).

The last one is something we stumbled over for the userspace side
where we wanted to have better tooling for documenting testcases.
Upstream gtkdoc is receptive, but getting anything in takes forever
and then there's the 6 month release schedule too. And there's also
the problem that we can't change the kerneldoc grammar (e.g. gtkdoc
has the code comment parser integrated and ofc it has slightly
different rules to mark up references to structs/enums).

I'm also trying to build up a doc team here with a full-time tech
writer for drm/i915 topics. So I'm very much interested in keeping
this all alive. But I don't have the time to mass-convert everything
else, and I'm also not sure it's a good idea: In my experience all the
doc toolchain suck somewhat, and it's better to be consistent within a
project instead of proliferating different solutions. It's hard enough
already to get developers to write docs and even harder to find
someone who writes good docs ...

Ofc we already have a big split between the ad-hoc text files in
Documentation and the docbook stuff in Documentation/DocBook. Otoh
docbook is the only way to extract the kerneldoc comments into
something readable and indexed.

>> > I will not make #2 be a precondition to getting some form of this work
>> > merged, but I would like to have a good answer for #1.  Adding such a
>> > heavyweight dependency (even as an optional one) needs to have a pretty
>> > good story behind it.
>>
>> Should we discuss #1 at ks? Imo as long as no one pipes up to do the
>> massive conversion away from the current toolchain (and subsystem
>> maintainers won't just kill that effort because it causes too much churn)
>> moving forward with the kerneldoc toolchain we have makes sense. Hence I'd
>> like to see those patches landed before we resolve #1 if possible.
>
> I've tossed the topic out there; the kernel summit agenda hasn't been made
> yet, though.  In any case, I'd be happy to resolve this particular issue
> before then if it were possible.

I don't mind much if we delay this part until KS. All the text markup
styles are pretty close. So fixing up one dialect for another for the
few patches I'll likely pull in for 4.4 won't be a lot of work if we
decide to go with something else. Or even if we decide to switch
markup styles much later, it should all be doable with a few sed jobs.

Only problem I see is that thus far no one else seems to care much,
and if we change the approach I need to convince management here again
that we need to invest some time to make it happen (and some excuses
why this approach here wasn't ok). That will probably suck down more
time than converting to whatever we decide on. And like I said I
personally don't care one bit what the toolchain looks like or the
exact flavour of text markup we pick.

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [PATCH 0/6] scripts/kernel-doc: Kernel-doc improvements

[Danilo Cesar Lemes de Paula]

On 09/13/2015 05:58 PM, Daniel Vetter wrote:
> On Sun, Sep 13, 2015 at 9:13 PM, Jonathan Corbet <corbet@lwn.net> wrote:
>> On Sun, 13 Sep 2015 12:36:07 +0200
>> Daniel Vetter <daniel@ffwll.ch> wrote:
>>
>>> Personally I don't care which kind of text markup we pick and wich
>>> converter, as long as the project looks reasonable far away from
>>> immeninent death (way too many one-person projects on github in this
>>> area).

I wouldn't care either. I choose pandoc as it's old, kind of stable and
has a decent community around it, so it won't probably die due the lack
of interested developers.

That said, cmark is not a option (at least for now). With the current
approach we need a tool capable of reading markdown text and output
DocBook tags, as we're actually still using docbook to spit all kinds of
Documentation formats (html, man, pdf). Looks like cmark can only do
markdown->html.

I did look into other options though, but they were all mostly one-man
job. But yeah, basically any tool capable of converting
markdown->docbook will work. I'm always listening to suggestions =)


>>>
>>> But if we have this discussion I'd like to decouple it from the other
>>> kerneldoc improvemnts in this series (patches 1, 5 and 6). If we cant
>>> agree on the text markup then drm docs will simply look a bit funny for
>>> everyone else. But if the inline struct stuff won't happen 0-day will
>>> scream around (and there's already patches which use the new layout).
>>
>> Those patches are:
>>
>> 1)  scripts/kernel-doc: Replacing highlights hash by an array
>> 5)  scripts/kernel-doc: Improve Markdown results
>> 6)  scripts/kernel-doc: Processing -nofunc for functions only
>>
>> #1 is fine, I'll merge that today.  #6 is already merged.  #5 is a markdown
>> patch, though, and doesn't make sense without the others?  Are you thinking
>> about #3 (drm/doc: Convert to markdown)?  That one would almost work (it
>> depends on #2 currently) and it nicely shows *why* I'd like to get away from
>> XML...

#5 is the DRM markdown conversion, which depends on #2 and #4. So if a
decision about this is going during KS, we need to hold those three patches.

> 
> Sorry I mixed things up, #5 is ok to leave out. I thought about
> "scripts/kernel-doc Allow struct arguments documentation in struct
> body" but you already merged that one for 4.2. Which I missed, but
> which is great since that's the one that would cause the big
> conflicts.
> 
>>>> 2 We're constructing an increasingly complicated document-processing
>>>>   mechanism with a lot of independently moving parts.  What if we
>>>>   converted the whole document to markdown and dispensed with the XML
>>>>   part altogether?  Making the source files simpler and dispensing with
>>>>   the xmlto requirement would be a big win, IMO.
>>>
>>> Who's going to convert the almost 30kloc of xml template (which often have
>>> large amounts of texts) and the over 60k kerneldoc comments that we have
>>> already?
>>
>> I thought you'd do that :)
>>
>> Seriously, though, a change would be a big job.  There's a reason I've said
>> several times that it would make no sense to delay the work at hand in the
>> hopes of somebody doing this whole job instead.  But we can certainly
>> ponder what might be better.
>>
>> Getting rid of the XML stuff may well simplify the whole process and make
>> the documentation much more accessible for those who would change it; that
>> could be a goal worth going for.  Oh, and anything requiring changes to the
>> kerneldoc comments is going nowhere, that was never something I'd
>> contemplated.  Those comments are fine.
> 
> Well my goal is to be able to have all the programmer docs in the
> code, including any high-level overview sections to tie everything
> together (hence hyperlinks and better formatting). But then you still
> need some skeleton to make a coherent whole out of all the parts, and
> I think the docbook xml templates we have serve rather fine for that
> role. Writing text in xml is indeed horrid, but if we can create
> in-line DOC: sections with decent formatting there's really no need
> for that. From that angle this work here already has the goal to get
> rid of the xml - I plane to move all the existing text in the drm.tmpl
> xml into inline DOC: kerneldoc commets.
> 
> And at least gtkdoc (which we use extensively for the userpace
> test/tools repo) relies on some xml thing to tie different topics
> together too. So that seems pretty standard.
> 
>> But any such change needs a lot of thought and a reasonable proof of
>> concept.  Meanwhile we have work that can make the docs better now, and I
>> want to merge it.  But we should choose the tools we use carefully, and I
>> anticipate a lot of opposition to one that has to drag in 70 Haskell
>> packages with it.
> 
> Well personally I don't care about the exact tooling, as long as:
> - it's a reasonable alive project
> - packages available on distros (works for me on both debian and fedora)
> - we can muck around with how things are integrated into overall docs
> (which is the case since pandoc is only used as a paragraph formatter
> here).
> 
> The last one is something we stumbled over for the userspace side
> where we wanted to have better tooling for documenting testcases.
> Upstream gtkdoc is receptive, but getting anything in takes forever
> and then there's the 6 month release schedule too. And there's also
> the problem that we can't change the kerneldoc grammar (e.g. gtkdoc
> has the code comment parser integrated and ofc it has slightly
> different rules to mark up references to structs/enums).
> 
> I'm also trying to build up a doc team here with a full-time tech
> writer for drm/i915 topics. So I'm very much interested in keeping
> this all alive. But I don't have the time to mass-convert everything
> else, and I'm also not sure it's a good idea: In my experience all the
> doc toolchain suck somewhat, and it's better to be consistent within a
> project instead of proliferating different solutions. It's hard enough
> already to get developers to write docs and even harder to find
> someone who writes good docs ...
> 
> Ofc we already have a big split between the ad-hoc text files in
> Documentation and the docbook stuff in Documentation/DocBook. Otoh
> docbook is the only way to extract the kerneldoc comments into
> something readable and indexed.
> 
>>>> I will not make #2 be a precondition to getting some form of this work
>>>> merged, but I would like to have a good answer for #1.  Adding such a
>>>> heavyweight dependency (even as an optional one) needs to have a pretty
>>>> good story behind it.
>>>
>>> Should we discuss #1 at ks? Imo as long as no one pipes up to do the
>>> massive conversion away from the current toolchain (and subsystem
>>> maintainers won't just kill that effort because it causes too much churn)
>>> moving forward with the kerneldoc toolchain we have makes sense. Hence I'd
>>> like to see those patches landed before we resolve #1 if possible.
>>
>> I've tossed the topic out there; the kernel summit agenda hasn't been made
>> yet, though.  In any case, I'd be happy to resolve this particular issue
>> before then if it were possible.
> 
> I don't mind much if we delay this part until KS. All the text markup
> styles are pretty close. So fixing up one dialect for another for the
> few patches I'll likely pull in for 4.4 won't be a lot of work if we
> decide to go with something else. Or even if we decide to switch
> markup styles much later, it should all be doable with a few sed jobs.
> 
> Only problem I see is that thus far no one else seems to care much,
> and if we change the approach I need to convince management here again
> that we need to invest some time to make it happen (and some excuses
> why this approach here wasn't ok). That will probably suck down more
> time than converting to whatever we decide on. And like I said I
> personally don't care one bit what the toolchain looks like or the
> exact flavour of text markup we pick.
> 
> Cheers, Daniel
>