[Qemu-devel] Introducing GSoC project: API Documentation Generation

[Qemu-devel] Introducing GSoC project: API Documentation Generation

[Eduardo Habkost]

Please welcome GSoC student Gabriel Barreto.  Gabriel is going to
work on QEMU API Documentation Generation[1].

Gabriel's first task is to evaluate our options for extract doc
comments from C source code and integrate them into Sphinx
documentation.  I saw that Peter has experimented with kernel-doc
in the past[2][3].  Has anybody evaluated other alternatives?
(e.g. Hawkmoth[4])

[1] https://wiki.qemu.org/Google_Summer_of_Code_2019#API_documentation_generation
[2] https://www.mail-archive.com/qemu-devel@nongnu.org/msg411643.html
[3] https://www.mail-archive.com/qemu-devel@nongnu.org/msg411841.html
[4] https://readthedocs.org/projects/hawkmoth/

-- 
Eduardo

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[John Snow]

On 5/20/19 2:41 PM, Eduardo Habkost wrote:
> Please welcome GSoC student Gabriel Barreto.  Gabriel is going to
> work on QEMU API Documentation Generation[1].
> 
> Gabriel's first task is to evaluate our options for extract doc
> comments from C source code and integrate them into Sphinx
> documentation.  I saw that Peter has experimented with kernel-doc
> in the past[2][3].  Has anybody evaluated other alternatives?
> (e.g. Hawkmoth[4])
> 
> [1] https://wiki.qemu.org/Google_Summer_of_Code_2019#API_documentation_generation
> [2] https://www.mail-archive.com/qemu-devel@nongnu.org/msg411643.html
> [3] https://www.mail-archive.com/qemu-devel@nongnu.org/msg411841.html
> [4] https://readthedocs.org/projects/hawkmoth/
> 

Gabriel, welcome! This has been something that has been bothering me a
LOT lately and I hope we'll be able to work together to improve the
state of things.

Please let us know if you need any help.

--js

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Daniel P. Berrangé]

On Mon, May 20, 2019 at 03:41:08PM -0300, Eduardo Habkost wrote:
> Please welcome GSoC student Gabriel Barreto.  Gabriel is going to
> work on QEMU API Documentation Generation[1].
> 
> Gabriel's first task is to evaluate our options for extract doc
> comments from C source code and integrate them into Sphinx
> documentation.  I saw that Peter has experimented with kernel-doc
> in the past[2][3].  Has anybody evaluated other alternatives?
> (e.g. Hawkmoth[4])

In the past I tested both GTK-DOC and Doxygen.

GTK-DOC has assumptions that you're following "normal" GTK/GLib style and
can't cope with many C type decls if you diverge, which rules it out on
practical grounds.

Doxygen can parse more or less anything which I really dislike the output
structure it generates for docs as it is not at all friendly to browser and
find the info you actually want compared to other docs tools

Hawkmoth seems pretty attractive in its output format, but doesn't appear
to be part of either Debian or Fedora distros, so we would have to bundle
it in QEMU I expect.  My big concern there is that there have only been
2 contributors to Hawkmoth in its entire 3 year existance, which makes
me fear for its long term viability if the main author gives up.

QEMU should pick a tool which is well established / widely used & thus
stands a good chance of being maintained for the long term, as we don't
want to end up relying on abandonware in 5 years time.  The kernel-doc
project is not widely used, but its main user is significant enough that
it isn't likely to die through lack of maintainers.

If we're using sphinx for the rest of our docs, then I think it is pretty
compelling to have a docs tool that integrates well with sphinx.

I personally don't care much about the source comment annotation syntax.
It is mostly just a matter of bikeshed colour picking, and no matter
which tool we pick will require updates to the source. The style /
layout / readability of the output is more important to me.

> [1] https://wiki.qemu.org/Google_Summer_of_Code_2019#API_documentation_generation
> [2] https://www.mail-archive.com/qemu-devel@nongnu.org/msg411643.html
> [3] https://www.mail-archive.com/qemu-devel@nongnu.org/msg411841.html
> [4] https://readthedocs.org/projects/hawkmoth/

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Peter Maydell]

On Tue, 21 May 2019 at 09:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
> Hawkmoth seems pretty attractive in its output format, but doesn't appear
> to be part of either Debian or Fedora distros, so we would have to bundle
> it in QEMU I expect.  My big concern there is that there have only been
> 2 contributors to Hawkmoth in its entire 3 year existance, which makes
> me fear for its long term viability if the main author gives up.

The dependency on clang 6 and the python bindings to it might also
be awkward for bundling it with QEMU...

thanks
-- PMM

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Daniel P. Berrangé]

On Tue, May 21, 2019 at 10:43:04AM +0100, Peter Maydell wrote:
> On Tue, 21 May 2019 at 09:54, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > Hawkmoth seems pretty attractive in its output format, but doesn't appear
> > to be part of either Debian or Fedora distros, so we would have to bundle
> > it in QEMU I expect.  My big concern there is that there have only been
> > 2 contributors to Hawkmoth in its entire 3 year existance, which makes
> > me fear for its long term viability if the main author gives up.
> 
> The dependency on clang 6 and the python bindings to it might also
> be awkward for bundling it with QEMU...

Requiring a second C compiler to build docs is rather unappealing :-(

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Paolo Bonzini]

On 21/05/19 10:53, Daniel P. Berrangé wrote:
> Hawkmoth seems pretty attractive in its output format, but doesn't appear
> to be part of either Debian or Fedora distros, so we would have to bundle
> it in QEMU I expect.  My big concern there is that there have only been
> 2 contributors to Hawkmoth in its entire 3 year existance, which makes
> me fear for its long term viability if the main author gives up.

On the plus side, I think the main author is among the people that
pushed rST and Sphinx in the kernel, so it's plausible that in the
future the kernel will pick Hawkmoth.  I agree that we should check with
him about his plans.

> QEMU should pick a tool which is well established / widely used & thus
> stands a good chance of being maintained for the long term, as we don't
> want to end up relying on abandonware in 5 years time.  The kernel-doc
> project is not widely used, but its main user is significant enough that
> it isn't likely to die through lack of maintainers.

A couple years ago I didn't have problems modifying kerneldoc for QEMU's
syntax, it was a 10 lines patch.  Unfortunately I cannot find it anymore.

Paolo

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 21/05/19 10:53, Daniel P. Berrangé wrote:
[...]
>> QEMU should pick a tool which is well established / widely used & thus
>> stands a good chance of being maintained for the long term, as we don't
>> want to end up relying on abandonware in 5 years time.  The kernel-doc
>> project is not widely used, but its main user is significant enough that
>> it isn't likely to die through lack of maintainers.
>
> A couple years ago I didn't have problems modifying kerneldoc for QEMU's
> syntax, it was a 10 lines patch.  Unfortunately I cannot find it anymore.

"QEMU's syntax" --- excuse me while I guffaw.

What you (quite charitably) call "syntax", I call a habit of imitating
examples.

Anyway.  What's so special about QEMU that justifies coming up with our
own doc syntax?  Other than "we made a hash of it, and cleaning it up
would be work".

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Daniel P. Berrangé]

On Tue, May 21, 2019 at 05:18:36PM +0200, Markus Armbruster wrote:
> Paolo Bonzini <pbonzini@redhat.com> writes:
> 
> > On 21/05/19 10:53, Daniel P. Berrangé wrote:
> [...]
> >> QEMU should pick a tool which is well established / widely used & thus
> >> stands a good chance of being maintained for the long term, as we don't
> >> want to end up relying on abandonware in 5 years time.  The kernel-doc
> >> project is not widely used, but its main user is significant enough that
> >> it isn't likely to die through lack of maintainers.
> >
> > A couple years ago I didn't have problems modifying kerneldoc for QEMU's
> > syntax, it was a 10 lines patch.  Unfortunately I cannot find it anymore.
> 
> "QEMU's syntax" --- excuse me while I guffaw.
> 
> What you (quite charitably) call "syntax", I call a habit of imitating
> examples.
> 
> Anyway.  What's so special about QEMU that justifies coming up with our
> own doc syntax?  Other than "we made a hash of it, and cleaning it up
> would be work".

There's really no such thing as "QEMU syntax" for docs comments right
now AFAICT. We have used at least 4 different syntaxes across the various
parts of the codebase and none seems a clear winner in terms of usage. So
I assume that whatever tool we pick, the majority of work will be updating
existing docs comments to follow whatever syntax is picked.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Peter Maydell]

On Tue, 21 May 2019 at 16:18, Markus Armbruster <armbru@redhat.com> wrote:
> Anyway.  What's so special about QEMU that justifies coming up with our
> own doc syntax?  Other than "we made a hash of it, and cleaning it up
> would be work".

The major problem as far as kernel-doc is concerned is that
it somewhat bakes in the kernel's style choice that the
'struct' keyword is not hidden behind typedefs, and so it
gets a bit confused by QEMU's "use typedefs for struct types"
style. The rest, as you say, is just a matter of fixing up
our syntax errors.

thanks
-- PMM

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Paolo Bonzini]

On 21/05/19 17:27, Peter Maydell wrote:
>> Anyway.  What's so special about QEMU that justifies coming up with our
>> own doc syntax?  Other than "we made a hash of it, and cleaning it up
>> would be work".
> The major problem as far as kernel-doc is concerned is that
> it somewhat bakes in the kernel's style choice that the
> 'struct' keyword is not hidden behind typedefs, and so it
> gets a bit confused by QEMU's "use typedefs for struct types"
> style. The rest, as you say, is just a matter of fixing up
> our syntax errors.

Exactly, "QEMU's syntax" is supposed to be actually gtkdoc, or inspired
by it, because of the similar typedef conventions.  The basic components
are:

- the head of the doc comment is either a CamelCase type or a function
name followed by parentheses

- @ introduces parameters, e.g. @path

- % introduces types, e.g. %DeviceState

- () terminate functions, e.g. memory_region_init()

- # introduces other C symbols, e.g. #NULL

and they map very well to what kerneldoc tries to parse, IIRC it only
requires some changes to the regular expression at the top of the file.

Paolo

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[John Snow]

On 5/21/19 11:27 AM, Peter Maydell wrote:
> On Tue, 21 May 2019 at 16:18, Markus Armbruster <armbru@redhat.com> wrote:
>> Anyway.  What's so special about QEMU that justifies coming up with our
>> own doc syntax?  Other than "we made a hash of it, and cleaning it up
>> would be work".
> 
> The major problem as far as kernel-doc is concerned is that
> it somewhat bakes in the kernel's style choice that the
> 'struct' keyword is not hidden behind typedefs, and so it
> gets a bit confused by QEMU's "use typedefs for struct types"
> style. The rest, as you say, is just a matter of fixing up
> our syntax errors.
> 
> thanks
> -- PMM
> 

But this is the one we're going with? Do we have a plan for teaching it
not to panic for our use of named custom types?

--js

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Eduardo Habkost]

On Tue, May 21, 2019 at 04:32:35PM -0400, John Snow wrote:
> 
> 
> On 5/21/19 11:27 AM, Peter Maydell wrote:
> > On Tue, 21 May 2019 at 16:18, Markus Armbruster <armbru@redhat.com> wrote:
> >> Anyway.  What's so special about QEMU that justifies coming up with our
> >> own doc syntax?  Other than "we made a hash of it, and cleaning it up
> >> would be work".
> > 
> > The major problem as far as kernel-doc is concerned is that
> > it somewhat bakes in the kernel's style choice that the
> > 'struct' keyword is not hidden behind typedefs, and so it
> > gets a bit confused by QEMU's "use typedefs for struct types"
> > style. The rest, as you say, is just a matter of fixing up
> > our syntax errors.
> > 
> > thanks
> > -- PMM
> > 
> 
> But this is the one we're going with? Do we have a plan for teaching it
> not to panic for our use of named custom types?

If I understood correctly, the patch from Paolo that I have
forwarded to this thread is all we need.  Are there other issues
with kernel-doc we would still need to address?

-- 
Eduardo

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Paolo Bonzini]

On 21/05/19 22:37, Eduardo Habkost wrote:
>> But this is the one we're going with? Do we have a plan for teaching it
>> not to panic for our use of named custom types?
>
> If I understood correctly, the patch from Paolo that I have
> forwarded to this thread is all we need.  Are there other issues
> with kernel-doc we would still need to address?

We can find out, but that patch is a good starting point to understand
that.  After all it's summer of code, not weekend of code. ;)

Paolo

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[John Snow]

On 5/22/19 4:20 AM, Paolo Bonzini wrote:
> On 21/05/19 22:37, Eduardo Habkost wrote:
>>> But this is the one we're going with? Do we have a plan for teaching it
>>> not to panic for our use of named custom types?
>>
>> If I understood correctly, the patch from Paolo that I have
>> forwarded to this thread is all we need.  Are there other issues
>> with kernel-doc we would still need to address?
> 
> We can find out, but that patch is a good starting point to understand
> that.  After all it's summer of code, not weekend of code. ;)
> 
> Paolo
> 

OK, if that's where we're at! I just saw the RFC from Peter Maydell and
assumed we were a little further along the decision making process, but
maybe not. I'll stay tuned.

--js

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Paolo Bonzini]

On 23/05/19 14:20, John Snow wrote:
> OK, if that's where we're at! I just saw the RFC from Peter Maydell and
> assumed we were a little further along the decision making process, but
> maybe not. I'll stay tuned.

For the decision making, yes; I think there's consensus to use
kerneldoc.  For the "debugging and seeing if anything has changed in 2.5
years", no.

Testing the patch that Eduardo posted will help Gabriel, Eduardo and
everyone else decide whether to patch kerneldoc or rather change the API
doc comments style.  (Personally I am in favor of patching; the
different coding conventions make using vanilla kerneldoc awkward, and
there are several thousands of lines of existing doc comments which
would require a transition.)

Paolo

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Eduardo Habkost]

On Fri, May 24, 2019 at 08:34:23PM +0200, Paolo Bonzini wrote:
> On 23/05/19 14:20, John Snow wrote:
> > OK, if that's where we're at! I just saw the RFC from Peter Maydell and
> > assumed we were a little further along the decision making process, but
> > maybe not. I'll stay tuned.
> 
> For the decision making, yes; I think there's consensus to use
> kerneldoc.  For the "debugging and seeing if anything has changed in 2.5
> years", no.
> 
> Testing the patch that Eduardo posted will help Gabriel, Eduardo and
> everyone else decide whether to patch kerneldoc or rather change the API
> doc comments style.  (Personally I am in favor of patching; the
> different coding conventions make using vanilla kerneldoc awkward, and
> there are several thousands of lines of existing doc comments which
> would require a transition.)

I'd prefer to fix our doc comments instead of patching kerneldoc,
whenever possible.  We don't even have a consistent doc comment
style in QEMU.

-- 
Eduardo

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Paolo Bonzini]

On 24/05/19 21:08, Eduardo Habkost wrote:
> On Fri, May 24, 2019 at 08:34:23PM +0200, Paolo Bonzini wrote:
>> On 23/05/19 14:20, John Snow wrote:
>>> OK, if that's where we're at! I just saw the RFC from Peter Maydell and
>>> assumed we were a little further along the decision making process, but
>>> maybe not. I'll stay tuned.
>>
>> For the decision making, yes; I think there's consensus to use
>> kerneldoc.  For the "debugging and seeing if anything has changed in 2.5
>> years", no.
>>
>> Testing the patch that Eduardo posted will help Gabriel, Eduardo and
>> everyone else decide whether to patch kerneldoc or rather change the API
>> doc comments style.  (Personally I am in favor of patching; the
>> different coding conventions make using vanilla kerneldoc awkward, and
>> there are several thousands of lines of existing doc comments which
>> would require a transition.)
> 
> I'd prefer to fix our doc comments instead of patching kerneldoc,
> whenever possible.  We don't even have a consistent doc comment
> style in QEMU.

I think we *mostly* do, at least as far as the @/#/% sigils are
concerned.  In particular, only "#" separates the QEMU doc comment style
from the kernel and it has 200+ instances vs. 6 for the kernel's
'&struct foo' (all in accel/tcg/translate-all.c), so it's clear that our
standard is different from the kernel in this respect.

The rest of the patch is to handle typedefed structs, which again is
more or less a necessity.

Paolo

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Eduardo Habkost]

On Tue, May 21, 2019 at 12:55:36PM +0200, Paolo Bonzini wrote:
> On 21/05/19 10:53, Daniel P. Berrangé wrote:
> > Hawkmoth seems pretty attractive in its output format, but doesn't appear
> > to be part of either Debian or Fedora distros, so we would have to bundle
> > it in QEMU I expect.  My big concern there is that there have only been
> > 2 contributors to Hawkmoth in its entire 3 year existance, which makes
> > me fear for its long term viability if the main author gives up.
> 
> On the plus side, I think the main author is among the people that
> pushed rST and Sphinx in the kernel, so it's plausible that in the
> future the kernel will pick Hawkmoth.  I agree that we should check with
> him about his plans.
> 
> > QEMU should pick a tool which is well established / widely used & thus
> > stands a good chance of being maintained for the long term, as we don't
> > want to end up relying on abandonware in 5 years time.  The kernel-doc
> > project is not widely used, but its main user is significant enough that
> > it isn't likely to die through lack of maintainers.
> 
> A couple years ago I didn't have problems modifying kerneldoc for QEMU's
> syntax, it was a 10 lines patch.  Unfortunately I cannot find it anymore.

Do you mean the following patch?

----- Forwarded message from Paolo Bonzini <pbonzini@redhat.com> -----

Date: Thu, 5 Jan 2017 17:47:30 +0100
From: Paolo Bonzini <pbonzini@redhat.com>
To: Peter Maydell <peter.maydell@linaro.org>, QEMU Developers <qemu-devel@nongnu.org>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Subject: Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)



On 07/11/2016 16:03, Peter Maydell wrote:
> 2) some of the doc-comment format differences are irritating:
>    . "function - short description" not "function: short description"
>    . "&struct.fieldname" not ".@fieldname"
>    . "&typename" not "#typename"
> 3) the most awkward part of kernel-doc syntax is that it bakes
>    in the kernel's style choice of always using "struct foo"
>    for types -- I don't think there's any way to document
>    'MemoryRegion' and 'AddressSpace' without the 'struct'
>    coming out in the documentation output.
> 
> We could fix (2) by loosening the kernel-doc script's
> parsing if we were happy to carry around a forked version
> of it. Fixing (3) requires more serious surgery on kernel-doc
> I suspect.

I've sent some changes to kernel-doc to simplify the implementation of
these changes (http://www.spinics.net/lists/linux-doc/msg42354.html) and
they were accepted.  So with 4.10 + those patches, the local changes to
kernel-doc for QEMU would be limited to the following:

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 4c9ada36fe6b..c43ac038398d 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -215,18 +215,18 @@ my $type_func = '(\w+)\(\)';
 my $type_param = '\@(\w+(\.\.\.)?)';
 my $type_fp_param = '\@(\w+)\(\)';  # Special RST handling for func ptr params
 my $type_env = '(\$\w+)';
-my $type_enum = '\&(enum\s*([_\w]+))';
-my $type_struct = '\&(struct\s*([_\w]+))';
-my $type_typedef = '\&(typedef\s*([_\w]+))';
-my $type_union = '\&(union\s*([_\w]+))';
-my $type_member = '\&([_\w]+)(\.|->)([_\w]+)';
-my $type_fallback = '\&([_\w]+)';
-my $type_enum_xml = '\&amp;(enum\s*([_\w]+))';
-my $type_struct_xml = '\&amp;(struct\s*([_\w]+))';
-my $type_typedef_xml = '\&amp;(typedef\s*([_\w]+))';
-my $type_union_xml = '\&amp;(union\s*([_\w]+))';
-my $type_member_xml = '\&amp;([_\w]+)(\.|-\&gt;)([_\w]+)';
-my $type_fallback_xml = '\&amp([_\w]+)';
+my $type_enum = '#(enum\s*([_\w]+))';
+my $type_struct = '#(struct\s*([_\w]+))';
+my $type_typedef = '#(([A-Z][_\w]*))';
+my $type_union = '#(union\s*([_\w]+))';
+my $type_member = '#([_\w]+)(\.|->)([_\w]+)';
+my $type_fallback = '(?!)';    # this never matches
+my $type_enum_xml = $type_enum;
+my $type_struct_xml = $type_struct;
+my $type_typedef_xml = $type_typedef;
+my $type_union_xml = $type_union;
+my $type_member_xml = $type_member;
+my $type_fallback_xml = $type_fallback;
 my $type_member_func = $type_member . '\(\)';
 
 # Output conversion substitutions.
@@ -2143,6 +2143,14 @@ sub output_blockhead {
 sub dump_declaration($$) {
     no strict 'refs';
     my ($prototype, $file) = @_;
+    if ($decl_type eq 'type name') {
+	if ($prototype =~ /^(enum|struct|union)\s+/) {
+	    $decl_type = $1;
+        } else {
+	    return;
+	}
+    }
+
     my $func = "dump_" . $decl_type;
     &$func(@_);
 }
@@ -2893,7 +2901,7 @@ sub process_file($) {
 	    }
 	    elsif (/$doc_decl/o) {
 		$identifier = $1;
-		if (/\s*([\w\s]+?)\s*-/) {
+		if (/\s*([\w\s]+?)(\s*-|:)/) {
 		    $identifier = $1;
 		}
 
@@ -2903,7 +2911,7 @@ sub process_file($) {
 		$contents = "";
 		$section = $section_default;
 		$new_start_line = $. + 1;
-		if (/-(.*)/) {
+		if (/[-:](.*)/) {
 		    # strip leading/trailing/multiple spaces
 		    $descr= $1;
 		    $descr =~ s/^\s*//;
@@ -2921,7 +2929,9 @@ sub process_file($) {
 			++$warnings;
 		}
 
-		if ($identifier =~ m/^struct/) {
+		if ($identifier =~ m/^[A-Z]/) {
+		    $decl_type = 'type name';
+	        } elsif ($identifier =~ m/^struct/) {
 		    $decl_type = 'struct';
 		} elsif ($identifier =~ m/^union/) {
 		    $decl_type = 'union';

which should be maintainable as a fork of Linux's kernel-doc.

I also worked a bit on support for Texinfo manuals in Sphinx.  My
current attempt is at http://people.redhat.com/pbonzini/qemu-test-doc/_build/.
Because this uses a Texinfo->Docbook->Sphinx pipeline, I also tried some
tools with native Docbook support (Publican), but despite Sphinx's quirks
the output was less usable, and the tools were slower and harder to use.

http://wiki.qemu-project.org/Features/Documentation is another place to
brainstorm ideas on this.

Paolo


----- End forwarded message -----

-- 
Eduardo

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Paolo Bonzini]

On 21/05/19 18:17, Eduardo Habkost wrote:
> On Tue, May 21, 2019 at 12:55:36PM +0200, Paolo Bonzini wrote:
>> On 21/05/19 10:53, Daniel P. Berrangé wrote:
>>> Hawkmoth seems pretty attractive in its output format, but doesn't appear
>>> to be part of either Debian or Fedora distros, so we would have to bundle
>>> it in QEMU I expect.  My big concern there is that there have only been
>>> 2 contributors to Hawkmoth in its entire 3 year existance, which makes
>>> me fear for its long term viability if the main author gives up.
>>
>> On the plus side, I think the main author is among the people that
>> pushed rST and Sphinx in the kernel, so it's plausible that in the
>> future the kernel will pick Hawkmoth.  I agree that we should check with
>> him about his plans.
>>
>>> QEMU should pick a tool which is well established / widely used & thus
>>> stands a good chance of being maintained for the long term, as we don't
>>> want to end up relying on abandonware in 5 years time.  The kernel-doc
>>> project is not widely used, but its main user is significant enough that
>>> it isn't likely to die through lack of maintainers.
>>
>> A couple years ago I didn't have problems modifying kerneldoc for QEMU's
>> syntax, it was a 10 lines patch.  Unfortunately I cannot find it anymore.
> 
> Do you mean the following patch?

You're awesome! :)

Paolo

> ----- Forwarded message from Paolo Bonzini <pbonzini@redhat.com> -----
> 
> Date: Thu, 5 Jan 2017 17:47:30 +0100
> From: Paolo Bonzini <pbonzini@redhat.com>
> To: Peter Maydell <peter.maydell@linaro.org>, QEMU Developers <qemu-devel@nongnu.org>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> Subject: Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)
> 
> 
> 
> On 07/11/2016 16:03, Peter Maydell wrote:
>> 2) some of the doc-comment format differences are irritating:
>>    . "function - short description" not "function: short description"
>>    . "&struct.fieldname" not ".@fieldname"
>>    . "&typename" not "#typename"
>> 3) the most awkward part of kernel-doc syntax is that it bakes
>>    in the kernel's style choice of always using "struct foo"
>>    for types -- I don't think there's any way to document
>>    'MemoryRegion' and 'AddressSpace' without the 'struct'
>>    coming out in the documentation output.
>>
>> We could fix (2) by loosening the kernel-doc script's
>> parsing if we were happy to carry around a forked version
>> of it. Fixing (3) requires more serious surgery on kernel-doc
>> I suspect.
> 
> I've sent some changes to kernel-doc to simplify the implementation of
> these changes (http://www.spinics.net/lists/linux-doc/msg42354.html) and
> they were accepted.  So with 4.10 + those patches, the local changes to
> kernel-doc for QEMU would be limited to the following:
> 
> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
> index 4c9ada36fe6b..c43ac038398d 100755
> --- a/scripts/kernel-doc
> +++ b/scripts/kernel-doc
> @@ -215,18 +215,18 @@ my $type_func = '(\w+)\(\)';
>  my $type_param = '\@(\w+(\.\.\.)?)';
>  my $type_fp_param = '\@(\w+)\(\)';  # Special RST handling for func ptr params
>  my $type_env = '(\$\w+)';
> -my $type_enum = '\&(enum\s*([_\w]+))';
> -my $type_struct = '\&(struct\s*([_\w]+))';
> -my $type_typedef = '\&(typedef\s*([_\w]+))';
> -my $type_union = '\&(union\s*([_\w]+))';
> -my $type_member = '\&([_\w]+)(\.|->)([_\w]+)';
> -my $type_fallback = '\&([_\w]+)';
> -my $type_enum_xml = '\&amp;(enum\s*([_\w]+))';
> -my $type_struct_xml = '\&amp;(struct\s*([_\w]+))';
> -my $type_typedef_xml = '\&amp;(typedef\s*([_\w]+))';
> -my $type_union_xml = '\&amp;(union\s*([_\w]+))';
> -my $type_member_xml = '\&amp;([_\w]+)(\.|-\&gt;)([_\w]+)';
> -my $type_fallback_xml = '\&amp([_\w]+)';
> +my $type_enum = '#(enum\s*([_\w]+))';
> +my $type_struct = '#(struct\s*([_\w]+))';
> +my $type_typedef = '#(([A-Z][_\w]*))';
> +my $type_union = '#(union\s*([_\w]+))';
> +my $type_member = '#([_\w]+)(\.|->)([_\w]+)';
> +my $type_fallback = '(?!)';    # this never matches
> +my $type_enum_xml = $type_enum;
> +my $type_struct_xml = $type_struct;
> +my $type_typedef_xml = $type_typedef;
> +my $type_union_xml = $type_union;
> +my $type_member_xml = $type_member;
> +my $type_fallback_xml = $type_fallback;
>  my $type_member_func = $type_member . '\(\)';
>  
>  # Output conversion substitutions.
> @@ -2143,6 +2143,14 @@ sub output_blockhead {
>  sub dump_declaration($$) {
>      no strict 'refs';
>      my ($prototype, $file) = @_;
> +    if ($decl_type eq 'type name') {
> +	if ($prototype =~ /^(enum|struct|union)\s+/) {
> +	    $decl_type = $1;
> +        } else {
> +	    return;
> +	}
> +    }
> +
>      my $func = "dump_" . $decl_type;
>      &$func(@_);
>  }
> @@ -2893,7 +2901,7 @@ sub process_file($) {
>  	    }
>  	    elsif (/$doc_decl/o) {
>  		$identifier = $1;
> -		if (/\s*([\w\s]+?)\s*-/) {
> +		if (/\s*([\w\s]+?)(\s*-|:)/) {
>  		    $identifier = $1;
>  		}
>  
> @@ -2903,7 +2911,7 @@ sub process_file($) {
>  		$contents = "";
>  		$section = $section_default;
>  		$new_start_line = $. + 1;
> -		if (/-(.*)/) {
> +		if (/[-:](.*)/) {
>  		    # strip leading/trailing/multiple spaces
>  		    $descr= $1;
>  		    $descr =~ s/^\s*//;
> @@ -2921,7 +2929,9 @@ sub process_file($) {
>  			++$warnings;
>  		}
>  
> -		if ($identifier =~ m/^struct/) {
> +		if ($identifier =~ m/^[A-Z]/) {
> +		    $decl_type = 'type name';
> +	        } elsif ($identifier =~ m/^struct/) {
>  		    $decl_type = 'struct';
>  		} elsif ($identifier =~ m/^union/) {
>  		    $decl_type = 'union';
> 
> which should be maintainable as a fork of Linux's kernel-doc.
> 
> I also worked a bit on support for Texinfo manuals in Sphinx.  My
> current attempt is at http://people.redhat.com/pbonzini/qemu-test-doc/_build/.
> Because this uses a Texinfo->Docbook->Sphinx pipeline, I also tried some
> tools with native Docbook support (Publican), but despite Sphinx's quirks
> the output was less usable, and the tools were slower and harder to use.
> 
> http://wiki.qemu-project.org/Features/Documentation is another place to
> brainstorm ideas on this.

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Peter Maydell]

On Mon, 20 May 2019 at 19:41, Eduardo Habkost <ehabkost@redhat.com> wrote:
>
> Please welcome GSoC student Gabriel Barreto.  Gabriel is going to
> work on QEMU API Documentation Generation[1].
>
> Gabriel's first task is to evaluate our options for extract doc
> comments from C source code and integrate them into Sphinx
> documentation.  I saw that Peter has experimented with kernel-doc
> in the past[2][3].  Has anybody evaluated other alternatives?
> (e.g. Hawkmoth[4])
>
> [1] https://wiki.qemu.org/Google_Summer_of_Code_2019#API_documentation_generation
> [2] https://www.mail-archive.com/qemu-devel@nongnu.org/msg411643.html
> [3] https://www.mail-archive.com/qemu-devel@nongnu.org/msg411841.html
> [4] https://readthedocs.org/projects/hawkmoth/

I think that kernel-doc is broadly where we'd decided we were
going with this. I have some in-progress patches that do
some integration of it: mostly just rebasing of the patches
in the git branch mentioned in your link [3] to be on top of
the sphinx support we now have in master. I'll just give
them a quick dusting off and push them out as an RFC, to
avoid duplication of work.

(The other interesting thing I'd wondered about with generation
of docs from code comments is whether we would get better
(ie more accurate, regularly updated) documentation of our
supported machine models if we generated those parts of the
docs from comments. But that's definitely much harder than just
getting API documentation, because it involves trying to
integrate them into a 'user documentation' manual which we
have not yet converted from texinfo.)

thanks
-- PMM

Re: [Qemu-devel] Introducing GSoC project: API Documentation Generation

[Paolo Bonzini]

On 21/05/19 11:42, Peter Maydell wrote:
> (The other interesting thing I'd wondered about with generation
> of docs from code comments is whether we would get better
> (ie more accurate, regularly updated) documentation of our
> supported machine models if we generated those parts of the
> docs from comments. But that's definitely much harder than just
> getting API documentation, because it involves trying to
> integrate them into a 'user documentation' manual which we
> have not yet converted from texinfo.)

For the user documentation, makeinfo can produce docbook, and that seems
to be the best way to convert out of Texinfo.  At that point you can
either pass docbook to sphinx (see
https://wiki.qemu.org/Features/Documentation though it's a bit out of
date), or convert it to rST using Pandoc.

Paolo