[Qemu-devel] [PATCH 0/5] RFC: generate QAPI doc from json

[Qemu-devel] [PATCH 0/5] RFC: generate QAPI doc from json

[Marc-André Lureau]

Hi,

This is a small series adding a script to generate documentation of
QAPI/QMP based on json "schema" files (man/pdf/html...)

According to the wiki page http://wiki.qemu.org/QMP [1], this is one
of the goal to get rid of duplicated documentation in qmp-commands.hx
by generating it from json.

A preview of the PDF result can be found there:
https://elmarco.fedorapeople.org/qemu-qapi.pdf

[1] Could someone create an account for me, so I can point to this
    series in the wiki? thanks

Marc-André Lureau (5):
  qapi: add comment block before ChardevDummy
  qapi: add missing @
  qapi: add qapi2texi script
  texi2pod: learn quotation, deftp and deftypefn
  build-sys: generate QAPI doc based on json

 Makefile             |  35 +++++++--
 qapi-schema.json     |   8 +-
 scripts/qapi.py      |  78 ++++++++++++++++++-
 scripts/qapi2texi.py | 212 +++++++++++++++++++++++++++++++++++++++++++++++++++
 scripts/texi2pod.pl  |  44 ++++++++++-
 5 files changed, 365 insertions(+), 12 deletions(-)
 create mode 100644 scripts/qapi2texi.py

-- 
2.4.3

[Qemu-devel] [PATCH 1/5] qapi: add comment block before ChardevDummy

[Marc-André Lureau]

This is mainly to please the doc generation that requires comment block
before the declaration.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 qapi-schema.json | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/qapi-schema.json b/qapi-schema.json
index 106008c..71c894e 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -3016,13 +3016,17 @@
 { 'struct': 'ChardevRingbuf', 'data': { '*size'  : 'int' } }
 
 ##
+# @ChardevDummy
+##
+{ 'struct': 'ChardevDummy', 'data': { } }
+
+##
 # @ChardevBackend:
 #
 # Configuration info for the new chardev backend.
 #
 # Since: 1.4 (testdev since 2.2)
 ##
-{ 'struct': 'ChardevDummy', 'data': { } }
 
 { 'union': 'ChardevBackend', 'data': { 'file'   : 'ChardevFile',
                                        'serial' : 'ChardevHostdev',
-- 
2.4.3

[Qemu-devel] [PATCH 2/5] qapi: add missing @

[Marc-André Lureau]

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 qapi-schema.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/qapi-schema.json b/qapi-schema.json
index 71c894e..1d3ee6b 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -15,7 +15,7 @@
 { 'include': 'qapi/trace.json' }
 
 ##
-# LostTickPolicy:
+# @LostTickPolicy:
 #
 # Policy for handling lost ticks in timer devices.
 #
-- 
2.4.3

[Qemu-devel] [PATCH 3/5] qapi: add qapi2texi script

[Marc-André Lureau]

From: Marc-André Lureau <marcandre.lureau@gmail.com>

As the name suggests, the qapi2texi script converts JSON QAPI
description into a standalone texi file suitable for different target
formats.

As the documentation format doesn't seem to be specified, it parses the
following blocks before each declaration with some variations:

  ##
  # @symbol
  #
  # body
  #
  # @arg: foo
  # @arg: #optional foo
  #
  # Returns: returns
  # Since: version
  # Notes: notes
  ##

Using the json declaration, it's able to give extra information about
the type of arguments and return value expected.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/qapi.py      |  78 ++++++++++++++++++-
 scripts/qapi2texi.py | 212 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 286 insertions(+), 4 deletions(-)
 create mode 100644 scripts/qapi2texi.py

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 06d7fc2..70208e8 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -103,6 +103,53 @@ class QAPIExprError(Exception):
         return error_path(self.info['parent']) + \
             "%s:%d: %s" % (self.info['file'], self.info['line'], self.msg)
 
+class QAPIDoc:
+    def __init__(self, comment):
+        self.symbol = None
+        self.comment = ""
+        self.args = OrderedDict()
+        self.meta = OrderedDict()
+        self.section = None
+
+        for line in comment.split('\n'):
+            line = ' '.join(line.split())
+            split = line.split(' ', 1)
+            key = split[0].rstrip(':')
+
+            if key.startswith("@"):
+                key = key[1:]
+                line = split[1] if len(split) > 1 else ""
+                if self.symbol == None:
+                    self.symbol = key
+                else:
+                    self.start_section(self.args, key)
+            elif key in ("Since", "Returns", "Notes"):
+                line = split[1] if len(split) > 1 else ""
+                self.start_section(self.meta, key)
+            self.append_comment(line)
+
+        self.end_section()
+
+    def append_comment(self, line):
+        if len(line) == 0:
+            return
+        if self.section != None:
+           self.section.append(line)
+        elif self.comment == "":
+            self.comment = line
+        else:
+            self.comment += line if line[0] == ' ' else " " + line
+
+    def end_section(self):
+        if self.section != None:
+            dic = self.section[0]
+            key = self.section[1]
+            dic[key] = " ".join(self.section[2:]).strip()
+
+    def start_section(self, dic, key):
+        self.end_section()
+        self.section = [dic, key, ""]
+
 class QAPISchema:
 
     def __init__(self, fp, previously_included = [], incl_info = None):
@@ -118,11 +165,14 @@ class QAPISchema:
         self.line = 1
         self.line_pos = 0
         self.exprs = []
+        self.comment = ""
+        self.apidoc = []
         self.accept()
 
         while self.tok != None:
             expr_info = {'file': fname, 'line': self.line,
-                         'parent': self.incl_info}
+                         'parent': self.incl_info, 'doc': self.apidoc}
+            self.apidoc = []
             expr = self.get_expr(False)
             if isinstance(expr, dict) and "include" in expr:
                 if len(expr) != 1:
@@ -154,7 +204,7 @@ class QAPISchema:
                 self.exprs.extend(exprs_include.exprs)
             else:
                 expr_elem = {'expr': expr,
-                             'info': expr_info}
+                             'info': expr_info }
                 self.exprs.append(expr_elem)
 
     def accept(self):
@@ -165,8 +215,19 @@ class QAPISchema:
             self.val = None
 
             if self.tok == '#':
-                self.cursor = self.src.find('\n', self.cursor)
-            elif self.tok in ['{', '}', ':', ',', '[', ']']:
+                end = self.src.find('\n', self.cursor)
+                if self.src[self.cursor] != "#":
+                    self.comment += self.src[self.cursor:end+1]
+                self.cursor = end+1
+                continue
+            else:
+                apidoc = QAPIDoc(self.comment)
+                if apidoc.symbol != None or \
+                   not apidoc.comment.find("*-*"):
+                    self.apidoc.append(apidoc)
+                self.comment = ""
+
+            if self.tok in ['{', '}', ':', ',', '[', ']']:
                 return
             elif self.tok == "'":
                 string = ''
@@ -762,6 +823,15 @@ def parse_schema(fname):
         print >>sys.stderr, e
         exit(1)
 
+def parse_schema_full(fname):
+    try:
+        schema = QAPISchema(open(fname, "r"))
+        check_exprs(schema.exprs)
+        return schema.exprs
+    except (QAPISchemaError, QAPIExprError), e:
+        print >>sys.stderr, e
+        exit(1)
+
 #
 # Code generation helpers
 #
diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
new file mode 100644
index 0000000..a3879a4
--- /dev/null
+++ b/scripts/qapi2texi.py
@@ -0,0 +1,212 @@
+# QAPI texi generator
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from qapi import *
+
+command_fmt = """
+@deftypefn {type} {{{ret}}} {name} @
+{{{args}}}
+
+{body}
+
+@end deftypefn
+
+""".format
+
+enum_fmt = """
+@deftp Enum {name}
+
+{body}
+
+@end deftp
+
+""".format
+
+struct_fmt = """
+@deftp {type} {name} @
+{{{attrs}}}
+
+{body}
+
+@end deftp
+
+""".format
+
+
+def subst_vars(doc):
+    return re.sub(r'@(\w*)', r'@var{\1}', doc)
+
+
+def texi_args(expr):
+    data = expr["data"] if "data" in expr else {}
+    if type(data) == str:
+        args = data
+    else:
+        args = []
+        for v, t in data.iteritems():
+            if type(t) == str and t.startswith("**"):
+                args.append("%s@dots{}" % v)
+            elif v.startswith("*"):
+                v = v[1:]
+                args.append("['%s': @var{%s}]" % (v, t))
+            else:
+                args.append("'%s': @var{%s}" % (v, t))
+        args = ", ".join(args)
+    return args
+
+
+def texi_body(doc, arg="@var"):
+    body = "@table %s\n" % arg
+    for arg, c in doc.args.iteritems():
+        if c.startswith("#optional"):
+            c = c[10:]
+            arg += "*"
+        body += "@item %s\n%s\n" % (arg, subst_vars(c))
+    body += "@end table\n"
+    body += subst_vars(doc.comment)
+
+    for k in ("Returns", "Notes", "Since"):
+        if k not in doc.meta:
+            continue
+        body += "\n@quotation %s\n%s\n@end quotation" % \
+                (k, subst_vars(doc.meta[k]))
+    return body
+
+
+def texi_alternate(expr, doc):
+    args = texi_args(expr)
+    body = texi_body(doc)
+    print struct_fmt(type="Alternate",
+                     name=doc.symbol,
+                     attrs="[ " + args + " ]",
+                     body=body)
+
+
+def texi_union(expr, doc):
+    args = texi_args(expr)
+    body = texi_body(doc)
+    print struct_fmt(type="Union",
+                     name=doc.symbol,
+                     attrs="[ " + args + " ]",
+                     body=body)
+
+
+def texi_enum(expr, doc):
+    body = texi_body(doc, "@samp")
+    print enum_fmt(name=doc.symbol,
+                   body=body)
+
+
+def texi_struct(expr, doc):
+    args = texi_args(expr)
+    body = texi_body(doc)
+    print struct_fmt(type="Struct",
+                     name=doc.symbol,
+                     attrs="@{ " + args + " @}",
+                     body=body)
+
+
+def texi_command(expr, doc):
+    args = texi_args(expr)
+    ret = expr["returns"] if "returns" in expr else ""
+    body = texi_body(doc)
+    print command_fmt(type="Command",
+                      name=doc.symbol,
+                      ret=ret,
+                      args="(" + args + ")",
+                      body=body)
+
+
+def texi_event(expr, doc):
+    args = texi_args(expr)
+    body = texi_body(doc)
+    print command_fmt(type="Event",
+                      name=doc.symbol,
+                      ret="",
+                      args="(" + args + ")",
+                      body=body)
+
+
+if len(sys.argv) != 5:
+    print >>sys.stderr, "%s: need exactly 4 arguments" % sys.argv[0]
+    sys.exit(1)
+
+exprs = parse_schema_full(sys.argv[4])
+
+print """
+\input texinfo
+@setfilename {filename}
+@documentlanguage en
+@exampleindent 0
+@paragraphindent 0
+
+@settitle {title}
+
+@ifinfo
+@direntry
+* QEMU: (qemu-doc).    {title}
+@end direntry
+@end ifinfo
+
+@titlepage
+@title {title} {version}
+@end titlepage
+
+@ifnottex
+@node Top
+@top
+
+This is the API reference for QEMU {version}.
+
+@menu
+* API Reference::
+* Commands and Events Index::
+* Data Types Index::
+@end menu
+
+@end ifnottex
+
+@contents
+
+@node API Reference
+@chapter API Reference
+
+@c man begin DESCRIPTION
+""".format(title=sys.argv[1], version=sys.argv[2], filename=sys.argv[3])
+
+for cmd in exprs:
+    expr = cmd['expr']
+    docs = cmd['info']['doc']
+    (kind, name) = expr.items()[0]
+
+    for d in docs[0:-1]:
+        print d.comment
+
+    texi = {"command": texi_command,
+            "struct": texi_struct,
+            "enum": texi_enum,
+            "union": texi_union,
+            "alternate": texi_alternate,
+            "event": texi_event}
+    try:
+        texi[kind](expr, docs[-1])
+    except KeyError:
+        raise ValueError("Unknown expression kind '%s'" % kind)
+
+print """
+@c man end
+
+@c man begin SEEALSO
+The HTML documentation of QEMU for more precise information.
+@c man end
+
+@node Commands and Events Index
+@unnumbered Commands and Events Index
+@printindex fn
+@node Data Types Index
+@unnumbered Data Types Index
+@printindex tp
+@bye
+"""
-- 
2.4.3

[Qemu-devel] [PATCH 4/5] texi2pod: learn quotation, deftp and deftypefn

[Marc-André Lureau]

From: Marc-André Lureau <marcandre.lureau@gmail.com>

Learn a few more markups used for API documentation.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 scripts/texi2pod.pl | 44 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 43 insertions(+), 1 deletion(-)

diff --git a/scripts/texi2pod.pl b/scripts/texi2pod.pl
index 94097fb..3590a2d 100755
--- a/scripts/texi2pod.pl
+++ b/scripts/texi2pod.pl
@@ -37,6 +37,7 @@ $inf = "";
 $ibase = "";
 @ipath = ();
 $encoding = undef;
+@args = ();
 
 while ($_ = shift) {
     if (/^-D(.*)$/) {
@@ -162,7 +163,8 @@ while(<$inf>) {
 	if ($ended =~ /^(?:ifset|ifclear|ignore|menu|iftex)$/) {
 	    $skipping = pop @skstack;
 	    next;
-	} elsif ($ended =~ /^(?:example|smallexample|display)$/) {
+	} elsif ($ended =~ /^(?:example|smallexample|display
+                            |quotation|deftp|deftypefn)$/x) {
 	    $shift = "";
 	    $_ = "";	# need a paragraph break
 	} elsif ($ended =~ /^(?:itemize|enumerate|[fv]?table)$/) {
@@ -323,6 +325,46 @@ while(<$inf>) {
 	$_ = "\n=item ".join (" : ", @columns)."\n";
     };
 
+    /^\@(quotation)\s*(.+)?$/ and do {
+        push @endwstack, $endw;
+        $endw = $1;
+        $_ = "\n$2:"
+    };
+
+    /^{(.*)}$|^(.*)$/ and $#args > 0 and do {
+        $kind = $args[0];
+        $arguments = $1 // "";
+        if ($endw eq "deftypefn") {
+            $ret = $args[1];
+            $fname = "B<$args[2]>";
+            $_ = $ret ? "$ret " : "";
+            $_ .= "$fname $arguments ($kind)";
+        } else {
+            $_ = "B<$args[1]> ($kind)\n\n$arguments";
+        }
+        @args = ();
+    };
+
+    /^\@(deftp)\s*(.+)?$/ and do {
+        push @endwstack, $endw;
+        $endw = $1;
+        $arg = $2;
+        $arg =~ s/{([^}]*)}/$1/g;
+        $arg =~ s/\@$//;
+        @args = split (/ /, $arg);
+        $_ = "";
+    };
+
+    /^\@(deftypefn)\s*(.+)?$/ and do {
+        push @endwstack, $endw;
+        $endw = $1;
+        $arg = $2;
+        $arg =~ s/{([^}]*)}/$1/g;
+        $arg =~ s/\@$//;
+        @args = split (/ /, $arg);
+        $_ = "";
+    };
+
     /^\@itemx?\s*(.+)?$/ and do {
 	if (defined $1) {
 	    # Entity escapes prevent munging by the <> processing below.
-- 
2.4.3

[Qemu-devel] [PATCH 5/5] build-sys: generate QAPI doc based on json

[Marc-André Lureau]

Learn to generate info/html/pdf/man documentation for QEMU and agent QMP
APIs.

This allows to provide missing agent documentation, and should help
getting rid of the duplicate documentation in qmp-commands.hx.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 Makefile | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/Makefile b/Makefile
index c9be643..c176650 100644
--- a/Makefile
+++ b/Makefile
@@ -89,6 +89,7 @@ HELPERS-$(CONFIG_LINUX) = qemu-bridge-helper$(EXESUF)
 
 ifdef BUILD_DOCS
 DOCS=qemu-doc.html qemu-tech.html qemu.1 qemu-img.1 qemu-nbd.8 qmp-commands.txt
+DOCS+=qemu-qapi.7 qemu-ga-qapi.7
 ifdef CONFIG_LINUX
 DOCS+=kvm_stat.1
 endif
@@ -244,6 +245,7 @@ qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated
 gen-out-type = $(subst .,-,$(suffix $@))
 
 qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
+qapi-py += $(SRC_PATH)/scripts/qapi2texi.py
 
 qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h :\
 $(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
@@ -354,7 +356,7 @@ distclean: clean
 	rm -f qemu-doc.vr
 	rm -f config.log
 	rm -f linux-headers/asm
-	rm -f qemu-tech.info qemu-tech.aux qemu-tech.cp qemu-tech.dvi qemu-tech.fn qemu-tech.info qemu-tech.ky qemu-tech.log qemu-tech.pdf qemu-tech.pg qemu-tech.toc qemu-tech.tp qemu-tech.vr
+	rm -f qemu-{tech,qapi,ga-qapi}.{info,aux,cp,dvi,fn,info,.ky,log,pdf,pg,toc,tp,vr}
 	for d in $(TARGET_DIRS); do \
 	rm -rf $$d || exit 1 ; \
         done
@@ -399,6 +401,7 @@ ifneq ($(TOOLS),)
 	$(INSTALL_DATA) qemu-img.1 "$(DESTDIR)$(mandir)/man1"
 	$(INSTALL_DIR) "$(DESTDIR)$(mandir)/man8"
 	$(INSTALL_DATA) qemu-nbd.8 "$(DESTDIR)$(mandir)/man8"
+	$(INSTALL_DATA) qemu-qapi.7 qemu-ga-qapi.7 "$(DESTDIR)$(mandir)/man7"
 endif
 endif
 ifdef CONFIG_VIRTFS
@@ -514,6 +517,16 @@ qmp-commands.txt: $(SRC_PATH)/qmp-commands.hx
 qemu-img-cmds.texi: $(SRC_PATH)/qemu-img-cmds.hx
 	$(call quiet-command,sh $(SRC_PATH)/scripts/hxtool -t < $< > $@,"  GEN   $@")
 
+qemu-qapi.texi: $(qapi-modules) $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py \
+	"QEMU QAPI Reference Manual" $(VERSION) qemu-qapi \
+	$< > $@,"  GEN   $@")
+
+qemu-ga-qapi.texi: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py \
+	"QEMU Guest Agent API Reference Manual" $(VERSION) qemu-ga-qapi \
+	$< > $@,"  GEN   $@")
+
 qemu.1: qemu-doc.texi qemu-options.texi qemu-monitor.texi
 	$(call quiet-command, \
 	  perl -Ww -- $(SRC_PATH)/scripts/texi2pod.pl $< qemu.pod && \
@@ -538,16 +551,28 @@ qemu-nbd.8: qemu-nbd.texi
 	  $(POD2MAN) --section=8 --center=" " --release=" " qemu-nbd.pod > $@, \
 	  "  GEN   $@")
 
+qemu-qapi.7: qemu-qapi.texi
+	$(call quiet-command, \
+	perl -Ww -- $(SRC_PATH)/scripts/texi2pod.pl $< qemu-qapi.pod && \
+	$(POD2MAN) --section=7 --center=" " --release=" " qemu-qapi.pod > $@, \
+	"  GEN   $@")
+
+qemu-ga-qapi.7: qemu-ga-qapi.texi
+	$(call quiet-command, \
+	perl -Ww -- $(SRC_PATH)/scripts/texi2pod.pl $< qemu-ga-qapi.pod && \
+	$(POD2MAN) --section=7 --center=" " --release=" " qemu-ga-qapi.pod > $@, \
+	"  GEN   $@")
+
 kvm_stat.1: scripts/kvm/kvm_stat.texi
 	$(call quiet-command, \
 	  perl -Ww -- $(SRC_PATH)/scripts/texi2pod.pl $< kvm_stat.pod && \
 	  $(POD2MAN) --section=1 --center=" " --release=" " kvm_stat.pod > $@, \
 	  "  GEN   $@")
 
-dvi: qemu-doc.dvi qemu-tech.dvi
-html: qemu-doc.html qemu-tech.html
-info: qemu-doc.info qemu-tech.info
-pdf: qemu-doc.pdf qemu-tech.pdf
+dvi: qemu-doc.dvi qemu-tech.dvi qemu-qapi.dvi
+html: qemu-doc.html qemu-tech.html qemu-qapi.html
+info: qemu-doc.info qemu-tech.info qemu-qapi.info
+pdf: qemu-doc.pdf qemu-tech.pdf qemu-qapi.pdf
 
 qemu-doc.dvi qemu-doc.html qemu-doc.info qemu-doc.pdf: \
 	qemu-img.texi qemu-nbd.texi qemu-options.texi \
-- 
2.4.3

Re: [Qemu-devel] [PATCH 2/5] qapi: add missing @

[Markus Armbruster]

Marc-André Lureau <marcandre.lureau@redhat.com> writes:

> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  qapi-schema.json | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 71c894e..1d3ee6b 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -15,7 +15,7 @@
>  { 'include': 'qapi/trace.json' }
>  
>  ##
> -# LostTickPolicy:
> +# @LostTickPolicy:
>  #
>  # Policy for handling lost ticks in timer devices.
>  #

Copying -trivial because it is, and I don't have anything queued up.

There's another one in qapi/block.json: BiosAtaTranslation.

Preferably with that one fixed, too:
Reviewed-by: Markus Armbruster <armbru@redhat.com>

Re: [Qemu-devel] [PATCH 2/5] qapi: add missing @

[Michael Tokarev]

03.07.2015 13:09, Markus Armbruster write:
> Marc-André Lureau <marcandre.lureau@redhat.com> writes:
> 
>> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
>> ---
>>  qapi-schema.json | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/qapi-schema.json b/qapi-schema.json
>> index 71c894e..1d3ee6b 100644
>> --- a/qapi-schema.json
>> +++ b/qapi-schema.json
>> @@ -15,7 +15,7 @@
>>  { 'include': 'qapi/trace.json' }
>>  
>>  ##
>> -# LostTickPolicy:
>> +# @LostTickPolicy:

Applied to -trivial, thanks!

/mjt

Re: [Qemu-devel] [PATCH 1/5] qapi: add comment block before ChardevDummy

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 869 bytes --]

On 07/03/2015 03:51 AM, Marc-André Lureau wrote:
> This is mainly to please the doc generation that requires comment block
> before the declaration.
> 
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  qapi-schema.json | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)

> 
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 106008c..71c894e 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -3016,13 +3016,17 @@
>  { 'struct': 'ChardevRingbuf', 'data': { '*size'  : 'int' } }
>  
>  ##
> +# @ChardevDummy

It's worth adding a 'Since: 1.4' comment while at it.

> +##
> +{ 'struct': 'ChardevDummy', 'data': { } }

With that added,
Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH 3/5] qapi: add qapi2texi script

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3566 bytes --]

On 07/03/2015 03:51 AM, Marc-André Lureau wrote:
> From: Marc-André Lureau <marcandre.lureau@gmail.com>
> 
> As the name suggests, the qapi2texi script converts JSON QAPI
> description into a standalone texi file suitable for different target
> formats.
> 
> As the documentation format doesn't seem to be specified, it parses the
> following blocks before each declaration with some variations:

docs/qapi-code-gen.txt tried to give a sample documentation.  Feel free
to formalize that, and to fix non-conforming uses, if you desire.  It'll
be a big one-time audit of the .json files, but getting things
consistent, _and keeping them that way by automatic conformance checks
in your conversion tool_, is fine by me.

> 
>   ##
>   # @symbol
>   #
>   # body
>   #
>   # @arg: foo
>   # @arg: #optional foo
>   #
>   # Returns: returns
>   # Since: version
>   # Notes: notes
>   ##
> 
> Using the json declaration, it's able to give extra information about
> the type of arguments and return value expected.
> 
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  scripts/qapi.py      |  78 ++++++++++++++++++-
>  scripts/qapi2texi.py | 212 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 286 insertions(+), 4 deletions(-)
>  create mode 100644 scripts/qapi2texi.py

Are you intending to apply this after Markus' big work on QMP Introspection?

> 
> diff --git a/scripts/qapi.py b/scripts/qapi.py
> index 06d7fc2..70208e8 100644
> --- a/scripts/qapi.py
> +++ b/scripts/qapi.py
> @@ -103,6 +103,53 @@ class QAPIExprError(Exception):
>          return error_path(self.info['parent']) + \
>              "%s:%d: %s" % (self.info['file'], self.info['line'], self.msg)
>  
> +class QAPIDoc:

In particular, this should probably be QAPIDoc(object) (new style class
declaration).

> @@ -762,6 +823,15 @@ def parse_schema(fname):
>          print >>sys.stderr, e
>          exit(1)
>  
> +def parse_schema_full(fname):
> +    try:
> +        schema = QAPISchema(open(fname, "r"))
> +        check_exprs(schema.exprs)
> +        return schema.exprs
> +    except (QAPISchemaError, QAPIExprError), e:
> +        print >>sys.stderr, e
> +        exit(1)

and this may need to be reworked on top of Markus' new parser class.

> +++ b/scripts/qapi2texi.py
> @@ -0,0 +1,212 @@
> +# QAPI texi generator
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2.
> +# See the COPYING file in the top-level directory.

Although I'm not a lawyer, my understanding is that for GPL to apply,
there has to be an explicit mention of Copyright.

> +exprs = parse_schema_full(sys.argv[4])

And you'll probably want to rewrite this in terms of the visitor interface.

> +for cmd in exprs:
> +    expr = cmd['expr']
> +    docs = cmd['info']['doc']
> +    (kind, name) = expr.items()[0]
> +
> +    for d in docs[0:-1]:
> +        print d.comment
> +
> +    texi = {"command": texi_command,
> +            "struct": texi_struct,
> +            "enum": texi_enum,
> +            "union": texi_union,
> +            "alternate": texi_alternate,
> +            "event": texi_event}
> +    try:
> +        texi[kind](expr, docs[-1])
> +    except KeyError:
> +        raise ValueError("Unknown expression kind '%s'" % kind)

although this is a rather cute way to polymorphically invoke the correct
helper function.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH 5/5] build-sys: generate QAPI doc based on json

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

On 07/03/2015 03:51 AM, Marc-André Lureau wrote:
> Learn to generate info/html/pdf/man documentation for QEMU and agent QMP
> APIs.
> 
> This allows to provide missing agent documentation, and should help

s/allows to provide/provides/

> getting rid of the duplicate documentation in qmp-commands.hx.
> 
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>  Makefile | 35 ++++++++++++++++++++++++++++++-----
>  1 file changed, 30 insertions(+), 5 deletions(-)
> 
> @@ -354,7 +356,7 @@ distclean: clean
>  	rm -f qemu-doc.vr
>  	rm -f config.log
>  	rm -f linux-headers/asm
> -	rm -f qemu-tech.info qemu-tech.aux qemu-tech.cp qemu-tech.dvi qemu-tech.fn qemu-tech.info qemu-tech.ky qemu-tech.log qemu-tech.pdf qemu-tech.pg qemu-tech.toc qemu-tech.tp qemu-tech.vr
> +	rm -f qemu-{tech,qapi,ga-qapi}.{info,aux,cp,dvi,fn,info,.ky,log,pdf,pg,toc,tp,vr}

{} expansion is a bashism; it will probably fail to do what you meant on
systems where /bin/sh is dash.  But you could probably come up with
makefile $() functions to generate the same sort of compact
representation in the Makefile without relying on bashism. (Hmm, $(join)
isn't pairwise, and $(addprefix) only adds a single prefix; I'm not sure
offhand how to get all possible pairwise combinations of two lists...)

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH 3/5] qapi: add qapi2texi script

[Marc-André Lureau]

Hi

----- Original Message -----
> On 07/03/2015 03:51 AM, Marc-André Lureau wrote:
> > From: Marc-André Lureau <marcandre.lureau@gmail.com>
> > 
> > As the name suggests, the qapi2texi script converts JSON QAPI
> > description into a standalone texi file suitable for different target
> > formats.
> > 
> > As the documentation format doesn't seem to be specified, it parses the
> > following blocks before each declaration with some variations:
> 
> docs/qapi-code-gen.txt tried to give a sample documentation.  Feel free
> to formalize that, and to fix non-conforming uses, if you desire.  It'll
> be a big one-time audit of the .json files, but getting things
> consistent, _and keeping them that way by automatic conformance checks
> in your conversion tool_, is fine by me.

Thanks, the patch series evolved quite a bit since this first series.

The current branch is: https://github.com/elmarco/qemu/commits/qapi

And I found about that doc, so I removed that comment in commit message.
 
> > 
> >   ##
> >   # @symbol
> >   #
> >   # body
> >   #
> >   # @arg: foo
> >   # @arg: #optional foo
> >   #
> >   # Returns: returns
> >   # Since: version
> >   # Notes: notes
> >   ##
> > 
> > Using the json declaration, it's able to give extra information about
> > the type of arguments and return value expected.
> > 
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  scripts/qapi.py      |  78 ++++++++++++++++++-
> >  scripts/qapi2texi.py | 212
> >  +++++++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 286 insertions(+), 4 deletions(-)
> >  create mode 100644 scripts/qapi2texi.py
> 
> Are you intending to apply this after Markus' big work on QMP Introspection?

Yes, I am in contact with Markus, I propose we revisit this series after his patches are merged.

Thanks for your comments so far, 

> > 
> > diff --git a/scripts/qapi.py b/scripts/qapi.py
> > index 06d7fc2..70208e8 100644
> > --- a/scripts/qapi.py
> > +++ b/scripts/qapi.py
> > @@ -103,6 +103,53 @@ class QAPIExprError(Exception):
> >          return error_path(self.info['parent']) + \
> >              "%s:%d: %s" % (self.info['file'], self.info['line'], self.msg)
> >  
> > +class QAPIDoc:
> 
> In particular, this should probably be QAPIDoc(object) (new style class
> declaration).

I just run pep8 fine on my file, I don't know about that new syntax.

> 
> > @@ -762,6 +823,15 @@ def parse_schema(fname):
> >          print >>sys.stderr, e
> >          exit(1)
> >  
> > +def parse_schema_full(fname):
> > +    try:
> > +        schema = QAPISchema(open(fname, "r"))
> > +        check_exprs(schema.exprs)
> > +        return schema.exprs
> > +    except (QAPISchemaError, QAPIExprError), e:
> > +        print >>sys.stderr, e
> > +        exit(1)
> 
> and this may need to be reworked on top of Markus' new parser class.
> 
> > +++ b/scripts/qapi2texi.py
> > @@ -0,0 +1,212 @@
> > +# QAPI texi generator
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2.
> > +# See the COPYING file in the top-level directory.
> 
> Although I'm not a lawyer, my understanding is that for GPL to apply,
> there has to be an explicit mention of Copyright.
> 

ah ok, I'll add it then


> > +exprs = parse_schema_full(sys.argv[4])
> 
> And you'll probably want to rewrite this in terms of the visitor interface.
> 
> > +for cmd in exprs:
> > +    expr = cmd['expr']
> > +    docs = cmd['info']['doc']
> > +    (kind, name) = expr.items()[0]
> > +
> > +    for d in docs[0:-1]:
> > +        print d.comment
> > +
> > +    texi = {"command": texi_command,
> > +            "struct": texi_struct,
> > +            "enum": texi_enum,
> > +            "union": texi_union,
> > +            "alternate": texi_alternate,
> > +            "event": texi_event}
> > +    try:
> > +        texi[kind](expr, docs[-1])
> > +    except KeyError:
> > +        raise ValueError("Unknown expression kind '%s'" % kind)
> 
> although this is a rather cute way to polymorphically invoke the correct
> helper function.
> 
> --
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
> 
> 

Re: [Qemu-devel] [PATCH 4/5] texi2pod: learn quotation, deftp and deftypefn

[Markus Armbruster]

Marc-André Lureau <marcandre.lureau@redhat.com> writes:

> From: Marc-André Lureau <marcandre.lureau@gmail.com>
>
> Learn a few more markups used for API documentation.
>
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>

I believe texi2pod.pl was originally stolen, most likely from GCC.  Have
you checked its current status there?

https://gcc.gnu.org/git/?p=gcc.git;a=history;f=contrib/texi2pod.pl;h=91bdbb5cea933d0381f2924ab94490fca31d5800;hb=HEAD

Should we steal the current version, then apply our own hacks (if any)
on top?

Re: [Qemu-devel] [PATCH 4/5] texi2pod: learn quotation, deftp and deftypefn

[Peter Maydell]

On 8 September 2015 at 12:45, Markus Armbruster <armbru@redhat.com> wrote:
> I believe texi2pod.pl was originally stolen, most likely from GCC.  Have
> you checked its current status there?
>
> https://gcc.gnu.org/git/?p=gcc.git;a=history;f=contrib/texi2pod.pl;h=91bdbb5cea933d0381f2924ab94490fca31d5800;hb=HEAD
>
> Should we steal the current version, then apply our own hacks (if any)
> on top?

The current version is GPLv3, so I don't think we should.

-- PMM

Re: [Qemu-devel] [PATCH 4/5] texi2pod: learn quotation, deftp and deftypefn

[Markus Armbruster]

Peter Maydell <peter.maydell@linaro.org> writes:

> On 8 September 2015 at 12:45, Markus Armbruster <armbru@redhat.com> wrote:
>> I believe texi2pod.pl was originally stolen, most likely from GCC.  Have
>> you checked its current status there?
>>
>> https://gcc.gnu.org/git/?p=gcc.git;a=history;f=contrib/texi2pod.pl;h=91bdbb5cea933d0381f2924ab94490fca31d5800;hb=HEAD
>>
>> Should we steal the current version, then apply our own hacks (if any)
>> on top?
>
> The current version is GPLv3, so I don't think we should.

You're right.

What about the latest version still under GPLv2+:
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=e3d2159eb818033b6e7590ce894cc8f964f3bcd0