Re: [PATCH 00/29] Convert QAPI doc comments to generate rST instead of texinfo

[Peter Maydell <peter.maydell@linaro.org>]

On Sat, 8 Feb 2020 at 14:16, Markus Armbruster <armbru@redhat.com> wrote:
> I understand the difficulty of parsing rST (Paolo called it "the Perl of
> markup languages" for a reason).  What I don't yet understand is (1) why
> we need to recognize the document structure of doc comments, and (2) why
> we can do that by recognizing '=' markup, but ignore the native rST
> document structure markup.

I think we're completely at cross purposes here, so there's
something I've not managed to communicate clearly.

We don't need to recognize the document structure of doc
comments, indeed my implementation does not -- it just
throws a doc comment at the rST parser to handle.

We *do* need to recognize the structure *of the document*
(ie that it does have a thing with a heading "Block devices"
that contains another thing with a heading "Background jobs"
that in turn contains documentation for JobType, JobStatus....)
so that when we're building up our tree of docutils node
objects we know when we need to create a new 'section'
node and give it a title 'Block devices' and which of
the various section nodes in the tree should have all
the nodes that make up the documentation of 'JobType'
added to it.

In order to achieve this separation (don't care about
document structure inside lumps of rST, but do want to
know what the overall section/subsection structure of
the document is), this patchset pulls the identification
of the document structure (heading/subheadings) completely
out of being something you might find in the middle of
a doc comment, and makes them their own
special kind of markup:

##
# = This is a heading
##

(In my head I find I'm thinking of this as "not actually
a doc comment", which is probably where some of my
lack of clarity is coming from, since syntactically
speaking and from the point of view of qapi/parser.py
that is a sort of doc comment.)

I suppose that there's an argument that the identification
of headings and subheadings should really be done in
qapi/parser.py, so that instead of

        vis = QAPISchemaGenRSTVisitor(self)
        vis.visit_begin(schema)
        for doc in schema.docs:
            if doc.symbol:
                vis.symbol(doc, schema.lookup_entity(doc.symbol))
            else:
                vis.freeform(doc)

you would have something more like

        vis = QAPISchemaGenRSTVisitor(self)
        vis.visit_begin(schema)
        for doc in schema.docs:
            if doc.symbol:
                vis.symbol(doc, schema.lookup_entity(doc.symbol))
            else if doc.is_section_header:
                vis.start_new_section(doc)
            else:
                vis.freeform(doc)

(with the identification of headers and pulling out of
what level of nesting this header is and what its text
is done in parser.py)

thanks
-- PMM