Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster <armbru@redhat.com>]

John Snow <jsnow@redhat.com> writes:

> On Thu, Mar 13, 2025 at 2:30 PM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > On Thu, Mar 13, 2025, 11:57 AM Markus Armbruster <armbru@redhat.com> wrote:
>> >
>> >> John Snow <jsnow@redhat.com> writes:
>> >>
>> >> > On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster <armbru@redhat.com wrote:

[...]

>> >> >> Regarding "2. If contextual information ...":
>> >> >>
>> >> >>     I guess "contextual information" is the context established by
>> >> >>     qapi:namespace and qapi:module directives, i.e. the current
>> >> >>     namespace and module, if any.
>> >> >>
>> >> >
>> >> > Yep!
>> >> >
>> >> >
>> >> >>
>> >> >>     If the cross reference lacks a namespace, we substitute the current
>> >> >>     namespace.  Same for module.
>> >> >>
>> >> >>     We then use that "to find a direct match to the implied target
>> >> >>     name".  Sounds greek to me.  Example(s) might help.
>> >> >>
>> >> >
>> >> > If namespace or module is missing from the link target, we try to fill in
>> >> > the blanks with the contextual information if present.
>> >> >
>> >> > Example, we are in the block-core section of the QEMU QMP reference manual
>> >> > document and we reference `block-dirty-bitmap-add`. With context, we are
>> >> > able to assemble a fully qualified name:
>> >> > "QMP:block-core.block-dirty-bitmap-add`. This matches an item in the
>> >> > registry directly, so it matches and no further search is performed.
>> >>
>> >> We try this lookup only when the reference lacks a namespace and we are
>> >> "in" a namespace, or when it lacks a module and we are "in" a module.
>> >> Correct?
>> >>
>> >
>> > or both: if we provided only a name but the context has both a namespace
>> > and module.
>>
>> So my or is inclusive :)
>>
>> > essentially the algorithm splits the explicit target into (ns, mod, name)
>> > and for any that are blank, we try to fill in those blanks with context
>> > info where available. Sometimes you have neither explicit nor contextual
>> > info for a component.
>> >
>> > Then we do a lookup for an exact match, in order;
>> >
>> > 1. explicit target name, whatever it was
>>
>> Fully qualified name.
>>
>
> Yes, for lookup to succeed it should be fully qualified, though if the
> target text is "ns:module", that's actually going to succeed here, too.
>
>
>>
>> If lookup succeeds, we're done.
>>
>> If lookup fails, we're also done.
>>
>
> If lookup fails, we actually continue on to #2, but whether or not this
> does anything useful depends on whether or not the original target text was
> fully qualified or not. If it was, #2 searches with the exact same text and
> will fail again and proceed to #3, where because we had a fully qualified
> name, none of the search conditions apply and we then just exit.
>
> (It just lacks an early return, but abstractly, if lookup on #1 fails with
> a fully qualified name, we are indeed done.)
>
> If lookup fails because it wasn't actually fully qualified, then #2 has
> some gaps to try to fill.

So, instead of

    if ref is fully qualified:
        try to look up ref
    else
        proceed to 2.

you do

    look up ref
    if found:
        ref must be fully qualified
    else
        proceed to 2.

Since "proceed to 2. won't do anything useful when ref is fully
qualified", the two should produce the same result.

>> *Except* for the ambiguous form NAMESPACE:MYSTERY.  If lookup fails for
>> that, the name is not fully qualified after all.  Probably.  Maybe.  We
>> assume it's missing a module, and proceed to 2.
>>
>> I'm mostly ignoring this exception from now on to keep things simple.
>>
>> > 2. FQN using contextual info
>>
>> Partially qualified name, but context can fill the gaps.
>>
>> If lookup succeeds, we're done.
>>
>> Else, we proceed to 3.
>>
>
> That's right.
>
>
>>
>> > and we stop after the first hit - no chance for multiple results here, just
>> > zero-or-one each step.
>> >
>> > i.e. any explicitly provided information is never "overwritten" with
>> > context, context only fills in the blanks where that info was not provided.
>> >
>> > If neither of these work, we move on to fuzzy searching.
>> >
>> >
>> >> We then subsitute current namespace / module for the lacking one(s), and
>> >> try the same lookup as in 1.  Correct?
>> >>
>> >
>> > Yes!
>> >
>> >
>> >> If we have a reference of the form MYSTERY, it could either be a
>> >> reference to module MYSTERY in the current namespace, or to definition
>> >> MYSTERY in the current namespace and module.  How do we decide?
>> >>
>> >
>> > fqn a: NS:MYSTERY
>> > fqn b: NS:MOD:MYSTERY
>> >
>> > Given we have a current ns/mod context, it's going to pick the second one.
>> >
>> > Hm. Maybe it ought to be ambiguous in this case... I'll have to revise
>> > this. (question is: how soon do we need it?)
>>
>> While we should try to put this on a more solid foundation, it is not a
>> blocker.
>>
>> >> >> Regarding "3. If both prior lookups fail ...":
>> >> >>
>> >> >>     I guess we get here when namespace or module are absent, and
>> >> >>     substituting the current namespace or module doesn't resolve.  We
>> >> >>     then substitute a wildcard, so to speak, i.e. look in all namespaces
>> >> >>     / modules, and succeed if we find exactly one resolution.  Fair?
>> >> >>
>> >> >
>> >> > More or less, though the mechanics are quite a bit more complex than your
>> >> > overview (and what I wrote in qapi-domain.rst.) We can get here for a few
>> >> > reasons:
>> >> >
>> >> > (1) We didn't provide a fully qualified target, and we don't have full
>> >> > context to construct one.
>>
>> We skipped 1. because not fully qualified, and we skipped 2. because
>> context can't fill the gaps.
>>
>
> we tried #1 and failed, then we tried #2 and failed again.

I think we're describing the same behavior with different words

Behavior:

    try to fill the gaps from context
    if all filled:
        look up
        if found:
            done

When you say "tried #2 and failed", you're referring to the entire
thing.

When I say "skipped 2.", I'm referring to the lookup.

>> >> >                           For example, we are not "in" a namespace and/or
>> >> > not "in" a module. This is quite likely to happen when writing simple
>> >> > references to a definition name from outside of the transmogfrified QAPI
>> >> > documentation, e.g. from qapi-domain.rst itself, or dirty-bitmaps.rst, etc.
>>
>> Yes.
>>
>> >> > (2) We didn't provide a fully qualified target, and we are referencing
>> >> > something from outside of the local context. For example, we are "in" a
>> >> > module but we are trying to link to a different module's definition. e.g.
>> >> > we are in QMP:transaction and we reference `block-dirty-bitmap-add`. The
>> >> > implied FQN will be QMP:transaction.block-dirty-bitmap.add, which will not
>> >> > resolve.
>>
>> We skipped 1. because not fully qualified, and we failed 2. because
>> context filled the gaps, but lookup failed.
>>
>
> Failed #1 and Failed #2.
>
>
>>
>> >> >
>> >> > The fuzzy search portion has an order of precedence for how it searches -
>> >> > and not all searches are tried universally, they are conditional to what
>> >> > was provided in the reference target and what context is available.
>> >> >
>> >> > 1. match against the explicitly provided namespace (module was not
>> >> > explicitly provided)
>> >>
>> >> Look for the name in all of the namespace's modules?
>> >>
>> >
>> > Yeah. search for "ns:*.name" and "ns:name" basically.
>>
>> Got it.
>>
>> >> > 2. match against the explicitly provided module (namespace was not
>> >> > explicitly provided)
>> >>
>> >> Look for the name in all modules so named in all namespaces?
>> >>
>> >
>> > Yes.
>>
>> Got it.
>>
>> >> > 3. match against the implied namespace (neither namespace/module was
>> >> > explicitly provided)
>> >>
>> >> ?
>> >>
>> >
>> > User gave `foo`, but we have a namespace from context, so we look for
>> > ns:*.foo or ns:foo.
>>
>> Got it.
>>
>> Detail I had not considered until now: a namespace contains modules that
>> contain definitions, but it also contains definitions directly.
>>
>> I can't recall offhand how schema.py represents this.  I'll figure it
>> out and report back.
>>
>
> I think it gets charged to a module named "qapi-schema". Silly, but it
> doesn't really break anything.

Alright, here's how it works.

A QAPI schema consists of the main schema file and optionally included
files.

Each file's definitions go into a module named like the file.  The
builtin definitions go into a special module './builtin'.

A definition is *always* in a module.  Even when the schema is
monolithic, i.e. includes nothing.

Why do you need to support definitions directly in the namespace?

[...]