Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster <armbru@redhat.com>]

John Snow <jsnow@redhat.com> writes:

> On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster <armbru@redhat.com>
> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster <armbru@redhat.com> wrote:
>> >
>> >> John Snow <jsnow@redhat.com> writes:
>> >>
>> >> > This patch does three things:
>> >> >
>> >> > 1. Record the current namespace context in pending_xrefs so it can be
>> >> >    used for link resolution later,
>> >> > 2. Pass that recorded namespace context to find_obj() when resolving a
>> >> >    reference, and
>> >> > 3. Wildly and completely rewrite find_obj().
>> >> >
>> >> > cross-reference support is expanded to tolerate the presence or absence
>> >> > of either namespace or module, and to cope with the presence or absence
>> >> > of contextual information for either.
>> >> >
>> >> > References now work like this:
>> >> >
>> >> > 1. If the explicit reference target is recorded in the domain's object
>> >> >    registry, we link to that target and stop looking. We do this lookup
>> >> >    regardless of how fully qualified the target is, which allows direct
>> >> >    references to modules (which don't have a module component to their
>> >> >    names) or direct references to definitions that may or may not belong
>> >> >    to a namespace or module.
>> >> >
>> >> > 2. If contextual information is available from qapi:namespace or
>> >> >    qapi:module directives, try using those components to find a direct
>> >> >    match to the implied target name.
>> >> >
>> >> > 3. If both prior lookups fail, generate a series of regular expressions
>> >> >    looking for wildcard matches in order from most to least
>> >> >    specific. Any explicitly provided components (namespace, module)
>> >> >    *must* match exactly, but both contextual and entirely omitted
>> >> >    components are allowed to differ from the search result. Note that if
>> >> >    more than one result is found, Sphinx will emit a warning (a build
>> >> >    error for QEMU) and list all of the candidate references.
>> >> >
>> >> > The practical upshot is that in the large majority of cases, namespace
>> >> > and module information is not required when creating simple `references`
>> >> > to definitions from within the same context -- even when identical
>> >> > definitions exist in other contexts.
>> >>
>> >> Can you illustrate this this examples?
>> >>
>> >
>> > do wha?
>>
>> Sorry, I went into the curve too fast.
>>
>> The stuff under "References now work like this" confuses me.  I guess it
>> describes a series of lookups to try one after the other.
>>
>> I understand a cross-reference consists of namespace (optional), module
>> (optional), name, and role.
>>
>> Let's assume role is "any" for simplicity's sake.
>>
>> Regarding "1. If the explicit ...":
>>
>>     What is a reference's "explicit reference target"?  Examples might
>>     help me understand.
>>
>
> explicit lookup: `QMP:block-core:block-dirty-bitmap-add`
>
> If that explicit target matches an object in the object database
> *directly*, we match immediately and don't consider other potential
> targets. This also applies to things like modules, e.g. `QMP:block-core`
> even though the "module" is absent (it IS the module)
>
> We always search for the explicit target no matter how un/fully qualified
> it is.
>
>
>>
>>     What is "recorded in the domain's object registry"?
>>
>
> domain.objects{} - essentially a record of every ObjectDefinition's
> "fullname" - the return value from QAPIDefinition._get_fqn().
>
>
>>
>>     Can you show me a reference where this lookup succeeds?
>>
>
> `QMP:block-core`
> `QMP:block-core.block-dirty-bitmap-add`

So, for this lookup to work, the reference must either be of the form
NAMESPACE:MODULE and resolve to that module in that namespace, or of the
form NAMESPACE:MODULE:DEFN and resolve to that definition in that module
in that namespace.  Correct?

These a "fully qualified names (FQN)" in your parlance, right?

Note that the first form is syntactically indistinguishable from
NAMESPACE:DEFN, i.e. a reference to a definition that specifies the
namespace, but not the module.

If the NAMESPACE:MODULE interpretation resolves, we never try the
NAMESPACE:DEFN interpretation, because that happens in later steps.
Correct?

The first form is fully qualified only if it resolves as FQN.  So,
whether such a reference is fully qualified is not syntactically
decidable.  Hmm.

>> Regarding "2. If contextual information ...":
>>
>>     I guess "contextual information" is the context established by
>>     qapi:namespace and qapi:module directives, i.e. the current
>>     namespace and module, if any.
>>
>
> Yep!
>
>
>>
>>     If the cross reference lacks a namespace, we substitute the current
>>     namespace.  Same for module.
>>
>>     We then use that "to find a direct match to the implied target
>>     name".  Sounds greek to me.  Example(s) might help.
>>
>
> If namespace or module is missing from the link target, we try to fill in
> the blanks with the contextual information if present.
>
> Example, we are in the block-core section of the QEMU QMP reference manual
> document and we reference `block-dirty-bitmap-add`. With context, we are
> able to assemble a fully qualified name:
> "QMP:block-core.block-dirty-bitmap-add`. This matches an item in the
> registry directly, so it matches and no further search is performed.

We try this lookup only when the reference lacks a namespace and we are
"in" a namespace, or when it lacks a module and we are "in" a module.
Correct?

We then subsitute current namespace / module for the lacking one(s), and
try the same lookup as in 1.  Correct?

If we have a reference of the form MYSTERY, it could either be a
reference to module MYSTERY in the current namespace, or to definition
MYSTERY in the current namespace and module.  How do we decide?

>> Regarding "3. If both prior lookups fail ...":
>>
>>     I guess we get here when namespace or module are absent, and
>>     substituting the current namespace or module doesn't resolve.  We
>>     then substitute a wildcard, so to speak, i.e. look in all namespaces
>>     / modules, and succeed if we find exactly one resolution.  Fair?
>>
>
> More or less, though the mechanics are quite a bit more complex than your
> overview (and what I wrote in qapi-domain.rst.) We can get here for a few
> reasons:
>
> (1) We didn't provide a fully qualified target, and we don't have full
> context to construct one. For example, we are not "in" a namespace and/or
> not "in" a module. This is quite likely to happen when writing simple
> references to a definition name from outside of the transmogfrified QAPI
> documentation, e.g. from qapi-domain.rst itself, or dirty-bitmaps.rst, etc.
>
> (2) We didn't provide a fully qualified target, and we are referencing
> something from outside of the local context. For example, we are "in" a
> module but we are trying to link to a different module's definition. e.g.
> we are in QMP:transaction and we reference `block-dirty-bitmap-add`. The
> implied FQN will be QMP:transaction.block-dirty-bitmap.add, which will not
> resolve.
>
> The fuzzy search portion has an order of precedence for how it searches -
> and not all searches are tried universally, they are conditional to what
> was provided in the reference target and what context is available.
>
> 1. match against the explicitly provided namespace (module was not
> explicitly provided)

Look for the name in all of the namespace's modules?

> 2. match against the explicitly provided module (namespace was not
> explicitly provided)

Look for the name in all modules so named in all namespaces?

> 3. match against the implied namespace (neither namespace/module was
> explicitly provided)

?

> 4. match against the implied module (neither namespace/module was
> explicitly provided)

?

> 5. match against the definition name only, from anywhere (neither
> namespace/module was explicitly provided)

Look for the name anywhere?

I need examples :)

> The searches are performed in order: if a search returns zero results, the
> next search is tried. If any search returns one or more results, those
> results are returned and we stop searching down the list. The priority
> order ensures that any explicitly provided information is *always* used to
> find a match, but contextually provided information is merely a "hint" and
> can be ignored for the sake of a match.
>
> If find_obj() as a whole returns zero results, Sphinx emits a warning for a
> dangling reference. if find_obj() as a whole returns multiple results,
> Sphinx emits a warning for the ambiguous cross-reference.
>
> QEMU errors out on any such warnings under our normal build settings.
>
> Clear as mud?

Clearer, but not quite mud, yet.

> --js
>
>
>> [...]
>>
>>