* [PATCH v4 01/11] qapi: golang: first level unmarshalling type
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
@ 2025-02-14 20:29 ` Victor Toso
2025-02-14 20:29 ` [PATCH v4 02/11] qapi: golang: Generate enum type Victor Toso
` (11 subsequent siblings)
12 siblings, 0 replies; 18+ messages in thread
From: Victor Toso @ 2025-02-14 20:29 UTC (permalink / raw)
To: qemu-devel
Cc: Markus Armbruster, John Snow, Daniel P . Berrangé,
Andrea Bolognani
This first patch introduces protocol.go. It introduces the Message Go
struct type that can unmarshall any QMP message.
It does not handle deeper than 1st layer of the JSON object, that is,
with:
1. {
"execute": "query-machines",
"arguments": { "compat-props": true }
}
2. {
"event": "BALLOON_CHANGE",
"data": { "actual": 944766976 },
"timestamp": {
"seconds": 1267020223,
"microseconds": 435656
}
}
We will be able to know it is a query-machine command or a
balloon-change event. Specific data type to handle arguments/data will
be introduced further in the series.
This patch also introduces the Visitor skeleton with a proper write()
function to copy-over the protocol.go to the target destination.
Note, you can execute any patch of this series with:
python3 ./scripts/qapi-gen.py -o /tmp/out qapi/qapi-schema.json
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang/__init__.py | 0
scripts/qapi/golang/golang.py | 135 ++++++++++++++++++++++++++++++++
scripts/qapi/golang/protocol.go | 48 ++++++++++++
scripts/qapi/main.py | 2 +
4 files changed, 185 insertions(+)
create mode 100644 scripts/qapi/golang/__init__.py
create mode 100644 scripts/qapi/golang/golang.py
create mode 100644 scripts/qapi/golang/protocol.go
diff --git a/scripts/qapi/golang/__init__.py b/scripts/qapi/golang/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py
new file mode 100644
index 0000000000..333741b47b
--- /dev/null
+++ b/scripts/qapi/golang/golang.py
@@ -0,0 +1,135 @@
+"""
+Golang QAPI generator
+"""
+
+# Copyright (c) 2025 Red Hat Inc.
+#
+# Authors:
+# Victor Toso <victortoso@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+# Just for type hint on self
+from __future__ import annotations
+
+import os, shutil
+from typing import List, Optional
+
+from ..schema import (
+ QAPISchema,
+ QAPISchemaBranches,
+ QAPISchemaEnumMember,
+ QAPISchemaFeature,
+ QAPISchemaIfCond,
+ QAPISchemaObjectType,
+ QAPISchemaObjectTypeMember,
+ QAPISchemaType,
+ QAPISchemaVariants,
+ QAPISchemaVisitor,
+)
+from ..source import QAPISourceInfo
+
+
+def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None:
+ vis = QAPISchemaGenGolangVisitor(prefix)
+ schema.visit(vis)
+ vis.write(output_dir)
+
+
+class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
+ # pylint: disable=too-many-arguments
+ def __init__(self, _: str):
+ super().__init__()
+ gofiles = ("protocol.go",)
+ self.schema: QAPISchema
+ self.golang_package_name = "qapi"
+ self.duplicate = list(gofiles)
+
+ def visit_begin(self, schema: QAPISchema) -> None:
+ self.schema = schema
+
+ def visit_end(self) -> None:
+ del self.schema
+
+ def visit_object_type(
+ self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: QAPISchemaIfCond,
+ features: List[QAPISchemaFeature],
+ base: Optional[QAPISchemaObjectType],
+ members: List[QAPISchemaObjectTypeMember],
+ branches: Optional[QAPISchemaBranches],
+ ) -> None:
+ pass
+
+ def visit_alternate_type(
+ self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: QAPISchemaIfCond,
+ features: List[QAPISchemaFeature],
+ variants: QAPISchemaVariants,
+ ) -> None:
+ pass
+
+ def visit_enum_type(
+ self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: QAPISchemaIfCond,
+ features: List[QAPISchemaFeature],
+ members: List[QAPISchemaEnumMember],
+ prefix: Optional[str],
+ ) -> None:
+ pass
+
+ def visit_array_type(
+ self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: QAPISchemaIfCond,
+ element_type: QAPISchemaType,
+ ) -> None:
+ pass
+
+ def visit_command(
+ self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: QAPISchemaIfCond,
+ features: List[QAPISchemaFeature],
+ arg_type: Optional[QAPISchemaObjectType],
+ ret_type: Optional[QAPISchemaType],
+ gen: bool,
+ success_response: bool,
+ boxed: bool,
+ allow_oob: bool,
+ allow_preconfig: bool,
+ coroutine: bool,
+ ) -> None:
+ pass
+
+ def visit_event(
+ self,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ ifcond: QAPISchemaIfCond,
+ features: List[QAPISchemaFeature],
+ arg_type: Optional[QAPISchemaObjectType],
+ boxed: bool,
+ ) -> None:
+ pass
+
+ def write(self, outdir: str) -> None:
+ godir = "go"
+ targetpath = os.path.join(outdir, godir)
+ os.makedirs(targetpath, exist_ok=True)
+
+ # Content to be copied over
+ srcdir = os.path.dirname(os.path.realpath(__file__))
+ for filename in self.duplicate:
+ srcpath = os.path.join(srcdir, filename)
+ dstpath = os.path.join(targetpath, filename)
+ shutil.copyfile(srcpath, dstpath)
diff --git a/scripts/qapi/golang/protocol.go b/scripts/qapi/golang/protocol.go
new file mode 100644
index 0000000000..4ff8d2f3fb
--- /dev/null
+++ b/scripts/qapi/golang/protocol.go
@@ -0,0 +1,48 @@
+/*
+ * Copyright 2025 Red Hat, Inc.
+ * SPDX-License-Identifier: MIT-0
+ *
+ * Authors:
+ * Victor Toso <victortoso@redhat.com>
+ * Daniel P. Berrange <berrange@redhat.com>
+ */
+package qapi
+
+import (
+ "encoding/json"
+ "time"
+)
+
+/* Union of data for command, response, error, or event,
+ * since when receiving we don't know upfront which we
+ * must deserialize */
+type Message struct {
+ QMP *json.RawMessage `json:"QMP,omitempty"`
+ Execute string `json:"execute,omitempty"`
+ ExecOOB string `json:"exec-oob,omitempty"`
+ Event string `json:"event,omitempty"`
+ Error *json.RawMessage `json:"error,omitempty"`
+ Return *json.RawMessage `json:"return,omitempty"`
+ ID string `json:"id,omitempty"`
+ Timestamp *Timestamp `json:"timestamp,omitempty"`
+ Data *json.RawMessage `json:"data,omitempty"`
+ Arguments *json.RawMessage `json:"arguments,omitempty"`
+}
+
+type QAPIError struct {
+ Class string `json:"class"`
+ Description string `json:"desc"`
+}
+
+func (err *QAPIError) Error() string {
+ return err.Description
+}
+
+type Timestamp struct {
+ Seconds int `json:"seconds"`
+ MicroSeconds int `json:"microseconds"`
+}
+
+func (t *Timestamp) AsTime() time.Time {
+ return time.Unix(int64(t.Seconds), int64(t.MicroSeconds)*1000)
+}
diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
index 324081b9fc..af315c1ad1 100644
--- a/scripts/qapi/main.py
+++ b/scripts/qapi/main.py
@@ -16,6 +16,7 @@
from .error import QAPIError
from .events import gen_events
from .features import gen_features
+from .golang import golang
from .introspect import gen_introspect
from .schema import QAPISchema
from .types import gen_types
@@ -55,6 +56,7 @@ def generate(schema_file: str,
gen_commands(schema, output_dir, prefix, gen_tracing)
gen_events(schema, output_dir, prefix)
gen_introspect(schema, output_dir, prefix, unmask)
+ golang.gen_golang(schema, output_dir, prefix)
def main() -> int:
--
2.48.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v4 02/11] qapi: golang: Generate enum type
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
2025-02-14 20:29 ` [PATCH v4 01/11] qapi: golang: first level unmarshalling type Victor Toso
@ 2025-02-14 20:29 ` Victor Toso
2025-02-14 20:29 ` [PATCH v4 03/11] qapi: golang: Generate alternate types Victor Toso
` (10 subsequent siblings)
12 siblings, 0 replies; 18+ messages in thread
From: Victor Toso @ 2025-02-14 20:29 UTC (permalink / raw)
To: qemu-devel
Cc: Markus Armbruster, John Snow, Daniel P . Berrangé,
Andrea Bolognani
This patch handles QAPI enum types and generates its equivalent in Go.
We sort the output based on enum's type name.
Enums are being handled as strings in Golang.
1. For each QAPI enum, we will define a string type in Go to be the
assigned type of this specific enum.
2. Naming: CamelCase will be used in any identifier that we want to
export, which is everything.
Example:
qapi:
| ##
| # @DisplayProtocol:
| #
| # Display protocols which support changing password options.
| #
| # Since: 7.0
| ##
| { 'enum': 'DisplayProtocol',
| 'data': [ 'vnc', 'spice' ] }
go:
| // Display protocols which support changing password options.
| //
| // Since: 7.0
| type DisplayProtocol string
|
| const (
| DisplayProtocolVnc DisplayProtocol = "vnc"
| DisplayProtocolSpice DisplayProtocol = "spice"
| )
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang/golang.py | 185 +++++++++++++++++++++++++++++++++-
1 file changed, 183 insertions(+), 2 deletions(-)
diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py
index 333741b47b..f074ec1f6f 100644
--- a/scripts/qapi/golang/golang.py
+++ b/scripts/qapi/golang/golang.py
@@ -13,7 +13,7 @@
# Just for type hint on self
from __future__ import annotations
-import os, shutil
+import os, shutil, textwrap
from typing import List, Optional
from ..schema import (
@@ -30,6 +30,65 @@
)
from ..source import QAPISourceInfo
+TEMPLATE_GENERATED_HEADER = """
+/*
+ * Copyright 2025 Red Hat, Inc.
+ * SPDX-License-Identifier: (MIT-0 and GPL-2.0-or-later)
+ */
+
+/****************************************************************************
+ * THIS CODE HAS BEEN GENERATED. DO NOT CHANGE IT DIRECTLY *
+ ****************************************************************************/
+package {package_name}
+"""
+
+TEMPLATE_GO_IMPORTS = """
+import (
+{imports}
+)
+"""
+
+TEMPLATE_ENUM = """
+type {name} string
+
+const (
+{fields}
+)
+"""
+
+
+# Takes the documentation object of a specific type and returns
+# that type's documentation and its member's docs.
+def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]):
+ if doc is None:
+ return "", {}
+
+ cmt = "// "
+ fmt = textwrap.TextWrapper(
+ width=70, initial_indent=cmt, subsequent_indent=cmt
+ )
+ main = fmt.fill(doc.body.text)
+
+ for section in doc.sections:
+ # TODO is not a relevant section to Go applications
+ if section.tag in ["TODO"]:
+ continue
+
+ if main != "":
+ # Give empty line as space for the tag.
+ main += "\n//\n"
+
+ tag = "" if section.tag is None else f"{section.tag}: "
+ text = section.text.replace(" ", " ")
+ main += fmt.fill(f"{tag}{text}")
+
+ fields = {}
+ for key, value in doc.args.items():
+ if len(value.text) > 0:
+ fields[key] = " ".join(value.text.replace("\n", " ").split())
+
+ return main, fields
+
def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None:
vis = QAPISchemaGenGolangVisitor(prefix)
@@ -37,20 +96,90 @@ def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None:
vis.write(output_dir)
+def qapi_to_field_name_enum(name: str) -> str:
+ return name.title().replace("-", "")
+
+
+def fetch_indent_blocks_over_enum_with_docs(
+ name: str, members: List[QAPISchemaEnumMember], docfields: Dict[str, str]
+) -> Tuple[int]:
+ maxname = 0
+ blocks: List[int] = [0]
+ for member in members:
+ # For simplicity, every time we have doc, we add a new indent block
+ hasdoc = member.name is not None and member.name in docfields
+
+ enum_name = f"{name}{qapi_to_field_name_enum(member.name)}"
+ maxname = (
+ max(maxname, len(enum_name)) if not hasdoc else len(enum_name)
+ )
+
+ if hasdoc:
+ blocks.append(maxname)
+ else:
+ blocks[-1] = maxname
+
+ return blocks
+
+
+def generate_content_from_dict(data: dict[str, str]) -> str:
+ content = ""
+
+ for name in sorted(data):
+ content += data[name]
+
+ return content.replace("\n\n\n", "\n\n")
+
+
+def generate_template_imports(words: List[str]) -> str:
+ if len(words) == 0:
+ return ""
+
+ if len(words) == 1:
+ return '\nimport "{words[0]}"\n'
+
+ return TEMPLATE_GO_IMPORTS.format(
+ imports="\n".join(f'\t"{w}"' for w in words)
+ )
+
+
class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
# pylint: disable=too-many-arguments
def __init__(self, _: str):
super().__init__()
gofiles = ("protocol.go",)
+ # Map each qapi type to the necessary Go imports
+ types = {
+ "enum": [],
+ }
+
self.schema: QAPISchema
self.golang_package_name = "qapi"
self.duplicate = list(gofiles)
+ self.enums: dict[str, str] = {}
+ self.docmap = {}
+
+ self.types = dict.fromkeys(types, "")
+ self.types_import = types
def visit_begin(self, schema: QAPISchema) -> None:
self.schema = schema
+ # iterate once in schema.docs to map doc objects to its name
+ for doc in schema.docs:
+ if doc.symbol is None:
+ continue
+ self.docmap[doc.symbol] = doc
+
+ for qapitype, imports in self.types_import.items():
+ self.types[qapitype] = TEMPLATE_GENERATED_HEADER[1:].format(
+ package_name=self.golang_package_name
+ )
+ self.types[qapitype] += generate_template_imports(imports)
+
def visit_end(self) -> None:
del self.schema
+ self.types["enum"] += generate_content_from_dict(self.enums)
def visit_object_type(
self,
@@ -83,7 +212,51 @@ def visit_enum_type(
members: List[QAPISchemaEnumMember],
prefix: Optional[str],
) -> None:
- pass
+ assert name not in self.enums
+ doc = self.docmap.get(name, None)
+ maindoc, docfields = qapi_to_golang_struct_docs(doc)
+
+ # The logic below is to generate QAPI enums as blocks of Go consts
+ # each with its own type for type safety inside Go applications.
+ #
+ # Block of const() blocks are vertically indented so we have to
+ # first iterate over all names to calculate space between
+ # $var_name and $var_type. This is achieved by helper function
+ # @fetch_indent_blocks_over_enum_with_docs()
+ #
+ # A new indentation block is defined by empty line or a comment.
+
+ indent_block = iter(
+ fetch_indent_blocks_over_enum_with_docs(name, members, docfields)
+ )
+ maxname = next(indent_block)
+ fields = ""
+ for index, member in enumerate(members):
+ # For simplicity, every time we have doc, we go to next indent block
+ hasdoc = member.name is not None and member.name in docfields
+
+ if hasdoc:
+ maxname = next(indent_block)
+
+ enum_name = f"{name}{qapi_to_field_name_enum(member.name)}"
+ name2type = " " * (maxname - len(enum_name) + 1)
+
+ if hasdoc:
+ docstr = (
+ textwrap.TextWrapper(width=80)
+ .fill(docfields[member.name])
+ .replace("\n", "\n\t// ")
+ )
+ fields += f"""\t// {docstr}\n"""
+
+ fields += f"""\t{enum_name}{name2type}{name} = "{member.name}"\n"""
+
+ if maindoc != "":
+ maindoc = f"\n{maindoc}"
+
+ self.enums[name] = maindoc + TEMPLATE_ENUM.format(
+ name=name, fields=fields[:-1]
+ )
def visit_array_type(
self,
@@ -133,3 +306,11 @@ def write(self, outdir: str) -> None:
srcpath = os.path.join(srcdir, filename)
dstpath = os.path.join(targetpath, filename)
shutil.copyfile(srcpath, dstpath)
+
+ # Types to be generated
+ for qapitype, content in self.types.items():
+ gofile = f"gen_type_{qapitype}.go"
+ pathname = os.path.join(targetpath, gofile)
+
+ with open(pathname, "w", encoding="utf8") as outfile:
+ outfile.write(content)
--
2.48.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v4 03/11] qapi: golang: Generate alternate types
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
2025-02-14 20:29 ` [PATCH v4 01/11] qapi: golang: first level unmarshalling type Victor Toso
2025-02-14 20:29 ` [PATCH v4 02/11] qapi: golang: Generate enum type Victor Toso
@ 2025-02-14 20:29 ` Victor Toso
2025-02-14 20:29 ` [PATCH v4 04/11] qapi: golang: Generate struct types Victor Toso
` (9 subsequent siblings)
12 siblings, 0 replies; 18+ messages in thread
From: Victor Toso @ 2025-02-14 20:29 UTC (permalink / raw)
To: qemu-devel
Cc: Markus Armbruster, John Snow, Daniel P . Berrangé,
Andrea Bolognani
This patch handles QAPI alternate types and generates data structures
in Go that handles it.
Alternate types are similar to Union but without a discriminator that
can be used to identify the underlying value on the wire.
1. Over the wire, we need to infer underlying value by its type
2. Pointer to types are mapped as optional. Absent value can be a
valid value.
3. We use Go's standard 'encoding/json' library with its Marshal
and Unmarshal interfaces.
4. As an exceptional but valid case, there are types that accept
JSON NULL as value. Due to limitations with Go's standard library
(point 3) combined with Absent being a possibility (point 2), we
translante NULL values to a boolean field called 'IsNull'. See the
second example and docs/devel/qapi-golang-code-gen.rst under
Alternate section.
* First example:
qapi:
| ##
| # @BlockdevRef:
| #
| # Reference to a block device.
| #
| # @definition: defines a new block device inline
| #
| # @reference: references the ID of an existing block device
| #
| # Since: 2.9
| ##
| { 'alternate': 'BlockdevRef',
| 'data': { 'definition': 'BlockdevOptions',
| 'reference': 'str' } }
go:
| // Reference to a block device.
| //
| // Since: 2.9
| type BlockdevRef struct {
| // defines a new block device inline
| Definition *BlockdevOptions
| // references the ID of an existing block device
| Reference *string
| }
|
| func (s BlockdevRef) MarshalJSON() ([]byte, error) {
| ...
| }
|
| func (s *BlockdevRef) UnmarshalJSON(data []byte) error {
| ...
| }
usage:
| input := `{"driver":"qcow2","data-file":"/some/place/my-image"}`
| k := BlockdevRef{}
| err := json.Unmarshal([]byte(input), &k)
| if err != nil {
| panic(err)
| }
| // *k.Definition.Qcow2.DataFile.Reference == "/some/place/my-image"
* Second example:
qapi:
| { 'alternate': 'StrOrNull',
| 'data': { 's': 'str',
| 'n': 'null' } }
| // This is a string value or the explicit lack of a string (null
| // pointer in C). Intended for cases when 'optional absent' already
| // has a different meaning.
| //
| // Since: 2.10
| type StrOrNull struct {
| // the string value
| S *string
| // no string value
| IsNull bool
| }
|
| // Helper function to get its underlying Go value or absent of value
| func (s *StrOrNull) ToAnyOrAbsent() (any, bool) {
| ...
| }
|
| func (s StrOrNull) MarshalJSON() ([]byte, error) {
| ...
| }
|
| func (s *StrOrNull) UnmarshalJSON(data []byte) error {
| ...
| }
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang/golang.py | 306 +++++++++++++++++++++++++++++++++-
scripts/qapi/golang/utils.go | 26 +++
2 files changed, 329 insertions(+), 3 deletions(-)
create mode 100644 scripts/qapi/golang/utils.go
diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py
index f074ec1f6f..aa1a18a501 100644
--- a/scripts/qapi/golang/golang.py
+++ b/scripts/qapi/golang/golang.py
@@ -14,10 +14,11 @@
from __future__ import annotations
import os, shutil, textwrap
-from typing import List, Optional
+from typing import List, Optional, Tuple
from ..schema import (
QAPISchema,
+ QAPISchemaAlternateType,
QAPISchemaBranches,
QAPISchemaEnumMember,
QAPISchemaFeature,
@@ -30,6 +31,8 @@
)
from ..source import QAPISourceInfo
+FOUR_SPACES = " "
+
TEMPLATE_GENERATED_HEADER = """
/*
* Copyright 2025 Red Hat, Inc.
@@ -56,6 +59,57 @@
)
"""
+TEMPLATE_ALTERNATE_CHECK_INVALID_JSON_NULL = """
+ // Check for json-null first
+ if string(data) == "null" {{
+ return errors.New(`null not supported for {name}`)
+ }}"""
+
+TEMPLATE_ALTERNATE_NULLABLE_CHECK = """
+ }} else if s.{var_name} != nil {{
+ return *s.{var_name}, false"""
+
+TEMPLATE_ALTERNATE_MARSHAL_CHECK = """
+ if s.{var_name} != nil {{
+ return json.Marshal(s.{var_name})
+ }} else """
+
+TEMPLATE_ALTERNATE_UNMARSHAL_CHECK = """
+ // Check for {var_type}
+ {{
+ s.{var_name} = new({var_type})
+ if err := strictDecode(s.{var_name}, data); err == nil {{
+ return nil
+ }}
+ s.{var_name} = nil
+ }}
+
+"""
+
+TEMPLATE_ALTERNATE_NULLABLE_MARSHAL_CHECK = """
+ if s.IsNull {
+ return []byte("null"), nil
+ } else """
+
+TEMPLATE_ALTERNATE_NULLABLE_UNMARSHAL_CHECK = """
+ // Check for json-null first
+ if string(data) == "null" {
+ s.IsNull = true
+ return nil
+ }"""
+
+TEMPLATE_ALTERNATE_METHODS = """
+func (s {name}) MarshalJSON() ([]byte, error) {{
+{marshal_check_fields}
+ return {marshal_return_default}
+}}
+
+func (s *{name}) UnmarshalJSON(data []byte) error {{
+{unmarshal_check_fields}
+ return fmt.Errorf("Can't convert to {name}: %s", string(data))
+}}
+"""
+
# Takes the documentation object of a specific type and returns
# that type's documentation and its member's docs.
@@ -96,10 +150,88 @@ def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None:
vis.write(output_dir)
+def qapi_to_field_name(name: str) -> str:
+ return name.title().replace("_", "").replace("-", "")
+
+
def qapi_to_field_name_enum(name: str) -> str:
return name.title().replace("-", "")
+def qapi_schema_type_to_go_type(qapitype: str) -> str:
+ schema_types_to_go = {
+ "str": "string",
+ "null": "nil",
+ "bool": "bool",
+ "number": "float64",
+ "size": "uint64",
+ "int": "int64",
+ "int8": "int8",
+ "int16": "int16",
+ "int32": "int32",
+ "int64": "int64",
+ "uint8": "uint8",
+ "uint16": "uint16",
+ "uint32": "uint32",
+ "uint64": "uint64",
+ "any": "any",
+ "QType": "QType",
+ }
+
+ prefix = ""
+ if qapitype.endswith("List"):
+ prefix = "[]"
+ qapitype = qapitype[:-4]
+
+ qapitype = schema_types_to_go.get(qapitype, qapitype)
+ return prefix + qapitype
+
+
+# Helper for Alternate generation
+def qapi_field_to_alternate_go_field(
+ member_name: str, type_name: str
+) -> Tuple[str, str, str]:
+ # Nothing to generate on null types. We update some
+ # variables to handle json-null on marshalling methods.
+ if type_name == "null":
+ return "IsNull", "bool", ""
+
+ # On Alternates, fields are optional represented in Go as pointer
+ return (
+ qapi_to_field_name(member_name),
+ qapi_schema_type_to_go_type(type_name),
+ "*",
+ )
+
+
+def fetch_indent_blocks_over_args(
+ args: List[dict[str:str]],
+) -> Tuple[int, int]:
+ maxname, maxtype = 0, 0
+ blocks: tuple(int, int) = []
+ for arg in args:
+ if "comment" in arg or "doc" in arg:
+ blocks.append((maxname, maxtype))
+ maxname, maxtype = 0, 0
+
+ if "comment" in arg:
+ # They are single blocks
+ continue
+
+ if "type" not in arg:
+ # Embed type are on top of the struct and the following
+ # fields do not consider it for formatting
+ blocks.append((maxname, maxtype))
+ maxname, maxtype = 0, 0
+ continue
+
+ maxname = max(maxname, len(arg.get("name", "")))
+ maxtype = max(maxtype, len(arg.get("type", "")))
+
+ blocks.append((maxname, maxtype))
+ return blocks
+
+
def fetch_indent_blocks_over_enum_with_docs(
name: str, members: List[QAPISchemaEnumMember], docfields: Dict[str, str]
) -> Tuple[int]:
@@ -122,6 +254,137 @@ def fetch_indent_blocks_over_enum_with_docs(
return blocks
+# Helper function for boxed or self contained structures.
+def generate_struct_type(
+ type_name,
+ type_doc: str = "",
+ args: List[dict[str:str]] = None,
+ indent: int = 0,
+) -> str:
+ base_indent = FOUR_SPACES * indent
+
+ with_type = ""
+ if type_name != "":
+ with_type = f"\n{base_indent}type {type_name}"
+
+ if type_doc != "":
+ # Append line jump only if type_doc exists
+ type_doc = f"\n{type_doc}"
+
+ if args is None:
+ # No args, early return
+ return f"""{type_doc}{with_type} struct{{}}"""
+
+ # The logic below is to generate fields of the struct.
+ # We have to be mindful of the different indentation possibilities between
+ # $var_name $var_type $var_tag that are vertically indented with gofmt.
+ #
+ # So, we first have to iterate over all args and find all indent blocks
+ # by calculating the spaces between (1) member and type and between (2)
+ # the type and tag. (1) and (2) is the tuple present in List returned
+ # by the helper function fetch_indent_blocks_over_args.
+ inner_indent = base_indent + FOUR_SPACES
+ doc_indent = inner_indent + "// "
+ fmt = textwrap.TextWrapper(
+ width=70, initial_indent=doc_indent, subsequent_indent=doc_indent
+ )
+
+ indent_block = iter(fetch_indent_blocks_over_args(args))
+ maxname, maxtype = next(indent_block)
+ members = " {\n"
+ for index, arg in enumerate(args):
+ if "comment" in arg:
+ maxname, maxtype = next(indent_block)
+ members += f""" // {arg["comment"]}\n"""
+ # comments are single blocks, so we can skip to next arg
+ continue
+
+ name2type = ""
+ if "doc" in arg:
+ maxname, maxtype = next(indent_block)
+ members += fmt.fill(arg["doc"])
+ members += "\n"
+
+ name = arg["name"]
+ if "type" in arg:
+ namelen = len(name)
+ name2type = " " * max(1, (maxname - namelen + 1))
+
+ type2tag = ""
+ if "tag" in arg:
+ typelen = len(arg["type"])
+ type2tag = " " * max(1, (maxtype - typelen + 1))
+
+ gotype = arg.get("type", "")
+ tag = arg.get("tag", "")
+ members += (
+ f"""{inner_indent}{name}{name2type}{gotype}{type2tag}{tag}\n"""
+ )
+
+ members += f"{base_indent}}}\n"
+ return f"""{type_doc}{with_type} struct{members}"""
+
+
+def generate_template_alternate(
+ self: QAPISchemaGenGolangVisitor,
+ name: str,
+ variants: Optional[QAPISchemaVariants],
+) -> str:
+ args: List[dict[str:str]] = []
+ nullable = name in self.accept_null_types
+ if nullable:
+ # Examples in QEMU QAPI schema: StrOrNull and BlockdevRefOrNull
+ marshal_return_default = """[]byte("{}"), nil"""
+ marshal_check_fields = TEMPLATE_ALTERNATE_NULLABLE_MARSHAL_CHECK[1:]
+ unmarshal_check_fields = TEMPLATE_ALTERNATE_NULLABLE_UNMARSHAL_CHECK
+ else:
+ marshal_return_default = f'nil, errors.New("{name} has empty fields")'
+ marshal_check_fields = ""
+ unmarshal_check_fields = (
+ TEMPLATE_ALTERNATE_CHECK_INVALID_JSON_NULL.format(name=name)
+ )
+
+ doc = self.docmap.get(name, None)
+ content, docfields = qapi_to_golang_struct_docs(doc)
+ if variants:
+ for var in variants.variants:
+ var_name, var_type, isptr = qapi_field_to_alternate_go_field(
+ var.name, var.type.name
+ )
+ args.append(
+ {
+ "name": f"{var_name}",
+ "type": f"{isptr}{var_type}",
+ "doc": docfields.get(var.name, ""),
+ }
+ )
+ # Null is special, handled first
+ if var.type.name == "null":
+ assert nullable
+ continue
+
+ skip_indent = 1 + len(FOUR_SPACES)
+ if marshal_check_fields == "":
+ skip_indent = 1
+ marshal_check_fields += TEMPLATE_ALTERNATE_MARSHAL_CHECK[
+ skip_indent:
+ ].format(var_name=var_name)
+ unmarshal_check_fields += TEMPLATE_ALTERNATE_UNMARSHAL_CHECK[
+ :-1
+ ].format(var_name=var_name, var_type=var_type)
+
+ content += string_to_code(generate_struct_type(name, args=args))
+ content += string_to_code(
+ TEMPLATE_ALTERNATE_METHODS.format(
+ name=name,
+ marshal_check_fields=marshal_check_fields[:-6],
+ marshal_return_default=marshal_return_default,
+ unmarshal_check_fields=unmarshal_check_fields[1:],
+ )
+ )
+ return "\n" + content
+
+
def generate_content_from_dict(data: dict[str, str]) -> str:
content = ""
@@ -131,6 +394,25 @@ def generate_content_from_dict(data: dict[str, str]) -> str:
return content.replace("\n\n\n", "\n\n")
+def string_to_code(text: str) -> str:
+ DOUBLE_BACKTICK = "``"
+ result = ""
+ for line in text.splitlines():
+ # replace left four spaces with tabs
+ limit = len(line) - len(line.lstrip())
+ result += line[:limit].replace(FOUR_SPACES, "\t")
+
+ # work with the rest of the line
+ if line[limit : limit + 2] == "//":
+ # gofmt tool does not like comments with backticks.
+ result += line[limit:].replace(DOUBLE_BACKTICK, '"')
+ else:
+ result += line[limit:]
+ result += "\n"
+
+ return result
+
+
def generate_template_imports(words: List[str]) -> str:
if len(words) == 0:
return ""
@@ -147,9 +429,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
# pylint: disable=too-many-arguments
def __init__(self, _: str):
super().__init__()
- gofiles = ("protocol.go",)
+ gofiles = ("protocol.go", "utils.go")
# Map each qapi type to the necessary Go imports
types = {
+ "alternate": ["encoding/json", "errors", "fmt"],
"enum": [],
}
@@ -157,6 +440,8 @@ def __init__(self, _: str):
self.golang_package_name = "qapi"
self.duplicate = list(gofiles)
self.enums: dict[str, str] = {}
+ self.alternates: dict[str, str] = {}
+ self.accept_null_types = []
self.docmap = {}
self.types = dict.fromkeys(types, "")
@@ -165,6 +450,17 @@ def __init__(self, _: str):
def visit_begin(self, schema: QAPISchema) -> None:
self.schema = schema
+ # We need to be aware of any types that accept JSON NULL
+ for name, entity in self.schema._entity_dict.items():
+ if not isinstance(entity, QAPISchemaAlternateType):
+ # Assume that only Alternate types accept JSON NULL
+ continue
+
+ for var in entity.alternatives.variants:
+ if var.type.name == "null":
+ self.accept_null_types.append(name)
+ break
+
# iterate once in schema.docs to map doc objects to its name
for doc in schema.docs:
if doc.symbol is None:
@@ -180,6 +476,7 @@ def visit_begin(self, schema: QAPISchema) -> None:
def visit_end(self) -> None:
del self.schema
self.types["enum"] += generate_content_from_dict(self.enums)
+ self.types["alternate"] += generate_content_from_dict(self.alternates)
def visit_object_type(
self,
@@ -201,7 +498,10 @@ def visit_alternate_type(
features: List[QAPISchemaFeature],
variants: QAPISchemaVariants,
) -> None:
- pass
+ assert name not in self.alternates
+ self.alternates[name] = generate_template_alternate(
+ self, name, variants
+ )
def visit_enum_type(
self,
diff --git a/scripts/qapi/golang/utils.go b/scripts/qapi/golang/utils.go
new file mode 100644
index 0000000000..f00c0a5d83
--- /dev/null
+++ b/scripts/qapi/golang/utils.go
@@ -0,0 +1,26 @@
+/*
+ * Copyright 2025 Red Hat, Inc.
+ * SPDX-License-Identifier: MIT-0
+ *
+ * Authors:
+ * Victor Toso <victortoso@redhat.com>
+ */
+package qapi
+
+import (
+ "encoding/json"
+ "strings"
+)
+
+// Creates a decoder that errors on unknown Fields
+// Returns nil if successfully decoded @from payload to @into type
+// Returns error if failed to decode @from payload to @into type
+func strictDecode(into interface{}, from []byte) error {
+ dec := json.NewDecoder(strings.NewReader(string(from)))
+ dec.DisallowUnknownFields()
+
+ if err := dec.Decode(into); err != nil {
+ return err
+ }
+ return nil
+}
--
2.48.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v4 04/11] qapi: golang: Generate struct types
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
` (2 preceding siblings ...)
2025-02-14 20:29 ` [PATCH v4 03/11] qapi: golang: Generate alternate types Victor Toso
@ 2025-02-14 20:29 ` Victor Toso
2025-02-14 20:29 ` [PATCH v4 05/11] qapi: golang: structs: Address nullable members Victor Toso
` (8 subsequent siblings)
12 siblings, 0 replies; 18+ messages in thread
From: Victor Toso @ 2025-02-14 20:29 UTC (permalink / raw)
To: qemu-devel
Cc: Markus Armbruster, John Snow, Daniel P . Berrangé,
Andrea Bolognani
This patch handles QAPI struct types and generates the equivalent
types in Go. The following patch adds extra logic when a member of the
struct has a Type that can take JSON Null value (e.g: StrOrNull in
QEMU)
The highlights of this implementation are:
1. Generating a Go struct that requires a @base type, the @base type
fields are copied over to the Go struct. The advantage of this
approach is to not have embed structs in any of the QAPI types.
Note that embedding a @base type is recursive, that is, if the
@base type has a @base, all of those fields will be copied over.
2. About the Go struct's fields:
i) They can be either by Value or Reference.
ii) Every field that is marked as optional in the QAPI specification
are translated to Reference fields in its Go structure. This
design decision is the most straightforward way to check if a
given field was set or not. Exception only for types that can
take JSON Null value.
iii) Mandatory fields are always by Value with the exception of QAPI
arrays, which are handled by Reference (to a block of memory) by
Go.
iv) All the fields are named with Uppercase due Golang's export
convention.
Example:
qapi:
| ##
| # @BlockdevCreateOptionsFile:
| #
| # Driver specific image creation options for file.
| #
| # @filename: Filename for the new image file
| #
| # @size: Size of the virtual disk in bytes
| #
| # @preallocation: Preallocation mode for the new image (default: off;
| # allowed values: off, falloc (if CONFIG_POSIX_FALLOCATE), full
| # (if CONFIG_POSIX))
| #
| # @nocow: Turn off copy-on-write (valid only on btrfs; default: off)
| #
| # @extent-size-hint: Extent size hint to add to the image file; 0 for
| # not adding an extent size hint (default: 1 MB, since 5.1)
| #
| # Since: 2.12
| ##
| { 'struct': 'BlockdevCreateOptionsFile',
| 'data': { 'filename': 'str',
| 'size': 'size',
| '*preallocation': 'PreallocMode',
| '*nocow': 'bool',
| '*extent-size-hint': 'size'} }
go:
| // Driver specific image creation options for file.
| //
| // Since: 2.12
| type BlockdevCreateOptionsFile struct {
| // Filename for the new image file
| Filename string `json:"filename"`
| // Size of the virtual disk in bytes
| Size uint64 `json:"size"`
| // Preallocation mode for the new image (default: off; allowed
| // values: off, falloc (if CONFIG_POSIX_FALLOCATE), full (if
| // CONFIG_POSIX))
| Preallocation *PreallocMode `json:"preallocation,omitempty"`
| // Turn off copy-on-write (valid only on btrfs; default: off)
| Nocow *bool `json:"nocow,omitempty"`
| // Extent size hint to add to the image file; 0 for not adding an
| // extent size hint (default: 1 MB, since 5.1)
| ExtentSizeHint *uint64 `json:"extent-size-hint,omitempty"`
| }
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang/golang.py | 193 +++++++++++++++++++++++++++++++++-
1 file changed, 192 insertions(+), 1 deletion(-)
diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py
index aa1a18a501..e8a47b4a1e 100644
--- a/scripts/qapi/golang/golang.py
+++ b/scripts/qapi/golang/golang.py
@@ -150,6 +150,14 @@ def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None:
vis.write(output_dir)
+def qapi_name_is_base(name: str) -> bool:
+ return qapi_name_is_object(name) and name.endswith("-base")
+
+
+def qapi_name_is_object(name: str) -> bool:
+ return name.startswith("q_obj_")
+
+
def qapi_to_field_name(name: str) -> str:
return name.title().replace("_", "").replace("-", "")
@@ -158,6 +166,27 @@ def qapi_to_field_name_enum(name: str) -> str:
return name.title().replace("-", "")
+def qapi_to_go_type_name(name: str) -> str:
+ # We want to keep CamelCase for Golang types. We want to avoid removing
+ # already set CameCase names while fixing uppercase ones, eg:
+ # 1) q_obj_SocketAddress_base -> SocketAddressBase
+ # 2) q_obj_WATCHDOG-arg -> WatchdogArg
+
+ if qapi_name_is_object(name):
+ # Remove q_obj_ prefix
+ name = name[6:]
+
+ # Handle CamelCase
+ words = list(name.replace("_", "-").split("-"))
+ name = words[0]
+ if name.islower() or name.isupper():
+ name = name.title()
+
+ name += "".join(word.title() for word in words[1:])
+
+ return name
+
+
def qapi_schema_type_to_go_type(qapitype: str) -> str:
schema_types_to_go = {
"str": "string",
@@ -325,6 +354,131 @@ def generate_struct_type(
return f"""{type_doc}{with_type} struct{members}"""
+def get_struct_field(
+ self: QAPISchemaGenGolangVisitor,
+ qapi_name: str,
+ qapi_type_name: str,
+ field_doc: str,
+ is_optional: bool,
+ is_variant: bool,
+) -> dict[str:str]:
+ field = qapi_to_field_name(qapi_name)
+ member_type = qapi_schema_type_to_go_type(qapi_type_name)
+
+ optional = ""
+ if is_optional:
+ if member_type not in self.accept_null_types:
+ optional = ",omitempty"
+
+ # Use pointer to type when field is optional
+ isptr = "*" if is_optional and member_type[0] not in "*[" else ""
+
+ fieldtag = (
+ '`json:"-"`' if is_variant else f'`json:"{qapi_name}{optional}"`'
+ )
+ arg = {
+ "name": f"{field}",
+ "type": f"{isptr}{member_type}",
+ "tag": f"{fieldtag}",
+ }
+ if field_doc != "":
+ arg["doc"] = field_doc
+
+ return arg
+
+
+def recursive_base(
+ self: QAPISchemaGenGolangVisitor,
+ base: Optional[QAPISchemaObjectType],
+) -> List[dict[str:str]]:
+ fields: List[dict[str:str]] = []
+
+ if not base:
+ return fields
+
+ if base.base is not None:
+ embed_base = self.schema.lookup_entity(base.base.name)
+ fields = recursive_base(self, embed_base)
+
+ doc = self.docmap.get(base.name, None)
+ _, docfields = qapi_to_golang_struct_docs(doc)
+
+ for member in base.local_members:
+ field_doc = docfields.get(member.name, "")
+ field = get_struct_field(
+ self,
+ member.name,
+ member.type.name,
+ field_doc,
+ member.optional,
+ False,
+ )
+ fields.append(field)
+
+ return fields
+
+
+# Helper function that is used for most of QAPI types
+def qapi_to_golang_struct(
+ self: QAPISchemaGenGolangVisitor,
+ name: str,
+ info: Optional[QAPISourceInfo],
+ __: QAPISchemaIfCond,
+ ___: List[QAPISchemaFeature],
+ base: Optional[QAPISchemaObjectType],
+ members: List[QAPISchemaObjectTypeMember],
+ variants: Optional[QAPISchemaVariants],
+ indent: int = 0,
+ doc_enabled: bool = True,
+) -> str:
+ fields = recursive_base(self, base)
+
+ doc = self.docmap.get(name, None)
+ type_doc, docfields = qapi_to_golang_struct_docs(doc)
+ if not doc_enabled:
+ type_doc = ""
+
+ if members:
+ for member in members:
+ field_doc = docfields.get(member.name, "") if doc_enabled else ""
+ field = get_struct_field(
+ self,
+ member.name,
+ member.type.name,
+ field_doc,
+ member.optional,
+ False,
+ )
+ fields.append(field)
+
+ exists = {}
+ if variants:
+ fields.append({"comment": "Variants fields"})
+ for variant in variants.variants:
+ if variant.type.is_implicit():
+ continue
+
+ exists[variant.name] = True
+ field_doc = docfields.get(variant.name, "") if doc_enabled else ""
+ field = get_struct_field(
+ self,
+ variant.name,
+ variant.type.name,
+ field_doc,
+ True,
+ True,
+ )
+ fields.append(field)
+
+ type_name = qapi_to_go_type_name(name)
+ content = string_to_code(
+ generate_struct_type(
+ type_name, type_doc=type_doc, args=fields, indent=indent
+ )
+ )
+ return content
+
+
def generate_template_alternate(
self: QAPISchemaGenGolangVisitor,
name: str,
@@ -434,6 +588,7 @@ def __init__(self, _: str):
types = {
"alternate": ["encoding/json", "errors", "fmt"],
"enum": [],
+ "struct": ["encoding/json"],
}
self.schema: QAPISchema
@@ -441,6 +596,7 @@ def __init__(self, _: str):
self.duplicate = list(gofiles)
self.enums: dict[str, str] = {}
self.alternates: dict[str, str] = {}
+ self.structs: dict[str, str] = {}
self.accept_null_types = []
self.docmap = {}
@@ -477,6 +633,7 @@ def visit_end(self) -> None:
del self.schema
self.types["enum"] += generate_content_from_dict(self.enums)
self.types["alternate"] += generate_content_from_dict(self.alternates)
+ self.types["struct"] += generate_content_from_dict(self.structs)
def visit_object_type(
self,
@@ -488,7 +645,41 @@ def visit_object_type(
members: List[QAPISchemaObjectTypeMember],
branches: Optional[QAPISchemaBranches],
) -> None:
- pass
+ # Do not handle anything besides struct.
+ if (
+ name == self.schema.the_empty_object_type.name
+ or not isinstance(name, str)
+ or info.defn_meta not in ["struct"]
+ ):
+ return
+
+ # Base structs are embed
+ if qapi_name_is_base(name):
+ return
+
+ # visit all inner objects as well, they are not going to be
+ # called by python's generator.
+ if branches:
+ for branch in branches.variants:
+ assert isinstance(branch.type, QAPISchemaObjectType)
+ self.visit_object_type(
+ self,
+ branch.type.name,
+ branch.type.info,
+ branch.type.ifcond,
+ branch.type.base,
+ branch.type.local_members,
+ branch.type.branches,
+ )
+
+ # Save generated Go code to be written later
+ if info.defn_meta == "struct":
+ assert name not in self.structs
+ self.structs[name] = string_to_code(
+ qapi_to_golang_struct(
+ self, name, info, ifcond, features, base, members, branches
+ )
+ )
def visit_alternate_type(
self,
--
2.48.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v4 05/11] qapi: golang: structs: Address nullable members
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
` (3 preceding siblings ...)
2025-02-14 20:29 ` [PATCH v4 04/11] qapi: golang: Generate struct types Victor Toso
@ 2025-02-14 20:29 ` Victor Toso
2025-02-14 20:29 ` [PATCH v4 06/11] qapi: golang: Generate union type Victor Toso
` (7 subsequent siblings)
12 siblings, 0 replies; 18+ messages in thread
From: Victor Toso @ 2025-02-14 20:29 UTC (permalink / raw)
To: qemu-devel
Cc: Markus Armbruster, John Snow, Daniel P . Berrangé,
Andrea Bolognani
Explaining why this is needed needs some context, so taking the
example of StrOrNull alternate type and considering a simplified
struct that has two fields:
qapi:
| { 'struct': 'MigrationExample',
| 'data': { '*label': 'StrOrNull',
| 'target': 'StrOrNull' } }
We have an optional member 'label' which can have three JSON values:
1. A string: { "target": "a.host.com", "label": "happy" }
2. A null : { "target": "a.host.com", "label": null }
3. Absent : { "target": null}
The member 'target' is not optional, hence it can't be absent.
A Go struct that contains an optional type that can be JSON Null like
'label' in the example above, will need extra care when Marshaling and
Unmarshaling from JSON.
This patch handles this very specific case:
- It implements the Marshaler interface for these structs to properly
handle these values.
- It adds the interface AbsentAlternate() and implement it for any
Alternate that can be JSON Null. See its uses in map_and_set()
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang/golang.py | 290 ++++++++++++++++++++++++++++++++--
1 file changed, 279 insertions(+), 11 deletions(-)
diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py
index e8a47b4a1e..0637bb3e3e 100644
--- a/scripts/qapi/golang/golang.py
+++ b/scripts/qapi/golang/golang.py
@@ -59,6 +59,17 @@
)
"""
+TEMPLATE_ALTERNATE = """
+// Only implemented on Alternate types that can take JSON NULL as value.
+//
+// This is a helper for the marshalling code. It should return true only when
+// the Alternate is empty (no members are set), otherwise it returns false and
+// the member set to be Marshalled.
+type AbsentAlternate interface {
+ ToAnyOrAbsent() (any, bool)
+}
+"""
+
TEMPLATE_ALTERNATE_CHECK_INVALID_JSON_NULL = """
// Check for json-null first
if string(data) == "null" {{
@@ -98,6 +109,19 @@
return nil
}"""
+TEMPLATE_ALTERNATE_NULLABLE = """
+func (s *{name}) ToAnyOrAbsent() (any, bool) {{
+ if s != nil {{
+ if s.IsNull {{
+ return nil, false
+{absent_check_fields}
+ }}
+ }}
+
+ return nil, true
+}}
+"""
+
TEMPLATE_ALTERNATE_METHODS = """
func (s {name}) MarshalJSON() ([]byte, error) {{
{marshal_check_fields}
@@ -111,6 +135,26 @@
"""
+TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL = """
+func (s {type_name}) MarshalJSON() ([]byte, error) {{
+ m := make(map[string]any)
+{map_members}{map_special}
+ return json.Marshal(&m)
+}}
+
+func (s *{type_name}) UnmarshalJSON(data []byte) error {{
+ tmp := {struct}{{}}
+
+ if err := json.Unmarshal(data, &tmp); err != nil {{
+ return err
+ }}
+
+{set_members}{set_special}
+ return nil
+}}
+"""
+
+
# Takes the documentation object of a specific type and returns
# that type's documentation and its member's docs.
def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]):
@@ -359,20 +403,30 @@ def get_struct_field(
qapi_name: str,
qapi_type_name: str,
field_doc: str,
+ within_nullable_struct: bool,
is_optional: bool,
is_variant: bool,
-) -> dict[str:str]:
+) -> Tuple[dict[str:str], bool]:
field = qapi_to_field_name(qapi_name)
member_type = qapi_schema_type_to_go_type(qapi_type_name)
+ is_nullable = False
optional = ""
if is_optional:
- if member_type not in self.accept_null_types:
+ if member_type in self.accept_null_types:
+ is_nullable = True
+ else:
optional = ",omitempty"
# Use pointer to type when field is optional
isptr = "*" if is_optional and member_type[0] not in "*[" else ""
+ if within_nullable_struct:
+ # Within a struct which has a field of type that can hold JSON NULL,
+ # we have to _not_ use a pointer, otherwise the Marshal methods are
+ # not called.
+ isptr = "" if member_type in self.accept_null_types else isptr
+
fieldtag = (
'`json:"-"`' if is_variant else f'`json:"{qapi_name}{optional}"`'
)
@@ -384,38 +438,228 @@ def get_struct_field(
if field_doc != "":
arg["doc"] = field_doc
- return arg
+ return arg, is_nullable
+
+
+# This helper is used whithin a struct that has members that accept JSON NULL.
+def map_and_set(
+ is_nullable: bool, field: str, field_is_optional: bool, name: str
+) -> Tuple[str, str]:
+ mapstr = ""
+ setstr = ""
+ if is_nullable:
+ mapstr = f"""
+ if val, absent := s.{field}.ToAnyOrAbsent(); !absent {{
+ m["{name}"] = val
+ }}
+"""
+ setstr += f"""
+ if _, absent := (&tmp.{field}).ToAnyOrAbsent(); !absent {{
+ s.{field} = &tmp.{field}
+ }}
+"""
+ elif field_is_optional:
+ mapstr = f"""
+ if s.{field} != nil {{
+ m["{name}"] = s.{field}
+ }}
+"""
+ setstr = f""" s.{field} = tmp.{field}\n"""
+ else:
+ mapstr = f""" m["{name}"] = s.{field}\n"""
+ setstr = f""" s.{field} = tmp.{field}\n"""
+
+ return mapstr, setstr
+
+
+def recursive_base_nullable(
+ self: QAPISchemaGenGolangVisitor, base: Optional[QAPISchemaObjectType]
+) -> Tuple[List[dict[str:str]], str, str, str, str]:
+ fields: List[dict[str:str]] = []
+ map_members = ""
+ set_members = ""
+ map_special = ""
+ set_special = ""
+
+ if not base:
+ return fields, map_members, set_members, map_special, set_special
+
+ doc = self.docmap.get(base.name, None)
+ _, docfields = qapi_to_golang_struct_docs(doc)
+
+ if base.base is not None:
+ embed_base = self.schema.lookup_entity(base.base.name)
+ (
+ fields,
+ map_members,
+ set_members,
+ map_special,
+ set_special,
+ ) = recursive_base_nullable(self, embed_base)
+
+ for member in base.local_members:
+ field_doc = docfields.get(member.name, "")
+ field, _ = get_struct_field(
+ self,
+ member.name,
+ member.type.name,
+ field_doc,
+ True,
+ member.optional,
+ False,
+ )
+ fields.append(field)
+
+ member_type = qapi_schema_type_to_go_type(member.type.name)
+ nullable = member_type in self.accept_null_types
+ field_name = qapi_to_field_name(member.name)
+ tomap, toset = map_and_set(
+ nullable, field_name, member.optional, member.name
+ )
+ if nullable:
+ map_special += tomap
+ set_special += toset
+ else:
+ map_members += tomap
+ set_members += toset
+
+ return fields, map_members, set_members, map_special, set_special
+
+
+# Helper function. This is executed when the QAPI schema has members
+# that could accept JSON NULL (e.g: StrOrNull in QEMU"s QAPI schema).
+# This struct will need to be extended with Marshal/Unmarshal methods to
+# properly handle such atypical members.
+#
+# Only the Marshallaing methods are generated but we do need to iterate over
+# all the members to properly set/check them in those methods.
+def struct_with_nullable_generate_marshal(
+ self: QAPISchemaGenGolangVisitor,
+ name: str,
+ base: Optional[QAPISchemaObjectType],
+ members: List[QAPISchemaObjectTypeMember],
+ variants: Optional[QAPISchemaVariants],
+) -> str:
+ (
+ fields,
+ map_members,
+ set_members,
+ map_special,
+ set_special,
+ ) = recursive_base_nullable(self, base)
+
+ doc = self.docmap.get(name, None)
+ _, docfields = qapi_to_golang_struct_docs(doc)
+
+ if members:
+ for member in members:
+ field_doc = docfields.get(member.name, "")
+ field, _ = get_struct_field(
+ self,
+ member.name,
+ member.type.name,
+ field_doc,
+ True,
+ member.optional,
+ False,
+ )
+ fields.append(field)
+
+ member_type = qapi_schema_type_to_go_type(member.type.name)
+ nullable = member_type in self.accept_null_types
+ tomap, toset = map_and_set(
+ nullable,
+ qapi_to_field_name(member.name),
+ member.optional,
+ member.name,
+ )
+ if nullable:
+ map_special += tomap
+ set_special += toset
+ else:
+ map_members += tomap
+ set_members += toset
+
+ if variants:
+ for variant in variants.variants:
+ if variant.type.is_implicit():
+ continue
+
+ field, _ = get_struct_field(
+ self,
+ variant.name,
+ variant.type.name,
+ True,
+ variant.optional,
+ True,
+ )
+ fields.append(field)
+
+ member_type = qapi_schema_type_to_go_type(variant.type.name)
+ nullable = member_type in self.accept_null_types
+ tomap, toset = map_and_set(
+ nullable,
+ qapi_to_field_name(variant.name),
+ variant.optional,
+ variant.name,
+ )
+ if nullable:
+ map_special += tomap
+ set_special += toset
+ else:
+ map_members += tomap
+ set_members += toset
+
+ type_name = qapi_to_go_type_name(name)
+ struct = generate_struct_type("", args=fields, indent=1)
+ return string_to_code(
+ TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL.format(
+ struct=struct[1:-1],
+ type_name=type_name,
+ map_members=map_members,
+ map_special=map_special,
+ set_members=set_members,
+ set_special=set_special,
+ )
+ )
def recursive_base(
self: QAPISchemaGenGolangVisitor,
base: Optional[QAPISchemaObjectType],
-) -> List[dict[str:str]]:
+ discriminator: Optional[str] = None,
+) -> Tuple[List[dict[str:str]], bool]:
fields: List[dict[str:str]] = []
+ with_nullable = False
if not base:
- return fields
+ return fields, with_nullable
if base.base is not None:
embed_base = self.schema.lookup_entity(base.base.name)
- fields = recursive_base(self, embed_base)
+ fields, with_nullable = recursive_base(self, embed_base, discriminator)
doc = self.docmap.get(base.name, None)
_, docfields = qapi_to_golang_struct_docs(doc)
for member in base.local_members:
+ if discriminator and member.name == discriminator:
+ continue
+
field_doc = docfields.get(member.name, "")
- field = get_struct_field(
+ field, nullable = get_struct_field(
self,
member.name,
member.type.name,
field_doc,
+ False,
member.optional,
False,
)
fields.append(field)
+ with_nullable = True if nullable else with_nullable
- return fields
+ return fields, with_nullable
# Helper function that is used for most of QAPI types
@@ -431,7 +675,8 @@ def qapi_to_golang_struct(
indent: int = 0,
doc_enabled: bool = True,
) -> str:
- fields = recursive_base(self, base)
+ discriminator = None if not variants else variants.tag_member.name
+ fields, with_nullable = recursive_base(self, base, discriminator)
doc = self.docmap.get(name, None)
type_doc, docfields = qapi_to_golang_struct_docs(doc)
@@ -441,15 +686,17 @@ def qapi_to_golang_struct(
if members:
for member in members:
field_doc = docfields.get(member.name, "") if doc_enabled else ""
- field = get_struct_field(
+ field, nullable = get_struct_field(
self,
member.name,
member.type.name,
field_doc,
+ False,
member.optional,
False,
)
fields.append(field)
+ with_nullable = True if nullable else with_nullable
exists = {}
if variants:
@@ -460,15 +707,17 @@ def qapi_to_golang_struct(
exists[variant.name] = True
field_doc = docfields.get(variant.name, "") if doc_enabled else ""
- field = get_struct_field(
+ field, nullable = get_struct_field(
self,
variant.name,
variant.type.name,
field_doc,
+ False,
True,
True,
)
fields.append(field)
+ with_nullable = True if nullable else with_nullable
type_name = qapi_to_go_type_name(name)
content = string_to_code(
@@ -476,6 +725,10 @@ def qapi_to_golang_struct(
type_name, type_doc=type_doc, args=fields, indent=indent
)
)
+ if with_nullable:
+ content += struct_with_nullable_generate_marshal(
+ self, name, base, members, variants
+ )
return content
@@ -484,6 +737,7 @@ def generate_template_alternate(
name: str,
variants: Optional[QAPISchemaVariants],
) -> str:
+ absent_check_fields = ""
args: List[dict[str:str]] = []
nullable = name in self.accept_null_types
if nullable:
@@ -517,6 +771,12 @@ def generate_template_alternate(
assert nullable
continue
+ if nullable:
+ absent_check_fields += string_to_code(
+ TEMPLATE_ALTERNATE_NULLABLE_CHECK[1:].format(
+ var_name=var_name
+ )
+ )
skip_indent = 1 + len(FOUR_SPACES)
if marshal_check_fields == "":
skip_indent = 1
@@ -528,6 +788,12 @@ def generate_template_alternate(
].format(var_name=var_name, var_type=var_type)
content += string_to_code(generate_struct_type(name, args=args))
+ if nullable:
+ content += string_to_code(
+ TEMPLATE_ALTERNATE_NULLABLE.format(
+ name=name, absent_check_fields=absent_check_fields[:-1]
+ )
+ )
content += string_to_code(
TEMPLATE_ALTERNATE_METHODS.format(
name=name,
@@ -629,6 +895,8 @@ def visit_begin(self, schema: QAPISchema) -> None:
)
self.types[qapitype] += generate_template_imports(imports)
+ self.types["alternate"] += string_to_code(TEMPLATE_ALTERNATE)
+
def visit_end(self) -> None:
del self.schema
self.types["enum"] += generate_content_from_dict(self.enums)
--
2.48.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v4 06/11] qapi: golang: Generate union type
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
` (4 preceding siblings ...)
2025-02-14 20:29 ` [PATCH v4 05/11] qapi: golang: structs: Address nullable members Victor Toso
@ 2025-02-14 20:29 ` Victor Toso
2025-02-14 20:29 ` [PATCH v4 07/11] qapi: golang: Generate event type Victor Toso
` (6 subsequent siblings)
12 siblings, 0 replies; 18+ messages in thread
From: Victor Toso @ 2025-02-14 20:29 UTC (permalink / raw)
To: qemu-devel
Cc: Markus Armbruster, John Snow, Daniel P . Berrangé,
Andrea Bolognani
This patch handles QAPI union types and generates the equivalent data
structures and methods in Go to handle it.
The QAPI union type has two types of fields: The @base and the
@Variants members. The @base fields can be considered common members
for the union while only one field maximum is set for the @Variants.
In the QAPI specification, it defines a @discriminator field, which is
an Enum type. The purpose of the @discriminator is to identify which
@variant type is being used.
For the @discriminator's enum that are not handled by the QAPI Union,
we add in the Go struct a separate block as "Unbranched enum fields".
The rationale for this extra block is to allow the user to pass that
enum value under the discriminator, without extra payload.
The union types implement the Marshaler and Unmarshaler interfaces to
seamless decode from JSON objects to Golang structs and vice versa.
qapi:
| ##
| # @SetPasswordOptions:
| #
| # Options for set_password.
| #
| # @protocol:
| # - 'vnc' to modify the VNC server password
| # - 'spice' to modify the Spice server password
| #
| # @password: the new password
| #
| # @connected: How to handle existing clients when changing the
| # password. If nothing is specified, defaults to 'keep'. For
| # VNC, only 'keep' is currently implemented.
| #
| # Since: 7.0
| ##
| { 'union': 'SetPasswordOptions',
| 'base': { 'protocol': 'DisplayProtocol',
| 'password': 'str',
| '*connected': 'SetPasswordAction' },
| 'discriminator': 'protocol',
| 'data': { 'vnc': 'SetPasswordOptionsVnc' } }
go:
| // Options for set_password.
| //
| // Since: 7.0
| type SetPasswordOptions struct {
| // the new password
| Password string `json:"password"`
| // How to handle existing clients when changing the password. If
| // nothing is specified, defaults to 'keep'. For VNC, only 'keep'
| // is currently implemented.
| Connected *SetPasswordAction `json:"connected,omitempty"`
| // Variants fields
| Vnc *SetPasswordOptionsVnc `json:"-"`
| // Unbranched enum fields
| Spice bool `json:"-"`
| }
|
| func (s SetPasswordOptions) MarshalJSON() ([]byte, error) {
| ...
| }
|
| func (s *SetPasswordOptions) UnmarshalJSON(data []byte) error {
| ...
| }
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang/golang.py | 208 +++++++++++++++++++++++++++++++++-
scripts/qapi/golang/utils.go | 12 ++
2 files changed, 217 insertions(+), 3 deletions(-)
diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py
index 0637bb3e3e..59e9b1f644 100644
--- a/scripts/qapi/golang/golang.py
+++ b/scripts/qapi/golang/golang.py
@@ -155,6 +155,81 @@
"""
+TEMPLATE_UNION_CHECK_VARIANT_FIELD = """
+ if s.{field} != nil && err == nil {{
+ if len(bytes) != 0 {{
+ err = errors.New(`multiple variant fields set`)
+ }} else if err = unwrapToMap(m, s.{field}); err == nil {{
+ m["{discriminator}"] = {go_enum_value}
+ bytes, err = json.Marshal(m)
+ }}
+ }}
+"""
+
+TEMPLATE_UNION_CHECK_UNBRANCHED_FIELD = """
+ if s.{field} && err == nil {{
+ if len(bytes) != 0 {{
+ err = errors.New(`multiple variant fields set`)
+ }} else {{
+ m["{discriminator}"] = {go_enum_value}
+ bytes, err = json.Marshal(m)
+ }}
+ }}
+"""
+
+TEMPLATE_UNION_DRIVER_VARIANT_CASE = """
+ case {go_enum_value}:
+ s.{field} = new({member_type})
+ if err := json.Unmarshal(data, s.{field}); err != nil {{
+ s.{field} = nil
+ return err
+ }}"""
+
+TEMPLATE_UNION_DRIVER_UNBRANCHED_CASE = """
+ case {go_enum_value}:
+ s.{field} = true
+"""
+
+TEMPLATE_UNION_METHODS = """
+func (s {type_name}) MarshalJSON() ([]byte, error) {{
+ var bytes []byte
+ var err error
+ m := make(map[string]any)
+ {{
+ type Alias {type_name}
+ v := Alias(s)
+ unwrapToMap(m, &v)
+ }}
+{check_fields}
+ if err != nil {{
+ return nil, fmt.Errorf("marshal {type_name} due:'%s' struct='%+v'", err, s)
+ }} else if len(bytes) == 0 {{
+ return nil, fmt.Errorf("marshal {type_name} unsupported, struct='%+v'", s)
+ }}
+ return bytes, nil
+}}
+
+func (s *{type_name}) UnmarshalJSON(data []byte) error {{
+{base_type_def}
+ tmp := struct {{
+ {base_type_name}
+ }}{{}}
+
+ if err := json.Unmarshal(data, &tmp); err != nil {{
+ return err
+ }}
+{base_type_assign_unmarshal}
+ switch tmp.{discriminator} {{
+{driver_cases}
+ default:
+ return fmt.Errorf("unmarshal {type_name} received unrecognized value '%s'",
+ tmp.{discriminator})
+ }}
+ return nil
+}}
+"""
+
+
# Takes the documentation object of a specific type and returns
# that type's documentation and its member's docs.
def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]):
@@ -202,6 +277,12 @@ def qapi_name_is_object(name: str) -> bool:
return name.startswith("q_obj_")
+def qapi_base_name_to_parent(name: str) -> str:
+ if qapi_name_is_base(name):
+ name = name[6:-5]
+ return name
+
+
def qapi_to_field_name(name: str) -> str:
return name.title().replace("_", "").replace("-", "")
@@ -639,7 +720,7 @@ def recursive_base(
embed_base = self.schema.lookup_entity(base.base.name)
fields, with_nullable = recursive_base(self, embed_base, discriminator)
- doc = self.docmap.get(base.name, None)
+ doc = self.docmap.get(qapi_base_name_to_parent(base.name), None)
_, docfields = qapi_to_golang_struct_docs(doc)
for member in base.local_members:
@@ -719,6 +800,24 @@ def qapi_to_golang_struct(
fields.append(field)
with_nullable = True if nullable else with_nullable
+ if info.defn_meta == "union" and variants:
+ enum_name = variants.tag_member.type.name
+ enum_obj = self.schema.lookup_entity(enum_name)
+ if len(exists) != len(enum_obj.members):
+ fields.append({"comment": "Unbranched enum fields"})
+ for member in enum_obj.members:
+ if member.name in exists:
+ continue
+
+ field_doc = (
+ docfields.get(member.name, "") if doc_enabled else ""
+ )
+ field, nullable = get_struct_field(
+ self, member.name, "bool", field_doc, False, False, True
+ )
+ fields.append(field)
+ with_nullable = True if nullable else with_nullable
+
type_name = qapi_to_go_type_name(name)
content = string_to_code(
generate_struct_type(
@@ -732,6 +831,98 @@ def qapi_to_golang_struct(
return content
+def qapi_to_golang_methods_union(
+ self: QAPISchemaGenGolangVisitor,
+ name: str,
+ base: Optional[QAPISchemaObjectType],
+ variants: Optional[QAPISchemaVariants],
+) -> str:
+ type_name = qapi_to_go_type_name(name)
+
+ assert base
+ base_type_assign_unmarshal = ""
+ base_type_name = qapi_to_go_type_name(base.name)
+ base_type_def = qapi_to_golang_struct(
+ self,
+ base.name,
+ base.info,
+ base.ifcond,
+ base.features,
+ base.base,
+ base.members,
+ base.branches,
+ indent=1,
+ doc_enabled=False,
+ )
+
+ discriminator = qapi_to_field_name(variants.tag_member.name)
+ for member in base.local_members:
+ field = qapi_to_field_name(member.name)
+ if field == discriminator:
+ continue
+ base_type_assign_unmarshal += f"""
+ s.{field} = tmp.{field}"""
+
+ driver_cases = ""
+ check_fields = ""
+ exists = {}
+ enum_name = variants.tag_member.type.name
+ if variants:
+ for var in variants.variants:
+ if var.type.is_implicit():
+ continue
+
+ field = qapi_to_field_name(var.name)
+ enum_value = qapi_to_field_name_enum(var.name)
+ member_type = qapi_schema_type_to_go_type(var.type.name)
+ go_enum_value = f"""{enum_name}{enum_value}"""
+ exists[go_enum_value] = True
+
+ check_fields += TEMPLATE_UNION_CHECK_VARIANT_FIELD.format(
+ field=field,
+ discriminator=variants.tag_member.name,
+ go_enum_value=go_enum_value,
+ )
+ driver_cases += TEMPLATE_UNION_DRIVER_VARIANT_CASE.format(
+ go_enum_value=go_enum_value,
+ field=field,
+ member_type=member_type,
+ )
+
+ enum_obj = self.schema.lookup_entity(enum_name)
+ if len(exists) != len(enum_obj.members):
+ for member in enum_obj.members:
+ value = qapi_to_field_name_enum(member.name)
+ go_enum_value = f"""{enum_name}{value}"""
+
+ if go_enum_value in exists:
+ continue
+
+ field = qapi_to_field_name(member.name)
+
+ check_fields += TEMPLATE_UNION_CHECK_UNBRANCHED_FIELD.format(
+ field=field,
+ discriminator=variants.tag_member.name,
+ go_enum_value=go_enum_value,
+ )
+ driver_cases += TEMPLATE_UNION_DRIVER_UNBRANCHED_CASE.format(
+ go_enum_value=go_enum_value,
+ field=field,
+ )
+
+ return string_to_code(
+ TEMPLATE_UNION_METHODS.format(
+ type_name=type_name,
+ check_fields=check_fields[1:],
+ base_type_def=base_type_def[1:],
+ base_type_name=base_type_name,
+ base_type_assign_unmarshal=base_type_assign_unmarshal,
+ discriminator=discriminator,
+ driver_cases=driver_cases[1:],
+ )
+ )
+
+
def generate_template_alternate(
self: QAPISchemaGenGolangVisitor,
name: str,
@@ -855,6 +1046,7 @@ def __init__(self, _: str):
"alternate": ["encoding/json", "errors", "fmt"],
"enum": [],
"struct": ["encoding/json"],
+ "union": ["encoding/json", "errors", "fmt"],
}
self.schema: QAPISchema
@@ -863,6 +1055,7 @@ def __init__(self, _: str):
self.enums: dict[str, str] = {}
self.alternates: dict[str, str] = {}
self.structs: dict[str, str] = {}
+ self.unions: dict[str, str] = {}
self.accept_null_types = []
self.docmap = {}
@@ -902,6 +1095,7 @@ def visit_end(self) -> None:
self.types["enum"] += generate_content_from_dict(self.enums)
self.types["alternate"] += generate_content_from_dict(self.alternates)
self.types["struct"] += generate_content_from_dict(self.structs)
+ self.types["union"] += generate_content_from_dict(self.unions)
def visit_object_type(
self,
@@ -913,11 +1107,11 @@ def visit_object_type(
members: List[QAPISchemaObjectTypeMember],
branches: Optional[QAPISchemaBranches],
) -> None:
- # Do not handle anything besides struct.
+ # Do not handle anything besides struct and unions.
if (
name == self.schema.the_empty_object_type.name
or not isinstance(name, str)
- or info.defn_meta not in ["struct"]
+ or info.defn_meta not in ["struct", "union"]
):
return
@@ -948,6 +1142,14 @@ def visit_object_type(
self, name, info, ifcond, features, base, members, branches
)
)
+ else:
+ assert name not in self.unions
+ self.unions[name] = qapi_to_golang_struct(
+ self, name, info, ifcond, features, base, members, branches
+ )
+ self.unions[name] += qapi_to_golang_methods_union(
+ self, name, base, branches
+ )
def visit_alternate_type(
self,
diff --git a/scripts/qapi/golang/utils.go b/scripts/qapi/golang/utils.go
index f00c0a5d83..193e0c53bb 100644
--- a/scripts/qapi/golang/utils.go
+++ b/scripts/qapi/golang/utils.go
@@ -9,6 +9,7 @@ package qapi
import (
"encoding/json"
+ "fmt"
"strings"
)
@@ -24,3 +25,14 @@ func strictDecode(into interface{}, from []byte) error {
}
return nil
}
+
+// This helper is used to move struct's fields into a map.
+// This function is useful to merge JSON objects.
+func unwrapToMap(m map[string]any, data any) error {
+ if bytes, err := json.Marshal(&data); err != nil {
+ return fmt.Errorf("unwrapToMap: %s", err)
+ } else if err := json.Unmarshal(bytes, &m); err != nil {
+ return fmt.Errorf("unwrapToMap: %s, data=%s", err, string(bytes))
+ }
+ return nil
+}
--
2.48.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v4 07/11] qapi: golang: Generate event type
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
` (5 preceding siblings ...)
2025-02-14 20:29 ` [PATCH v4 06/11] qapi: golang: Generate union type Victor Toso
@ 2025-02-14 20:29 ` Victor Toso
2025-02-14 20:29 ` [PATCH v4 08/11] qapi: golang: Generate Event interface Victor Toso
` (5 subsequent siblings)
12 siblings, 0 replies; 18+ messages in thread
From: Victor Toso @ 2025-02-14 20:29 UTC (permalink / raw)
To: qemu-devel
Cc: Markus Armbruster, John Snow, Daniel P . Berrangé,
Andrea Bolognani
This patch handles QAPI event types and generates data structures in
Go that handles it.
Note that the timestamp is part of the first layer of unmarshal, so it
s a member of protocol.go's Message type.
Example:
qapi:
| ##
| # @MEMORY_DEVICE_SIZE_CHANGE:
| #
| # Emitted when the size of a memory device changes. Only emitted for
| # memory devices that can actually change the size (e.g., virtio-mem
| # due to guest action).
| #
| # @id: device's ID
| #
| # @size: the new size of memory that the device provides
| #
| # @qom-path: path to the device object in the QOM tree (since 6.2)
| #
| # .. note:: This event is rate-limited.
| #
| # Since: 5.1
| #
| # .. qmp-example::
| #
| # <- { "event": "MEMORY_DEVICE_SIZE_CHANGE",
| # "data": { "id": "vm0", "size": 1073741824,
| # "qom-path": "/machine/unattached/device[2]" },
| # "timestamp": { "seconds": 1588168529, "microseconds": 201316 } }
| ##
| { 'event': 'MEMORY_DEVICE_SIZE_CHANGE',
| 'data': { '*id': 'str', 'size': 'size', 'qom-path' : 'str'} }
go:
| // Emitted when the size of a memory device changes. Only emitted for
| // memory devices that can actually change the size (e.g., virtio-mem
| // due to guest action).
| //
| // .. note:: This event is rate-limited.
| //
| // Since: 5.1
| //
| // .. qmp-example:: <- { "event": "MEMORY_DEVICE_SIZE_CHANGE",
| // "data": { "id": "vm0", "size": 1073741824, "qom-path":
| // "/machine/unattached/device[2]" }, "timestamp": { "seconds":
| // 1588168529, "microseconds": 201316 } }
| type MemoryDeviceSizeChangeEvent struct {
| // device's ID
| Id *string `json:"id,omitempty"`
| // the new size of memory that the device provides
| Size uint64 `json:"size"`
| // path to the device object in the QOM tree (since 6.2)
| QomPath string `json:"qom-path"`
| }
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang/golang.py | 49 ++++++++++++++++++++++++++++++++---
1 file changed, 46 insertions(+), 3 deletions(-)
diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py
index 59e9b1f644..63d55ca950 100644
--- a/scripts/qapi/golang/golang.py
+++ b/scripts/qapi/golang/golang.py
@@ -291,7 +291,7 @@ def qapi_to_field_name_enum(name: str) -> str:
return name.title().replace("-", "")
-def qapi_to_go_type_name(name: str) -> str:
+def qapi_to_go_type_name(name: str, meta: Optional[str] = None) -> str:
# We want to keep CamelCase for Golang types. We want to avoid removing
# already set CameCase names while fixing uppercase ones, eg:
# 1) q_obj_SocketAddress_base -> SocketAddressBase
@@ -309,6 +309,12 @@ def qapi_to_go_type_name(name: str) -> str:
name += "".join(word.title() for word in words[1:])
+ # Handle specific meta suffix
+ types = ["event"]
+ if meta in types:
+ name = name[:-3] if name.endswith("Arg") else name
+ name += meta.title().replace(" ", "")
+
return name
@@ -818,7 +824,8 @@ def qapi_to_golang_struct(
fields.append(field)
with_nullable = True if nullable else with_nullable
- type_name = qapi_to_go_type_name(name)
+ type_name = qapi_to_go_type_name(name, info.defn_meta)
+
content = string_to_code(
generate_struct_type(
type_name, type_doc=type_doc, args=fields, indent=indent
@@ -996,6 +1003,15 @@ def generate_template_alternate(
return "\n" + content
+def generate_template_event(events: dict[str, Tuple[str, str]]) -> str:
+ content = ""
+ for name in sorted(events):
+ type_name, gocode = events[name]
+ content += gocode
+
+ return content
+
+
def generate_content_from_dict(data: dict[str, str]) -> str:
content = ""
@@ -1045,11 +1061,13 @@ def __init__(self, _: str):
types = {
"alternate": ["encoding/json", "errors", "fmt"],
"enum": [],
+ "event": [],
"struct": ["encoding/json"],
"union": ["encoding/json", "errors", "fmt"],
}
self.schema: QAPISchema
+ self.events: dict[str, Tuple[str, str]] = {}
self.golang_package_name = "qapi"
self.duplicate = list(gofiles)
self.enums: dict[str, str] = {}
@@ -1096,6 +1114,7 @@ def visit_end(self) -> None:
self.types["alternate"] += generate_content_from_dict(self.alternates)
self.types["struct"] += generate_content_from_dict(self.structs)
self.types["union"] += generate_content_from_dict(self.unions)
+ self.types["event"] += generate_template_event(self.events)
def visit_object_type(
self,
@@ -1254,7 +1273,31 @@ def visit_event(
arg_type: Optional[QAPISchemaObjectType],
boxed: bool,
) -> None:
- pass
+ assert name == info.defn_name
+ assert name not in self.events
+ type_name = qapi_to_go_type_name(name, info.defn_meta)
+
+ if isinstance(arg_type, QAPISchemaObjectType):
+ content = string_to_code(
+ qapi_to_golang_struct(
+ self,
+ name,
+ info,
+ arg_type.ifcond,
+ arg_type.features,
+ arg_type.base,
+ arg_type.members,
+ arg_type.branches,
+ )
+ )
+ else:
+ doc = self.docmap.get(name, None)
+ type_doc, _ = qapi_to_golang_struct_docs(doc)
+ content = string_to_code(
+ generate_struct_type(type_name, type_doc=type_doc)
+ )
+
+ self.events[name] = (type_name, content)
def write(self, outdir: str) -> None:
godir = "go"
--
2.48.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v4 08/11] qapi: golang: Generate Event interface
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
` (6 preceding siblings ...)
2025-02-14 20:29 ` [PATCH v4 07/11] qapi: golang: Generate event type Victor Toso
@ 2025-02-14 20:29 ` Victor Toso
2025-02-14 20:29 ` [PATCH v4 09/11] qapi: golang: Generate command type Victor Toso
` (4 subsequent siblings)
12 siblings, 0 replies; 18+ messages in thread
From: Victor Toso @ 2025-02-14 20:29 UTC (permalink / raw)
To: qemu-devel
Cc: Markus Armbruster, John Snow, Daniel P . Berrangé,
Andrea Bolognani
The Event interface is an abstraction that can be used by client and
server to the manager the Event types albeit with a different
implementation for sending and receiving.
The implementation of client/server is not part of this series.
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang/golang.py | 38 ++++++++++++++++++++++++++++++++---
1 file changed, 35 insertions(+), 3 deletions(-)
diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py
index 63d55ca950..b9a2c47137 100644
--- a/scripts/qapi/golang/golang.py
+++ b/scripts/qapi/golang/golang.py
@@ -229,6 +229,12 @@
}}
"""
+TEMPLATE_EVENT = """
+type Event interface {{
+{methods}
+}}
+"""
+
# Takes the documentation object of a specific type and returns
# that type's documentation and its member's docs.
@@ -1003,13 +1009,16 @@ def generate_template_alternate(
return "\n" + content
-def generate_template_event(events: dict[str, Tuple[str, str]]) -> str:
+def generate_template_event(events: dict[str, Tuple[str, str]]) -> (str, str):
content = ""
+ methods = ""
for name in sorted(events):
type_name, gocode = events[name]
+ methods += f"\t{type_name}({type_name}, time.Time) error\n"
content += gocode
- return content
+ iface = string_to_code(TEMPLATE_EVENT.format(methods=methods[:-1]))
+ return content, iface
def generate_content_from_dict(data: dict[str, str]) -> str:
@@ -1065,6 +1074,9 @@ def __init__(self, _: str):
"struct": ["encoding/json"],
"union": ["encoding/json", "errors", "fmt"],
}
+ interfaces = {
+ "event": ["time"],
+ }
self.schema: QAPISchema
self.events: dict[str, Tuple[str, str]] = {}
@@ -1080,6 +1092,9 @@ def __init__(self, _: str):
self.types = dict.fromkeys(types, "")
self.types_import = types
+ self.interfaces = dict.fromkeys(interfaces, "")
+ self.interface_imports = interfaces
+
def visit_begin(self, schema: QAPISchema) -> None:
self.schema = schema
@@ -1100,6 +1115,12 @@ def visit_begin(self, schema: QAPISchema) -> None:
continue
self.docmap[doc.symbol] = doc
+ for qapitype, imports in self.interface_imports.items():
+ self.interfaces[qapitype] = TEMPLATE_GENERATED_HEADER[1:].format(
+ package_name=self.golang_package_name
+ )
+ self.interfaces[qapitype] += generate_template_imports(imports)
+
for qapitype, imports in self.types_import.items():
self.types[qapitype] = TEMPLATE_GENERATED_HEADER[1:].format(
package_name=self.golang_package_name
@@ -1114,7 +1135,10 @@ def visit_end(self) -> None:
self.types["alternate"] += generate_content_from_dict(self.alternates)
self.types["struct"] += generate_content_from_dict(self.structs)
self.types["union"] += generate_content_from_dict(self.unions)
- self.types["event"] += generate_template_event(self.events)
+
+ evtype, eviface = generate_template_event(self.events)
+ self.types["event"] += evtype
+ self.interfaces["event"] += eviface
def visit_object_type(
self,
@@ -1318,3 +1342,11 @@ def write(self, outdir: str) -> None:
with open(pathname, "w", encoding="utf8") as outfile:
outfile.write(content)
+
+ # Interfaces to be generated
+ for qapitype, content in self.interfaces.items():
+ gofile = f"gen_iface_{qapitype}.go"
+ pathname = os.path.join(targetpath, gofile)
+
+ with open(pathname, "w", encoding="utf8") as outfile:
+ outfile.write(content)
--
2.48.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v4 09/11] qapi: golang: Generate command type
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
` (7 preceding siblings ...)
2025-02-14 20:29 ` [PATCH v4 08/11] qapi: golang: Generate Event interface Victor Toso
@ 2025-02-14 20:29 ` Victor Toso
2025-02-14 20:29 ` [PATCH v4 10/11] qapi: golang: Generate Command sync/async interfaces Victor Toso
` (3 subsequent siblings)
12 siblings, 0 replies; 18+ messages in thread
From: Victor Toso @ 2025-02-14 20:29 UTC (permalink / raw)
To: qemu-devel
Cc: Markus Armbruster, John Snow, Daniel P . Berrangé,
Andrea Bolognani
This patch handles QAPI command types and generates data structures in
Go that handles it.
Note that command's id is part of the first layer of unmarshal, so it
is a member of protocol.go's Message type.
qapi:
| ##
| # @add-fd:
| #
| # Add a file descriptor, that was passed via SCM rights, to an fd set.
| #
| # @fdset-id: The ID of the fd set to add the file descriptor to.
| #
| # @opaque: A free-form string that can be used to describe the fd.
| #
| # Returns:
| # @AddfdInfo
| #
| # Errors:
| # - If file descriptor was not received, GenericError
| # - If @fdset-id is a negative value, GenericError
| #
| # .. note:: The list of fd sets is shared by all monitor connections.
| #
| # .. note:: If @fdset-id is not specified, a new fd set will be
| # created.
| #
| # Since: 1.2
| #
| # .. qmp-example::
| #
| # -> { "execute": "add-fd", "arguments": { "fdset-id": 1 } }
| # <- { "return": { "fdset-id": 1, "fd": 3 } }
| ##
| { 'command': 'add-fd',
| 'data': { '*fdset-id': 'int',
| '*opaque': 'str' },
| 'returns': 'AddfdInfo' }
go:
| // Add a file descriptor, that was passed via SCM rights, to an fd
| // set.
| //
| // Returns: @AddfdInfo
| //
| // Errors: - If file descriptor was not received, GenericError -
| // If @fdset-id is a negative value, GenericError
| //
| // .. note:: The list of fd sets is shared by all monitor connections.
| // .. note:: If @fdset-id is not specified, a new fd set will be
| // created.
| //
| // Since: 1.2
| //
| // .. qmp-example:: -> { "execute": "add-fd", "arguments": {
| // "fdset-id": 1 } } <- { "return": { "fdset-id": 1, "fd": 3 } }
| type AddFdCommand struct {
| // The ID of the fd set to add the file descriptor to.
| FdsetId *int64 `json:"fdset-id,omitempty"`
| // A free-form string that can be used to describe the fd.
| Opaque *string `json:"opaque,omitempty"`
| }
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang/golang.py | 52 +++++++++++++++++++++++++++++++++--
1 file changed, 50 insertions(+), 2 deletions(-)
diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py
index b9a2c47137..a14970fb1f 100644
--- a/scripts/qapi/golang/golang.py
+++ b/scripts/qapi/golang/golang.py
@@ -316,7 +316,7 @@ def qapi_to_go_type_name(name: str, meta: Optional[str] = None) -> str:
name += "".join(word.title() for word in words[1:])
# Handle specific meta suffix
- types = ["event"]
+ types = ["event", "command"]
if meta in types:
name = name[:-3] if name.endswith("Arg") else name
name += meta.title().replace(" ", "")
@@ -1009,6 +1009,15 @@ def generate_template_alternate(
return "\n" + content
+def generate_template_command(commands: dict[str, Tuple[str, str]]) -> str:
+ content = ""
+ for name in sorted(commands):
+ type_name, gocode = commands[name]
+ content += gocode
+
+ return content
+
+
def generate_template_event(events: dict[str, Tuple[str, str]]) -> (str, str):
content = ""
methods = ""
@@ -1069,6 +1078,7 @@ def __init__(self, _: str):
# Map each qapi type to the necessary Go imports
types = {
"alternate": ["encoding/json", "errors", "fmt"],
+ "command": [],
"enum": [],
"event": [],
"struct": ["encoding/json"],
@@ -1080,6 +1090,7 @@ def __init__(self, _: str):
self.schema: QAPISchema
self.events: dict[str, Tuple[str, str]] = {}
+ self.commands: dict[str, Tuple[str, str]] = {}
self.golang_package_name = "qapi"
self.duplicate = list(gofiles)
self.enums: dict[str, str] = {}
@@ -1140,6 +1151,8 @@ def visit_end(self) -> None:
self.types["event"] += evtype
self.interfaces["event"] += eviface
+ self.types["command"] += generate_template_command(self.commands)
+
def visit_object_type(
self,
name: str,
@@ -1286,7 +1299,42 @@ def visit_command(
allow_preconfig: bool,
coroutine: bool,
) -> None:
- pass
+ assert name == info.defn_name
+ assert name not in self.commands
+
+ type_name = qapi_to_go_type_name(name, info.defn_meta)
+
+ doc = self.docmap.get(name, None)
+ type_doc, _ = qapi_to_golang_struct_docs(doc)
+
+ content = ""
+ if boxed or not arg_type or not qapi_name_is_object(arg_type.name):
+ args: List[dict[str:str]] = []
+ if arg_type:
+ args.append(
+ {
+ "name": f"{arg_type.name}",
+ }
+ )
+ content += string_to_code(
+ generate_struct_type(type_name, type_doc=type_doc, args=args)
+ )
+ else:
+ assert isinstance(arg_type, QAPISchemaObjectType)
+ content += string_to_code(
+ qapi_to_golang_struct(
+ self,
+ name,
+ arg_type.info,
+ arg_type.ifcond,
+ arg_type.features,
+ arg_type.base,
+ arg_type.members,
+ arg_type.branches,
+ )
+ )
+
+ self.commands[name] = (type_name, content)
def visit_event(
self,
--
2.48.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v4 10/11] qapi: golang: Generate Command sync/async interfaces
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
` (8 preceding siblings ...)
2025-02-14 20:29 ` [PATCH v4 09/11] qapi: golang: Generate command type Victor Toso
@ 2025-02-14 20:29 ` Victor Toso
2025-02-14 20:29 ` [PATCH v4 11/11] docs: add notes on Golang code generator Victor Toso
` (2 subsequent siblings)
12 siblings, 0 replies; 18+ messages in thread
From: Victor Toso @ 2025-02-14 20:29 UTC (permalink / raw)
To: qemu-devel
Cc: Markus Armbruster, John Snow, Daniel P . Berrangé,
Andrea Bolognani
The Command interface is an abstraction that can be used by client and
server to the manager the Command types albeit with a different
implementation for sending and receiving.
The proposal for Sync is defined as Command while the Async is named
CommandAsync.
The implementation of client/server is not part of this series.
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
scripts/qapi/golang/golang.py | 56 +++++++++++++++++++++++++++++++----
1 file changed, 50 insertions(+), 6 deletions(-)
diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py
index a14970fb1f..0b7dadff41 100644
--- a/scripts/qapi/golang/golang.py
+++ b/scripts/qapi/golang/golang.py
@@ -235,6 +235,20 @@
}}
"""
+TEMPLATE_COMMAND = """
+// Synchronous interface of all available commands
+type Commands interface {{
+{methods}
+}}
+
+{callbacks}
+
+// Asynchronous interface of all available commands
+type CommandsAsync interface {{
+{async_methods}
+}}
+"""
+
# Takes the documentation object of a specific type and returns
# that type's documentation and its member's docs.
@@ -1009,13 +1023,37 @@ def generate_template_alternate(
return "\n" + content
-def generate_template_command(commands: dict[str, Tuple[str, str]]) -> str:
+def generate_template_command(
+ commands: dict[str, Tuple[str, str, str]]
+) -> (str, str):
content = ""
+ methods = ""
+ async_methods = ""
+ callbacks = ""
+
for name in sorted(commands):
- type_name, gocode = commands[name]
+ type_name, gocode, retarg = commands[name]
content += gocode
- return content
+ name = type_name[:-7]
+ cbname = f"{name}Complete"
+ syncret = "error"
+ cbarg = "error"
+ if retarg != "":
+ cbarg = f"{retarg}, error"
+ syncret = f"({retarg}, error)"
+ methods += f"\t{name}({type_name}) {syncret}\n"
+ async_methods += f"\t{name}({type_name}, {cbname}) error\n"
+ callbacks += f"type {cbname} func({cbarg})\n"
+
+ iface = string_to_code(
+ TEMPLATE_COMMAND.format(
+ methods=methods[:-1],
+ callbacks=callbacks[:-1],
+ async_methods=async_methods[:-1],
+ )
+ )
+ return content, iface
def generate_template_event(events: dict[str, Tuple[str, str]]) -> (str, str):
@@ -1085,12 +1123,13 @@ def __init__(self, _: str):
"union": ["encoding/json", "errors", "fmt"],
}
interfaces = {
+ "command": [],
"event": ["time"],
}
self.schema: QAPISchema
self.events: dict[str, Tuple[str, str]] = {}
- self.commands: dict[str, Tuple[str, str]] = {}
+ self.commands: dict[str, Tuple[str, str, str]] = {}
self.golang_package_name = "qapi"
self.duplicate = list(gofiles)
self.enums: dict[str, str] = {}
@@ -1151,7 +1190,9 @@ def visit_end(self) -> None:
self.types["event"] += evtype
self.interfaces["event"] += eviface
- self.types["command"] += generate_template_command(self.commands)
+ cmdtype, cmdiface = generate_template_command(self.commands)
+ self.types["command"] += cmdtype
+ self.interfaces["command"] += cmdiface
def visit_object_type(
self,
@@ -1334,7 +1375,10 @@ def visit_command(
)
)
- self.commands[name] = (type_name, content)
+ retarg = ""
+ if ret_type:
+ retarg = qapi_schema_type_to_go_type(ret_type.name)
+ self.commands[name] = (type_name, content, retarg)
def visit_event(
self,
--
2.48.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH v4 11/11] docs: add notes on Golang code generator
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
` (9 preceding siblings ...)
2025-02-14 20:29 ` [PATCH v4 10/11] qapi: golang: Generate Command sync/async interfaces Victor Toso
@ 2025-02-14 20:29 ` Victor Toso
2025-02-17 13:13 ` [PATCH v4 00/11] Daniel P. Berrangé
2025-02-17 14:58 ` Daniel P. Berrangé
12 siblings, 0 replies; 18+ messages in thread
From: Victor Toso @ 2025-02-14 20:29 UTC (permalink / raw)
To: qemu-devel
Cc: Markus Armbruster, John Snow, Daniel P . Berrangé,
Andrea Bolognani
The goal of this patch is converge discussions into a documentation,
to make it easy and explicit design decisions, known issues and what
else might help a person interested in how the Go module is generated.
Signed-off-by: Victor Toso <victortoso@redhat.com>
---
docs/devel/index-build.rst | 1 +
docs/devel/qapi-golang-code-gen.rst | 420 ++++++++++++++++++++++++++++
2 files changed, 421 insertions(+)
create mode 100644 docs/devel/qapi-golang-code-gen.rst
diff --git a/docs/devel/index-build.rst b/docs/devel/index-build.rst
index 0745c81a26..13cc0646c6 100644
--- a/docs/devel/index-build.rst
+++ b/docs/devel/index-build.rst
@@ -12,4 +12,5 @@ some of the basics if you are adding new files and targets to the build.
kconfig
docs
qapi-code-gen
+ qapi-golang-code-gen
control-flow-integrity
diff --git a/docs/devel/qapi-golang-code-gen.rst b/docs/devel/qapi-golang-code-gen.rst
new file mode 100644
index 0000000000..b39c14cfd5
--- /dev/null
+++ b/docs/devel/qapi-golang-code-gen.rst
@@ -0,0 +1,420 @@
+==========================
+QAPI Golang code generator
+==========================
+
+..
+ Copyright (C) 2025 Red Hat, Inc.
+
+ This work is licensed under the terms of the GNU GPL, version 2 or
+ later. See the COPYING file in the top-level directory.
+
+
+Introduction
+============
+
+This document provides information of how the generated Go code maps
+with the QAPI specification, clarifying design decisions when needed.
+
+
+Scope of the generated Go code
+==============================
+
+The scope is to provide data structures that can interpret and be used
+to generate valid QMP messages. These data structures are generated
+from a QAPI schema and should be able to handle QMP messages from the
+same schema.
+
+We also provide interfaces for Commands and Events which allows an
+abstraction for client and server applications with the possibility of
+custom back end implantations.
+
+The generated Go code is a Go module with data structs that uses Go
+standard library ``encoding/json``, implementing its field tags and
+Marshal interface whenever needed.
+
+
+QAPI Documentation
+==================
+
+The documentation included in QAPI schema such as type and type's
+fields information, comments, examples and more, they are converted
+and embed in the Go generated source code. Metadata information that
+might not be relevant to developers are excluded (e.g: TODOs)
+
+
+QAPI types to Go structs
+========================
+
+Enum
+----
+
+Enums are mapped as strings in Go, using a specified string type per
+Enum to help with type safety in the Go application.
+
+::
+
+ { 'enum': 'HostMemPolicy',
+ 'data': [ 'default', 'preferred', 'bind', 'interleave' ] }
+
+.. code-block:: go
+
+ // Host memory policy types
+ //
+ // Since: 2.1
+ type HostMemPolicy string
+
+ const (
+ // restore default policy, remove any nondefault policy
+ HostMemPolicyDefault HostMemPolicy = "default"
+ // set the preferred host nodes for allocation
+ HostMemPolicyPreferred HostMemPolicy = "preferred"
+ // a strict policy that restricts memory allocation to the host nodes specified
+ HostMemPolicyBind HostMemPolicy = "bind"
+ // memory allocations are interleaved across the set of host nodes specified
+ HostMemPolicyInterleave HostMemPolicy = "interleave"
+ )
+
+
+Struct
+------
+
+The mapping between a QAPI struct in Go struct is very straightforward.
+ - Each member of the QAPI struct has its own field in a Go struct.
+ - Optional members are pointers type with 'omitempty' field tag set
+
+One important design decision was to _not_ embed base struct, copying
+the base members to the original struct. This reduces the complexity
+for the Go application.
+
+::
+
+ { 'struct': 'BlockExportOptionsNbdBase',
+ 'data': { '*name': 'str', '*description': 'str' } }
+
+ { 'struct': 'BlockExportOptionsNbd',
+ 'base': 'BlockExportOptionsNbdBase',
+ 'data': { '*bitmaps': ['BlockDirtyBitmapOrStr'],
+ '*allocation-depth': 'bool' } }
+
+.. code-block:: go
+
+ // An NBD block export (distinct options used in the NBD branch of
+ // block-export-add).
+ //
+ // Since: 5.2
+ type BlockExportOptionsNbd struct {
+ // Export name. If unspecified, the @device parameter is used as
+ // the export name. (Since 2.12)
+ Name *string `json:"name,omitempty"`
+ // Free-form description of the export, up to 4096 bytes. (Since
+ // 5.0)
+ Description *string `json:"description,omitempty"`
+ // Also export each of the named dirty bitmaps reachable from
+ // @device, so the NBD client can use NBD_OPT_SET_META_CONTEXT
+ // with the metadata context name "qemu:dirty-bitmap:BITMAP" to
+ // inspect each bitmap. Since 7.1 bitmap may be specified by
+ // node/name pair.
+ Bitmaps []BlockDirtyBitmapOrStr `json:"bitmaps,omitempty"`
+ // Also export the allocation depth map for @device, so the NBD
+ // client can use NBD_OPT_SET_META_CONTEXT with the metadata
+ // context name "qemu:allocation-depth" to inspect allocation
+ // details. (since 5.2)
+ AllocationDepth *bool `json:"allocation-depth,omitempty"`
+ }
+
+
+Union
+-----
+
+Unions in QAPI are bounded to a Enum type which provides all possible
+branches of the union. The most important caveat here is that the Union
+does not need to have a complex type implemented for all possible
+branches of the Enum. Receiving a enum value of a empty branch is valid.
+
+The generated Go struct will then define a field for each
+Enum value. The type for Enum values of empty branch is bool. Only one
+field can be set at time.
+
+::
+
+ { 'union': 'ImageInfoSpecificQCow2Encryption',
+ 'base': 'ImageInfoSpecificQCow2EncryptionBase',
+ 'discriminator': 'format',
+ 'data': { 'luks': 'QCryptoBlockInfoLUKS' } }
+
+ { 'struct': 'ImageInfoSpecificQCow2EncryptionBase',
+ 'data': { 'format': 'BlockdevQcow2EncryptionFormat'}}
+
+ { 'enum': 'BlockdevQcow2EncryptionFormat',
+ 'data': [ 'aes', 'luks' ] }
+
+.. code-block:: go
+
+ type ImageInfoSpecificQCow2Encryption struct {
+ // Variants fields
+ Luks *QCryptoBlockInfoLUKS `json:"-"`
+ // Empty branched enum fields
+ Aes bool `json:"-"`
+ }
+
+ func (s ImageInfoSpecificQCow2Encryption) MarshalJSON() ([]byte, error) {
+ // ...
+ // Logic for branched Enum
+ if s.Luks != nil && err == nil {
+ if len(bytes) != 0 {
+ err = errors.New(`multiple variant fields set`)
+ } else if err = unwrapToMap(m, s.Luks); err == nil {
+ m["format"] = BlockdevQcow2EncryptionFormatLuks
+ bytes, err = json.Marshal(m)
+ }
+ }
+
+ // Logic for unbranched Enum
+ if s.Aes && err == nil {
+ if len(bytes) != 0 {
+ err = errors.New(`multiple variant fields set`)
+ } else {
+ m["format"] = BlockdevQcow2EncryptionFormatAes
+ bytes, err = json.Marshal(m)
+ }
+ }
+
+ // ...
+ // Handle errors
+ }
+
+
+ func (s *ImageInfoSpecificQCow2Encryption) UnmarshalJSON(data []byte) error {
+ // ...
+
+ switch tmp.Format {
+ case BlockdevQcow2EncryptionFormatLuks:
+ s.Luks = new(QCryptoBlockInfoLUKS)
+ if err := json.Unmarshal(data, s.Luks); err != nil {
+ s.Luks = nil
+ return err
+ }
+ case BlockdevQcow2EncryptionFormatAes:
+ s.Aes = true
+
+ default:
+ return fmt.Errorf("error: unmarshal: ImageInfoSpecificQCow2Encryption: received unrecognized value: '%s'",
+ tmp.Format)
+ }
+ return nil
+ }
+
+
+Alternate
+---------
+
+Like Unions, alternates can have branches. Unlike Unions, they don't
+have a discriminator field and each branch should be a different class
+of Type entirely (e.g: You can't have two branches of type int in one
+Alternate).
+
+While the marshalling is similar to Unions, the unmarshalling uses a
+try-and-error approach, trying to fit the data payload in one of the
+Alternate fields.
+
+The biggest caveat is handling Alternates that can take JSON Null as
+value. The issue lies on ``encoding/json`` library limitation where
+unmarshalling JSON Null data to a Go struct which has the 'omitempty'
+field as it will bypass the Marshal interface. The same happens when
+marshalling, if the field tag 'omitempty' is used, a nil pointer would
+never be translated to null JSON value. The problem here is that we do
+use pointer to type plus ``omitempty`` field to express a QAPI
+optional member.
+
+In order to handle JSON Null, the generator needs to do the following:
+ - Read the QAPI schema prior to generate any code and cache
+ all alternate types that can take JSON Null
+ - For all Go structs that should be considered optional and they type
+ are one of those alternates, do not set ``omitempty`` and implement
+ Marshal interface for this Go struct, to properly handle JSON Null
+ - In the Alternate, uses a boolean 'IsNull' to express a JSON Null
+ and implement the AbsentAlternate interface, to help structs know
+ if a given Alternate type should be considered Absent (not set) or
+ any other possible Value, including JSON Null.
+
+::
+
+ { 'alternate': 'BlockdevRefOrNull',
+ 'data': { 'definition': 'BlockdevOptions',
+ 'reference': 'str',
+ 'null': 'null' } }
+
+.. code-block:: go
+
+ // Reference to a block device.
+ //
+ // Since: 2.9
+ type BlockdevRefOrNull struct {
+ // defines a new block device inline
+ Definition *BlockdevOptions
+ // references the ID of an existing block device. An empty string
+ // means that no block device should be referenced. Deprecated;
+ // use null instead.
+ Reference *string
+ // No block device should be referenced (since 2.10)
+ IsNull bool
+ }
+
+ func (s *BlockdevRefOrNull) ToAnyOrAbsent() (any, bool) {
+ if s != nil {
+ if s.IsNull {
+ return nil, false
+ } else if s.Definition != nil {
+ return *s.Definition, false
+ } else if s.Reference != nil {
+ return *s.Reference, false
+ }
+ }
+
+ return nil, true
+ }
+
+ func (s BlockdevRefOrNull) MarshalJSON() ([]byte, error) {
+ if s.IsNull {
+ return []byte("null"), nil
+ } else if s.Definition != nil {
+ return json.Marshal(s.Definition)
+ } else if s.Reference != nil {
+ return json.Marshal(s.Reference)
+ }
+ return []byte("{}"), nil
+ }
+
+ func (s *BlockdevRefOrNull) UnmarshalJSON(data []byte) error {
+ // Check for json-null first
+ if string(data) == "null" {
+ s.IsNull = true
+ return nil
+ }
+ // Check for BlockdevOptions
+ {
+ s.Definition = new(BlockdevOptions)
+ if err := StrictDecode(s.Definition, data); err == nil {
+ return nil
+ }
+ s.Definition = nil
+ }
+
+ // Check for string
+ {
+ s.Reference = new(string)
+ if err := StrictDecode(s.Reference, data); err == nil {
+ return nil
+ }
+ s.Reference = nil
+ }
+
+ return fmt.Errorf("Can't convert to BlockdevRefOrNull: %s", string(data))
+ }
+
+
+Event
+-----
+
+Each event is mapped to its own struct with.
+
+::
+
+ { 'event': 'SHUTDOWN',
+ 'data': { 'guest': 'bool',
+ 'reason': 'ShutdownCause' } }
+
+.. code-block:: go
+
+ // Emitted when the virtual machine has shut down, indicating that
+ // qemu is about to exit.
+ //
+ // .. note:: If the command-line option "-no-shutdown" has been
+ // specified, qemu will not exit, and a STOP event will eventually
+ // follow the SHUTDOWN event.
+ //
+ // Since: 0.12
+ //
+ // .. qmp-example:: <- { "event": "SHUTDOWN", "data": {
+ // "guest": true, "reason": "guest-shutdown" }, "timestamp": {
+ // "seconds": 1267040730, "microseconds": 682951 } }
+ type ShutdownEvent struct {
+ // If true, the shutdown was triggered by a guest request (such as
+ // a guest-initiated ACPI shutdown request or other hardware-
+ // specific action) rather than a host request (such as sending
+ // qemu a SIGINT). (since 2.10)
+ Guest bool `json:"guest"`
+ // The @ShutdownCause which resulted in the SHUTDOWN. (since 4.0)
+ Reason ShutdownCause `json:"reason"`
+ }
+
+
+Command
+-------
+
+Each commands is mapped to its own struct. If the command has a boxed
+data struct, the option struct will be embed in the command struct.
+
+The return value is always a well defined type and it is part of first
+layer unmarshalling type, Message.
+
+::
+
+ { 'command': 'set_password',
+ 'boxed': true,
+ 'data': 'SetPasswordOptions' }
+
+ { 'union': 'SetPasswordOptions',
+ 'base': { 'protocol': 'DisplayProtocol',
+ 'password': 'str',
+ '*connected': 'SetPasswordAction' },
+ 'discriminator': 'protocol',
+ 'data': { 'vnc': 'SetPasswordOptionsVnc' } }
+
+.. code-block:: go
+
+ // Set the password of a remote display server.
+ // Errors: - If Spice is not enabled, DeviceNotFound
+ //
+ // Since: 0.14
+ //
+ // .. qmp-example:: -> { "execute": "set_password", "arguments": {
+ // "protocol": "vnc", "password": "secret" }
+ // } <- { "return": {} }
+ type SetPasswordCommand struct {
+ SetPasswordOptions
+ }
+
+Now an example of a command without boxed type.
+
+::
+
+ { 'command': 'set_link',
+ 'data': {'name': 'str', 'up': 'bool'} }
+
+.. code-block:: go
+
+ // Sets the link status of a virtual network adapter.
+ //
+ // Errors: - If @name is not a valid network device, DeviceNotFound
+ //
+ // Since: 0.14
+ //
+ // .. note:: Not all network adapters support setting link status.
+ // This command will succeed even if the network adapter does not
+ // support link status notification. .. qmp-example:: -> {
+ // "execute": "set_link", "arguments": { "name": "e1000.0", "up":
+ // false } } <- { "return": {} }
+ type SetLinkCommand struct {
+ // the device name of the virtual network adapter
+ Name string `json:"name"`
+ // true to set the link status to be up
+ Up bool `json:"up"`
+ }
+
+Known issues
+============
+
+- Type names might not follow proper Go convention. Andrea suggested an
+ annotation to the QAPI schema that could solve it.
+ https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg00127.html
--
2.48.1
^ permalink raw reply related [flat|nested] 18+ messages in thread* Re: [PATCH v4 00/11]
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
` (10 preceding siblings ...)
2025-02-14 20:29 ` [PATCH v4 11/11] docs: add notes on Golang code generator Victor Toso
@ 2025-02-17 13:13 ` Daniel P. Berrangé
2025-02-20 8:06 ` Markus Armbruster
2025-02-17 14:58 ` Daniel P. Berrangé
12 siblings, 1 reply; 18+ messages in thread
From: Daniel P. Berrangé @ 2025-02-17 13:13 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, Markus Armbruster, John Snow, Andrea Bolognani
On Fri, Feb 14, 2025 at 09:29:33PM +0100, Victor Toso wrote:
> Hi again,
>
> This patch series intent is to introduce a generator that produces a Go
> module for Go applications to interact over QMP with QEMU.
>
> Previous version (10 Jan 2025)
> https://lists.gnu.org/archive/html/qemu-devel/2025-01/msg01530.html
>
> The generated code was mostly tested using existing examples in the QAPI
> documentation, 192 instances that might have multiple QMP messages each.
snip
> 1. Daniel wrote a demo on top of v3 and proposed changes that would
> result in more interesting module to build on top:
> https://lists.gnu.org/archive/html/qemu-devel/2025-01/msg03052.html
>
> I've implemented all the suggestions that are relevant for this
> introductory series, they are:
>
> a. New struct type Message, that shall be used for a 1st level
> unmarshalling of the JSON message.
> b. Removal of Marshal/Unmarshal code in both Events and Comands,
> together with utility code that is not relevant anymore.
> c. Declaration of 3 new interfaces:
> i. Events
> ii. Commands
> iii. CommandsAsync
>
> 2. I've moved the code to a new folder: scripts/qapi/golang. This
> allowed me to move templates out of golang.py, keeping go related
> code self-contained in the new directory.
When I think about the code generator and how this will all
evolve over time, I have a strong feeling that none of this
should be in qemu.git.
Taking those three interfaces in (1)(c), when we come to actually
generate implementations of them, the generated code is going to
be intimately tied to the client/server API framework we need to
plumb into.
IMHO qemu.git should not have any knowledge of this, as it will
create a bidirectional dependency betweeen qemu.git and the
qemu-go.git repo, requiring them to evolve in lockstep.
We'll need 3 releases of the Go code per year, to match the
introduction the new QAPI schema versions, but outside that I
think there ought to be the freedom to have other Go releases
independently.
The need for this to be part of qemu.git is fairly narrow. It
provides a new hook into the QAPI generator code, and the QAPI
generator is not installed by 'make install', it is in-tree
only.
Can we solve this linkage ? We would need to be able to:
* Have a mechanism for the QAPI code generator to load
new genrator backends
* Have a mechanism to tell the QAPI code generator to
only run a single backend.
* Have a mechanism to consume the QAPI code genrfator
from outside qemu.git
The first point there could be addressable using the python "entry-points"
concept
https://packaging.python.org/en/latest/specifications/entry-points/
https://setuptools.pypa.io/en/latest/pkg_resources.html#entry-points
Or, alternatively by passing the name of a python module on the CLI.
The second point is just a bit of glue code to wire up a new CLI arg
The third point either involves having 'make install' put the QAPI
code generator into /usr/share/qemu, or /usr/lib/python3.x. The
latter would be if QAPI code generator were a formally release python
module on pypi. The former would be if we're just trying a QAPI code
generator as a local tool, not for pypi. A fallback would be to
consume qemu.git as a git submodule from qapi-go.git. The gitmodule
commit would not have to match the version of the QAPI schema we are
generating code for.
The least effort approach would be making QAPI code genrator accept
a python module name, and call it via a git submodule. This would
avoid declaring a long term support policy for the QAPI code gen
python APIs as a blocker.
>
> 3. As mentioned in (2), created protocol.go and utils.go that are 100%
> hand generated Go code. Message mentioned in (1a) is under
> protocol.go
>
> 4. Defined license using SPDX-License-Identifier.
> a. Every Go source code written by hand is 100% MIT-0
> b. Every Go source code generated is dual licensed as MIT-0 and
> GPL-2.0-or-later
> c. The binary code is expected to be MIT-0 only but not really
> relevant for this series.
>
> If you want more information, please check the thread:
> https://lists.gnu.org/archive/html/qemu-devel/2024-11/msg01621.html
>
> 5. I've renamed the generated files.
> a. Any type related file is now prefixed with "gen_type_"
> b. Any interface related file is prefixed as "gen_iface_"
>
> 6. Relevant changes were made to the doc but it is not complete. I plan
> that follow-up proposals would add to the documentation.
>
> 7. Improvements to the generator were made to.
>
> 8. Also worth to mention that resulting generated code does not have any
> diff with gofmt and goimport tools, as requested in the past.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH v4 00/11]
2025-02-17 13:13 ` [PATCH v4 00/11] Daniel P. Berrangé
@ 2025-02-20 8:06 ` Markus Armbruster
0 siblings, 0 replies; 18+ messages in thread
From: Markus Armbruster @ 2025-02-20 8:06 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Victor Toso, qemu-devel, John Snow, Andrea Bolognani
Daniel P. Berrangé <berrange@redhat.com> writes:
[...]
> When I think about the code generator and how this will all
> evolve over time, I have a strong feeling that none of this
> should be in qemu.git.
Yes, keeping it in qemu.git has its drawbacks. Testing is awkward
there. The coupling could cause friction.
I'll gladly support exploring alternatives. I saw your "[PATCH] qapi:
pluggable backend code generators", and just sent my review.
Thanks!
[...]
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v4 00/11]
2025-02-14 20:29 [PATCH v4 00/11] Victor Toso
` (11 preceding siblings ...)
2025-02-17 13:13 ` [PATCH v4 00/11] Daniel P. Berrangé
@ 2025-02-17 14:58 ` Daniel P. Berrangé
2025-02-17 16:52 ` Victor Toso
12 siblings, 1 reply; 18+ messages in thread
From: Daniel P. Berrangé @ 2025-02-17 14:58 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, Markus Armbruster, John Snow, Andrea Bolognani
On Fri, Feb 14, 2025 at 09:29:33PM +0100, Victor Toso wrote:
> Hi again,
>
> This patch series intent is to introduce a generator that produces a Go
> module for Go applications to interact over QMP with QEMU.
>
> Previous version (10 Jan 2025)
> https://lists.gnu.org/archive/html/qemu-devel/2025-01/msg01530.html
>
> The generated code was mostly tested using existing examples in the QAPI
> documentation, 192 instances that might have multiple QMP messages each.
>
> You can find the the tests and the generated code in my personal repo,
> main branch:
>
> https://gitlab.com/victortoso/qapi-go
>
> If you want to see the generated code from QEMU's master but per patch:
>
> https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v4-by-patch
In terms of generated code, my only real feedback is that the
re-wrapping of docs comments is having undesirable effets
on formatting
##
# @add_client:
#
# Allow client connections for VNC, Spice and socket based character
# devices to be passed in to QEMU via SCM_RIGHTS.
#
# If the FD associated with @fdname is not a socket, the command will
# fail and the FD will be closed.
#
# @protocol: protocol name. Valid names are "vnc", "spice",
# "@dbus-display" or the name of a character device (e.g. from
# -chardev id=XXXX)
#
# @fdname: file descriptor name previously passed via 'getfd' command
#
# @skipauth: whether to skip authentication. Only applies to "vnc"
# and "spice" protocols
#
# @tls: whether to perform TLS. Only applies to the "spice" protocol
#
# Since: 0.14
#
# .. qmp-example::
#
# -> { "execute": "add_client", "arguments": { "protocol": "vnc",
# "fdname": "myclient" } }
# <- { "return": {} }
##
is getting turned into
// Allow client connections for VNC, Spice and socket based character
// devices to be passed in to QEMU via SCM_RIGHTS. If the FD
// associated with @fdname is not a socket, the command will fail and
// the FD will be closed.
//
// Since: 0.14
//
// .. qmp-example:: -> { "execute": "add_client", "arguments": {
// "protocol": "vnc", "fdname": "myclient" }
// } <- { "return": {} }
the '.. qmp-example' bit is what's particularly badly affected.
If we assume that the input QAPI schemas are nicely lined wrapped,
we could probably just preserve the docs lines as-is with no change
in wrapping.
That said I'm not sure if we'll need some docs syntax changes to
make it render nicely - hard to say until the code appears up on
pkg.go.dev, so can probably worry about that aspect later.
> ################
> # Expectations #
> ################
>
> As is, this still is a PoC that works. I'd like to have the generated
> code included in QEMU's gitlab [0] in order to write library and tools
> on top. Initial version should be considered alpha. Moving to
> beta/stable would require functional libraries and tools, but this work
> needs to be merged before one commit to that.
We don't need to overthink this. I don't think we're best served by
continuing to post many more rounds of this series. Better to just
get it into a dedicated git repo and iterate via pull requests IMHO.
Golang uses semver, so we could start publishing the generated code in
a Go module as it is today, as long as we pick a v0.XXX.0 version number.
'XXX' would be a packing of QEMU's 3 digit version into the semver 2nd
digit. This lets us indicate this is not considered a stable API, letting
us iterate on further imlp details, while also getting us in the habit of
publishing releases to track schema updates for each new QEMU.
We just need the patch for qapi-gen.py to support plugins for code
generation to make this happen, so we can decouple ongoing development
from QEMU's main git repo & release cycle.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH v4 00/11]
2025-02-17 14:58 ` Daniel P. Berrangé
@ 2025-02-17 16:52 ` Victor Toso
2025-03-07 11:25 ` Victor Toso
0 siblings, 1 reply; 18+ messages in thread
From: Victor Toso @ 2025-02-17 16:52 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, Markus Armbruster, John Snow, Andrea Bolognani
[-- Attachment #1: Type: text/plain, Size: 4729 bytes --]
Hi,
On Mon, Feb 17, 2025 at 02:58:22PM +0000, Daniel P. Berrangé wrote:
> On Fri, Feb 14, 2025 at 09:29:33PM +0100, Victor Toso wrote:
> > Hi again,
> >
> > This patch series intent is to introduce a generator that produces a Go
> > module for Go applications to interact over QMP with QEMU.
> >
> > Previous version (10 Jan 2025)
> > https://lists.gnu.org/archive/html/qemu-devel/2025-01/msg01530.html
> >
> > The generated code was mostly tested using existing examples in the QAPI
> > documentation, 192 instances that might have multiple QMP messages each.
> >
> > You can find the the tests and the generated code in my personal repo,
> > main branch:
> >
> > https://gitlab.com/victortoso/qapi-go
> >
> > If you want to see the generated code from QEMU's master but per patch:
> >
> > https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v4-by-patch
>
> In terms of generated code, my only real feedback is that the
> re-wrapping of docs comments is having undesirable effets
> on formatting
>
> ##
> # @add_client:
> #
> # Allow client connections for VNC, Spice and socket based character
> # devices to be passed in to QEMU via SCM_RIGHTS.
> #
> # If the FD associated with @fdname is not a socket, the command will
> # fail and the FD will be closed.
> #
> # @protocol: protocol name. Valid names are "vnc", "spice",
> # "@dbus-display" or the name of a character device (e.g. from
> # -chardev id=XXXX)
> #
> # @fdname: file descriptor name previously passed via 'getfd' command
> #
> # @skipauth: whether to skip authentication. Only applies to "vnc"
> # and "spice" protocols
> #
> # @tls: whether to perform TLS. Only applies to the "spice" protocol
> #
> # Since: 0.14
> #
> # .. qmp-example::
> #
> # -> { "execute": "add_client", "arguments": { "protocol": "vnc",
> # "fdname": "myclient" } }
> # <- { "return": {} }
> ##
>
>
> is getting turned into
>
>
> // Allow client connections for VNC, Spice and socket based character
> // devices to be passed in to QEMU via SCM_RIGHTS. If the FD
> // associated with @fdname is not a socket, the command will fail and
> // the FD will be closed.
> //
> // Since: 0.14
> //
> // .. qmp-example:: -> { "execute": "add_client", "arguments": {
> // "protocol": "vnc", "fdname": "myclient" }
> // } <- { "return": {} }
>
>
> the '.. qmp-example' bit is what's particularly badly affected.
>
> If we assume that the input QAPI schemas are nicely lined wrapped,
> we could probably just preserve the docs lines as-is with no change
> in wrapping.
>
> That said I'm not sure if we'll need some docs syntax changes to
> make it render nicely - hard to say until the code appears up on
> pkg.go.dev, so can probably worry about that aspect later.
My preference is that the Go code has nicely formatted sections,
like the qmp-example one. The decision to not work on this now
was made together with Markus as he pointed out this formatting
on documentation part is still a work in progress, besides the
fact that it can be done as a follow-up.
Having examples is a nice thing even if the format is not great.
> > ################
> > # Expectations #
> > ################
> >
> > As is, this still is a PoC that works. I'd like to have the generated
> > code included in QEMU's gitlab [0] in order to write library and tools
> > on top. Initial version should be considered alpha. Moving to
> > beta/stable would require functional libraries and tools, but this work
> > needs to be merged before one commit to that.
>
> We don't need to overthink this. I don't think we're best served by
> continuing to post many more rounds of this series. Better to just
> get it into a dedicated git repo and iterate via pull requests IMHO.
Well, I'm happy to hear it. How the repo get created so we can
move the discussion and patches there?
> Golang uses semver, so we could start publishing the generated code in
> a Go module as it is today, as long as we pick a v0.XXX.0 version number.
> 'XXX' would be a packing of QEMU's 3 digit version into the semver 2nd
> digit. This lets us indicate this is not considered a stable API, letting
> us iterate on further imlp details, while also getting us in the habit of
> publishing releases to track schema updates for each new QEMU.
Sure.
> We just need the patch for qapi-gen.py to support plugins for
> code generation to make this happen, so we can decouple ongoing
> development from QEMU's main git repo & release cycle.
Looking forward to it.
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH v4 00/11]
2025-02-17 16:52 ` Victor Toso
@ 2025-03-07 11:25 ` Victor Toso
2025-03-07 11:33 ` Daniel P. Berrangé
0 siblings, 1 reply; 18+ messages in thread
From: Victor Toso @ 2025-03-07 11:25 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, Markus Armbruster, John Snow, Andrea Bolognani
[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]
Hi,
On Mon, Feb 17, 2025 at 05:52:49PM +0100, Victor Toso wrote:
> On Mon, Feb 17, 2025 at 02:58:22PM +0000, Daniel P. Berrangé wrote:
> > On Fri, Feb 14, 2025 at 09:29:33PM +0100, Victor Toso wrote:
> > > ################
> > > # Expectations #
> > > ################
> > >
> > > As is, this still is a PoC that works. I'd like to have the
> > > generated code included in QEMU's gitlab [0] in order to
> > > write library and tools on top. Initial version should be
> > > considered alpha. Moving to beta/stable would require
> > > functional libraries and tools, but this work needs to be
> > > merged before one commit to that.
> >
> > We don't need to overthink this. I don't think we're best
> > served by continuing to post many more rounds of this series.
> > Better to just get it into a dedicated git repo and iterate
> > via pull requests IMHO.
>
> Well, I'm happy to hear it. How the repo get created so we can
> move the discussion and patches there?
Now that pluggable backend [0] was merged, I take that we should
create a repo so I can submit a PR for to Go bindings. Who should
we ping for this?
[0] https://gitlab.com/qemu-project/qemu/-/commit/dde279925c97b
Cheers,
Victor
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v4 00/11]
2025-03-07 11:25 ` Victor Toso
@ 2025-03-07 11:33 ` Daniel P. Berrangé
0 siblings, 0 replies; 18+ messages in thread
From: Daniel P. Berrangé @ 2025-03-07 11:33 UTC (permalink / raw)
To: Victor Toso; +Cc: qemu-devel, Markus Armbruster, John Snow, Andrea Bolognani
On Fri, Mar 07, 2025 at 12:25:32PM +0100, Victor Toso wrote:
> Hi,
>
> On Mon, Feb 17, 2025 at 05:52:49PM +0100, Victor Toso wrote:
> > On Mon, Feb 17, 2025 at 02:58:22PM +0000, Daniel P. Berrangé wrote:
> > > On Fri, Feb 14, 2025 at 09:29:33PM +0100, Victor Toso wrote:
> > > > ################
> > > > # Expectations #
> > > > ################
> > > >
> > > > As is, this still is a PoC that works. I'd like to have the
> > > > generated code included in QEMU's gitlab [0] in order to
> > > > write library and tools on top. Initial version should be
> > > > considered alpha. Moving to beta/stable would require
> > > > functional libraries and tools, but this work needs to be
> > > > merged before one commit to that.
> > >
> > > We don't need to overthink this. I don't think we're best
> > > served by continuing to post many more rounds of this series.
> > > Better to just get it into a dedicated git repo and iterate
> > > via pull requests IMHO.
> >
> > Well, I'm happy to hear it. How the repo get created so we can
> > move the discussion and patches there?
>
> Now that pluggable backend [0] was merged, I take that we should
> create a repo so I can submit a PR for to Go bindings. Who should
> we ping for this?
People who can create new repos are:
https://gitlab.com/groups/qemu-project/-/group_members
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 18+ messages in thread