[PATCH v1 0/9] qapi-go: add generator for Golang interface

[PATCH v1 0/9] qapi-go: add generator for Golang interface

[Victor Toso]

Hi, long time no see!

This patch series intent is to introduce a generator that produces a Go
module for Go applications to interact over QMP with QEMU.

This idea was discussed before, as RFC:
 (RFC v1) https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00226.html
 (RFC v2) https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00226.html

The work got stuck due to changes needed around types that can take JSON
Null as value, but that's now fixed.

I've pushed this series in my gitlab fork:
    https://gitlab.com/victortoso/qemu/-/tree/qapi-golang-v1

I've also generated the qapi-go module over QEMU tags: v7.0.0, v7.1.0,
v7.2.6, v8.0.0 and v8.1.1, see the commits history here:
    https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v1-by-tags

I've also generated the qapi-go module over each commit of this series,
see the commits history here (using previous refered qapi-golang-v1)
    https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v1-by-patch


 * Why this?

My main goal is to allow Go applications that interact with QEMU to have
a native way of doing so.

Ideally, we can merge a new QAPI command, update qapi-go module to allow
Go applications to consume the new command in no time (e.g: if
development of said applications are using latest QEMU)


 * Expectations

From previous discussions, there are things that are still missing. One
simple example is Andrea's annotation suggestion to fix type names. My
proposal is to have a qapi-go module in a formal non-stable version till
some of those tasks get addressed or we declare it a non-problem.

I've created a docs/devel/qapi-golang-code-gen.rst to add information
from the discussions we might have in this series. Suggestions always
welcome.

P.S: Sorry about my broken python :)

Cheers,
Victor

Victor Toso (9):
  qapi: golang: Generate qapi's enum types in Go
  qapi: golang: Generate qapi's alternate types in Go
  qapi: golang: Generate qapi's struct types in Go
  qapi: golang: structs: Address 'null' members
  qapi: golang: Generate qapi's union types in Go
  qapi: golang: Generate qapi's event types in Go
  qapi: golang: Generate qapi's command types in Go
  qapi: golang: Add CommandResult type to Go
  docs: add notes on Golang code generator

 docs/devel/qapi-golang-code-gen.rst |  341 +++++++++
 scripts/qapi/golang.py              | 1047 +++++++++++++++++++++++++++
 scripts/qapi/main.py                |    2 +
 3 files changed, 1390 insertions(+)
 create mode 100644 docs/devel/qapi-golang-code-gen.rst
 create mode 100644 scripts/qapi/golang.py

-- 
2.41.0

[PATCH v1 1/9] qapi: golang: Generate qapi's enum types in Go

[Victor Toso]

This patch handles QAPI enum types and generates its equivalent in Go.

Basically, Enums are being handled as strings in Golang.

1. For each QAPI enum, we will define a string type in Go to be the
   assigned type of this specific enum.

2. Naming: CamelCase will be used in any identifier that we want to
   export [0], which is everything.

[0] https://go.dev/ref/spec#Exported_identifiers

Example:

qapi:
  | { 'enum': 'DisplayProtocol',
  |   'data': [ 'vnc', 'spice' ] }

go:
  | type DisplayProtocol string
  |
  | const (
  |     DisplayProtocolVnc   DisplayProtocol = "vnc"
  |     DisplayProtocolSpice DisplayProtocol = "spice"
  | )

Signed-off-by: Victor Toso <victortoso@redhat.com>
---
 scripts/qapi/golang.py | 140 +++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/main.py   |   2 +
 2 files changed, 142 insertions(+)
 create mode 100644 scripts/qapi/golang.py

diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
new file mode 100644
index 0000000000..87081cdd05
--- /dev/null
+++ b/scripts/qapi/golang.py
@@ -0,0 +1,140 @@
+"""
+Golang QAPI generator
+"""
+# Copyright (c) 2023 Red Hat Inc.
+#
+# Authors:
+#  Victor Toso <victortoso@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+# due QAPISchemaVisitor interface
+# pylint: disable=too-many-arguments
+
+# Just for type hint on self
+from __future__ import annotations
+
+import os
+from typing import List, Optional
+
+from .schema import (
+    QAPISchema,
+    QAPISchemaType,
+    QAPISchemaVisitor,
+    QAPISchemaEnumMember,
+    QAPISchemaFeature,
+    QAPISchemaIfCond,
+    QAPISchemaObjectType,
+    QAPISchemaObjectTypeMember,
+    QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
+
+TEMPLATE_ENUM = '''
+type {name} string
+const (
+{fields}
+)
+'''
+
+
+def gen_golang(schema: QAPISchema,
+               output_dir: str,
+               prefix: str) -> None:
+    vis = QAPISchemaGenGolangVisitor(prefix)
+    schema.visit(vis)
+    vis.write(output_dir)
+
+
+def qapi_to_field_name_enum(name: str) -> str:
+    return name.title().replace("-", "")
+
+
+class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
+
+    def __init__(self, _: str):
+        super().__init__()
+        types = ["enum"]
+        self.target = {name: "" for name in types}
+        self.schema = None
+        self.golang_package_name = "qapi"
+
+    def visit_begin(self, schema):
+        self.schema = schema
+
+        # Every Go file needs to reference its package name
+        for target in self.target:
+            self.target[target] = f"package {self.golang_package_name}\n"
+
+    def visit_end(self):
+        self.schema = None
+
+    def visit_object_type(self: QAPISchemaGenGolangVisitor,
+                          name: str,
+                          info: Optional[QAPISourceInfo],
+                          ifcond: QAPISchemaIfCond,
+                          features: List[QAPISchemaFeature],
+                          base: Optional[QAPISchemaObjectType],
+                          members: List[QAPISchemaObjectTypeMember],
+                          variants: Optional[QAPISchemaVariants]
+                          ) -> None:
+        pass
+
+    def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
+                             name: str,
+                             info: Optional[QAPISourceInfo],
+                             ifcond: QAPISchemaIfCond,
+                             features: List[QAPISchemaFeature],
+                             variants: QAPISchemaVariants
+                             ) -> None:
+        pass
+
+    def visit_enum_type(self: QAPISchemaGenGolangVisitor,
+                        name: str,
+                        info: Optional[QAPISourceInfo],
+                        ifcond: QAPISchemaIfCond,
+                        features: List[QAPISchemaFeature],
+                        members: List[QAPISchemaEnumMember],
+                        prefix: Optional[str]
+                        ) -> None:
+
+        value = qapi_to_field_name_enum(members[0].name)
+        fields = ""
+        for member in members:
+            value = qapi_to_field_name_enum(member.name)
+            fields += f'''\t{name}{value} {name} = "{member.name}"\n'''
+
+        self.target["enum"] += TEMPLATE_ENUM.format(name=name, fields=fields[:-1])
+
+    def visit_array_type(self, name, info, ifcond, element_type):
+        pass
+
+    def visit_command(self,
+                      name: str,
+                      info: Optional[QAPISourceInfo],
+                      ifcond: QAPISchemaIfCond,
+                      features: List[QAPISchemaFeature],
+                      arg_type: Optional[QAPISchemaObjectType],
+                      ret_type: Optional[QAPISchemaType],
+                      gen: bool,
+                      success_response: bool,
+                      boxed: bool,
+                      allow_oob: bool,
+                      allow_preconfig: bool,
+                      coroutine: bool) -> None:
+        pass
+
+    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+        pass
+
+    def write(self, output_dir: str) -> None:
+        for module_name, content in self.target.items():
+            go_module = module_name + "s.go"
+            go_dir = "go"
+            pathname = os.path.join(output_dir, go_dir, go_module)
+            odir = os.path.dirname(pathname)
+            os.makedirs(odir, exist_ok=True)
+
+            with open(pathname, "w", encoding="ascii") as outfile:
+                outfile.write(content)
diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
index 316736b6a2..cdbb3690fd 100644
--- a/scripts/qapi/main.py
+++ b/scripts/qapi/main.py
@@ -15,6 +15,7 @@
 from .common import must_match
 from .error import QAPIError
 from .events import gen_events
+from .golang import gen_golang
 from .introspect import gen_introspect
 from .schema import QAPISchema
 from .types import gen_types
@@ -54,6 +55,7 @@ def generate(schema_file: str,
     gen_events(schema, output_dir, prefix)
     gen_introspect(schema, output_dir, prefix, unmask)
 
+    gen_golang(schema, output_dir, prefix)
 
 def main() -> int:
     """
-- 
2.41.0

Re: [PATCH v1 1/9] qapi: golang: Generate qapi's enum types in Go

[Daniel P. Berrangé]

On Wed, Sep 27, 2023 at 01:25:36PM +0200, Victor Toso wrote:
> This patch handles QAPI enum types and generates its equivalent in Go.
> 
> Basically, Enums are being handled as strings in Golang.
> 
> 1. For each QAPI enum, we will define a string type in Go to be the
>    assigned type of this specific enum.
> 
> 2. Naming: CamelCase will be used in any identifier that we want to
>    export [0], which is everything.
> 
> [0] https://go.dev/ref/spec#Exported_identifiers
> 
> Example:
> 
> qapi:
>   | { 'enum': 'DisplayProtocol',
>   |   'data': [ 'vnc', 'spice' ] }
> 
> go:
>   | type DisplayProtocol string
>   |
>   | const (
>   |     DisplayProtocolVnc   DisplayProtocol = "vnc"
>   |     DisplayProtocolSpice DisplayProtocol = "spice"
>   | )
> 
> Signed-off-by: Victor Toso <victortoso@redhat.com>
> ---
>  scripts/qapi/golang.py | 140 +++++++++++++++++++++++++++++++++++++++++
>  scripts/qapi/main.py   |   2 +
>  2 files changed, 142 insertions(+)
>  create mode 100644 scripts/qapi/golang.py
> 
> diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> new file mode 100644
> index 0000000000..87081cdd05
> --- /dev/null
> +++ b/scripts/qapi/golang.py
> @@ -0,0 +1,140 @@
> +"""
> +Golang QAPI generator
> +"""
> +# Copyright (c) 2023 Red Hat Inc.
> +#
> +# Authors:
> +#  Victor Toso <victortoso@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2.
> +# See the COPYING file in the top-level directory.
> +
> +# due QAPISchemaVisitor interface
> +# pylint: disable=too-many-arguments
> +
> +# Just for type hint on self
> +from __future__ import annotations
> +
> +import os
> +from typing import List, Optional
> +
> +from .schema import (
> +    QAPISchema,
> +    QAPISchemaType,
> +    QAPISchemaVisitor,
> +    QAPISchemaEnumMember,
> +    QAPISchemaFeature,
> +    QAPISchemaIfCond,
> +    QAPISchemaObjectType,
> +    QAPISchemaObjectTypeMember,
> +    QAPISchemaVariants,
> +)
> +from .source import QAPISourceInfo
> +
> +TEMPLATE_ENUM = '''
> +type {name} string
> +const (
> +{fields}
> +)
> +'''
> +
> +
> +def gen_golang(schema: QAPISchema,
> +               output_dir: str,
> +               prefix: str) -> None:
> +    vis = QAPISchemaGenGolangVisitor(prefix)
> +    schema.visit(vis)
> +    vis.write(output_dir)
> +
> +
> +def qapi_to_field_name_enum(name: str) -> str:
> +    return name.title().replace("-", "")
> +
> +
> +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> +
> +    def __init__(self, _: str):
> +        super().__init__()
> +        types = ["enum"]
> +        self.target = {name: "" for name in types}
> +        self.schema = None
> +        self.golang_package_name = "qapi"
> +
> +    def visit_begin(self, schema):
> +        self.schema = schema
> +
> +        # Every Go file needs to reference its package name
> +        for target in self.target:
> +            self.target[target] = f"package {self.golang_package_name}\n"
> +
> +    def visit_end(self):
> +        self.schema = None
> +
> +    def visit_object_type(self: QAPISchemaGenGolangVisitor,
> +                          name: str,
> +                          info: Optional[QAPISourceInfo],
> +                          ifcond: QAPISchemaIfCond,
> +                          features: List[QAPISchemaFeature],
> +                          base: Optional[QAPISchemaObjectType],
> +                          members: List[QAPISchemaObjectTypeMember],
> +                          variants: Optional[QAPISchemaVariants]
> +                          ) -> None:
> +        pass
> +
> +    def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
> +                             name: str,
> +                             info: Optional[QAPISourceInfo],
> +                             ifcond: QAPISchemaIfCond,
> +                             features: List[QAPISchemaFeature],
> +                             variants: QAPISchemaVariants
> +                             ) -> None:
> +        pass
> +
> +    def visit_enum_type(self: QAPISchemaGenGolangVisitor,
> +                        name: str,
> +                        info: Optional[QAPISourceInfo],
> +                        ifcond: QAPISchemaIfCond,
> +                        features: List[QAPISchemaFeature],
> +                        members: List[QAPISchemaEnumMember],
> +                        prefix: Optional[str]
> +                        ) -> None:
> +
> +        value = qapi_to_field_name_enum(members[0].name)
> +        fields = ""
> +        for member in members:
> +            value = qapi_to_field_name_enum(member.name)
> +            fields += f'''\t{name}{value} {name} = "{member.name}"\n'''
> +
> +        self.target["enum"] += TEMPLATE_ENUM.format(name=name, fields=fields[:-1])

Here you are formatting the enums as you visit them, appending to
the output buffer. The resulting enums appear in whatever order we
visited them with, which is pretty arbitrary.

Browsing the generated Go code to understand it, I find myself
wishing that it was emitted in alphabetical order.

This could be done if we worked in two phase. In the visit phase,
we collect the bits of data we need, and then add a format phase
then generates the formatted output, having first sorted by enum
name.

Same thought for the other types/commands.

> +
> +    def visit_array_type(self, name, info, ifcond, element_type):
> +        pass
> +
> +    def visit_command(self,
> +                      name: str,
> +                      info: Optional[QAPISourceInfo],
> +                      ifcond: QAPISchemaIfCond,
> +                      features: List[QAPISchemaFeature],
> +                      arg_type: Optional[QAPISchemaObjectType],
> +                      ret_type: Optional[QAPISchemaType],
> +                      gen: bool,
> +                      success_response: bool,
> +                      boxed: bool,
> +                      allow_oob: bool,
> +                      allow_preconfig: bool,
> +                      coroutine: bool) -> None:
> +        pass
> +
> +    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
> +        pass
> +
> +    def write(self, output_dir: str) -> None:
> +        for module_name, content in self.target.items():
> +            go_module = module_name + "s.go"
> +            go_dir = "go"
> +            pathname = os.path.join(output_dir, go_dir, go_module)
> +            odir = os.path.dirname(pathname)
> +            os.makedirs(odir, exist_ok=True)
> +
> +            with open(pathname, "w", encoding="ascii") as outfile:

IIUC, we defacto consider the .qapi json files to be UTF-8, and thus
in theory we could have non-ascii characters in there somewhere. I'd
suggest we using utf8 encoding when outputting to avoid surprises.

> +                outfile.write(content)


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 1/9] qapi: golang: Generate qapi's enum types in Go

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Wed, Sep 27, 2023 at 01:25:36PM +0200, Victor Toso wrote:
>> This patch handles QAPI enum types and generates its equivalent in Go.
>> 
>> Basically, Enums are being handled as strings in Golang.
>> 
>> 1. For each QAPI enum, we will define a string type in Go to be the
>>    assigned type of this specific enum.
>> 
>> 2. Naming: CamelCase will be used in any identifier that we want to
>>    export [0], which is everything.
>> 
>> [0] https://go.dev/ref/spec#Exported_identifiers
>> 
>> Example:
>> 
>> qapi:
>>   | { 'enum': 'DisplayProtocol',
>>   |   'data': [ 'vnc', 'spice' ] }
>> 
>> go:
>>   | type DisplayProtocol string
>>   |
>>   | const (
>>   |     DisplayProtocolVnc   DisplayProtocol = "vnc"
>>   |     DisplayProtocolSpice DisplayProtocol = "spice"
>>   | )
>> 
>> Signed-off-by: Victor Toso <victortoso@redhat.com>
>> ---
>>  scripts/qapi/golang.py | 140 +++++++++++++++++++++++++++++++++++++++++
>>  scripts/qapi/main.py   |   2 +
>>  2 files changed, 142 insertions(+)
>>  create mode 100644 scripts/qapi/golang.py
>> 
>> diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
>> new file mode 100644
>> index 0000000000..87081cdd05
>> --- /dev/null
>> +++ b/scripts/qapi/golang.py
>> @@ -0,0 +1,140 @@
>> +"""
>> +Golang QAPI generator
>> +"""
>> +# Copyright (c) 2023 Red Hat Inc.
>> +#
>> +# Authors:
>> +#  Victor Toso <victortoso@redhat.com>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2.
>> +# See the COPYING file in the top-level directory.
>> +
>> +# due QAPISchemaVisitor interface
>> +# pylint: disable=too-many-arguments
>> +
>> +# Just for type hint on self
>> +from __future__ import annotations
>> +
>> +import os
>> +from typing import List, Optional
>> +
>> +from .schema import (
>> +    QAPISchema,
>> +    QAPISchemaType,
>> +    QAPISchemaVisitor,
>> +    QAPISchemaEnumMember,
>> +    QAPISchemaFeature,
>> +    QAPISchemaIfCond,
>> +    QAPISchemaObjectType,
>> +    QAPISchemaObjectTypeMember,
>> +    QAPISchemaVariants,
>> +)
>> +from .source import QAPISourceInfo
>> +
>> +TEMPLATE_ENUM = '''
>> +type {name} string
>> +const (
>> +{fields}
>> +)
>> +'''
>> +
>> +
>> +def gen_golang(schema: QAPISchema,
>> +               output_dir: str,
>> +               prefix: str) -> None:
>> +    vis = QAPISchemaGenGolangVisitor(prefix)
>> +    schema.visit(vis)
>> +    vis.write(output_dir)
>> +
>> +
>> +def qapi_to_field_name_enum(name: str) -> str:
>> +    return name.title().replace("-", "")
>> +
>> +
>> +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
>> +
>> +    def __init__(self, _: str):
>> +        super().__init__()
>> +        types = ["enum"]
>> +        self.target = {name: "" for name in types}
>> +        self.schema = None
>> +        self.golang_package_name = "qapi"
>> +
>> +    def visit_begin(self, schema):
>> +        self.schema = schema
>> +
>> +        # Every Go file needs to reference its package name
>> +        for target in self.target:
>> +            self.target[target] = f"package {self.golang_package_name}\n"
>> +
>> +    def visit_end(self):
>> +        self.schema = None
>> +
>> +    def visit_object_type(self: QAPISchemaGenGolangVisitor,
>> +                          name: str,
>> +                          info: Optional[QAPISourceInfo],
>> +                          ifcond: QAPISchemaIfCond,
>> +                          features: List[QAPISchemaFeature],
>> +                          base: Optional[QAPISchemaObjectType],
>> +                          members: List[QAPISchemaObjectTypeMember],
>> +                          variants: Optional[QAPISchemaVariants]
>> +                          ) -> None:
>> +        pass
>> +
>> +    def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
>> +                             name: str,
>> +                             info: Optional[QAPISourceInfo],
>> +                             ifcond: QAPISchemaIfCond,
>> +                             features: List[QAPISchemaFeature],
>> +                             variants: QAPISchemaVariants
>> +                             ) -> None:
>> +        pass
>> +
>> +    def visit_enum_type(self: QAPISchemaGenGolangVisitor,
>> +                        name: str,
>> +                        info: Optional[QAPISourceInfo],
>> +                        ifcond: QAPISchemaIfCond,
>> +                        features: List[QAPISchemaFeature],
>> +                        members: List[QAPISchemaEnumMember],
>> +                        prefix: Optional[str]
>> +                        ) -> None:
>> +
>> +        value = qapi_to_field_name_enum(members[0].name)
>> +        fields = ""
>> +        for member in members:
>> +            value = qapi_to_field_name_enum(member.name)
>> +            fields += f'''\t{name}{value} {name} = "{member.name}"\n'''
>> +
>> +        self.target["enum"] += TEMPLATE_ENUM.format(name=name, fields=fields[:-1])
>
> Here you are formatting the enums as you visit them, appending to
> the output buffer. The resulting enums appear in whatever order we
> visited them with, which is pretty arbitrary.

We visit in source order, not in arbitrary order.

> Browsing the generated Go code to understand it, I find myself
> wishing that it was emitted in alphabetical order.

If that's easier to read in generated Go, then I suspect it would also
be easier to read in the QAPI schema and in generated C.

> This could be done if we worked in two phase. In the visit phase,
> we collect the bits of data we need, and then add a format phase
> then generates the formatted output, having first sorted by enum
> name.
>
> Same thought for the other types/commands.
>
>> +
>> +    def visit_array_type(self, name, info, ifcond, element_type):
>> +        pass
>> +
>> +    def visit_command(self,
>> +                      name: str,
>> +                      info: Optional[QAPISourceInfo],
>> +                      ifcond: QAPISchemaIfCond,
>> +                      features: List[QAPISchemaFeature],
>> +                      arg_type: Optional[QAPISchemaObjectType],
>> +                      ret_type: Optional[QAPISchemaType],
>> +                      gen: bool,
>> +                      success_response: bool,
>> +                      boxed: bool,
>> +                      allow_oob: bool,
>> +                      allow_preconfig: bool,
>> +                      coroutine: bool) -> None:
>> +        pass
>> +
>> +    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
>> +        pass
>> +
>> +    def write(self, output_dir: str) -> None:
>> +        for module_name, content in self.target.items():
>> +            go_module = module_name + "s.go"
>> +            go_dir = "go"
>> +            pathname = os.path.join(output_dir, go_dir, go_module)
>> +            odir = os.path.dirname(pathname)
>> +            os.makedirs(odir, exist_ok=True)
>> +
>> +            with open(pathname, "w", encoding="ascii") as outfile:
>
> IIUC, we defacto consider the .qapi json files to be UTF-8, and thus
> in theory we could have non-ascii characters in there somewhere. I'd
> suggest we using utf8 encoding when outputting to avoid surprises.

Seconded.  QAPIGen.write() already uses encoding='utf-8' for writing
generated files.

>> +                outfile.write(content)
>
>
> With regards,
> Daniel

Re: [PATCH v1 1/9] qapi: golang: Generate qapi's enum types in Go

[Daniel P. Berrangé]

On Thu, Sep 28, 2023 at 04:20:55PM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Wed, Sep 27, 2023 at 01:25:36PM +0200, Victor Toso wrote:
> >> This patch handles QAPI enum types and generates its equivalent in Go.
> >> 
> >> Basically, Enums are being handled as strings in Golang.
> >> 
> >> 1. For each QAPI enum, we will define a string type in Go to be the
> >>    assigned type of this specific enum.
> >> 
> >> 2. Naming: CamelCase will be used in any identifier that we want to
> >>    export [0], which is everything.
> >> 
> >> [0] https://go.dev/ref/spec#Exported_identifiers
> >> 
> >> Example:
> >> 
> >> qapi:
> >>   | { 'enum': 'DisplayProtocol',
> >>   |   'data': [ 'vnc', 'spice' ] }
> >> 
> >> go:
> >>   | type DisplayProtocol string
> >>   |
> >>   | const (
> >>   |     DisplayProtocolVnc   DisplayProtocol = "vnc"
> >>   |     DisplayProtocolSpice DisplayProtocol = "spice"
> >>   | )
> >> 
> >> Signed-off-by: Victor Toso <victortoso@redhat.com>
> >> ---
> >>  scripts/qapi/golang.py | 140 +++++++++++++++++++++++++++++++++++++++++
> >>  scripts/qapi/main.py   |   2 +
> >>  2 files changed, 142 insertions(+)
> >>  create mode 100644 scripts/qapi/golang.py
> >> 
> >> diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> >> new file mode 100644
> >> index 0000000000..87081cdd05
> >> --- /dev/null
> >> +++ b/scripts/qapi/golang.py
> >> @@ -0,0 +1,140 @@
> >> +"""
> >> +Golang QAPI generator
> >> +"""
> >> +# Copyright (c) 2023 Red Hat Inc.
> >> +#
> >> +# Authors:
> >> +#  Victor Toso <victortoso@redhat.com>
> >> +#
> >> +# This work is licensed under the terms of the GNU GPL, version 2.
> >> +# See the COPYING file in the top-level directory.
> >> +
> >> +# due QAPISchemaVisitor interface
> >> +# pylint: disable=too-many-arguments
> >> +
> >> +# Just for type hint on self
> >> +from __future__ import annotations
> >> +
> >> +import os
> >> +from typing import List, Optional
> >> +
> >> +from .schema import (
> >> +    QAPISchema,
> >> +    QAPISchemaType,
> >> +    QAPISchemaVisitor,
> >> +    QAPISchemaEnumMember,
> >> +    QAPISchemaFeature,
> >> +    QAPISchemaIfCond,
> >> +    QAPISchemaObjectType,
> >> +    QAPISchemaObjectTypeMember,
> >> +    QAPISchemaVariants,
> >> +)
> >> +from .source import QAPISourceInfo
> >> +
> >> +TEMPLATE_ENUM = '''
> >> +type {name} string
> >> +const (
> >> +{fields}
> >> +)
> >> +'''
> >> +
> >> +
> >> +def gen_golang(schema: QAPISchema,
> >> +               output_dir: str,
> >> +               prefix: str) -> None:
> >> +    vis = QAPISchemaGenGolangVisitor(prefix)
> >> +    schema.visit(vis)
> >> +    vis.write(output_dir)
> >> +
> >> +
> >> +def qapi_to_field_name_enum(name: str) -> str:
> >> +    return name.title().replace("-", "")
> >> +
> >> +
> >> +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> >> +
> >> +    def __init__(self, _: str):
> >> +        super().__init__()
> >> +        types = ["enum"]
> >> +        self.target = {name: "" for name in types}
> >> +        self.schema = None
> >> +        self.golang_package_name = "qapi"
> >> +
> >> +    def visit_begin(self, schema):
> >> +        self.schema = schema
> >> +
> >> +        # Every Go file needs to reference its package name
> >> +        for target in self.target:
> >> +            self.target[target] = f"package {self.golang_package_name}\n"
> >> +
> >> +    def visit_end(self):
> >> +        self.schema = None
> >> +
> >> +    def visit_object_type(self: QAPISchemaGenGolangVisitor,
> >> +                          name: str,
> >> +                          info: Optional[QAPISourceInfo],
> >> +                          ifcond: QAPISchemaIfCond,
> >> +                          features: List[QAPISchemaFeature],
> >> +                          base: Optional[QAPISchemaObjectType],
> >> +                          members: List[QAPISchemaObjectTypeMember],
> >> +                          variants: Optional[QAPISchemaVariants]
> >> +                          ) -> None:
> >> +        pass
> >> +
> >> +    def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
> >> +                             name: str,
> >> +                             info: Optional[QAPISourceInfo],
> >> +                             ifcond: QAPISchemaIfCond,
> >> +                             features: List[QAPISchemaFeature],
> >> +                             variants: QAPISchemaVariants
> >> +                             ) -> None:
> >> +        pass
> >> +
> >> +    def visit_enum_type(self: QAPISchemaGenGolangVisitor,
> >> +                        name: str,
> >> +                        info: Optional[QAPISourceInfo],
> >> +                        ifcond: QAPISchemaIfCond,
> >> +                        features: List[QAPISchemaFeature],
> >> +                        members: List[QAPISchemaEnumMember],
> >> +                        prefix: Optional[str]
> >> +                        ) -> None:
> >> +
> >> +        value = qapi_to_field_name_enum(members[0].name)
> >> +        fields = ""
> >> +        for member in members:
> >> +            value = qapi_to_field_name_enum(member.name)
> >> +            fields += f'''\t{name}{value} {name} = "{member.name}"\n'''
> >> +
> >> +        self.target["enum"] += TEMPLATE_ENUM.format(name=name, fields=fields[:-1])
> >
> > Here you are formatting the enums as you visit them, appending to
> > the output buffer. The resulting enums appear in whatever order we
> > visited them with, which is pretty arbitrary.
> 
> We visit in source order, not in arbitrary order.

I meant arbitrary in the sense that us developers just add new
QAPI types pretty much anywhere we feel like it in the .qapi
files.

> 
> > Browsing the generated Go code to understand it, I find myself
> > wishing that it was emitted in alphabetical order.
> 
> If that's easier to read in generated Go, then I suspect it would also
> be easier to read in the QAPI schema and in generated C.

Yes, although C has some ordering constraints on things being
declared, so it would be a bit harder to do this in C.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 1/9] qapi: golang: Generate qapi's enum types in Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 7698 bytes --]

Hi,

On Thu, Sep 28, 2023 at 02:52:08PM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 27, 2023 at 01:25:36PM +0200, Victor Toso wrote:
> > This patch handles QAPI enum types and generates its equivalent in Go.
> > 
> > Basically, Enums are being handled as strings in Golang.
> > 
> > 1. For each QAPI enum, we will define a string type in Go to be the
> >    assigned type of this specific enum.
> > 
> > 2. Naming: CamelCase will be used in any identifier that we want to
> >    export [0], which is everything.
> > 
> > [0] https://go.dev/ref/spec#Exported_identifiers
> > 
> > Example:
> > 
> > qapi:
> >   | { 'enum': 'DisplayProtocol',
> >   |   'data': [ 'vnc', 'spice' ] }
> > 
> > go:
> >   | type DisplayProtocol string
> >   |
> >   | const (
> >   |     DisplayProtocolVnc   DisplayProtocol = "vnc"
> >   |     DisplayProtocolSpice DisplayProtocol = "spice"
> >   | )
> > 
> > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > ---
> >  scripts/qapi/golang.py | 140 +++++++++++++++++++++++++++++++++++++++++
> >  scripts/qapi/main.py   |   2 +
> >  2 files changed, 142 insertions(+)
> >  create mode 100644 scripts/qapi/golang.py
> > 
> > diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> > new file mode 100644
> > index 0000000000..87081cdd05
> > --- /dev/null
> > +++ b/scripts/qapi/golang.py
> > @@ -0,0 +1,140 @@
> > +"""
> > +Golang QAPI generator
> > +"""
> > +# Copyright (c) 2023 Red Hat Inc.
> > +#
> > +# Authors:
> > +#  Victor Toso <victortoso@redhat.com>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2.
> > +# See the COPYING file in the top-level directory.
> > +
> > +# due QAPISchemaVisitor interface
> > +# pylint: disable=too-many-arguments
> > +
> > +# Just for type hint on self
> > +from __future__ import annotations
> > +
> > +import os
> > +from typing import List, Optional
> > +
> > +from .schema import (
> > +    QAPISchema,
> > +    QAPISchemaType,
> > +    QAPISchemaVisitor,
> > +    QAPISchemaEnumMember,
> > +    QAPISchemaFeature,
> > +    QAPISchemaIfCond,
> > +    QAPISchemaObjectType,
> > +    QAPISchemaObjectTypeMember,
> > +    QAPISchemaVariants,
> > +)
> > +from .source import QAPISourceInfo
> > +
> > +TEMPLATE_ENUM = '''
> > +type {name} string
> > +const (
> > +{fields}
> > +)
> > +'''
> > +
> > +
> > +def gen_golang(schema: QAPISchema,
> > +               output_dir: str,
> > +               prefix: str) -> None:
> > +    vis = QAPISchemaGenGolangVisitor(prefix)
> > +    schema.visit(vis)
> > +    vis.write(output_dir)
> > +
> > +
> > +def qapi_to_field_name_enum(name: str) -> str:
> > +    return name.title().replace("-", "")
> > +
> > +
> > +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> > +
> > +    def __init__(self, _: str):
> > +        super().__init__()
> > +        types = ["enum"]
> > +        self.target = {name: "" for name in types}
> > +        self.schema = None
> > +        self.golang_package_name = "qapi"
> > +
> > +    def visit_begin(self, schema):
> > +        self.schema = schema
> > +
> > +        # Every Go file needs to reference its package name
> > +        for target in self.target:
> > +            self.target[target] = f"package {self.golang_package_name}\n"
> > +
> > +    def visit_end(self):
> > +        self.schema = None
> > +
> > +    def visit_object_type(self: QAPISchemaGenGolangVisitor,
> > +                          name: str,
> > +                          info: Optional[QAPISourceInfo],
> > +                          ifcond: QAPISchemaIfCond,
> > +                          features: List[QAPISchemaFeature],
> > +                          base: Optional[QAPISchemaObjectType],
> > +                          members: List[QAPISchemaObjectTypeMember],
> > +                          variants: Optional[QAPISchemaVariants]
> > +                          ) -> None:
> > +        pass
> > +
> > +    def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
> > +                             name: str,
> > +                             info: Optional[QAPISourceInfo],
> > +                             ifcond: QAPISchemaIfCond,
> > +                             features: List[QAPISchemaFeature],
> > +                             variants: QAPISchemaVariants
> > +                             ) -> None:
> > +        pass
> > +
> > +    def visit_enum_type(self: QAPISchemaGenGolangVisitor,
> > +                        name: str,
> > +                        info: Optional[QAPISourceInfo],
> > +                        ifcond: QAPISchemaIfCond,
> > +                        features: List[QAPISchemaFeature],
> > +                        members: List[QAPISchemaEnumMember],
> > +                        prefix: Optional[str]
> > +                        ) -> None:
> > +
> > +        value = qapi_to_field_name_enum(members[0].name)
> > +        fields = ""
> > +        for member in members:
> > +            value = qapi_to_field_name_enum(member.name)
> > +            fields += f'''\t{name}{value} {name} = "{member.name}"\n'''
> > +
> > +        self.target["enum"] += TEMPLATE_ENUM.format(name=name, fields=fields[:-1])
> 
> Here you are formatting the enums as you visit them, appending to
> the output buffer. The resulting enums appear in whatever order we
> visited them with, which is pretty arbitrary.
> 
> Browsing the generated Go code to understand it, I find myself
> wishing that it was emitted in alphabetical order.
> 
> This could be done if we worked in two phase. In the visit phase,
> we collect the bits of data we need, and then add a format phase
> then generates the formatted output, having first sorted by enum
> name.
> 
> Same thought for the other types/commands.

I cared for sorted in some places [0] but not all of them indeed.
I'll include your request/suggestion in the next version.

[0] https://gitlab.com/victortoso/qemu/-/blob/qapi-golang-v1/scripts/qapi/golang.py?ref_type=heads#L804

> 
> > +
> > +    def visit_array_type(self, name, info, ifcond, element_type):
> > +        pass
> > +
> > +    def visit_command(self,
> > +                      name: str,
> > +                      info: Optional[QAPISourceInfo],
> > +                      ifcond: QAPISchemaIfCond,
> > +                      features: List[QAPISchemaFeature],
> > +                      arg_type: Optional[QAPISchemaObjectType],
> > +                      ret_type: Optional[QAPISchemaType],
> > +                      gen: bool,
> > +                      success_response: bool,
> > +                      boxed: bool,
> > +                      allow_oob: bool,
> > +                      allow_preconfig: bool,
> > +                      coroutine: bool) -> None:
> > +        pass
> > +
> > +    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
> > +        pass
> > +
> > +    def write(self, output_dir: str) -> None:
> > +        for module_name, content in self.target.items():
> > +            go_module = module_name + "s.go"
> > +            go_dir = "go"
> > +            pathname = os.path.join(output_dir, go_dir, go_module)
> > +            odir = os.path.dirname(pathname)
> > +            os.makedirs(odir, exist_ok=True)
> > +
> > +            with open(pathname, "w", encoding="ascii") as outfile:
> 
> IIUC, we defacto consider the .qapi json files to be UTF-8, and thus
> in theory we could have non-ascii characters in there somewhere. I'd
> suggest we using utf8 encoding when outputting to avoid surprises.

Sure thing.

Cheers,
Victor

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1 1/9] qapi: golang: Generate qapi's enum types in Go

[John Snow]

On Wed, Sep 27, 2023 at 7:25 AM Victor Toso <victortoso@redhat.com> wrote:
>
> This patch handles QAPI enum types and generates its equivalent in Go.
>
> Basically, Enums are being handled as strings in Golang.
>
> 1. For each QAPI enum, we will define a string type in Go to be the
>    assigned type of this specific enum.
>
> 2. Naming: CamelCase will be used in any identifier that we want to
>    export [0], which is everything.
>
> [0] https://go.dev/ref/spec#Exported_identifiers
>
> Example:
>
> qapi:
>   | { 'enum': 'DisplayProtocol',
>   |   'data': [ 'vnc', 'spice' ] }
>
> go:
>   | type DisplayProtocol string
>   |
>   | const (
>   |     DisplayProtocolVnc   DisplayProtocol = "vnc"
>   |     DisplayProtocolSpice DisplayProtocol = "spice"
>   | )
>
> Signed-off-by: Victor Toso <victortoso@redhat.com>
> ---
>  scripts/qapi/golang.py | 140 +++++++++++++++++++++++++++++++++++++++++
>  scripts/qapi/main.py   |   2 +
>  2 files changed, 142 insertions(+)
>  create mode 100644 scripts/qapi/golang.py
>
> diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> new file mode 100644
> index 0000000000..87081cdd05
> --- /dev/null
> +++ b/scripts/qapi/golang.py
> @@ -0,0 +1,140 @@
> +"""
> +Golang QAPI generator
> +"""
> +# Copyright (c) 2023 Red Hat Inc.
> +#
> +# Authors:
> +#  Victor Toso <victortoso@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2.
> +# See the COPYING file in the top-level directory.
> +
> +# due QAPISchemaVisitor interface
> +# pylint: disable=too-many-arguments
> +
> +# Just for type hint on self
> +from __future__ import annotations
> +
> +import os
> +from typing import List, Optional
> +
> +from .schema import (
> +    QAPISchema,
> +    QAPISchemaType,
> +    QAPISchemaVisitor,
> +    QAPISchemaEnumMember,
> +    QAPISchemaFeature,
> +    QAPISchemaIfCond,
> +    QAPISchemaObjectType,
> +    QAPISchemaObjectTypeMember,
> +    QAPISchemaVariants,
> +)
> +from .source import QAPISourceInfo
> +
> +TEMPLATE_ENUM = '''
> +type {name} string
> +const (
> +{fields}
> +)
> +'''
> +
> +
> +def gen_golang(schema: QAPISchema,
> +               output_dir: str,
> +               prefix: str) -> None:
> +    vis = QAPISchemaGenGolangVisitor(prefix)
> +    schema.visit(vis)
> +    vis.write(output_dir)
> +
> +
> +def qapi_to_field_name_enum(name: str) -> str:
> +    return name.title().replace("-", "")
> +
> +
> +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> +
> +    def __init__(self, _: str):
> +        super().__init__()
> +        types = ["enum"]
> +        self.target = {name: "" for name in types}
> +        self.schema = None
> +        self.golang_package_name = "qapi"
> +
> +    def visit_begin(self, schema):
> +        self.schema = schema
> +
> +        # Every Go file needs to reference its package name
> +        for target in self.target:
> +            self.target[target] = f"package {self.golang_package_name}\n"
> +
> +    def visit_end(self):
> +        self.schema = None
> +
> +    def visit_object_type(self: QAPISchemaGenGolangVisitor,
> +                          name: str,
> +                          info: Optional[QAPISourceInfo],
> +                          ifcond: QAPISchemaIfCond,
> +                          features: List[QAPISchemaFeature],
> +                          base: Optional[QAPISchemaObjectType],
> +                          members: List[QAPISchemaObjectTypeMember],
> +                          variants: Optional[QAPISchemaVariants]
> +                          ) -> None:
> +        pass
> +
> +    def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
> +                             name: str,
> +                             info: Optional[QAPISourceInfo],
> +                             ifcond: QAPISchemaIfCond,
> +                             features: List[QAPISchemaFeature],
> +                             variants: QAPISchemaVariants
> +                             ) -> None:
> +        pass
> +
> +    def visit_enum_type(self: QAPISchemaGenGolangVisitor,
> +                        name: str,
> +                        info: Optional[QAPISourceInfo],
> +                        ifcond: QAPISchemaIfCond,
> +                        features: List[QAPISchemaFeature],
> +                        members: List[QAPISchemaEnumMember],
> +                        prefix: Optional[str]
> +                        ) -> None:
> +
> +        value = qapi_to_field_name_enum(members[0].name)

Unsure if this was addressed on the mailing list yet, but in our call
we discussed how this call was vestigial and was causing the QAPI
tests to fail. Actually, I can't quite run "make check-qapi-schema"
and see the failure, I'm seeing it when I run "make check" and I'm not
sure how to find the failure more efficiently/quickly:

jsnow@scv ~/s/q/build (review)> make
[1/60] Generating subprojects/dtc/version_gen.h with a custom command
[2/60] Generating qemu-version.h with a custom command (wrapped by
meson to capture output)
[3/44] Generating tests/Test QAPI files with a custom command
FAILED: tests/qapi-builtin-types.c tests/qapi-builtin-types.h
tests/qapi-builtin-visit.c tests/qapi-builtin-visit.h
tests/test-qapi-commands-sub-sub-module.c
tests/test-qapi-commands-sub-sub-module.h tests/test-qapi-commands.c
tests/test-qapi-commands.h tests/test-qapi-emit-events.c
tests/test-qapi-emit-events.h tests/test-qapi-events-sub-sub-module.c
tests/test-qapi-events-sub-sub-module.h tests/test-qapi-events.c
tests/test-qapi-events.h tests/test-qapi-init-commands.c
tests/test-qapi-init-commands.h tests/test-qapi-introspect.c
tests/test-qapi-introspect.h tests/test-qapi-types-sub-sub-module.c
tests/test-qapi-types-sub-sub-module.h tests/test-qapi-types.c
tests/test-qapi-types.h tests/test-qapi-visit-sub-sub-module.c
tests/test-qapi-visit-sub-sub-module.h tests/test-qapi-visit.c
tests/test-qapi-visit.h
/home/jsnow/src/qemu/build/pyvenv/bin/python3
/home/jsnow/src/qemu/scripts/qapi-gen.py -o
/home/jsnow/src/qemu/build/tests -b -p test-
../tests/qapi-schema/qapi-schema-test.json --suppress-tracing
Traceback (most recent call last):
  File "/home/jsnow/src/qemu/scripts/qapi-gen.py", line 19, in <module>
    sys.exit(main.main())
             ^^^^^^^^^^^
  File "/home/jsnow/src/qemu/scripts/qapi/main.py", line 96, in main
    generate(args.schema,
  File "/home/jsnow/src/qemu/scripts/qapi/main.py", line 58, in generate
    gen_golang(schema, output_dir, prefix)
  File "/home/jsnow/src/qemu/scripts/qapi/golang.py", line 46, in gen_golang
    schema.visit(vis)
  File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 1227, in visit
    mod.visit(visitor)
  File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 209, in visit
    entity.visit(visitor)
  File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 346, in visit
    visitor.visit_enum_type(
  File "/home/jsnow/src/qemu/scripts/qapi/golang.py", line 102, in
visit_enum_type
    value = qapi_to_field_name_enum(members[0].name)
                                    ~~~~~~~^^^
IndexError: list index out of range
ninja: build stopped: subcommand failed.
make: *** [Makefile:162: run-ninja] Error 1


For the rest of my review, I commented this line out and continued on.

> +        fields = ""
> +        for member in members:
> +            value = qapi_to_field_name_enum(member.name)
> +            fields += f'''\t{name}{value} {name} = "{member.name}"\n'''
> +
> +        self.target["enum"] += TEMPLATE_ENUM.format(name=name, fields=fields[:-1])
> +
> +    def visit_array_type(self, name, info, ifcond, element_type):
> +        pass
> +
> +    def visit_command(self,
> +                      name: str,
> +                      info: Optional[QAPISourceInfo],
> +                      ifcond: QAPISchemaIfCond,
> +                      features: List[QAPISchemaFeature],
> +                      arg_type: Optional[QAPISchemaObjectType],
> +                      ret_type: Optional[QAPISchemaType],
> +                      gen: bool,
> +                      success_response: bool,
> +                      boxed: bool,
> +                      allow_oob: bool,
> +                      allow_preconfig: bool,
> +                      coroutine: bool) -> None:
> +        pass
> +
> +    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
> +        pass
> +
> +    def write(self, output_dir: str) -> None:
> +        for module_name, content in self.target.items():
> +            go_module = module_name + "s.go"
> +            go_dir = "go"
> +            pathname = os.path.join(output_dir, go_dir, go_module)
> +            odir = os.path.dirname(pathname)
> +            os.makedirs(odir, exist_ok=True)
> +
> +            with open(pathname, "w", encoding="ascii") as outfile:
> +                outfile.write(content)
> diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
> index 316736b6a2..cdbb3690fd 100644
> --- a/scripts/qapi/main.py
> +++ b/scripts/qapi/main.py
> @@ -15,6 +15,7 @@
>  from .common import must_match
>  from .error import QAPIError
>  from .events import gen_events
> +from .golang import gen_golang
>  from .introspect import gen_introspect
>  from .schema import QAPISchema
>  from .types import gen_types
> @@ -54,6 +55,7 @@ def generate(schema_file: str,
>      gen_events(schema, output_dir, prefix)
>      gen_introspect(schema, output_dir, prefix, unmask)
>
> +    gen_golang(schema, output_dir, prefix)
>
>  def main() -> int:
>      """
> --
> 2.41.0
>

Re: [PATCH v1 1/9] qapi: golang: Generate qapi's enum types in Go

[John Snow]

On Mon, Oct 2, 2023 at 3:07 PM John Snow <jsnow@redhat.com> wrote:
>
> On Wed, Sep 27, 2023 at 7:25 AM Victor Toso <victortoso@redhat.com> wrote:
> >
> > This patch handles QAPI enum types and generates its equivalent in Go.
> >
> > Basically, Enums are being handled as strings in Golang.
> >
> > 1. For each QAPI enum, we will define a string type in Go to be the
> >    assigned type of this specific enum.
> >
> > 2. Naming: CamelCase will be used in any identifier that we want to
> >    export [0], which is everything.
> >
> > [0] https://go.dev/ref/spec#Exported_identifiers
> >
> > Example:
> >
> > qapi:
> >   | { 'enum': 'DisplayProtocol',
> >   |   'data': [ 'vnc', 'spice' ] }
> >
> > go:
> >   | type DisplayProtocol string
> >   |
> >   | const (
> >   |     DisplayProtocolVnc   DisplayProtocol = "vnc"
> >   |     DisplayProtocolSpice DisplayProtocol = "spice"
> >   | )
> >
> > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > ---
> >  scripts/qapi/golang.py | 140 +++++++++++++++++++++++++++++++++++++++++
> >  scripts/qapi/main.py   |   2 +
> >  2 files changed, 142 insertions(+)
> >  create mode 100644 scripts/qapi/golang.py
> >
> > diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> > new file mode 100644
> > index 0000000000..87081cdd05
> > --- /dev/null
> > +++ b/scripts/qapi/golang.py
> > @@ -0,0 +1,140 @@
> > +"""
> > +Golang QAPI generator
> > +"""
> > +# Copyright (c) 2023 Red Hat Inc.
> > +#
> > +# Authors:
> > +#  Victor Toso <victortoso@redhat.com>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2.
> > +# See the COPYING file in the top-level directory.
> > +
> > +# due QAPISchemaVisitor interface
> > +# pylint: disable=too-many-arguments

"due to" - also, you could more selectively disable this warning by
putting this comment in the body of the QAPISchemaVisitor class which
would make your exemption from the linter more locally obvious.

> > +
> > +# Just for type hint on self
> > +from __future__ import annotations

Oh, you know - it's been so long since I worked on QAPI I didn't
realize we had access to this now. That's awesome!

(It was introduced in Python 3.7+)

> > +
> > +import os
> > +from typing import List, Optional
> > +
> > +from .schema import (
> > +    QAPISchema,
> > +    QAPISchemaType,
> > +    QAPISchemaVisitor,
> > +    QAPISchemaEnumMember,
> > +    QAPISchemaFeature,
> > +    QAPISchemaIfCond,
> > +    QAPISchemaObjectType,
> > +    QAPISchemaObjectTypeMember,
> > +    QAPISchemaVariants,
> > +)
> > +from .source import QAPISourceInfo
> > +

Try running isort here:

> cd ~/src/qemu/scripts
> isort -c qapi/golang.py

ERROR: /home/jsnow/src/qemu/scripts/qapi/golang.py Imports are
incorrectly sorted and/or formatted.

you can have it fix the import order for you:

> isort qapi/golang.py

It's very pedantic stuff, but luckily there's a tool to just handle it for you.

> > +TEMPLATE_ENUM = '''
> > +type {name} string
> > +const (
> > +{fields}
> > +)
> > +'''
> > +
> > +
> > +def gen_golang(schema: QAPISchema,
> > +               output_dir: str,
> > +               prefix: str) -> None:
> > +    vis = QAPISchemaGenGolangVisitor(prefix)
> > +    schema.visit(vis)
> > +    vis.write(output_dir)
> > +
> > +
> > +def qapi_to_field_name_enum(name: str) -> str:
> > +    return name.title().replace("-", "")
> > +
> > +
> > +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> > +
> > +    def __init__(self, _: str):
> > +        super().__init__()
> > +        types = ["enum"]
> > +        self.target = {name: "" for name in types}

you *could* say:

types = ("enum",)
self.target = dict.fromkeys(types, "")

1. We don't need a list because we won't be modifying it, so a tuple suffices
2. There's an idiom for doing what you want that reads a little better
3. None of it really matters, though.

Also keep in mind you don't *need* to initialize a dict in this way,
you can just arbitrarily assign into it whenever you'd like.

sellf.target['enum'] = foo

I don't know if that makes things easier or not with however the
subsequent patches are written.

> > +        self.schema = None
> > +        self.golang_package_name = "qapi"
> > +
> > +    def visit_begin(self, schema):
> > +        self.schema = schema
> > +
> > +        # Every Go file needs to reference its package name
> > +        for target in self.target:
> > +            self.target[target] = f"package {self.golang_package_name}\n"
> > +
> > +    def visit_end(self):
> > +        self.schema = None
> > +
> > +    def visit_object_type(self: QAPISchemaGenGolangVisitor,
> > +                          name: str,
> > +                          info: Optional[QAPISourceInfo],
> > +                          ifcond: QAPISchemaIfCond,
> > +                          features: List[QAPISchemaFeature],
> > +                          base: Optional[QAPISchemaObjectType],
> > +                          members: List[QAPISchemaObjectTypeMember],
> > +                          variants: Optional[QAPISchemaVariants]
> > +                          ) -> None:
> > +        pass
> > +
> > +    def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
> > +                             name: str,
> > +                             info: Optional[QAPISourceInfo],
> > +                             ifcond: QAPISchemaIfCond,
> > +                             features: List[QAPISchemaFeature],
> > +                             variants: QAPISchemaVariants
> > +                             ) -> None:
> > +        pass
> > +
> > +    def visit_enum_type(self: QAPISchemaGenGolangVisitor,

Was there a problem when you omitted the type for 'self'? Usually that
can be inferred. As of this patch, at least, I think this can be
safely dropped. (Maybe it becomes important later.)

> > +                        name: str,
> > +                        info: Optional[QAPISourceInfo],
> > +                        ifcond: QAPISchemaIfCond,
> > +                        features: List[QAPISchemaFeature],
> > +                        members: List[QAPISchemaEnumMember],
> > +                        prefix: Optional[str]
> > +                        ) -> None:
> > +
> > +        value = qapi_to_field_name_enum(members[0].name)
>
> Unsure if this was addressed on the mailing list yet, but in our call
> we discussed how this call was vestigial and was causing the QAPI
> tests to fail. Actually, I can't quite run "make check-qapi-schema"
> and see the failure, I'm seeing it when I run "make check" and I'm not
> sure how to find the failure more efficiently/quickly:
>
> jsnow@scv ~/s/q/build (review)> make
> [1/60] Generating subprojects/dtc/version_gen.h with a custom command
> [2/60] Generating qemu-version.h with a custom command (wrapped by
> meson to capture output)
> [3/44] Generating tests/Test QAPI files with a custom command
> FAILED: tests/qapi-builtin-types.c tests/qapi-builtin-types.h
> tests/qapi-builtin-visit.c tests/qapi-builtin-visit.h
> tests/test-qapi-commands-sub-sub-module.c
> tests/test-qapi-commands-sub-sub-module.h tests/test-qapi-commands.c
> tests/test-qapi-commands.h tests/test-qapi-emit-events.c
> tests/test-qapi-emit-events.h tests/test-qapi-events-sub-sub-module.c
> tests/test-qapi-events-sub-sub-module.h tests/test-qapi-events.c
> tests/test-qapi-events.h tests/test-qapi-init-commands.c
> tests/test-qapi-init-commands.h tests/test-qapi-introspect.c
> tests/test-qapi-introspect.h tests/test-qapi-types-sub-sub-module.c
> tests/test-qapi-types-sub-sub-module.h tests/test-qapi-types.c
> tests/test-qapi-types.h tests/test-qapi-visit-sub-sub-module.c
> tests/test-qapi-visit-sub-sub-module.h tests/test-qapi-visit.c
> tests/test-qapi-visit.h
> /home/jsnow/src/qemu/build/pyvenv/bin/python3
> /home/jsnow/src/qemu/scripts/qapi-gen.py -o
> /home/jsnow/src/qemu/build/tests -b -p test-
> ../tests/qapi-schema/qapi-schema-test.json --suppress-tracing
> Traceback (most recent call last):
>   File "/home/jsnow/src/qemu/scripts/qapi-gen.py", line 19, in <module>
>     sys.exit(main.main())
>              ^^^^^^^^^^^
>   File "/home/jsnow/src/qemu/scripts/qapi/main.py", line 96, in main
>     generate(args.schema,
>   File "/home/jsnow/src/qemu/scripts/qapi/main.py", line 58, in generate
>     gen_golang(schema, output_dir, prefix)
>   File "/home/jsnow/src/qemu/scripts/qapi/golang.py", line 46, in gen_golang
>     schema.visit(vis)
>   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 1227, in visit
>     mod.visit(visitor)
>   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 209, in visit
>     entity.visit(visitor)
>   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 346, in visit
>     visitor.visit_enum_type(
>   File "/home/jsnow/src/qemu/scripts/qapi/golang.py", line 102, in
> visit_enum_type
>     value = qapi_to_field_name_enum(members[0].name)
>                                     ~~~~~~~^^^
> IndexError: list index out of range
> ninja: build stopped: subcommand failed.
> make: *** [Makefile:162: run-ninja] Error 1
>
>
> For the rest of my review, I commented this line out and continued on.
>
> > +        fields = ""
> > +        for member in members:
> > +            value = qapi_to_field_name_enum(member.name)
> > +            fields += f'''\t{name}{value} {name} = "{member.name}"\n'''
> > +
> > +        self.target["enum"] += TEMPLATE_ENUM.format(name=name, fields=fields[:-1])

This line is a little too long. (sorry)

try:

cd ~/src/qemu/scripts
flake8 qapi/

jsnow@scv ~/s/q/scripts (review)> flake8 qapi/
qapi/main.py:60:1: E302 expected 2 blank lines, found 1
qapi/golang.py:106:80: E501 line too long (82 > 79 characters)

> > +
> > +    def visit_array_type(self, name, info, ifcond, element_type):
> > +        pass
> > +
> > +    def visit_command(self,
> > +                      name: str,
> > +                      info: Optional[QAPISourceInfo],
> > +                      ifcond: QAPISchemaIfCond,
> > +                      features: List[QAPISchemaFeature],
> > +                      arg_type: Optional[QAPISchemaObjectType],
> > +                      ret_type: Optional[QAPISchemaType],
> > +                      gen: bool,
> > +                      success_response: bool,
> > +                      boxed: bool,
> > +                      allow_oob: bool,
> > +                      allow_preconfig: bool,
> > +                      coroutine: bool) -> None:
> > +        pass
> > +
> > +    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
> > +        pass
> > +
> > +    def write(self, output_dir: str) -> None:
> > +        for module_name, content in self.target.items():
> > +            go_module = module_name + "s.go"
> > +            go_dir = "go"
> > +            pathname = os.path.join(output_dir, go_dir, go_module)
> > +            odir = os.path.dirname(pathname)
> > +            os.makedirs(odir, exist_ok=True)
> > +
> > +            with open(pathname, "w", encoding="ascii") as outfile:
> > +                outfile.write(content)
> > diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
> > index 316736b6a2..cdbb3690fd 100644
> > --- a/scripts/qapi/main.py
> > +++ b/scripts/qapi/main.py
> > @@ -15,6 +15,7 @@
> >  from .common import must_match
> >  from .error import QAPIError
> >  from .events import gen_events
> > +from .golang import gen_golang
> >  from .introspect import gen_introspect
> >  from .schema import QAPISchema
> >  from .types import gen_types
> > @@ -54,6 +55,7 @@ def generate(schema_file: str,
> >      gen_events(schema, output_dir, prefix)
> >      gen_introspect(schema, output_dir, prefix, unmask)
> >
> > +    gen_golang(schema, output_dir, prefix)
> >
> >  def main() -> int:
> >      """
> > --
> > 2.41.0
> >

Re: [PATCH v1 1/9] qapi: golang: Generate qapi's enum types in Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 13340 bytes --]

Hi,

On Mon, Oct 02, 2023 at 04:09:29PM -0400, John Snow wrote:
> On Mon, Oct 2, 2023 at 3:07 PM John Snow <jsnow@redhat.com> wrote:
> >
> > On Wed, Sep 27, 2023 at 7:25 AM Victor Toso <victortoso@redhat.com> wrote:
> > >
> > > This patch handles QAPI enum types and generates its equivalent in Go.
> > >
> > > Basically, Enums are being handled as strings in Golang.
> > >
> > > 1. For each QAPI enum, we will define a string type in Go to be the
> > >    assigned type of this specific enum.
> > >
> > > 2. Naming: CamelCase will be used in any identifier that we want to
> > >    export [0], which is everything.
> > >
> > > [0] https://go.dev/ref/spec#Exported_identifiers
> > >
> > > Example:
> > >
> > > qapi:
> > >   | { 'enum': 'DisplayProtocol',
> > >   |   'data': [ 'vnc', 'spice' ] }
> > >
> > > go:
> > >   | type DisplayProtocol string
> > >   |
> > >   | const (
> > >   |     DisplayProtocolVnc   DisplayProtocol = "vnc"
> > >   |     DisplayProtocolSpice DisplayProtocol = "spice"
> > >   | )
> > >
> > > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > > ---
> > >  scripts/qapi/golang.py | 140 +++++++++++++++++++++++++++++++++++++++++
> > >  scripts/qapi/main.py   |   2 +
> > >  2 files changed, 142 insertions(+)
> > >  create mode 100644 scripts/qapi/golang.py
> > >
> > > diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> > > new file mode 100644
> > > index 0000000000..87081cdd05
> > > --- /dev/null
> > > +++ b/scripts/qapi/golang.py
> > > @@ -0,0 +1,140 @@
> > > +"""
> > > +Golang QAPI generator
> > > +"""
> > > +# Copyright (c) 2023 Red Hat Inc.
> > > +#
> > > +# Authors:
> > > +#  Victor Toso <victortoso@redhat.com>
> > > +#
> > > +# This work is licensed under the terms of the GNU GPL, version 2.
> > > +# See the COPYING file in the top-level directory.
> > > +
> > > +# due QAPISchemaVisitor interface
> > > +# pylint: disable=too-many-arguments
> 
> "due to" - also, you could more selectively disable this warning by
> putting this comment in the body of the QAPISchemaVisitor class which
> would make your exemption from the linter more locally obvious.
> 
> > > +
> > > +# Just for type hint on self
> > > +from __future__ import annotations
> 
> Oh, you know - it's been so long since I worked on QAPI I didn't
> realize we had access to this now. That's awesome!
> 
> (It was introduced in Python 3.7+)
> 
> > > +
> > > +import os
> > > +from typing import List, Optional
> > > +
> > > +from .schema import (
> > > +    QAPISchema,
> > > +    QAPISchemaType,
> > > +    QAPISchemaVisitor,
> > > +    QAPISchemaEnumMember,
> > > +    QAPISchemaFeature,
> > > +    QAPISchemaIfCond,
> > > +    QAPISchemaObjectType,
> > > +    QAPISchemaObjectTypeMember,
> > > +    QAPISchemaVariants,
> > > +)
> > > +from .source import QAPISourceInfo
> > > +
> 
> Try running isort here:
> 
> > cd ~/src/qemu/scripts
> > isort -c qapi/golang.py
> 
> ERROR: /home/jsnow/src/qemu/scripts/qapi/golang.py Imports are
> incorrectly sorted and/or formatted.
> 
> you can have it fix the import order for you:
> 
> > isort qapi/golang.py
> 
> It's very pedantic stuff, but luckily there's a tool to just handle it for you.

Thanks! Also fixed for the next iteration.
 
> > > +TEMPLATE_ENUM = '''
> > > +type {name} string
> > > +const (
> > > +{fields}
> > > +)
> > > +'''
> > > +
> > > +
> > > +def gen_golang(schema: QAPISchema,
> > > +               output_dir: str,
> > > +               prefix: str) -> None:
> > > +    vis = QAPISchemaGenGolangVisitor(prefix)
> > > +    schema.visit(vis)
> > > +    vis.write(output_dir)
> > > +
> > > +
> > > +def qapi_to_field_name_enum(name: str) -> str:
> > > +    return name.title().replace("-", "")
> > > +
> > > +
> > > +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> > > +
> > > +    def __init__(self, _: str):
> > > +        super().__init__()
> > > +        types = ["enum"]
> > > +        self.target = {name: "" for name in types}
> 
> you *could* say:
> 
> types = ("enum",)
> self.target = dict.fromkeys(types, "")
> 
> 1. We don't need a list because we won't be modifying it, so a tuple suffices
> 2. There's an idiom for doing what you want that reads a little better
> 3. None of it really matters, though.

No complains with moving it to a tuple.

> Also keep in mind you don't *need* to initialize a dict in this way,
> you can just arbitrarily assign into it whenever you'd like.
> 
> sellf.target['enum'] = foo

I think it is a problem with += operator when not initialized.

    self.target['enum'] = foo

At least I recall having errors around dict not being
initialized.
 
> I don't know if that makes things easier or not with however the
> subsequent patches are written.
> 
> > > +        self.schema = None
> > > +        self.golang_package_name = "qapi"
> > > +
> > > +    def visit_begin(self, schema):
> > > +        self.schema = schema
> > > +
> > > +        # Every Go file needs to reference its package name
> > > +        for target in self.target:
> > > +            self.target[target] = f"package {self.golang_package_name}\n"
> > > +
> > > +    def visit_end(self):
> > > +        self.schema = None
> > > +
> > > +    def visit_object_type(self: QAPISchemaGenGolangVisitor,
> > > +                          name: str,
> > > +                          info: Optional[QAPISourceInfo],
> > > +                          ifcond: QAPISchemaIfCond,
> > > +                          features: List[QAPISchemaFeature],
> > > +                          base: Optional[QAPISchemaObjectType],
> > > +                          members: List[QAPISchemaObjectTypeMember],
> > > +                          variants: Optional[QAPISchemaVariants]
> > > +                          ) -> None:
> > > +        pass
> > > +
> > > +    def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
> > > +                             name: str,
> > > +                             info: Optional[QAPISourceInfo],
> > > +                             ifcond: QAPISchemaIfCond,
> > > +                             features: List[QAPISchemaFeature],
> > > +                             variants: QAPISchemaVariants
> > > +                             ) -> None:
> > > +        pass
> > > +
> > > +    def visit_enum_type(self: QAPISchemaGenGolangVisitor,
> 
> Was there a problem when you omitted the type for 'self'?
> Usually that can be inferred. As of this patch, at least, I
> think this can be safely dropped. (Maybe it becomes important
> later.)

I don't think I tried removing the type for self. I actually
tried to keep all types expressed, just for the sake of knowing
what types they were.

Yes, it can be easily inferred and removed.

> > > +                        name: str,
> > > +                        info: Optional[QAPISourceInfo],
> > > +                        ifcond: QAPISchemaIfCond,
> > > +                        features: List[QAPISchemaFeature],
> > > +                        members: List[QAPISchemaEnumMember],
> > > +                        prefix: Optional[str]
> > > +                        ) -> None:
> > > +
> > > +        value = qapi_to_field_name_enum(members[0].name)
> >
> > Unsure if this was addressed on the mailing list yet, but in our call
> > we discussed how this call was vestigial and was causing the QAPI
> > tests to fail. Actually, I can't quite run "make check-qapi-schema"
> > and see the failure, I'm seeing it when I run "make check" and I'm not
> > sure how to find the failure more efficiently/quickly:
> >
> > jsnow@scv ~/s/q/build (review)> make
> > [1/60] Generating subprojects/dtc/version_gen.h with a custom command
> > [2/60] Generating qemu-version.h with a custom command (wrapped by
> > meson to capture output)
> > [3/44] Generating tests/Test QAPI files with a custom command
> > FAILED: tests/qapi-builtin-types.c tests/qapi-builtin-types.h
> > tests/qapi-builtin-visit.c tests/qapi-builtin-visit.h
> > tests/test-qapi-commands-sub-sub-module.c
> > tests/test-qapi-commands-sub-sub-module.h tests/test-qapi-commands.c
> > tests/test-qapi-commands.h tests/test-qapi-emit-events.c
> > tests/test-qapi-emit-events.h tests/test-qapi-events-sub-sub-module.c
> > tests/test-qapi-events-sub-sub-module.h tests/test-qapi-events.c
> > tests/test-qapi-events.h tests/test-qapi-init-commands.c
> > tests/test-qapi-init-commands.h tests/test-qapi-introspect.c
> > tests/test-qapi-introspect.h tests/test-qapi-types-sub-sub-module.c
> > tests/test-qapi-types-sub-sub-module.h tests/test-qapi-types.c
> > tests/test-qapi-types.h tests/test-qapi-visit-sub-sub-module.c
> > tests/test-qapi-visit-sub-sub-module.h tests/test-qapi-visit.c
> > tests/test-qapi-visit.h
> > /home/jsnow/src/qemu/build/pyvenv/bin/python3
> > /home/jsnow/src/qemu/scripts/qapi-gen.py -o
> > /home/jsnow/src/qemu/build/tests -b -p test-
> > ../tests/qapi-schema/qapi-schema-test.json --suppress-tracing
> > Traceback (most recent call last):
> >   File "/home/jsnow/src/qemu/scripts/qapi-gen.py", line 19, in <module>
> >     sys.exit(main.main())
> >              ^^^^^^^^^^^
> >   File "/home/jsnow/src/qemu/scripts/qapi/main.py", line 96, in main
> >     generate(args.schema,
> >   File "/home/jsnow/src/qemu/scripts/qapi/main.py", line 58, in generate
> >     gen_golang(schema, output_dir, prefix)
> >   File "/home/jsnow/src/qemu/scripts/qapi/golang.py", line 46, in gen_golang
> >     schema.visit(vis)
> >   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 1227, in visit
> >     mod.visit(visitor)
> >   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 209, in visit
> >     entity.visit(visitor)
> >   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 346, in visit
> >     visitor.visit_enum_type(
> >   File "/home/jsnow/src/qemu/scripts/qapi/golang.py", line 102, in
> > visit_enum_type
> >     value = qapi_to_field_name_enum(members[0].name)
> >                                     ~~~~~~~^^^
> > IndexError: list index out of range
> > ninja: build stopped: subcommand failed.
> > make: *** [Makefile:162: run-ninja] Error 1
> >
> >
> > For the rest of my review, I commented this line out and continued on.
> >
> > > +        fields = ""
> > > +        for member in members:
> > > +            value = qapi_to_field_name_enum(member.name)
> > > +            fields += f'''\t{name}{value} {name} = "{member.name}"\n'''
> > > +
> > > +        self.target["enum"] += TEMPLATE_ENUM.format(name=name, fields=fields[:-1])
> 
> This line is a little too long. (sorry)
> 
> try:
> 
> cd ~/src/qemu/scripts
> flake8 qapi/


toso@tapioca ~/s/qemu > flake8 scripts/qapi | wc
     89     734    6260

Yep, I'll fix them.

> jsnow@scv ~/s/q/scripts (review)> flake8 qapi/
> qapi/main.py:60:1: E302 expected 2 blank lines, found 1
> qapi/golang.py:106:80: E501 line too long (82 > 79 characters)

Cheers,
Victor

> 
> > > +
> > > +    def visit_array_type(self, name, info, ifcond, element_type):
> > > +        pass
> > > +
> > > +    def visit_command(self,
> > > +                      name: str,
> > > +                      info: Optional[QAPISourceInfo],
> > > +                      ifcond: QAPISchemaIfCond,
> > > +                      features: List[QAPISchemaFeature],
> > > +                      arg_type: Optional[QAPISchemaObjectType],
> > > +                      ret_type: Optional[QAPISchemaType],
> > > +                      gen: bool,
> > > +                      success_response: bool,
> > > +                      boxed: bool,
> > > +                      allow_oob: bool,
> > > +                      allow_preconfig: bool,
> > > +                      coroutine: bool) -> None:
> > > +        pass
> > > +
> > > +    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
> > > +        pass
> > > +
> > > +    def write(self, output_dir: str) -> None:
> > > +        for module_name, content in self.target.items():
> > > +            go_module = module_name + "s.go"
> > > +            go_dir = "go"
> > > +            pathname = os.path.join(output_dir, go_dir, go_module)
> > > +            odir = os.path.dirname(pathname)
> > > +            os.makedirs(odir, exist_ok=True)
> > > +
> > > +            with open(pathname, "w", encoding="ascii") as outfile:
> > > +                outfile.write(content)
> > > diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
> > > index 316736b6a2..cdbb3690fd 100644
> > > --- a/scripts/qapi/main.py
> > > +++ b/scripts/qapi/main.py
> > > @@ -15,6 +15,7 @@
> > >  from .common import must_match
> > >  from .error import QAPIError
> > >  from .events import gen_events
> > > +from .golang import gen_golang
> > >  from .introspect import gen_introspect
> > >  from .schema import QAPISchema
> > >  from .types import gen_types
> > > @@ -54,6 +55,7 @@ def generate(schema_file: str,
> > >      gen_events(schema, output_dir, prefix)
> > >      gen_introspect(schema, output_dir, prefix, unmask)
> > >
> > > +    gen_golang(schema, output_dir, prefix)
> > >
> > >  def main() -> int:
> > >      """
> > > --
> > > 2.41.0
> > >
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1 1/9] qapi: golang: Generate qapi's enum types in Go

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 14646 bytes --]

On Wed, Oct 4, 2023, 8:43 AM Victor Toso <victortoso@redhat.com> wrote:

> Hi,
>
> On Mon, Oct 02, 2023 at 04:09:29PM -0400, John Snow wrote:
> > On Mon, Oct 2, 2023 at 3:07 PM John Snow <jsnow@redhat.com> wrote:
> > >
> > > On Wed, Sep 27, 2023 at 7:25 AM Victor Toso <victortoso@redhat.com>
> wrote:
> > > >
> > > > This patch handles QAPI enum types and generates its equivalent in
> Go.
> > > >
> > > > Basically, Enums are being handled as strings in Golang.
> > > >
> > > > 1. For each QAPI enum, we will define a string type in Go to be the
> > > >    assigned type of this specific enum.
> > > >
> > > > 2. Naming: CamelCase will be used in any identifier that we want to
> > > >    export [0], which is everything.
> > > >
> > > > [0] https://go.dev/ref/spec#Exported_identifiers
> > > >
> > > > Example:
> > > >
> > > > qapi:
> > > >   | { 'enum': 'DisplayProtocol',
> > > >   |   'data': [ 'vnc', 'spice' ] }
> > > >
> > > > go:
> > > >   | type DisplayProtocol string
> > > >   |
> > > >   | const (
> > > >   |     DisplayProtocolVnc   DisplayProtocol = "vnc"
> > > >   |     DisplayProtocolSpice DisplayProtocol = "spice"
> > > >   | )
> > > >
> > > > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > > > ---
> > > >  scripts/qapi/golang.py | 140
> +++++++++++++++++++++++++++++++++++++++++
> > > >  scripts/qapi/main.py   |   2 +
> > > >  2 files changed, 142 insertions(+)
> > > >  create mode 100644 scripts/qapi/golang.py
> > > >
> > > > diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> > > > new file mode 100644
> > > > index 0000000000..87081cdd05
> > > > --- /dev/null
> > > > +++ b/scripts/qapi/golang.py
> > > > @@ -0,0 +1,140 @@
> > > > +"""
> > > > +Golang QAPI generator
> > > > +"""
> > > > +# Copyright (c) 2023 Red Hat Inc.
> > > > +#
> > > > +# Authors:
> > > > +#  Victor Toso <victortoso@redhat.com>
> > > > +#
> > > > +# This work is licensed under the terms of the GNU GPL, version 2.
> > > > +# See the COPYING file in the top-level directory.
> > > > +
> > > > +# due QAPISchemaVisitor interface
> > > > +# pylint: disable=too-many-arguments
> >
> > "due to" - also, you could more selectively disable this warning by
> > putting this comment in the body of the QAPISchemaVisitor class which
> > would make your exemption from the linter more locally obvious.
> >
> > > > +
> > > > +# Just for type hint on self
> > > > +from __future__ import annotations
> >
> > Oh, you know - it's been so long since I worked on QAPI I didn't
> > realize we had access to this now. That's awesome!
> >
> > (It was introduced in Python 3.7+)
> >
> > > > +
> > > > +import os
> > > > +from typing import List, Optional
> > > > +
> > > > +from .schema import (
> > > > +    QAPISchema,
> > > > +    QAPISchemaType,
> > > > +    QAPISchemaVisitor,
> > > > +    QAPISchemaEnumMember,
> > > > +    QAPISchemaFeature,
> > > > +    QAPISchemaIfCond,
> > > > +    QAPISchemaObjectType,
> > > > +    QAPISchemaObjectTypeMember,
> > > > +    QAPISchemaVariants,
> > > > +)
> > > > +from .source import QAPISourceInfo
> > > > +
> >
> > Try running isort here:
> >
> > > cd ~/src/qemu/scripts
> > > isort -c qapi/golang.py
> >
> > ERROR: /home/jsnow/src/qemu/scripts/qapi/golang.py Imports are
> > incorrectly sorted and/or formatted.
> >
> > you can have it fix the import order for you:
> >
> > > isort qapi/golang.py
> >
> > It's very pedantic stuff, but luckily there's a tool to just handle it
> for you.
>
> Thanks! Also fixed for the next iteration.
>
> > > > +TEMPLATE_ENUM = '''
> > > > +type {name} string
> > > > +const (
> > > > +{fields}
> > > > +)
> > > > +'''
> > > > +
> > > > +
> > > > +def gen_golang(schema: QAPISchema,
> > > > +               output_dir: str,
> > > > +               prefix: str) -> None:
> > > > +    vis = QAPISchemaGenGolangVisitor(prefix)
> > > > +    schema.visit(vis)
> > > > +    vis.write(output_dir)
> > > > +
> > > > +
> > > > +def qapi_to_field_name_enum(name: str) -> str:
> > > > +    return name.title().replace("-", "")
> > > > +
> > > > +
> > > > +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> > > > +
> > > > +    def __init__(self, _: str):
> > > > +        super().__init__()
> > > > +        types = ["enum"]
> > > > +        self.target = {name: "" for name in types}
> >
> > you *could* say:
> >
> > types = ("enum",)
> > self.target = dict.fromkeys(types, "")
> >
> > 1. We don't need a list because we won't be modifying it, so a tuple
> suffices
> > 2. There's an idiom for doing what you want that reads a little better
> > 3. None of it really matters, though.
>
> No complains with moving it to a tuple.
>
> > Also keep in mind you don't *need* to initialize a dict in this way,
> > you can just arbitrarily assign into it whenever you'd like.
> >
> > sellf.target['enum'] = foo
>
> I think it is a problem with += operator when not initialized.
>
>     self.target['enum'] = foo
>
> At least I recall having errors around dict not being
> initialized.
>

ah, okay.

You can also do:

self.target.setdefault("enum", "") += "blah"

but it's also fine to initialize up front. just teaching you a trick in
case it helps.


> > I don't know if that makes things easier or not with however the
> > subsequent patches are written.
> >
> > > > +        self.schema = None
> > > > +        self.golang_package_name = "qapi"
> > > > +
> > > > +    def visit_begin(self, schema):
> > > > +        self.schema = schema
> > > > +
> > > > +        # Every Go file needs to reference its package name
> > > > +        for target in self.target:
> > > > +            self.target[target] = f"package
> {self.golang_package_name}\n"
> > > > +
> > > > +    def visit_end(self):
> > > > +        self.schema = None
> > > > +
> > > > +    def visit_object_type(self: QAPISchemaGenGolangVisitor,
> > > > +                          name: str,
> > > > +                          info: Optional[QAPISourceInfo],
> > > > +                          ifcond: QAPISchemaIfCond,
> > > > +                          features: List[QAPISchemaFeature],
> > > > +                          base: Optional[QAPISchemaObjectType],
> > > > +                          members: List[QAPISchemaObjectTypeMember],
> > > > +                          variants: Optional[QAPISchemaVariants]
> > > > +                          ) -> None:
> > > > +        pass
> > > > +
> > > > +    def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
> > > > +                             name: str,
> > > > +                             info: Optional[QAPISourceInfo],
> > > > +                             ifcond: QAPISchemaIfCond,
> > > > +                             features: List[QAPISchemaFeature],
> > > > +                             variants: QAPISchemaVariants
> > > > +                             ) -> None:
> > > > +        pass
> > > > +
> > > > +    def visit_enum_type(self: QAPISchemaGenGolangVisitor,
> >
> > Was there a problem when you omitted the type for 'self'?
> > Usually that can be inferred. As of this patch, at least, I
> > think this can be safely dropped. (Maybe it becomes important
> > later.)
>
> I don't think I tried removing the type for self. I actually
> tried to keep all types expressed, just for the sake of knowing
> what types they were.
>
> Yes, it can be easily inferred and removed.
>

Normally I'm also in favor of being explicit, but where python is concerned
this may interfere with inheritance.

Usually it's best to leave self untyped because it avoids cyclical
references and it also behaves correctly in the type tree.

There are idioms for how to express a return type of "self" if that becomes
needed.


> > > > +                        name: str,
> > > > +                        info: Optional[QAPISourceInfo],
> > > > +                        ifcond: QAPISchemaIfCond,
> > > > +                        features: List[QAPISchemaFeature],
> > > > +                        members: List[QAPISchemaEnumMember],
> > > > +                        prefix: Optional[str]
> > > > +                        ) -> None:
> > > > +
> > > > +        value = qapi_to_field_name_enum(members[0].name)
> > >
> > > Unsure if this was addressed on the mailing list yet, but in our call
> > > we discussed how this call was vestigial and was causing the QAPI
> > > tests to fail. Actually, I can't quite run "make check-qapi-schema"
> > > and see the failure, I'm seeing it when I run "make check" and I'm not
> > > sure how to find the failure more efficiently/quickly:
> > >
> > > jsnow@scv ~/s/q/build (review)> make
> > > [1/60] Generating subprojects/dtc/version_gen.h with a custom command
> > > [2/60] Generating qemu-version.h with a custom command (wrapped by
> > > meson to capture output)
> > > [3/44] Generating tests/Test QAPI files with a custom command
> > > FAILED: tests/qapi-builtin-types.c tests/qapi-builtin-types.h
> > > tests/qapi-builtin-visit.c tests/qapi-builtin-visit.h
> > > tests/test-qapi-commands-sub-sub-module.c
> > > tests/test-qapi-commands-sub-sub-module.h tests/test-qapi-commands.c
> > > tests/test-qapi-commands.h tests/test-qapi-emit-events.c
> > > tests/test-qapi-emit-events.h tests/test-qapi-events-sub-sub-module.c
> > > tests/test-qapi-events-sub-sub-module.h tests/test-qapi-events.c
> > > tests/test-qapi-events.h tests/test-qapi-init-commands.c
> > > tests/test-qapi-init-commands.h tests/test-qapi-introspect.c
> > > tests/test-qapi-introspect.h tests/test-qapi-types-sub-sub-module.c
> > > tests/test-qapi-types-sub-sub-module.h tests/test-qapi-types.c
> > > tests/test-qapi-types.h tests/test-qapi-visit-sub-sub-module.c
> > > tests/test-qapi-visit-sub-sub-module.h tests/test-qapi-visit.c
> > > tests/test-qapi-visit.h
> > > /home/jsnow/src/qemu/build/pyvenv/bin/python3
> > > /home/jsnow/src/qemu/scripts/qapi-gen.py -o
> > > /home/jsnow/src/qemu/build/tests -b -p test-
> > > ../tests/qapi-schema/qapi-schema-test.json --suppress-tracing
> > > Traceback (most recent call last):
> > >   File "/home/jsnow/src/qemu/scripts/qapi-gen.py", line 19, in <module>
> > >     sys.exit(main.main())
> > >              ^^^^^^^^^^^
> > >   File "/home/jsnow/src/qemu/scripts/qapi/main.py", line 96, in main
> > >     generate(args.schema,
> > >   File "/home/jsnow/src/qemu/scripts/qapi/main.py", line 58, in
> generate
> > >     gen_golang(schema, output_dir, prefix)
> > >   File "/home/jsnow/src/qemu/scripts/qapi/golang.py", line 46, in
> gen_golang
> > >     schema.visit(vis)
> > >   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 1227, in
> visit
> > >     mod.visit(visitor)
> > >   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 209, in
> visit
> > >     entity.visit(visitor)
> > >   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 346, in
> visit
> > >     visitor.visit_enum_type(
> > >   File "/home/jsnow/src/qemu/scripts/qapi/golang.py", line 102, in
> > > visit_enum_type
> > >     value = qapi_to_field_name_enum(members[0].name)
> > >                                     ~~~~~~~^^^
> > > IndexError: list index out of range
> > > ninja: build stopped: subcommand failed.
> > > make: *** [Makefile:162: run-ninja] Error 1
> > >
> > >
> > > For the rest of my review, I commented this line out and continued on.
> > >
> > > > +        fields = ""
> > > > +        for member in members:
> > > > +            value = qapi_to_field_name_enum(member.name)
> > > > +            fields += f'''\t{name}{value} {name} = "{member.name
> }"\n'''
> > > > +
> > > > +        self.target["enum"] += TEMPLATE_ENUM.format(name=name,
> fields=fields[:-1])
> >
> > This line is a little too long. (sorry)
> >
> > try:
> >
> > cd ~/src/qemu/scripts
> > flake8 qapi/
>
>
> toso@tapioca ~/s/qemu > flake8 scripts/qapi | wc
>      89     734    6260
>
> Yep, I'll fix them.
>
> > jsnow@scv ~/s/q/scripts (review)> flake8 qapi/
> > qapi/main.py:60:1: E302 expected 2 blank lines, found 1
> > qapi/golang.py:106:80: E501 line too long (82 > 79 characters)
>
> Cheers,
> Victor
>
> >
> > > > +
> > > > +    def visit_array_type(self, name, info, ifcond, element_type):
> > > > +        pass
> > > > +
> > > > +    def visit_command(self,
> > > > +                      name: str,
> > > > +                      info: Optional[QAPISourceInfo],
> > > > +                      ifcond: QAPISchemaIfCond,
> > > > +                      features: List[QAPISchemaFeature],
> > > > +                      arg_type: Optional[QAPISchemaObjectType],
> > > > +                      ret_type: Optional[QAPISchemaType],
> > > > +                      gen: bool,
> > > > +                      success_response: bool,
> > > > +                      boxed: bool,
> > > > +                      allow_oob: bool,
> > > > +                      allow_preconfig: bool,
> > > > +                      coroutine: bool) -> None:
> > > > +        pass
> > > > +
> > > > +    def visit_event(self, name, info, ifcond, features, arg_type,
> boxed):
> > > > +        pass
> > > > +
> > > > +    def write(self, output_dir: str) -> None:
> > > > +        for module_name, content in self.target.items():
> > > > +            go_module = module_name + "s.go"
> > > > +            go_dir = "go"
> > > > +            pathname = os.path.join(output_dir, go_dir, go_module)
> > > > +            odir = os.path.dirname(pathname)
> > > > +            os.makedirs(odir, exist_ok=True)
> > > > +
> > > > +            with open(pathname, "w", encoding="ascii") as outfile:
> > > > +                outfile.write(content)
> > > > diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
> > > > index 316736b6a2..cdbb3690fd 100644
> > > > --- a/scripts/qapi/main.py
> > > > +++ b/scripts/qapi/main.py
> > > > @@ -15,6 +15,7 @@
> > > >  from .common import must_match
> > > >  from .error import QAPIError
> > > >  from .events import gen_events
> > > > +from .golang import gen_golang
> > > >  from .introspect import gen_introspect
> > > >  from .schema import QAPISchema
> > > >  from .types import gen_types
> > > > @@ -54,6 +55,7 @@ def generate(schema_file: str,
> > > >      gen_events(schema, output_dir, prefix)
> > > >      gen_introspect(schema, output_dir, prefix, unmask)
> > > >
> > > > +    gen_golang(schema, output_dir, prefix)
> > > >
> > > >  def main() -> int:
> > > >      """
> > > > --
> > > > 2.41.0
> > > >
> >
>

[-- Attachment #2: Type: text/html, Size: 20599 bytes --]

Re: [PATCH v1 1/9] qapi: golang: Generate qapi's enum types in Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 10499 bytes --]

Hi,

On Mon, Oct 02, 2023 at 03:07:49PM -0400, John Snow wrote:
> On Wed, Sep 27, 2023 at 7:25 AM Victor Toso <victortoso@redhat.com> wrote:
> >
> > This patch handles QAPI enum types and generates its equivalent in Go.
> >
> > Basically, Enums are being handled as strings in Golang.
> >
> > 1. For each QAPI enum, we will define a string type in Go to be the
> >    assigned type of this specific enum.
> >
> > 2. Naming: CamelCase will be used in any identifier that we want to
> >    export [0], which is everything.
> >
> > [0] https://go.dev/ref/spec#Exported_identifiers
> >
> > Example:
> >
> > qapi:
> >   | { 'enum': 'DisplayProtocol',
> >   |   'data': [ 'vnc', 'spice' ] }
> >
> > go:
> >   | type DisplayProtocol string
> >   |
> >   | const (
> >   |     DisplayProtocolVnc   DisplayProtocol = "vnc"
> >   |     DisplayProtocolSpice DisplayProtocol = "spice"
> >   | )
> >
> > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > ---
> >  scripts/qapi/golang.py | 140 +++++++++++++++++++++++++++++++++++++++++
> >  scripts/qapi/main.py   |   2 +
> >  2 files changed, 142 insertions(+)
> >  create mode 100644 scripts/qapi/golang.py
> >
> > diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> > new file mode 100644
> > index 0000000000..87081cdd05
> > --- /dev/null
> > +++ b/scripts/qapi/golang.py
> > @@ -0,0 +1,140 @@
> > +"""
> > +Golang QAPI generator
> > +"""
> > +# Copyright (c) 2023 Red Hat Inc.
> > +#
> > +# Authors:
> > +#  Victor Toso <victortoso@redhat.com>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2.
> > +# See the COPYING file in the top-level directory.
> > +
> > +# due QAPISchemaVisitor interface
> > +# pylint: disable=too-many-arguments
> > +
> > +# Just for type hint on self
> > +from __future__ import annotations
> > +
> > +import os
> > +from typing import List, Optional
> > +
> > +from .schema import (
> > +    QAPISchema,
> > +    QAPISchemaType,
> > +    QAPISchemaVisitor,
> > +    QAPISchemaEnumMember,
> > +    QAPISchemaFeature,
> > +    QAPISchemaIfCond,
> > +    QAPISchemaObjectType,
> > +    QAPISchemaObjectTypeMember,
> > +    QAPISchemaVariants,
> > +)
> > +from .source import QAPISourceInfo
> > +
> > +TEMPLATE_ENUM = '''
> > +type {name} string
> > +const (
> > +{fields}
> > +)
> > +'''
> > +
> > +
> > +def gen_golang(schema: QAPISchema,
> > +               output_dir: str,
> > +               prefix: str) -> None:
> > +    vis = QAPISchemaGenGolangVisitor(prefix)
> > +    schema.visit(vis)
> > +    vis.write(output_dir)
> > +
> > +
> > +def qapi_to_field_name_enum(name: str) -> str:
> > +    return name.title().replace("-", "")
> > +
> > +
> > +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> > +
> > +    def __init__(self, _: str):
> > +        super().__init__()
> > +        types = ["enum"]
> > +        self.target = {name: "" for name in types}
> > +        self.schema = None
> > +        self.golang_package_name = "qapi"
> > +
> > +    def visit_begin(self, schema):
> > +        self.schema = schema
> > +
> > +        # Every Go file needs to reference its package name
> > +        for target in self.target:
> > +            self.target[target] = f"package {self.golang_package_name}\n"
> > +
> > +    def visit_end(self):
> > +        self.schema = None
> > +
> > +    def visit_object_type(self: QAPISchemaGenGolangVisitor,
> > +                          name: str,
> > +                          info: Optional[QAPISourceInfo],
> > +                          ifcond: QAPISchemaIfCond,
> > +                          features: List[QAPISchemaFeature],
> > +                          base: Optional[QAPISchemaObjectType],
> > +                          members: List[QAPISchemaObjectTypeMember],
> > +                          variants: Optional[QAPISchemaVariants]
> > +                          ) -> None:
> > +        pass
> > +
> > +    def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
> > +                             name: str,
> > +                             info: Optional[QAPISourceInfo],
> > +                             ifcond: QAPISchemaIfCond,
> > +                             features: List[QAPISchemaFeature],
> > +                             variants: QAPISchemaVariants
> > +                             ) -> None:
> > +        pass
> > +
> > +    def visit_enum_type(self: QAPISchemaGenGolangVisitor,
> > +                        name: str,
> > +                        info: Optional[QAPISourceInfo],
> > +                        ifcond: QAPISchemaIfCond,
> > +                        features: List[QAPISchemaFeature],
> > +                        members: List[QAPISchemaEnumMember],
> > +                        prefix: Optional[str]
> > +                        ) -> None:
> > +
> > +        value = qapi_to_field_name_enum(members[0].name)
> 
> Unsure if this was addressed on the mailing list yet, but in our call
> we discussed how this call was vestigial and was causing the QAPI
> tests to fail. Actually, I can't quite run "make check-qapi-schema"
> and see the failure, I'm seeing it when I run "make check" and I'm not
> sure how to find the failure more efficiently/quickly:
> 
> jsnow@scv ~/s/q/build (review)> make
> [1/60] Generating subprojects/dtc/version_gen.h with a custom command
> [2/60] Generating qemu-version.h with a custom command (wrapped by
> meson to capture output)
> [3/44] Generating tests/Test QAPI files with a custom command
> FAILED: tests/qapi-builtin-types.c tests/qapi-builtin-types.h
> tests/qapi-builtin-visit.c tests/qapi-builtin-visit.h
> tests/test-qapi-commands-sub-sub-module.c
> tests/test-qapi-commands-sub-sub-module.h tests/test-qapi-commands.c
> tests/test-qapi-commands.h tests/test-qapi-emit-events.c
> tests/test-qapi-emit-events.h tests/test-qapi-events-sub-sub-module.c
> tests/test-qapi-events-sub-sub-module.h tests/test-qapi-events.c
> tests/test-qapi-events.h tests/test-qapi-init-commands.c
> tests/test-qapi-init-commands.h tests/test-qapi-introspect.c
> tests/test-qapi-introspect.h tests/test-qapi-types-sub-sub-module.c
> tests/test-qapi-types-sub-sub-module.h tests/test-qapi-types.c
> tests/test-qapi-types.h tests/test-qapi-visit-sub-sub-module.c
> tests/test-qapi-visit-sub-sub-module.h tests/test-qapi-visit.c
> tests/test-qapi-visit.h
> /home/jsnow/src/qemu/build/pyvenv/bin/python3
> /home/jsnow/src/qemu/scripts/qapi-gen.py -o
> /home/jsnow/src/qemu/build/tests -b -p test-
> ../tests/qapi-schema/qapi-schema-test.json --suppress-tracing
> Traceback (most recent call last):
>   File "/home/jsnow/src/qemu/scripts/qapi-gen.py", line 19, in <module>
>     sys.exit(main.main())
>              ^^^^^^^^^^^
>   File "/home/jsnow/src/qemu/scripts/qapi/main.py", line 96, in main
>     generate(args.schema,
>   File "/home/jsnow/src/qemu/scripts/qapi/main.py", line 58, in generate
>     gen_golang(schema, output_dir, prefix)
>   File "/home/jsnow/src/qemu/scripts/qapi/golang.py", line 46, in gen_golang
>     schema.visit(vis)
>   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 1227, in visit
>     mod.visit(visitor)
>   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 209, in visit
>     entity.visit(visitor)
>   File "/home/jsnow/src/qemu/scripts/qapi/schema.py", line 346, in visit
>     visitor.visit_enum_type(
>   File "/home/jsnow/src/qemu/scripts/qapi/golang.py", line 102, in
> visit_enum_type
>     value = qapi_to_field_name_enum(members[0].name)
>                                     ~~~~~~~^^^
> IndexError: list index out of range
> ninja: build stopped: subcommand failed.
> make: *** [Makefile:162: run-ninja] Error 1
> 
> 
> For the rest of my review, I commented this line out and continued on.

Yes, it was a leftover when I was reorganizing the patches. You
can see this value is not actually used.

I removed it in my branch for v2.

Cheers,
Victor

> > +        fields = ""
> > +        for member in members:
> > +            value = qapi_to_field_name_enum(member.name)
> > +            fields += f'''\t{name}{value} {name} = "{member.name}"\n'''
> > +
> > +        self.target["enum"] += TEMPLATE_ENUM.format(name=name, fields=fields[:-1])
> > +
> > +    def visit_array_type(self, name, info, ifcond, element_type):
> > +        pass
> > +
> > +    def visit_command(self,
> > +                      name: str,
> > +                      info: Optional[QAPISourceInfo],
> > +                      ifcond: QAPISchemaIfCond,
> > +                      features: List[QAPISchemaFeature],
> > +                      arg_type: Optional[QAPISchemaObjectType],
> > +                      ret_type: Optional[QAPISchemaType],
> > +                      gen: bool,
> > +                      success_response: bool,
> > +                      boxed: bool,
> > +                      allow_oob: bool,
> > +                      allow_preconfig: bool,
> > +                      coroutine: bool) -> None:
> > +        pass
> > +
> > +    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
> > +        pass
> > +
> > +    def write(self, output_dir: str) -> None:
> > +        for module_name, content in self.target.items():
> > +            go_module = module_name + "s.go"
> > +            go_dir = "go"
> > +            pathname = os.path.join(output_dir, go_dir, go_module)
> > +            odir = os.path.dirname(pathname)
> > +            os.makedirs(odir, exist_ok=True)
> > +
> > +            with open(pathname, "w", encoding="ascii") as outfile:
> > +                outfile.write(content)
> > diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
> > index 316736b6a2..cdbb3690fd 100644
> > --- a/scripts/qapi/main.py
> > +++ b/scripts/qapi/main.py
> > @@ -15,6 +15,7 @@
> >  from .common import must_match
> >  from .error import QAPIError
> >  from .events import gen_events
> > +from .golang import gen_golang
> >  from .introspect import gen_introspect
> >  from .schema import QAPISchema
> >  from .types import gen_types
> > @@ -54,6 +55,7 @@ def generate(schema_file: str,
> >      gen_events(schema, output_dir, prefix)
> >      gen_introspect(schema, output_dir, prefix, unmask)
> >
> > +    gen_golang(schema, output_dir, prefix)
> >
> >  def main() -> int:
> >      """
> > --
> > 2.41.0
> >
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v1 2/9] qapi: golang: Generate qapi's alternate types in Go

[Victor Toso]

This patch handles QAPI alternate types and generates data structures
in Go that handles it.

Alternate types are similar to Union but without a discriminator that
can be used to identify the underlying value on the wire. It is needed
to infer it. In Go, most of the types [*] are mapped as optional
fields and Marshal and Unmarshal methods will be handling the data
checks.

Example:

qapi:
  | { 'alternate': 'BlockdevRef',
  |   'data': { 'definition': 'BlockdevOptions',
  |             'reference': 'str' } }

go:
  | type BlockdevRef struct {
  |         Definition *BlockdevOptions
  |         Reference  *string
  | }

usage:
  | input := `{"driver":"qcow2","data-file":"/some/place/my-image"}`
  | k := BlockdevRef{}
  | err := json.Unmarshal([]byte(input), &k)
  | if err != nil {
  |     panic(err)
  | }
  | // *k.Definition.Qcow2.DataFile.Reference == "/some/place/my-image"

[*] The exception for optional fields as default is to Types that can
accept JSON Null as a value like StrOrNull and BlockdevRefOrNull. For
this case, we translate Null with a boolean value in a field called
IsNull. This will be explained better in the documentation patch of
this series but the main rationale is around Marshaling to and from
JSON and Go data structures.

Example:

qapi:
 | { 'alternate': 'StrOrNull',
 |   'data': { 's': 'str',
 |             'n': 'null' } }

go:
 | type StrOrNull struct {
 |     S      *string
 |     IsNull bool
 | }

Signed-off-by: Victor Toso <victortoso@redhat.com>
---
 scripts/qapi/golang.py | 188 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 185 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 87081cdd05..43dbdde14c 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -16,10 +16,11 @@
 from __future__ import annotations
 
 import os
-from typing import List, Optional
+from typing import Tuple, List, Optional
 
 from .schema import (
     QAPISchema,
+    QAPISchemaAlternateType,
     QAPISchemaType,
     QAPISchemaVisitor,
     QAPISchemaEnumMember,
@@ -38,6 +39,76 @@
 )
 '''
 
+TEMPLATE_HELPER = '''
+// Creates a decoder that errors on unknown Fields
+// Returns nil if successfully decoded @from payload to @into type
+// Returns error if failed to decode @from payload to @into type
+func StrictDecode(into interface{}, from []byte) error {
+    dec := json.NewDecoder(strings.NewReader(string(from)))
+    dec.DisallowUnknownFields()
+
+    if err := dec.Decode(into); err != nil {
+        return err
+    }
+    return nil
+}
+'''
+
+TEMPLATE_ALTERNATE = '''
+// Only implemented on Alternate types that can take JSON NULL as value.
+//
+// This is a helper for the marshalling code. It should return true only when
+// the Alternate is empty (no members are set), otherwise it returns false and
+// the member set to be Marshalled.
+type AbsentAlternate interface {
+	ToAnyOrAbsent() (any, bool)
+}
+'''
+
+TEMPLATE_ALTERNATE_NULLABLE_CHECK = '''
+    }} else if s.{var_name} != nil {{
+        return *s.{var_name}, false'''
+
+TEMPLATE_ALTERNATE_MARSHAL_CHECK = '''
+    if s.{var_name} != nil {{
+        return json.Marshal(s.{var_name})
+    }} else '''
+
+TEMPLATE_ALTERNATE_UNMARSHAL_CHECK = '''
+    // Check for {var_type}
+    {{
+        s.{var_name} = new({var_type})
+        if err := StrictDecode(s.{var_name}, data); err == nil {{
+            return nil
+        }}
+        s.{var_name} = nil
+    }}
+'''
+
+TEMPLATE_ALTERNATE_NULLABLE = '''
+func (s *{name}) ToAnyOrAbsent() (any, bool) {{
+    if s != nil {{
+        if s.IsNull {{
+            return nil, false
+{absent_check_fields}
+        }}
+    }}
+
+    return nil, true
+}}
+'''
+
+TEMPLATE_ALTERNATE_METHODS = '''
+func (s {name}) MarshalJSON() ([]byte, error) {{
+    {marshal_check_fields}
+    return {marshal_return_default}
+}}
+
+func (s *{name}) UnmarshalJSON(data []byte) error {{
+    {unmarshal_check_fields}
+    return fmt.Errorf("Can't convert to {name}: %s", string(data))
+}}
+'''
 
 def gen_golang(schema: QAPISchema,
                output_dir: str,
@@ -46,27 +117,135 @@ def gen_golang(schema: QAPISchema,
     schema.visit(vis)
     vis.write(output_dir)
 
+def qapi_to_field_name(name: str) -> str:
+    return name.title().replace("_", "").replace("-", "")
 
 def qapi_to_field_name_enum(name: str) -> str:
     return name.title().replace("-", "")
 
+def qapi_schema_type_to_go_type(qapitype: str) -> str:
+    schema_types_to_go = {
+            'str': 'string', 'null': 'nil', 'bool': 'bool', 'number':
+            'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8',
+            'int16': 'int16', 'int32': 'int32', 'int64': 'int64', 'uint8':
+            'uint8', 'uint16': 'uint16', 'uint32': 'uint32', 'uint64':
+            'uint64', 'any': 'any', 'QType': 'QType',
+    }
+
+    prefix = ""
+    if qapitype.endswith("List"):
+        prefix = "[]"
+        qapitype = qapitype[:-4]
+
+    qapitype = schema_types_to_go.get(qapitype, qapitype)
+    return prefix + qapitype
+
+def qapi_field_to_go_field(member_name: str, type_name: str) -> Tuple[str, str, str]:
+    # Nothing to generate on null types. We update some
+    # variables to handle json-null on marshalling methods.
+    if type_name == "null":
+        return "IsNull", "bool", ""
+
+    # This function is called on Alternate, so fields should be ptrs
+    return qapi_to_field_name(member_name), qapi_schema_type_to_go_type(type_name), "*"
+
+# Helper function for boxed or self contained structures.
+def generate_struct_type(type_name, args="") -> str:
+    args = args if len(args) == 0 else f"\n{args}\n"
+    with_type = f"\ntype {type_name}" if len(type_name) > 0 else ""
+    return f'''{with_type} struct {{{args}}}
+'''
+
+def generate_template_alternate(self: QAPISchemaGenGolangVisitor,
+                                name: str,
+                                variants: Optional[QAPISchemaVariants]) -> str:
+    absent_check_fields = ""
+    variant_fields = ""
+    # to avoid having to check accept_null_types
+    nullable = False
+    if name in self.accept_null_types:
+        # In QEMU QAPI schema, only StrOrNull and BlockdevRefOrNull.
+        nullable = True
+        marshal_return_default = '''[]byte("{}"), nil'''
+        marshal_check_fields = '''
+        if s.IsNull {
+            return []byte("null"), nil
+        } else '''
+        unmarshal_check_fields = '''
+        // Check for json-null first
+            if string(data) == "null" {
+                s.IsNull = true
+                return nil
+            }
+        '''
+    else:
+        marshal_return_default = f'nil, errors.New("{name} has empty fields")'
+        marshal_check_fields = ""
+        unmarshal_check_fields = f'''
+            // Check for json-null first
+            if string(data) == "null" {{
+                return errors.New(`null not supported for {name}`)
+            }}
+        '''
+
+    for var in variants.variants:
+        var_name, var_type, isptr = qapi_field_to_go_field(var.name, var.type.name)
+        variant_fields += f"\t{var_name} {isptr}{var_type}\n"
+
+        # Null is special, handled first
+        if var.type.name == "null":
+            assert nullable
+            continue
+
+        if nullable:
+            absent_check_fields += TEMPLATE_ALTERNATE_NULLABLE_CHECK.format(var_name=var_name)[1:]
+        marshal_check_fields += TEMPLATE_ALTERNATE_MARSHAL_CHECK.format(var_name=var_name)
+        unmarshal_check_fields += TEMPLATE_ALTERNATE_UNMARSHAL_CHECK.format(var_name=var_name,
+                                                                            var_type=var_type)[1:]
+
+    content = generate_struct_type(name, variant_fields)
+    if nullable:
+        content += TEMPLATE_ALTERNATE_NULLABLE.format(name=name,
+                                                      absent_check_fields=absent_check_fields)
+    content += TEMPLATE_ALTERNATE_METHODS.format(name=name,
+                                                 marshal_check_fields=marshal_check_fields[1:-5],
+                                                 marshal_return_default=marshal_return_default,
+                                                 unmarshal_check_fields=unmarshal_check_fields[1:])
+    return content
+
 
 class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, _: str):
         super().__init__()
-        types = ["enum"]
+        types = ["alternate", "enum", "helper"]
         self.target = {name: "" for name in types}
+        self.objects_seen = {}
         self.schema = None
         self.golang_package_name = "qapi"
+        self.accept_null_types = []
 
     def visit_begin(self, schema):
         self.schema = schema
 
+        # We need to be aware of any types that accept JSON NULL
+        for name, entity in self.schema._entity_dict.items():
+            if not isinstance(entity, QAPISchemaAlternateType):
+                # Assume that only Alternate types accept JSON NULL
+                continue
+
+            for var in  entity.variants.variants:
+                if var.type.name == 'null':
+                    self.accept_null_types.append(name)
+                    break
+
         # Every Go file needs to reference its package name
         for target in self.target:
             self.target[target] = f"package {self.golang_package_name}\n"
 
+        self.target["helper"] += TEMPLATE_HELPER
+        self.target["alternate"] += TEMPLATE_ALTERNATE
+
     def visit_end(self):
         self.schema = None
 
@@ -88,7 +267,10 @@ def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
                              features: List[QAPISchemaFeature],
                              variants: QAPISchemaVariants
                              ) -> None:
-        pass
+        assert name not in self.objects_seen
+        self.objects_seen[name] = True
+
+        self.target["alternate"] += generate_template_alternate(self, name, variants)
 
     def visit_enum_type(self: QAPISchemaGenGolangVisitor,
                         name: str,
-- 
2.41.0

Re: [PATCH v1 2/9] qapi: golang: Generate qapi's alternate types in Go

[Daniel P. Berrangé]

On Wed, Sep 27, 2023 at 01:25:37PM +0200, Victor Toso wrote:
> This patch handles QAPI alternate types and generates data structures
> in Go that handles it.

This file (and most others) needs some imports added.
I found the following to be required in order to
actually compile this:

alternates.go:import (
alternates.go-	"encoding/json"
alternates.go-	"errors"
alternates.go-	"fmt"
alternates.go-)


commands.go:import (
commands.go-	"encoding/json"
commands.go-	"errors"
commands.go-	"fmt"
commands.go-)


events.go:import (
events.go-	"encoding/json"
events.go-	"errors"
events.go-	"fmt"
events.go-)


helpers.go:import (
helpers.go-	"encoding/json"
helpers.go-	"fmt"
helpers.go-	"strings"
helpers.go-)


structs.go:import (
structs.go-	"encoding/json"
structs.go-)


unions.go:import (
unions.go-	"encoding/json"
unions.go-	"errors"
unions.go-	"fmt"
unions.go-)




With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 2/9] qapi: golang: Generate qapi's alternate types in Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 2380 bytes --]

Hi,

On Thu, Sep 28, 2023 at 03:51:50PM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 27, 2023 at 01:25:37PM +0200, Victor Toso wrote:
> > This patch handles QAPI alternate types and generates data structures
> > in Go that handles it.
> 
> This file (and most others) needs some imports added.
> I found the following to be required in order to
> actually compile this:

This was by design, I mean, my preference. I decided that the
generator should output correct but not necessarly
formatted/buildable Go code. The consumer should still use
gofmt/goimports.

Do you think we should do this in QEMU? What about extra
dependencies in QEMU with go binaries?

This is how it is done in victortoso/qapi-go module:

# to generate
    toso@tapioca ~> QEMU_REPO=/home/toso/src/qemu go generate ./...

# the generation
    toso@tapioca ~> cat src/qapi-go/pkg/qapi/doc.go
    //go:generate ../../scripts/generate.sh
    //go:generate gofmt -w .
    //go:generate goimports -w .
    package qapi

# script
    URL="https://gitlab.com/victortoso/qemu.git"
    BRANCH="qapi-golang"

    if [[ -z "${QEMU_REPO}" ]]; then
        git clone --depth 1 --branch $BRANCH $URL
        QEMU_REPO="$PWD/qemu"
    fi

    python3 $QEMU_REPO/scripts/qapi-gen.py -o tmp $QEMU_REPO/qapi/qapi-schema.json
    mv tmp/go/* .
    rm -rf tmp qemu

Cheers,
Victor

> alternates.go:import (
> alternates.go-	"encoding/json"
> alternates.go-	"errors"
> alternates.go-	"fmt"
> alternates.go-)
> 
> 
> commands.go:import (
> commands.go-	"encoding/json"
> commands.go-	"errors"
> commands.go-	"fmt"
> commands.go-)
> 
> 
> events.go:import (
> events.go-	"encoding/json"
> events.go-	"errors"
> events.go-	"fmt"
> events.go-)
> 
> 
> helpers.go:import (
> helpers.go-	"encoding/json"
> helpers.go-	"fmt"
> helpers.go-	"strings"
> helpers.go-)
> 
> 
> structs.go:import (
> structs.go-	"encoding/json"
> structs.go-)
> 
> 
> unions.go:import (
> unions.go-	"encoding/json"
> unions.go-	"errors"
> unions.go-	"fmt"
> unions.go-)
> 
> 
> 
> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1 2/9] qapi: golang: Generate qapi's alternate types in Go

[Daniel P. Berrangé]

On Fri, Sep 29, 2023 at 02:23:22PM +0200, Victor Toso wrote:
> Hi,
> 
> On Thu, Sep 28, 2023 at 03:51:50PM +0100, Daniel P. Berrangé wrote:
> > On Wed, Sep 27, 2023 at 01:25:37PM +0200, Victor Toso wrote:
> > > This patch handles QAPI alternate types and generates data structures
> > > in Go that handles it.
> > 
> > This file (and most others) needs some imports added.
> > I found the following to be required in order to
> > actually compile this:
> 
> This was by design, I mean, my preference. I decided that the
> generator should output correct but not necessarly
> formatted/buildable Go code. The consumer should still use
> gofmt/goimports.
> 
> Do you think we should do this in QEMU? What about extra
> dependencies in QEMU with go binaries?

IMHO the generator should be omitting well formatted and buildable
code, otherwise we need to wrap the generator in a second generator
to do the extra missing bits.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 2/9] qapi: golang: Generate qapi's alternate types in Go

[John Snow]

On Fri, Sep 29, 2023 at 8:37 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, Sep 29, 2023 at 02:23:22PM +0200, Victor Toso wrote:
> > Hi,
> >
> > On Thu, Sep 28, 2023 at 03:51:50PM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Sep 27, 2023 at 01:25:37PM +0200, Victor Toso wrote:
> > > > This patch handles QAPI alternate types and generates data structures
> > > > in Go that handles it.
> > >
> > > This file (and most others) needs some imports added.
> > > I found the following to be required in order to
> > > actually compile this:
> >
> > This was by design, I mean, my preference. I decided that the
> > generator should output correct but not necessarly
> > formatted/buildable Go code. The consumer should still use
> > gofmt/goimports.
> >
> > Do you think we should do this in QEMU? What about extra
> > dependencies in QEMU with go binaries?
>
> IMHO the generator should be omitting well formatted and buildable
> code, otherwise we need to wrap the generator in a second generator
> to do the extra missing bits.
>

Unless there's some consideration I'm unaware of, I think I agree with
Dan here - I don't *think* there's a reason to need to do this in two
layers, unless there's some tool that magically fixes/handles
dependencies that you want to leverage as part of the pipeline here.

Re: [PATCH v1 2/9] qapi: golang: Generate qapi's alternate types in Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 2695 bytes --]

Hi,

On Mon, Oct 02, 2023 at 05:48:37PM -0400, John Snow wrote:
> On Fri, Sep 29, 2023 at 8:37 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Fri, Sep 29, 2023 at 02:23:22PM +0200, Victor Toso wrote:
> > > Hi,
> > >
> > > On Thu, Sep 28, 2023 at 03:51:50PM +0100, Daniel P. Berrangé wrote:
> > > > On Wed, Sep 27, 2023 at 01:25:37PM +0200, Victor Toso wrote:
> > > > > This patch handles QAPI alternate types and generates data structures
> > > > > in Go that handles it.
> > > >
> > > > This file (and most others) needs some imports added.
> > > > I found the following to be required in order to
> > > > actually compile this:
> > >
> > > This was by design, I mean, my preference. I decided that the
> > > generator should output correct but not necessarly
> > > formatted/buildable Go code. The consumer should still use
> > > gofmt/goimports.
> > >
> > > Do you think we should do this in QEMU? What about extra
> > > dependencies in QEMU with go binaries?
> >
> > IMHO the generator should be omitting well formatted and buildable
> > code, otherwise we need to wrap the generator in a second generator
> > to do the extra missing bits.
> >
> 
> Unless there's some consideration I'm unaware of, I think I agree with
> Dan here - I don't *think* there's a reason to need to do this in two
> layers, unless there's some tool that magically fixes/handles
> dependencies that you want to leverage as part of the pipeline here.

But there is. In the current qapi-go repository, I have 4 line
doc.go [0] file:

 1  //go:generate ../../scripts/generate.sh
 2  //go:generate gofmt -w .
 3  //go:generate goimports -w .
 4  package qapi

With this, anyone can run `go generate` which runs this generator
(1), runs gofmt (2) and goimports (3).

- gofmt fixes the formatting
- goimports adds the imports that are missing

Both things were mentioned by Dan as a problem to be fixed but
somewhat can be solved by another tool.

From what I can see, we have three options to chose:

 1. Let this be as is. That means that someone else validates
    and fixes the generator's output.

 2. Fix the indentation and the (very few) imports that are
    needed. This means gofmt and goimports should do 0 changes,
    so they would not be needed.

 3. Add a post-processing that runs gofmt and goimports from
    QEMU. I would like to avoid this just to no add go binaries
    as requirement for QEMU, but perhaps it isn't a big deal.

I'm planning to work on (2) for v2 and further discuss if (3)
will be needed on top of that.

[0] https://gitlab.com/victortoso/qapi-go/-/blob/main/pkg/qapi/doc.go

Cheers,
Victor

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1 2/9] qapi: golang: Generate qapi's alternate types in Go

[John Snow]

On Wed, Sep 27, 2023 at 7:25 AM Victor Toso <victortoso@redhat.com> wrote:
>
> This patch handles QAPI alternate types and generates data structures
> in Go that handles it.
>
> Alternate types are similar to Union but without a discriminator that
> can be used to identify the underlying value on the wire. It is needed
> to infer it. In Go, most of the types [*] are mapped as optional
> fields and Marshal and Unmarshal methods will be handling the data
> checks.
>
> Example:
>
> qapi:
>   | { 'alternate': 'BlockdevRef',
>   |   'data': { 'definition': 'BlockdevOptions',
>   |             'reference': 'str' } }
>
> go:
>   | type BlockdevRef struct {
>   |         Definition *BlockdevOptions
>   |         Reference  *string
>   | }
>
> usage:
>   | input := `{"driver":"qcow2","data-file":"/some/place/my-image"}`
>   | k := BlockdevRef{}
>   | err := json.Unmarshal([]byte(input), &k)
>   | if err != nil {
>   |     panic(err)
>   | }
>   | // *k.Definition.Qcow2.DataFile.Reference == "/some/place/my-image"
>
> [*] The exception for optional fields as default is to Types that can
> accept JSON Null as a value like StrOrNull and BlockdevRefOrNull. For
> this case, we translate Null with a boolean value in a field called
> IsNull. This will be explained better in the documentation patch of
> this series but the main rationale is around Marshaling to and from
> JSON and Go data structures.
>
> Example:
>
> qapi:
>  | { 'alternate': 'StrOrNull',
>  |   'data': { 's': 'str',
>  |             'n': 'null' } }
>
> go:
>  | type StrOrNull struct {
>  |     S      *string
>  |     IsNull bool
>  | }
>
> Signed-off-by: Victor Toso <victortoso@redhat.com>
> ---
>  scripts/qapi/golang.py | 188 ++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 185 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> index 87081cdd05..43dbdde14c 100644
> --- a/scripts/qapi/golang.py
> +++ b/scripts/qapi/golang.py
> @@ -16,10 +16,11 @@
>  from __future__ import annotations
>
>  import os
> -from typing import List, Optional
> +from typing import Tuple, List, Optional
>
>  from .schema import (
>      QAPISchema,
> +    QAPISchemaAlternateType,
>      QAPISchemaType,
>      QAPISchemaVisitor,
>      QAPISchemaEnumMember,
> @@ -38,6 +39,76 @@
>  )
>  '''
>
> +TEMPLATE_HELPER = '''
> +// Creates a decoder that errors on unknown Fields
> +// Returns nil if successfully decoded @from payload to @into type
> +// Returns error if failed to decode @from payload to @into type
> +func StrictDecode(into interface{}, from []byte) error {
> +    dec := json.NewDecoder(strings.NewReader(string(from)))
> +    dec.DisallowUnknownFields()
> +
> +    if err := dec.Decode(into); err != nil {
> +        return err
> +    }
> +    return nil
> +}
> +'''
> +
> +TEMPLATE_ALTERNATE = '''
> +// Only implemented on Alternate types that can take JSON NULL as value.
> +//
> +// This is a helper for the marshalling code. It should return true only when
> +// the Alternate is empty (no members are set), otherwise it returns false and
> +// the member set to be Marshalled.
> +type AbsentAlternate interface {
> +       ToAnyOrAbsent() (any, bool)
> +}
> +'''
> +
> +TEMPLATE_ALTERNATE_NULLABLE_CHECK = '''
> +    }} else if s.{var_name} != nil {{
> +        return *s.{var_name}, false'''
> +
> +TEMPLATE_ALTERNATE_MARSHAL_CHECK = '''
> +    if s.{var_name} != nil {{
> +        return json.Marshal(s.{var_name})
> +    }} else '''
> +
> +TEMPLATE_ALTERNATE_UNMARSHAL_CHECK = '''
> +    // Check for {var_type}
> +    {{
> +        s.{var_name} = new({var_type})
> +        if err := StrictDecode(s.{var_name}, data); err == nil {{
> +            return nil
> +        }}
> +        s.{var_name} = nil
> +    }}
> +'''
> +
> +TEMPLATE_ALTERNATE_NULLABLE = '''
> +func (s *{name}) ToAnyOrAbsent() (any, bool) {{
> +    if s != nil {{
> +        if s.IsNull {{
> +            return nil, false
> +{absent_check_fields}
> +        }}
> +    }}
> +
> +    return nil, true
> +}}
> +'''
> +
> +TEMPLATE_ALTERNATE_METHODS = '''
> +func (s {name}) MarshalJSON() ([]byte, error) {{
> +    {marshal_check_fields}
> +    return {marshal_return_default}
> +}}
> +
> +func (s *{name}) UnmarshalJSON(data []byte) error {{
> +    {unmarshal_check_fields}
> +    return fmt.Errorf("Can't convert to {name}: %s", string(data))
> +}}
> +'''
>
>  def gen_golang(schema: QAPISchema,
>                 output_dir: str,
> @@ -46,27 +117,135 @@ def gen_golang(schema: QAPISchema,
>      schema.visit(vis)
>      vis.write(output_dir)
>
> +def qapi_to_field_name(name: str) -> str:
> +    return name.title().replace("_", "").replace("-", "")
>
>  def qapi_to_field_name_enum(name: str) -> str:
>      return name.title().replace("-", "")
>
> +def qapi_schema_type_to_go_type(qapitype: str) -> str:
> +    schema_types_to_go = {
> +            'str': 'string', 'null': 'nil', 'bool': 'bool', 'number':
> +            'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8',
> +            'int16': 'int16', 'int32': 'int32', 'int64': 'int64', 'uint8':
> +            'uint8', 'uint16': 'uint16', 'uint32': 'uint32', 'uint64':
> +            'uint64', 'any': 'any', 'QType': 'QType',
> +    }
> +
> +    prefix = ""
> +    if qapitype.endswith("List"):
> +        prefix = "[]"
> +        qapitype = qapitype[:-4]
> +
> +    qapitype = schema_types_to_go.get(qapitype, qapitype)
> +    return prefix + qapitype
> +
> +def qapi_field_to_go_field(member_name: str, type_name: str) -> Tuple[str, str, str]:
> +    # Nothing to generate on null types. We update some
> +    # variables to handle json-null on marshalling methods.
> +    if type_name == "null":
> +        return "IsNull", "bool", ""
> +
> +    # This function is called on Alternate, so fields should be ptrs
> +    return qapi_to_field_name(member_name), qapi_schema_type_to_go_type(type_name), "*"
> +
> +# Helper function for boxed or self contained structures.
> +def generate_struct_type(type_name, args="") -> str:
> +    args = args if len(args) == 0 else f"\n{args}\n"
> +    with_type = f"\ntype {type_name}" if len(type_name) > 0 else ""
> +    return f'''{with_type} struct {{{args}}}
> +'''
> +
> +def generate_template_alternate(self: QAPISchemaGenGolangVisitor,
> +                                name: str,
> +                                variants: Optional[QAPISchemaVariants]) -> str:
> +    absent_check_fields = ""
> +    variant_fields = ""
> +    # to avoid having to check accept_null_types
> +    nullable = False
> +    if name in self.accept_null_types:
> +        # In QEMU QAPI schema, only StrOrNull and BlockdevRefOrNull.
> +        nullable = True
> +        marshal_return_default = '''[]byte("{}"), nil'''
> +        marshal_check_fields = '''
> +        if s.IsNull {
> +            return []byte("null"), nil
> +        } else '''
> +        unmarshal_check_fields = '''
> +        // Check for json-null first
> +            if string(data) == "null" {
> +                s.IsNull = true
> +                return nil
> +            }
> +        '''
> +    else:
> +        marshal_return_default = f'nil, errors.New("{name} has empty fields")'
> +        marshal_check_fields = ""
> +        unmarshal_check_fields = f'''
> +            // Check for json-null first
> +            if string(data) == "null" {{
> +                return errors.New(`null not supported for {name}`)
> +            }}
> +        '''
> +
> +    for var in variants.variants:

qapi/golang.py:213: error: Item "None" of "QAPISchemaVariants | None"
has no attribute "variants"  [union-attr]

if variants is None (    variants: Optional[QAPISchemaVariants]) we
can't iterate over it without checking to see if it's present first or
not. It may make more sense to change this field to always be an
Iterable and have it default to an empty iterable, but as it is
currently typed we need to check if it's None first.

> +        var_name, var_type, isptr = qapi_field_to_go_field(var.name, var.type.name)
> +        variant_fields += f"\t{var_name} {isptr}{var_type}\n"
> +
> +        # Null is special, handled first
> +        if var.type.name == "null":
> +            assert nullable
> +            continue
> +
> +        if nullable:
> +            absent_check_fields += TEMPLATE_ALTERNATE_NULLABLE_CHECK.format(var_name=var_name)[1:]
> +        marshal_check_fields += TEMPLATE_ALTERNATE_MARSHAL_CHECK.format(var_name=var_name)
> +        unmarshal_check_fields += TEMPLATE_ALTERNATE_UNMARSHAL_CHECK.format(var_name=var_name,
> +                                                                            var_type=var_type)[1:]
> +
> +    content = generate_struct_type(name, variant_fields)
> +    if nullable:
> +        content += TEMPLATE_ALTERNATE_NULLABLE.format(name=name,
> +                                                      absent_check_fields=absent_check_fields)
> +    content += TEMPLATE_ALTERNATE_METHODS.format(name=name,
> +                                                 marshal_check_fields=marshal_check_fields[1:-5],
> +                                                 marshal_return_default=marshal_return_default,
> +                                                 unmarshal_check_fields=unmarshal_check_fields[1:])
> +    return content
> +
>
>  class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
>
>      def __init__(self, _: str):
>          super().__init__()
> -        types = ["enum"]
> +        types = ["alternate", "enum", "helper"]
>          self.target = {name: "" for name in types}
> +        self.objects_seen = {}

qapi/golang.py:256: error: Need type annotation for "objects_seen"
(hint: "objects_seen: Dict[<type>, <type>] = ...")  [var-annotated]

self.objects_seen: Dict[str, bool] = {}

>          self.schema = None
>          self.golang_package_name = "qapi"
> +        self.accept_null_types = []

qapi/golang.py:259: error: Need type annotation for
"accept_null_types" (hint: "accept_null_types: List[<type>] = ...")
[var-annotated]

self.accept_null_types: List[str] = []

>
>      def visit_begin(self, schema):
>          self.schema = schema
>
> +        # We need to be aware of any types that accept JSON NULL
> +        for name, entity in self.schema._entity_dict.items():

We're reaching into the schema's private fields here. I know you
avoided touching the core which Markus liked, but we should look into
giving this a proper interface that we can rely on.

OR, if we all agree that this is fine, and all of this code is going
to *continue living in the same repository for the foreseeable
future*, you can just silence this warning.

jsnow@scv ~/s/q/scripts (review)> pylint qapi --rcfile=qapi/pylintrc
************* Module qapi.golang
qapi/golang.py:265:28: W0212: Access to a protected member
_entity_dict of a client class (protected-access)


for name, entity in self.schema._entity_dict.items():  # pylint:
disable=protected-access

> +            if not isinstance(entity, QAPISchemaAlternateType):
> +                # Assume that only Alternate types accept JSON NULL
> +                continue
> +
> +            for var in  entity.variants.variants:
> +                if var.type.name == 'null':
> +                    self.accept_null_types.append(name)
> +                    break
> +
>          # Every Go file needs to reference its package name
>          for target in self.target:
>              self.target[target] = f"package {self.golang_package_name}\n"
>
> +        self.target["helper"] += TEMPLATE_HELPER
> +        self.target["alternate"] += TEMPLATE_ALTERNATE
> +
>      def visit_end(self):
>          self.schema = None
>
> @@ -88,7 +267,10 @@ def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
>                               features: List[QAPISchemaFeature],
>                               variants: QAPISchemaVariants
>                               ) -> None:
> -        pass
> +        assert name not in self.objects_seen
> +        self.objects_seen[name] = True
> +
> +        self.target["alternate"] += generate_template_alternate(self, name, variants)
>
>      def visit_enum_type(self: QAPISchemaGenGolangVisitor,
>                          name: str,
> --
> 2.41.0
>

flake8 is a little unhappy on this patch. My line numbers here won't
match up because I've been splicing in my own fixes/edits, but here's
the gist:

qapi/golang.py:62:1: W191 indentation contains tabs
qapi/golang.py:62:1: E101 indentation contains mixed spaces and tabs
qapi/golang.py:111:1: E302 expected 2 blank lines, found 1
qapi/golang.py:118:1: E302 expected 2 blank lines, found 1
qapi/golang.py:121:1: E302 expected 2 blank lines, found 1
qapi/golang.py:124:1: E302 expected 2 blank lines, found 1
qapi/golang.py:141:1: E302 expected 2 blank lines, found 1
qapi/golang.py:141:80: E501 line too long (85 > 79 characters)
qapi/golang.py:148:80: E501 line too long (87 > 79 characters)
qapi/golang.py:151:1: E302 expected 2 blank lines, found 1
qapi/golang.py:157:1: E302 expected 2 blank lines, found 1
qapi/golang.py:190:80: E501 line too long (83 > 79 characters)
qapi/golang.py:199:80: E501 line too long (98 > 79 characters)
qapi/golang.py:200:80: E501 line too long (90 > 79 characters)
qapi/golang.py:201:80: E501 line too long (94 > 79 characters)
qapi/golang.py:202:80: E501 line too long (98 > 79 characters)
qapi/golang.py:207:80: E501 line too long (94 > 79 characters)
qapi/golang.py:209:80: E501 line too long (97 > 79 characters)
qapi/golang.py:210:80: E501 line too long (95 > 79 characters)
qapi/golang.py:211:80: E501 line too long (99 > 79 characters)
qapi/golang.py:236:23: E271 multiple spaces after keyword
qapi/golang.py:272:80: E501 line too long (85 > 79 characters)
qapi/golang.py:289:80: E501 line too long (82 > 79 characters)

Mostly just lines being too long and so forth, nothing functional. You
can fix all of this by running black - I didn't use black when I
toured qapi last, but it's grown on me since and I think it does a
reasonable job at applying a braindead standard that you don't have to
think about.

Try it:

jsnow@scv ~/s/q/scripts (review)> black --line-length 79 qapi/golang.py
reformatted qapi/golang.py

All done! ✨ 🍰 ✨
1 file reformatted.
jsnow@scv ~/s/q/scripts (review)> flake8 qapi/golang.py
qapi/golang.py:62:1: W191 indentation contains tabs
qapi/golang.py:62:1: E101 indentation contains mixed spaces and tabs

The remaining tab stuff happens in your templates/comments and doesn't
seem to bother anything, but I think you should fix it for the sake of
the linter tooling in Python if you can.

This is pretty disruptive to the formatting you've chosen so far, but
I think it's a reasonable standard for new code and removes a lot of
the thinking. (like gofmt, right?)

Just keep in mind that our line length limitation for QEMU is a bit
shorter than black's default, so you'll have to tell it the line
length you want each time. It can be made shorter with "-l 79".

Re: [PATCH v1 2/9] qapi: golang: Generate qapi's alternate types in Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 16747 bytes --]

Hi,

On Mon, Oct 02, 2023 at 04:36:11PM -0400, John Snow wrote:
> On Wed, Sep 27, 2023 at 7:25 AM Victor Toso <victortoso@redhat.com> wrote:
> >
> > This patch handles QAPI alternate types and generates data structures
> > in Go that handles it.
> >
> > Alternate types are similar to Union but without a discriminator that
> > can be used to identify the underlying value on the wire. It is needed
> > to infer it. In Go, most of the types [*] are mapped as optional
> > fields and Marshal and Unmarshal methods will be handling the data
> > checks.
> >
> > Example:
> >
> > qapi:
> >   | { 'alternate': 'BlockdevRef',
> >   |   'data': { 'definition': 'BlockdevOptions',
> >   |             'reference': 'str' } }
> >
> > go:
> >   | type BlockdevRef struct {
> >   |         Definition *BlockdevOptions
> >   |         Reference  *string
> >   | }
> >
> > usage:
> >   | input := `{"driver":"qcow2","data-file":"/some/place/my-image"}`
> >   | k := BlockdevRef{}
> >   | err := json.Unmarshal([]byte(input), &k)
> >   | if err != nil {
> >   |     panic(err)
> >   | }
> >   | // *k.Definition.Qcow2.DataFile.Reference == "/some/place/my-image"
> >
> > [*] The exception for optional fields as default is to Types that can
> > accept JSON Null as a value like StrOrNull and BlockdevRefOrNull. For
> > this case, we translate Null with a boolean value in a field called
> > IsNull. This will be explained better in the documentation patch of
> > this series but the main rationale is around Marshaling to and from
> > JSON and Go data structures.
> >
> > Example:
> >
> > qapi:
> >  | { 'alternate': 'StrOrNull',
> >  |   'data': { 's': 'str',
> >  |             'n': 'null' } }
> >
> > go:
> >  | type StrOrNull struct {
> >  |     S      *string
> >  |     IsNull bool
> >  | }
> >
> > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > ---
> >  scripts/qapi/golang.py | 188 ++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 185 insertions(+), 3 deletions(-)
> >
> > diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> > index 87081cdd05..43dbdde14c 100644
> > --- a/scripts/qapi/golang.py
> > +++ b/scripts/qapi/golang.py
> > @@ -16,10 +16,11 @@
> >  from __future__ import annotations
> >
> >  import os
> > -from typing import List, Optional
> > +from typing import Tuple, List, Optional
> >
> >  from .schema import (
> >      QAPISchema,
> > +    QAPISchemaAlternateType,
> >      QAPISchemaType,
> >      QAPISchemaVisitor,
> >      QAPISchemaEnumMember,
> > @@ -38,6 +39,76 @@
> >  )
> >  '''
> >
> > +TEMPLATE_HELPER = '''
> > +// Creates a decoder that errors on unknown Fields
> > +// Returns nil if successfully decoded @from payload to @into type
> > +// Returns error if failed to decode @from payload to @into type
> > +func StrictDecode(into interface{}, from []byte) error {
> > +    dec := json.NewDecoder(strings.NewReader(string(from)))
> > +    dec.DisallowUnknownFields()
> > +
> > +    if err := dec.Decode(into); err != nil {
> > +        return err
> > +    }
> > +    return nil
> > +}
> > +'''
> > +
> > +TEMPLATE_ALTERNATE = '''
> > +// Only implemented on Alternate types that can take JSON NULL as value.
> > +//
> > +// This is a helper for the marshalling code. It should return true only when
> > +// the Alternate is empty (no members are set), otherwise it returns false and
> > +// the member set to be Marshalled.
> > +type AbsentAlternate interface {
> > +       ToAnyOrAbsent() (any, bool)
> > +}
> > +'''
> > +
> > +TEMPLATE_ALTERNATE_NULLABLE_CHECK = '''
> > +    }} else if s.{var_name} != nil {{
> > +        return *s.{var_name}, false'''
> > +
> > +TEMPLATE_ALTERNATE_MARSHAL_CHECK = '''
> > +    if s.{var_name} != nil {{
> > +        return json.Marshal(s.{var_name})
> > +    }} else '''
> > +
> > +TEMPLATE_ALTERNATE_UNMARSHAL_CHECK = '''
> > +    // Check for {var_type}
> > +    {{
> > +        s.{var_name} = new({var_type})
> > +        if err := StrictDecode(s.{var_name}, data); err == nil {{
> > +            return nil
> > +        }}
> > +        s.{var_name} = nil
> > +    }}
> > +'''
> > +
> > +TEMPLATE_ALTERNATE_NULLABLE = '''
> > +func (s *{name}) ToAnyOrAbsent() (any, bool) {{
> > +    if s != nil {{
> > +        if s.IsNull {{
> > +            return nil, false
> > +{absent_check_fields}
> > +        }}
> > +    }}
> > +
> > +    return nil, true
> > +}}
> > +'''
> > +
> > +TEMPLATE_ALTERNATE_METHODS = '''
> > +func (s {name}) MarshalJSON() ([]byte, error) {{
> > +    {marshal_check_fields}
> > +    return {marshal_return_default}
> > +}}
> > +
> > +func (s *{name}) UnmarshalJSON(data []byte) error {{
> > +    {unmarshal_check_fields}
> > +    return fmt.Errorf("Can't convert to {name}: %s", string(data))
> > +}}
> > +'''
> >
> >  def gen_golang(schema: QAPISchema,
> >                 output_dir: str,
> > @@ -46,27 +117,135 @@ def gen_golang(schema: QAPISchema,
> >      schema.visit(vis)
> >      vis.write(output_dir)
> >
> > +def qapi_to_field_name(name: str) -> str:
> > +    return name.title().replace("_", "").replace("-", "")
> >
> >  def qapi_to_field_name_enum(name: str) -> str:
> >      return name.title().replace("-", "")
> >
> > +def qapi_schema_type_to_go_type(qapitype: str) -> str:
> > +    schema_types_to_go = {
> > +            'str': 'string', 'null': 'nil', 'bool': 'bool', 'number':
> > +            'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8',
> > +            'int16': 'int16', 'int32': 'int32', 'int64': 'int64', 'uint8':
> > +            'uint8', 'uint16': 'uint16', 'uint32': 'uint32', 'uint64':
> > +            'uint64', 'any': 'any', 'QType': 'QType',
> > +    }
> > +
> > +    prefix = ""
> > +    if qapitype.endswith("List"):
> > +        prefix = "[]"
> > +        qapitype = qapitype[:-4]
> > +
> > +    qapitype = schema_types_to_go.get(qapitype, qapitype)
> > +    return prefix + qapitype
> > +
> > +def qapi_field_to_go_field(member_name: str, type_name: str) -> Tuple[str, str, str]:
> > +    # Nothing to generate on null types. We update some
> > +    # variables to handle json-null on marshalling methods.
> > +    if type_name == "null":
> > +        return "IsNull", "bool", ""
> > +
> > +    # This function is called on Alternate, so fields should be ptrs
> > +    return qapi_to_field_name(member_name), qapi_schema_type_to_go_type(type_name), "*"
> > +
> > +# Helper function for boxed or self contained structures.
> > +def generate_struct_type(type_name, args="") -> str:
> > +    args = args if len(args) == 0 else f"\n{args}\n"
> > +    with_type = f"\ntype {type_name}" if len(type_name) > 0 else ""
> > +    return f'''{with_type} struct {{{args}}}
> > +'''
> > +
> > +def generate_template_alternate(self: QAPISchemaGenGolangVisitor,
> > +                                name: str,
> > +                                variants: Optional[QAPISchemaVariants]) -> str:
> > +    absent_check_fields = ""
> > +    variant_fields = ""
> > +    # to avoid having to check accept_null_types
> > +    nullable = False
> > +    if name in self.accept_null_types:
> > +        # In QEMU QAPI schema, only StrOrNull and BlockdevRefOrNull.
> > +        nullable = True
> > +        marshal_return_default = '''[]byte("{}"), nil'''
> > +        marshal_check_fields = '''
> > +        if s.IsNull {
> > +            return []byte("null"), nil
> > +        } else '''
> > +        unmarshal_check_fields = '''
> > +        // Check for json-null first
> > +            if string(data) == "null" {
> > +                s.IsNull = true
> > +                return nil
> > +            }
> > +        '''
> > +    else:
> > +        marshal_return_default = f'nil, errors.New("{name} has empty fields")'
> > +        marshal_check_fields = ""
> > +        unmarshal_check_fields = f'''
> > +            // Check for json-null first
> > +            if string(data) == "null" {{
> > +                return errors.New(`null not supported for {name}`)
> > +            }}
> > +        '''
> > +
> > +    for var in variants.variants:
> 
> qapi/golang.py:213: error: Item "None" of "QAPISchemaVariants | None"
> has no attribute "variants"  [union-attr]
> 
> if variants is None (    variants: Optional[QAPISchemaVariants]) we
> can't iterate over it without checking to see if it's present first or
> not. It may make more sense to change this field to always be an
> Iterable and have it default to an empty iterable, but as it is
> currently typed we need to check if it's None first.

Sure,
 
> > +        var_name, var_type, isptr = qapi_field_to_go_field(var.name, var.type.name)
> > +        variant_fields += f"\t{var_name} {isptr}{var_type}\n"
> > +
> > +        # Null is special, handled first
> > +        if var.type.name == "null":
> > +            assert nullable
> > +            continue
> > +
> > +        if nullable:
> > +            absent_check_fields += TEMPLATE_ALTERNATE_NULLABLE_CHECK.format(var_name=var_name)[1:]
> > +        marshal_check_fields += TEMPLATE_ALTERNATE_MARSHAL_CHECK.format(var_name=var_name)
> > +        unmarshal_check_fields += TEMPLATE_ALTERNATE_UNMARSHAL_CHECK.format(var_name=var_name,
> > +                                                                            var_type=var_type)[1:]
> > +
> > +    content = generate_struct_type(name, variant_fields)
> > +    if nullable:
> > +        content += TEMPLATE_ALTERNATE_NULLABLE.format(name=name,
> > +                                                      absent_check_fields=absent_check_fields)
> > +    content += TEMPLATE_ALTERNATE_METHODS.format(name=name,
> > +                                                 marshal_check_fields=marshal_check_fields[1:-5],
> > +                                                 marshal_return_default=marshal_return_default,
> > +                                                 unmarshal_check_fields=unmarshal_check_fields[1:])
> > +    return content
> > +
> >
> >  class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> >
> >      def __init__(self, _: str):
> >          super().__init__()
> > -        types = ["enum"]
> > +        types = ["alternate", "enum", "helper"]
> >          self.target = {name: "" for name in types}
> > +        self.objects_seen = {}
> 
> qapi/golang.py:256: error: Need type annotation for "objects_seen"
> (hint: "objects_seen: Dict[<type>, <type>] = ...")  [var-annotated]
> 
> self.objects_seen: Dict[str, bool] = {}
> 
> >          self.schema = None
> >          self.golang_package_name = "qapi"
> > +        self.accept_null_types = []
> 
> qapi/golang.py:259: error: Need type annotation for
> "accept_null_types" (hint: "accept_null_types: List[<type>] = ...")
> [var-annotated]
> 
> self.accept_null_types: List[str] = []
> 
> >
> >      def visit_begin(self, schema):
> >          self.schema = schema
> >
> > +        # We need to be aware of any types that accept JSON NULL
> > +        for name, entity in self.schema._entity_dict.items():
> 
> We're reaching into the schema's private fields here. I know you
> avoided touching the core which Markus liked, but we should look into
> giving this a proper interface that we can rely on.
> 
> OR, if we all agree that this is fine, and all of this code is
> going to *continue living in the same repository for the
> foreseeable future*, you can just silence this warning.
> 
> jsnow@scv ~/s/q/scripts (review)> pylint qapi --rcfile=qapi/pylintrc
> ************* Module qapi.golang
> qapi/golang.py:265:28: W0212: Access to a protected member
> _entity_dict of a client class (protected-access)
> 
> 
> for name, entity in self.schema._entity_dict.items():  # pylint:
> disable=protected-access

Right. Here I knew it was somewhat bad. It is up to you/Markus. I
can introduce a proper interface in the schema as a preparatory
patch to this one, or we mark this as a problem for the future,
for sure there is no intention to detach this from this repo,
specifically scripts/qapi.

It depends on what you think is best.

> > +            if not isinstance(entity, QAPISchemaAlternateType):
> > +                # Assume that only Alternate types accept JSON NULL
> > +                continue
> > +
> > +            for var in  entity.variants.variants:
> > +                if var.type.name == 'null':
> > +                    self.accept_null_types.append(name)
> > +                    break
> > +
> >          # Every Go file needs to reference its package name
> >          for target in self.target:
> >              self.target[target] = f"package {self.golang_package_name}\n"
> >
> > +        self.target["helper"] += TEMPLATE_HELPER
> > +        self.target["alternate"] += TEMPLATE_ALTERNATE
> > +
> >      def visit_end(self):
> >          self.schema = None
> >
> > @@ -88,7 +267,10 @@ def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
> >                               features: List[QAPISchemaFeature],
> >                               variants: QAPISchemaVariants
> >                               ) -> None:
> > -        pass
> > +        assert name not in self.objects_seen
> > +        self.objects_seen[name] = True
> > +
> > +        self.target["alternate"] += generate_template_alternate(self, name, variants)
> >
> >      def visit_enum_type(self: QAPISchemaGenGolangVisitor,
> >                          name: str,
> > --
> > 2.41.0
> >
> 
> flake8 is a little unhappy on this patch. My line numbers here won't
> match up because I've been splicing in my own fixes/edits, but here's
> the gist:
> 
> qapi/golang.py:62:1: W191 indentation contains tabs
> qapi/golang.py:62:1: E101 indentation contains mixed spaces and tabs
> qapi/golang.py:111:1: E302 expected 2 blank lines, found 1
> qapi/golang.py:118:1: E302 expected 2 blank lines, found 1
> qapi/golang.py:121:1: E302 expected 2 blank lines, found 1
> qapi/golang.py:124:1: E302 expected 2 blank lines, found 1
> qapi/golang.py:141:1: E302 expected 2 blank lines, found 1
> qapi/golang.py:141:80: E501 line too long (85 > 79 characters)
> qapi/golang.py:148:80: E501 line too long (87 > 79 characters)
> qapi/golang.py:151:1: E302 expected 2 blank lines, found 1
> qapi/golang.py:157:1: E302 expected 2 blank lines, found 1
> qapi/golang.py:190:80: E501 line too long (83 > 79 characters)
> qapi/golang.py:199:80: E501 line too long (98 > 79 characters)
> qapi/golang.py:200:80: E501 line too long (90 > 79 characters)
> qapi/golang.py:201:80: E501 line too long (94 > 79 characters)
> qapi/golang.py:202:80: E501 line too long (98 > 79 characters)
> qapi/golang.py:207:80: E501 line too long (94 > 79 characters)
> qapi/golang.py:209:80: E501 line too long (97 > 79 characters)
> qapi/golang.py:210:80: E501 line too long (95 > 79 characters)
> qapi/golang.py:211:80: E501 line too long (99 > 79 characters)
> qapi/golang.py:236:23: E271 multiple spaces after keyword
> qapi/golang.py:272:80: E501 line too long (85 > 79 characters)
> qapi/golang.py:289:80: E501 line too long (82 > 79 characters)
> 
> Mostly just lines being too long and so forth, nothing
> functional. You can fix all of this by running black - I didn't
> use black when I toured qapi last, but it's grown on me since
> and I think it does a reasonable job at applying a braindead
> standard that you don't have to think about.
> 
> Try it:
> 
> jsnow@scv ~/s/q/scripts (review)> black --line-length 79 qapi/golang.py
> reformatted qapi/golang.py
> 
> All done! ✨ 🍰 ✨
> 1 file reformatted.
> jsnow@scv ~/s/q/scripts (review)> flake8 qapi/golang.py
> qapi/golang.py:62:1: W191 indentation contains tabs
> qapi/golang.py:62:1: E101 indentation contains mixed spaces and tabs
> 
> The remaining tab stuff happens in your templates/comments and doesn't
> seem to bother anything, but I think you should fix it for the sake of
> the linter tooling in Python if you can.
> 
> This is pretty disruptive to the formatting you've chosen so far, but
> I think it's a reasonable standard for new code and removes a lot of
> the thinking. (like gofmt, right?)
> 
> Just keep in mind that our line length limitation for QEMU is a bit
> shorter than black's default, so you'll have to tell it the line
> length you want each time. It can be made shorter with "-l 79".

Awesome. I didn't know this tool. I'll apply it to all patches
for v2, fixing all python tooling that you've mention that throws
a warning at me.

Thanks for the patience!

Cheers,
Victor

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v1 3/9] qapi: golang: Generate qapi's struct types in Go

[Victor Toso]

This patch handles QAPI struct types and generates the equivalent
types in Go. The following patch adds extra logic when a member of the
struct has a Type that can take JSON Null value (e.g: StrOrNull in
QEMU)

The highlights of this implementation are:

1. Generating an Go struct that requires a @base type, the @base type
   fields are copied over to the Go struct. The advantage of this
   approach is to not have embed structs in any of the QAPI types.
   Note that embedding a @base type is recursive, that is, if the
   @base type has a @base, all of those fields will be copied over.

2. About the Go struct's fields:

   i) They can be either by Value or Reference.

  ii) Every field that is marked as optional in the QAPI specification
      are translated to Reference fields in its Go structure. This
      design decision is the most straightforward way to check if a
      given field was set or not. Exception only for types that can
      take JSON Null value.

 iii) Mandatory fields are always by Value with the exception of QAPI
      arrays, which are handled by Reference (to a block of memory) by
      Go.

  iv) All the fields are named with Uppercase due Golang's export
      convention.

   v) In order to avoid any kind of issues when encoding or decoding,
      to or from JSON, we mark all fields with its @name and, when it is
      optional, member, with @omitempty

Example:

qapi:
 | { 'struct': 'BlockdevCreateOptionsFile',
 |   'data': { 'filename': 'str',
 |             'size': 'size',
 |             '*preallocation': 'PreallocMode',
 |             '*nocow': 'bool',
 |             '*extent-size-hint': 'size'} }

go:
| type BlockdevCreateOptionsFile struct {
|     Filename       string        `json:"filename"`
|     Size           uint64        `json:"size"`
|     Preallocation  *PreallocMode `json:"preallocation,omitempty"`
|     Nocow          *bool         `json:"nocow,omitempty"`
|     ExtentSizeHint *uint64       `json:"extent-size-hint,omitempty"`
| }

Signed-off-by: Victor Toso <victortoso@redhat.com>
---
 scripts/qapi/golang.py | 138 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 136 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 43dbdde14c..1b19e4b232 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -117,12 +117,35 @@ def gen_golang(schema: QAPISchema,
     schema.visit(vis)
     vis.write(output_dir)
 
+def qapi_name_is_base(name: str) -> bool:
+    return qapi_name_is_object(name) and name.endswith("-base")
+
+def qapi_name_is_object(name: str) -> bool:
+    return name.startswith("q_obj_")
+
 def qapi_to_field_name(name: str) -> str:
     return name.title().replace("_", "").replace("-", "")
 
 def qapi_to_field_name_enum(name: str) -> str:
     return name.title().replace("-", "")
 
+def qapi_to_go_type_name(name: str) -> str:
+    if qapi_name_is_object(name):
+        name = name[6:]
+
+    # We want to keep CamelCase for Golang types. We want to avoid removing
+    # already set CameCase names while fixing uppercase ones, eg:
+    # 1) q_obj_SocketAddress_base -> SocketAddressBase
+    # 2) q_obj_WATCHDOG-arg -> WatchdogArg
+    words = list(name.replace("_", "-").split("-"))
+    name = words[0]
+    if name.islower() or name.isupper():
+        name = name.title()
+
+    name += ''.join(word.title() for word in words[1:])
+
+    return name
+
 def qapi_schema_type_to_go_type(qapitype: str) -> str:
     schema_types_to_go = {
             'str': 'string', 'null': 'nil', 'bool': 'bool', 'number':
@@ -156,6 +179,82 @@ def generate_struct_type(type_name, args="") -> str:
     return f'''{with_type} struct {{{args}}}
 '''
 
+def get_struct_field(self: QAPISchemaGenGolangVisitor,
+                     qapi_name: str,
+                     qapi_type_name: str,
+                     is_optional: bool,
+                     is_variant: bool) -> str:
+
+    field = qapi_to_field_name(qapi_name)
+    member_type = qapi_schema_type_to_go_type(qapi_type_name)
+
+    optional = ""
+    if is_optional:
+        if member_type not in self.accept_null_types:
+            optional = ",omitempty"
+
+    # Use pointer to type when field is optional
+    isptr = "*" if is_optional and member_type[0] not in "*[" else ""
+
+    fieldtag = '`json:"-"`' if is_variant else f'`json:"{qapi_name}{optional}"`'
+    return f"\t{field} {isptr}{member_type}{fieldtag}\n"
+
+def recursive_base(self: QAPISchemaGenGolangVisitor,
+                   base: Optional[QAPISchemaObjectType]) -> str:
+    fields = ""
+
+    if not base:
+        return fields
+
+    if base.base is not None:
+        embed_base = self.schema.lookup_entity(base.base.name)
+        fields = recursive_base(self, embed_base)
+
+    for member in base.local_members:
+        if base.variants and base.variants.tag_member.name == member.name:
+            fields += '''// Discriminator\n'''
+
+        field = get_struct_field(self, member.name, member.type.name, member.optional, False)
+        fields += field
+
+    if len(fields) > 0:
+        fields += "\n"
+
+    return fields
+
+# Helper function that is used for most of QAPI types
+def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
+                          name: str,
+                          _: Optional[QAPISourceInfo],
+                          __: QAPISchemaIfCond,
+                          ___: List[QAPISchemaFeature],
+                          base: Optional[QAPISchemaObjectType],
+                          members: List[QAPISchemaObjectTypeMember],
+                          variants: Optional[QAPISchemaVariants]) -> str:
+
+
+    fields = recursive_base(self, base)
+
+    if members:
+        for member in members:
+            field = get_struct_field(self, member.name, member.type.name, member.optional, False)
+            fields += field
+
+        fields += "\n"
+
+    if variants:
+        fields += "\t// Variants fields\n"
+        for variant in variants.variants:
+            if variant.type.is_implicit():
+                continue
+
+            field = get_struct_field(self, variant.name, variant.type.name, True, True)
+            fields += field
+
+    type_name = qapi_to_go_type_name(name)
+    content = generate_struct_type(type_name, fields)
+    return content
+
 def generate_template_alternate(self: QAPISchemaGenGolangVisitor,
                                 name: str,
                                 variants: Optional[QAPISchemaVariants]) -> str:
@@ -218,7 +317,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, _: str):
         super().__init__()
-        types = ["alternate", "enum", "helper"]
+        types = ["alternate", "enum", "helper", "struct"]
         self.target = {name: "" for name in types}
         self.objects_seen = {}
         self.schema = None
@@ -258,7 +357,42 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor,
                           members: List[QAPISchemaObjectTypeMember],
                           variants: Optional[QAPISchemaVariants]
                           ) -> None:
-        pass
+        # Do not handle anything besides struct.
+        if (name == self.schema.the_empty_object_type.name or
+                not isinstance(name, str) or
+                info.defn_meta not in ["struct"]):
+            return
+
+        # Base structs are embed
+        if qapi_name_is_base(name):
+            return
+
+        # Safety checks.
+        assert name not in self.objects_seen
+        self.objects_seen[name] = True
+
+        # visit all inner objects as well, they are not going to be
+        # called by python's generator.
+        if variants:
+            for var in variants.variants:
+                assert isinstance(var.type, QAPISchemaObjectType)
+                self.visit_object_type(self,
+                                       var.type.name,
+                                       var.type.info,
+                                       var.type.ifcond,
+                                       var.type.base,
+                                       var.type.local_members,
+                                       var.type.variants)
+
+        # Save generated Go code to be written later
+        self.target[info.defn_meta] += qapi_to_golang_struct(self,
+                                                             name,
+                                                             info,
+                                                             ifcond,
+                                                             features,
+                                                             base,
+                                                             members,
+                                                             variants)
 
     def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
                              name: str,
-- 
2.41.0

Re: [PATCH v1 3/9] qapi: golang: Generate qapi's struct types in Go

[Daniel P. Berrangé]

On Wed, Sep 27, 2023 at 01:25:38PM +0200, Victor Toso wrote:
> This patch handles QAPI struct types and generates the equivalent
> types in Go. The following patch adds extra logic when a member of the
> struct has a Type that can take JSON Null value (e.g: StrOrNull in
> QEMU)
> 
> The highlights of this implementation are:
> 
> 1. Generating an Go struct that requires a @base type, the @base type
>    fields are copied over to the Go struct. The advantage of this
>    approach is to not have embed structs in any of the QAPI types.
>    Note that embedding a @base type is recursive, that is, if the
>    @base type has a @base, all of those fields will be copied over.
> 
> 2. About the Go struct's fields:
> 
>    i) They can be either by Value or Reference.
> 
>   ii) Every field that is marked as optional in the QAPI specification
>       are translated to Reference fields in its Go structure. This
>       design decision is the most straightforward way to check if a
>       given field was set or not. Exception only for types that can
>       take JSON Null value.
> 
>  iii) Mandatory fields are always by Value with the exception of QAPI
>       arrays, which are handled by Reference (to a block of memory) by
>       Go.
> 
>   iv) All the fields are named with Uppercase due Golang's export
>       convention.
> 
>    v) In order to avoid any kind of issues when encoding or decoding,
>       to or from JSON, we mark all fields with its @name and, when it is
>       optional, member, with @omitempty
> 
> Example:
> 
> qapi:
>  | { 'struct': 'BlockdevCreateOptionsFile',
>  |   'data': { 'filename': 'str',
>  |             'size': 'size',
>  |             '*preallocation': 'PreallocMode',
>  |             '*nocow': 'bool',
>  |             '*extent-size-hint': 'size'} }
> 
> go:
> | type BlockdevCreateOptionsFile struct {
> |     Filename       string        `json:"filename"`
> |     Size           uint64        `json:"size"`
> |     Preallocation  *PreallocMode `json:"preallocation,omitempty"`
> |     Nocow          *bool         `json:"nocow,omitempty"`
> |     ExtentSizeHint *uint64       `json:"extent-size-hint,omitempty"`
> | }

Note, 'omitempty' shouldn't be used on pointer fields, only scalar
fields. The pointer fields are always omitted when nil.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 3/9] qapi: golang: Generate qapi's struct types in Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 3742 bytes --]

Hi,

On Thu, Sep 28, 2023 at 03:06:23PM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 27, 2023 at 01:25:38PM +0200, Victor Toso wrote:
> > This patch handles QAPI struct types and generates the equivalent
> > types in Go. The following patch adds extra logic when a member of the
> > struct has a Type that can take JSON Null value (e.g: StrOrNull in
> > QEMU)
> > 
> > The highlights of this implementation are:
> > 
> > 1. Generating an Go struct that requires a @base type, the @base type
> >    fields are copied over to the Go struct. The advantage of this
> >    approach is to not have embed structs in any of the QAPI types.
> >    Note that embedding a @base type is recursive, that is, if the
> >    @base type has a @base, all of those fields will be copied over.
> > 
> > 2. About the Go struct's fields:
> > 
> >    i) They can be either by Value or Reference.
> > 
> >   ii) Every field that is marked as optional in the QAPI specification
> >       are translated to Reference fields in its Go structure. This
> >       design decision is the most straightforward way to check if a
> >       given field was set or not. Exception only for types that can
> >       take JSON Null value.
> > 
> >  iii) Mandatory fields are always by Value with the exception of QAPI
> >       arrays, which are handled by Reference (to a block of memory) by
> >       Go.
> > 
> >   iv) All the fields are named with Uppercase due Golang's export
> >       convention.
> > 
> >    v) In order to avoid any kind of issues when encoding or decoding,
> >       to or from JSON, we mark all fields with its @name and, when it is
> >       optional, member, with @omitempty
> > 
> > Example:
> > 
> > qapi:
> >  | { 'struct': 'BlockdevCreateOptionsFile',
> >  |   'data': { 'filename': 'str',
> >  |             'size': 'size',
> >  |             '*preallocation': 'PreallocMode',
> >  |             '*nocow': 'bool',
> >  |             '*extent-size-hint': 'size'} }
> > 
> > go:
> > | type BlockdevCreateOptionsFile struct {
> > |     Filename       string        `json:"filename"`
> > |     Size           uint64        `json:"size"`
> > |     Preallocation  *PreallocMode `json:"preallocation,omitempty"`
> > |     Nocow          *bool         `json:"nocow,omitempty"`
> > |     ExtentSizeHint *uint64       `json:"extent-size-hint,omitempty"`
> > | }
> 
> Note, 'omitempty' shouldn't be used on pointer fields, only
> scalar fields. The pointer fields are always omitted when nil.

'omitempty' should be used with pointer fields unless you want to
Marshal JSON Null, which is not the expected output.

'omitempty' is used when QAPI member is said to be optional. This
is true for all optional members with the exception of two
Alternates: StrOrNull and BlockdevRefOrNull, as we _do_ want to
express JSON Null with them.

```Go
    package main

    import (
        "encoding/json"
        "fmt"
    )

    type Test1 struct {
        Foo *bool `json:"foo"`
        Bar *bool `json:"bar,omitempty"`
        Esc bool  `json:"esc"`
        Lar bool  `json:"lar,omitempty"`
    }

    type Test2 struct {
        Foo *uint64 `json:"foo"`
        Bar *uint64 `json:"bar,omitempty"`
        Esc uint64  `json:"esc"`
        Lar uint64  `json:"lar,omitempty"`
    }

    func printIt(s any) {
        if b, err := json.Marshal(s); err != nil {
            panic(err)
        } else {
            fmt.Println(string(b))
        }
    }

    func main() {
        printIt(Test1{})
        printIt(Test2{})
    }
```

```console
    toso@tapioca /tmp> go run main.go
    {"foo":null,"esc":false}
    {"foo":null,"esc":0}
```

Cheers,
Victor

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1 3/9] qapi: golang: Generate qapi's struct types in Go

[Daniel P. Berrangé]

On Fri, Sep 29, 2023 at 03:29:34PM +0200, Victor Toso wrote:
> Hi,
> 
> On Thu, Sep 28, 2023 at 03:06:23PM +0100, Daniel P. Berrangé wrote:
> > On Wed, Sep 27, 2023 at 01:25:38PM +0200, Victor Toso wrote:
> > > This patch handles QAPI struct types and generates the equivalent
> > > types in Go. The following patch adds extra logic when a member of the
> > > struct has a Type that can take JSON Null value (e.g: StrOrNull in
> > > QEMU)
> > > 
> > > The highlights of this implementation are:
> > > 
> > > 1. Generating an Go struct that requires a @base type, the @base type
> > >    fields are copied over to the Go struct. The advantage of this
> > >    approach is to not have embed structs in any of the QAPI types.
> > >    Note that embedding a @base type is recursive, that is, if the
> > >    @base type has a @base, all of those fields will be copied over.
> > > 
> > > 2. About the Go struct's fields:
> > > 
> > >    i) They can be either by Value or Reference.
> > > 
> > >   ii) Every field that is marked as optional in the QAPI specification
> > >       are translated to Reference fields in its Go structure. This
> > >       design decision is the most straightforward way to check if a
> > >       given field was set or not. Exception only for types that can
> > >       take JSON Null value.
> > > 
> > >  iii) Mandatory fields are always by Value with the exception of QAPI
> > >       arrays, which are handled by Reference (to a block of memory) by
> > >       Go.
> > > 
> > >   iv) All the fields are named with Uppercase due Golang's export
> > >       convention.
> > > 
> > >    v) In order to avoid any kind of issues when encoding or decoding,
> > >       to or from JSON, we mark all fields with its @name and, when it is
> > >       optional, member, with @omitempty
> > > 
> > > Example:
> > > 
> > > qapi:
> > >  | { 'struct': 'BlockdevCreateOptionsFile',
> > >  |   'data': { 'filename': 'str',
> > >  |             'size': 'size',
> > >  |             '*preallocation': 'PreallocMode',
> > >  |             '*nocow': 'bool',
> > >  |             '*extent-size-hint': 'size'} }
> > > 
> > > go:
> > > | type BlockdevCreateOptionsFile struct {
> > > |     Filename       string        `json:"filename"`
> > > |     Size           uint64        `json:"size"`
> > > |     Preallocation  *PreallocMode `json:"preallocation,omitempty"`
> > > |     Nocow          *bool         `json:"nocow,omitempty"`
> > > |     ExtentSizeHint *uint64       `json:"extent-size-hint,omitempty"`
> > > | }
> > 
> > Note, 'omitempty' shouldn't be used on pointer fields, only
> > scalar fields. The pointer fields are always omitted when nil.
> 
> 'omitempty' should be used with pointer fields unless you want to
> Marshal JSON Null, which is not the expected output.

Oh doh, did't notice that.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH v1 4/9] qapi: golang: structs: Address 'null' members

[Victor Toso]

Explaining why this is needed needs some context, so taking the
example of StrOrNull alternate type and considering a simplified
struct that has two fields:

qapi:
 | { 'struct': 'MigrationExample',
 |   'data': { '*label': 'StrOrNull',
 |             'target': 'StrOrNull' } }

We have a optional member 'label' which can have three JSON values:
 1. A string: { "label": "happy" }
 2. A null  : { "label": null }
 3. Absent  : {}

The member 'target' is not optional, hence it can't be absent.

A Go struct that contains a optional type that can be JSON Null like
'label' in the example above, will need extra care when Marshaling
and Unmarshaling from JSON.

This patch handles this very specific case:
 - It implements the Marshaler interface for these structs to properly
   handle these values.
 - It adds the interface AbsentAlternate() and implement it for any
   Alternate that can be JSON Null

Signed-off-by: Victor Toso <victortoso@redhat.com>
---
 scripts/qapi/golang.py | 195 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 184 insertions(+), 11 deletions(-)

diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 1b19e4b232..8320af99b6 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -110,6 +110,27 @@
 }}
 '''
 
+TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL = '''
+func (s {type_name}) MarshalJSON() ([]byte, error) {{
+    m := make(map[string]any)
+    {map_members}
+    {map_special}
+    return json.Marshal(&m)
+}}
+
+func (s *{type_name}) UnmarshalJSON(data []byte) error {{
+    tmp := {struct}{{}}
+
+    if err := json.Unmarshal(data, &tmp); err != nil {{
+        return err
+    }}
+
+    {set_members}
+    {set_special}
+    return nil
+}}
+'''
+
 def gen_golang(schema: QAPISchema,
                output_dir: str,
                prefix: str) -> None:
@@ -182,45 +203,187 @@ def generate_struct_type(type_name, args="") -> str:
 def get_struct_field(self: QAPISchemaGenGolangVisitor,
                      qapi_name: str,
                      qapi_type_name: str,
+                     within_nullable_struct: bool,
                      is_optional: bool,
-                     is_variant: bool) -> str:
+                     is_variant: bool) -> Tuple[str, bool]:
 
     field = qapi_to_field_name(qapi_name)
     member_type = qapi_schema_type_to_go_type(qapi_type_name)
+    is_nullable = False
 
     optional = ""
     if is_optional:
-        if member_type not in self.accept_null_types:
+        if member_type in self.accept_null_types:
+            is_nullable = True
+        else:
             optional = ",omitempty"
 
     # Use pointer to type when field is optional
     isptr = "*" if is_optional and member_type[0] not in "*[" else ""
 
+    if within_nullable_struct:
+        # Within a struct which has a field of type that can hold JSON NULL,
+        # we have to _not_ use a pointer, otherwise the Marshal methods are
+        # not called.
+        isptr = "" if member_type in self.accept_null_types else isptr
+
     fieldtag = '`json:"-"`' if is_variant else f'`json:"{qapi_name}{optional}"`'
-    return f"\t{field} {isptr}{member_type}{fieldtag}\n"
+    return f"\t{field} {isptr}{member_type}{fieldtag}\n", is_nullable
+
+# This helper is used whithin a struct that has members that accept JSON NULL.
+def map_and_set(is_nullable: bool,
+                field: str,
+                field_is_optional: bool,
+                name: str) -> Tuple[str, str]:
+
+    mapstr = ""
+    setstr = ""
+    if is_nullable:
+        mapstr = f'''
+    if val, absent := s.{field}.ToAnyOrAbsent(); !absent {{
+        m["{name}"] = val
+    }}
+'''
+        setstr += f'''
+    if _, absent := (&tmp.{field}).ToAnyOrAbsent(); !absent {{
+        s.{field} = &tmp.{field}
+    }}
+'''
+    elif field_is_optional:
+        mapstr = f'''
+    if s.{field} != nil {{
+        m["{name}"] = s.{field}
+    }}
+'''
+        setstr = f'''\ts.{field} = tmp.{field}\n'''
+    else:
+        mapstr = f'''\tm["{name}"] = s.{field}\n'''
+        setstr = f'''\ts.{field} = tmp.{field}\n'''
+
+    return mapstr, setstr
+
+def recursive_base_nullable(self: QAPISchemaGenGolangVisitor,
+                            base: Optional[QAPISchemaObjectType]) -> Tuple[str, str, str, str, str]:
+    fields = ""
+    map_members = ""
+    set_members = ""
+    map_special = ""
+    set_special = ""
+
+    if not base:
+        return fields, map_members, set_members, map_special, set_special
+
+    if base.base is not None:
+        embed_base = self.schema.lookup_entity(base.base.name)
+        fields, map_members, set_members, map_special, set_special = recursive_base_nullable(self, embed_base)
+
+    for member in base.local_members:
+        field, _ = get_struct_field(self, member.name, member.type.name,
+                                    True, member.optional, False)
+        fields += field
+
+        member_type = qapi_schema_type_to_go_type(member.type.name)
+        nullable = member_type in self.accept_null_types
+        field_name = qapi_to_field_name(member.name)
+        tomap, toset = map_and_set(nullable, field_name, member.optional, member.name)
+        if nullable:
+            map_special += tomap
+            set_special += toset
+        else:
+            map_members += tomap
+            set_members += toset
+
+    return fields, map_members, set_members, map_special, set_special
+
+# Helper function. This is executed when the QAPI schema has members
+# that could accept JSON NULL (e.g: StrOrNull in QEMU"s QAPI schema).
+# This struct will need to be extended with Marshal/Unmarshal methods to
+# properly handle such atypical members.
+#
+# Only the Marshallaing methods are generated but we do need to iterate over
+# all the members to properly set/check them in those methods.
+def struct_with_nullable_generate_marshal(self: QAPISchemaGenGolangVisitor,
+                                          name: str,
+                                          base: Optional[QAPISchemaObjectType],
+                                          members: List[QAPISchemaObjectTypeMember],
+                                          variants: Optional[QAPISchemaVariants]) -> str:
+
+
+    fields, map_members, set_members, map_special, set_special = recursive_base_nullable(self, base)
+
+    if members:
+        for member in members:
+            field, _ = get_struct_field(self, member.name, member.type.name,
+                                        True, member.optional, False)
+            fields += field
+
+            member_type = qapi_schema_type_to_go_type(member.type.name)
+            nullable = member_type in self.accept_null_types
+            tomap, toset = map_and_set(nullable, qapi_to_field_name(member.name),
+                                       member.optional, member.name)
+            if nullable:
+                map_special += tomap
+                set_special += toset
+            else:
+                map_members += tomap
+                set_members += toset
+
+        fields += "\n"
+
+    if variants:
+        for variant in variants.variants:
+            if variant.type.is_implicit():
+                continue
+
+            field, _ = get_struct_field(self, variant.name, variant.type.name,
+                                        True, variant.optional, True)
+            fields += field
+
+            member_type = qapi_schema_type_to_go_type(variant.type.name)
+            nullable = member_type in self.accept_null_types
+            tomap, toset = map_and_set(nullable, qapi_to_field_name(variant.name),
+                                       variant.optional, variant.name)
+            if nullable:
+                map_special += tomap
+                set_special += toset
+            else:
+                map_members += tomap
+                set_members += toset
+
+    type_name = qapi_to_go_type_name(name)
+    struct = generate_struct_type("", fields)[:-1]
+    return TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL.format(struct=struct,
+                                                        type_name=type_name,
+                                                        map_members=map_members,
+                                                        map_special=map_special,
+                                                        set_members=set_members,
+                                                        set_special=set_special)
 
 def recursive_base(self: QAPISchemaGenGolangVisitor,
-                   base: Optional[QAPISchemaObjectType]) -> str:
+                   base: Optional[QAPISchemaObjectType]) -> Tuple[str, bool]:
     fields = ""
+    with_nullable = False
 
     if not base:
-        return fields
+        return fields, with_nullable
 
     if base.base is not None:
         embed_base = self.schema.lookup_entity(base.base.name)
-        fields = recursive_base(self, embed_base)
+        fields, with_nullable = recursive_base(self, embed_base)
 
     for member in base.local_members:
         if base.variants and base.variants.tag_member.name == member.name:
             fields += '''// Discriminator\n'''
 
-        field = get_struct_field(self, member.name, member.type.name, member.optional, False)
+        field, nullable = get_struct_field(self, member.name, member.type.name,
+                                           False, member.optional, False)
         fields += field
+        with_nullable = True if nullable else with_nullable
 
     if len(fields) > 0:
         fields += "\n"
 
-    return fields
+    return fields, with_nullable
 
 # Helper function that is used for most of QAPI types
 def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
@@ -233,12 +396,14 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
                           variants: Optional[QAPISchemaVariants]) -> str:
 
 
-    fields = recursive_base(self, base)
+    fields, with_nullable = recursive_base(self, base)
 
     if members:
         for member in members:
-            field = get_struct_field(self, member.name, member.type.name, member.optional, False)
+            field, nullable = get_struct_field(self, member.name, member.type.name,
+                                               False, member.optional, False)
             fields += field
+            with_nullable = True if nullable else with_nullable
 
         fields += "\n"
 
@@ -248,11 +413,19 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
             if variant.type.is_implicit():
                 continue
 
-            field = get_struct_field(self, variant.name, variant.type.name, True, True)
+            field, nullable = get_struct_field(self, variant.name, variant.type.name,
+                                               False, True, True)
             fields += field
+            with_nullable = True if nullable else with_nullable
 
     type_name = qapi_to_go_type_name(name)
     content = generate_struct_type(type_name, fields)
+    if with_nullable:
+        content += struct_with_nullable_generate_marshal(self,
+                                                         name,
+                                                         base,
+                                                         members,
+                                                         variants)
     return content
 
 def generate_template_alternate(self: QAPISchemaGenGolangVisitor,
-- 
2.41.0

[PATCH v1 5/9] qapi: golang: Generate qapi's union types in Go

[Victor Toso]

This patch handles QAPI union types and generates the equivalent data
structures and methods in Go to handle it.

The QAPI union type has two types of fields: The @base and the
@Variants members. The @base fields can be considered common members
for the union while only one field maximum is set for the @Variants.

In the QAPI specification, it defines a @discriminator field, which is
an Enum type. The purpose of the  @discriminator is to identify which
@variant type is being used.

Not that @discriminator's enum might have more values than the union's
data struct. This is fine. The union does not need to handle all cases
of the enum, but it should accept them without error. For this
specific case, we keep the @discriminator field in every union type.

The union types implement the Marshaler and Unmarshaler interfaces to
seamless decode from JSON objects to Golang structs and vice versa.

qapi:
 | { 'union': 'SetPasswordOptions',
 |   'base': { 'protocol': 'DisplayProtocol',
 |             'password': 'str',
 |             '*connected': 'SetPasswordAction' },
 |   'discriminator': 'protocol',
 |   'data': { 'vnc': 'SetPasswordOptionsVnc' } }

go:
 | type SetPasswordOptions struct {
 |     Protocol  DisplayProtocol    `json:"protocol"`
 |     Password  string             `json:"password"`
 |     Connected *SetPasswordAction `json:"connected,omitempty"`
 |
 |     // Variants fields
 |     Vnc *SetPasswordOptionsVnc `json:"-"`
 | }

Signed-off-by: Victor Toso <victortoso@redhat.com>
---
 scripts/qapi/golang.py | 170 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 167 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 8320af99b6..343c9c9b95 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -52,6 +52,17 @@
     }
     return nil
 }
+
+// This helper is used to move struct's fields into a map.
+// This function is useful to merge JSON objects.
+func unwrapToMap(m map[string]any, data any) error {
+	if bytes, err := json.Marshal(&data); err != nil {
+		return fmt.Errorf("unwrapToMap: %s", err)
+	} else if err := json.Unmarshal(bytes, &m); err != nil {
+		return fmt.Errorf("unwrapToMap: %s, data=%s", err, string(bytes))
+	}
+	return nil
+}
 '''
 
 TEMPLATE_ALTERNATE = '''
@@ -131,6 +142,62 @@
 }}
 '''
 
+TEMPLATE_UNION_CHECK_FIELD = '''
+if s.{field} != nil && err == nil {{
+    if len(bytes) != 0 {{
+        err = errors.New(`multiple variant fields set`)
+    }} else if err = unwrapToMap(m, s.{field}); err == nil {{
+        s.{discriminator} = {go_enum_value}
+        m["{member_name}"] = {go_enum_value}
+        bytes, err = json.Marshal(m)
+    }}
+}}
+'''
+
+TEMPLATE_UNION_DRIVER_CASE = '''
+case {go_enum_value}:
+    s.{field} = new({member_type})
+    if err := json.Unmarshal(data, s.{field}); err != nil {{
+        s.{field} = nil
+        return err
+    }}'''
+
+TEMPLATE_UNION_METHODS = '''
+func (s {type_name}) MarshalJSON() ([]byte, error) {{
+    var bytes []byte
+    var err error
+    type Alias {type_name}
+    v := Alias(s)
+    m := make(map[string]any)
+    unwrapToMap(m, &v)
+    {check_fields}
+    {check_non_fields_marshal}
+    if err != nil {{
+        return nil, fmt.Errorf("error: marshal: {type_name}: reason='%s', struct='%+v'", err, s)
+    }} else if len(bytes) == 0 {{
+        return nil, fmt.Errorf("error: marshal: {type_name} unsupported, struct='%+v'", s)
+     }}
+     return bytes, nil
+}}
+
+func (s *{type_name}) UnmarshalJSON(data []byte) error {{{base_type_def}
+    tmp := struct {{
+        {base_type_name}
+    }}{{}}
+
+    if err := json.Unmarshal(data, &tmp); err != nil {{
+        return err
+    }}
+    {base_type_assign_unmarshal}
+    switch tmp.{discriminator} {{
+    {driver_cases}
+    {check_non_fields_unmarshal}
+    }}
+    return nil
+}}
+'''
+
+
 def gen_golang(schema: QAPISchema,
                output_dir: str,
                prefix: str) -> None:
@@ -428,6 +495,98 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
                                                          variants)
     return content
 
+def qapi_to_golang_methods_union(self: QAPISchemaGenGolangVisitor,
+                                 name: str,
+                                 base: Optional[QAPISchemaObjectType],
+                                 variants: Optional[QAPISchemaVariants]
+                                 ) -> str:
+
+    type_name = qapi_to_go_type_name(name)
+
+    assert base
+    base_type_assign_unmarshal = ""
+    base_type_name = qapi_to_go_type_name(base.name)
+    base_type_def = qapi_to_golang_struct(self,
+                                          base.name,
+                                          base.info,
+                                          base.ifcond,
+                                          base.features,
+                                          base.base,
+                                          base.members,
+                                          base.variants)
+    for member in base.local_members:
+        field = qapi_to_field_name(member.name)
+        base_type_assign_unmarshal += f'''s.{field} = tmp.{field}
+'''
+
+    driver_cases = ""
+    check_fields = ""
+    exists = {}
+    enum_name = variants.tag_member.type.name
+    discriminator = qapi_to_field_name(variants.tag_member.name)
+    if variants:
+        for var in variants.variants:
+            if var.type.is_implicit():
+                continue
+
+            field = qapi_to_field_name(var.name)
+            enum_value = qapi_to_field_name_enum(var.name)
+            member_type = qapi_schema_type_to_go_type(var.type.name)
+            go_enum_value = f'''{enum_name}{enum_value}'''
+            exists[go_enum_value] = True
+
+            check_fields += TEMPLATE_UNION_CHECK_FIELD.format(field=field,
+                                                             discriminator=discriminator,
+                                                             member_name=variants.tag_member.name,
+                                                             go_enum_value=go_enum_value)
+            driver_cases += TEMPLATE_UNION_DRIVER_CASE.format(go_enum_value=go_enum_value,
+                                                             field=field,
+                                                             member_type=member_type)
+
+    check_non_fields_marshal = ""
+    check_non_fields_unmarshal = ""
+    enum_obj = self.schema.lookup_entity(enum_name)
+    if len(exists) != len(enum_obj.members):
+        driver_cases += '''\ndefault:'''
+        check_non_fields_marshal = '''
+    // Check for valid values without field members
+    if len(bytes) == 0 && err == nil &&
+        ('''
+        check_non_fields_unmarshal = '''
+    // Check for valid values without field members
+    if '''
+
+        for member in enum_obj.members:
+            value = qapi_to_field_name_enum(member.name)
+            go_enum_value = f'''{enum_name}{value}'''
+
+            if go_enum_value in exists:
+                continue
+
+            check_non_fields_marshal += f'''s.{discriminator} == {go_enum_value} ||\n'''
+            check_non_fields_unmarshal += f'''tmp.{discriminator} != {go_enum_value} &&\n'''
+
+        check_non_fields_marshal = f'''{check_non_fields_marshal[:-3]}) {{
+        type Alias {type_name}
+        bytes, err = json.Marshal(Alias(s))
+    }}
+'''
+        check_non_fields_unmarshal = f'''{check_non_fields_unmarshal[1:-3]} {{
+        return fmt.Errorf("error: unmarshal: {type_name}: received unrecognized value: '%s'",
+            tmp.{discriminator})
+    }}
+'''
+
+    return TEMPLATE_UNION_METHODS.format(type_name=type_name,
+                                         check_fields=check_fields,
+                                         check_non_fields_marshal=check_non_fields_marshal,
+                                         base_type_def=base_type_def,
+                                         base_type_name=base_type_name,
+                                         base_type_assign_unmarshal=base_type_assign_unmarshal,
+                                         discriminator=discriminator,
+                                         driver_cases=driver_cases[1:],
+                                         check_non_fields_unmarshal=check_non_fields_unmarshal)
+
 def generate_template_alternate(self: QAPISchemaGenGolangVisitor,
                                 name: str,
                                 variants: Optional[QAPISchemaVariants]) -> str:
@@ -490,7 +649,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, _: str):
         super().__init__()
-        types = ["alternate", "enum", "helper", "struct"]
+        types = ["alternate", "enum", "helper", "struct", "union"]
         self.target = {name: "" for name in types}
         self.objects_seen = {}
         self.schema = None
@@ -530,10 +689,10 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor,
                           members: List[QAPISchemaObjectTypeMember],
                           variants: Optional[QAPISchemaVariants]
                           ) -> None:
-        # Do not handle anything besides struct.
+        # Do not handle anything besides struct and unions.
         if (name == self.schema.the_empty_object_type.name or
                 not isinstance(name, str) or
-                info.defn_meta not in ["struct"]):
+                info.defn_meta not in ["struct", "union"]):
             return
 
         # Base structs are embed
@@ -566,6 +725,11 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor,
                                                              base,
                                                              members,
                                                              variants)
+        if info.defn_meta == "union":
+            self.target[info.defn_meta] += qapi_to_golang_methods_union(self,
+                                                                        name,
+                                                                        base,
+                                                                        variants)
 
     def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
                              name: str,
-- 
2.41.0

Re: [PATCH v1 5/9] qapi: golang: Generate qapi's union types in Go

[Daniel P. Berrangé]

On Wed, Sep 27, 2023 at 01:25:40PM +0200, Victor Toso wrote:
> This patch handles QAPI union types and generates the equivalent data
> structures and methods in Go to handle it.
> 
> The QAPI union type has two types of fields: The @base and the
> @Variants members. The @base fields can be considered common members
> for the union while only one field maximum is set for the @Variants.
> 
> In the QAPI specification, it defines a @discriminator field, which is
> an Enum type. The purpose of the  @discriminator is to identify which
> @variant type is being used.
> 
> Not that @discriminator's enum might have more values than the union's
> data struct. This is fine. The union does not need to handle all cases
> of the enum, but it should accept them without error. For this
> specific case, we keep the @discriminator field in every union type.

I still tend think the @discriminator field should not be
present in the union structs. It feels like we're just trying
to directly copy the C code in Go and so smells wrong from a
Go POV.

For most of the unions the @discriminator field will be entirely
redundant, becasue the commonm case is that a @variant field
exists for every possible @discriminator value.

To take one example

  type SocketAddress struct {
        Type SocketAddressType `json:"type"`

        // Variants fields
        Inet  *InetSocketAddress  `json:"-"`
        Unix  *UnixSocketAddress  `json:"-"`
        Vsock *VsockSocketAddress `json:"-"`
        Fd    *String             `json:"-"`
  }

If one was just writing Go code without the pre-existing knowledge
of the QAPI C code, 'Type' is not something a Go programmer would
be inclined add IMHO.

And yet you are right that we need a way to represent a @discriminator
value that has no corresponding @variant, since QAPI allows for that
scenario. To deal with that I would suggest we just use an empty
interface type. eg


  type SocketAddress struct {
        Type SocketAddressType `json:"type"`

        // Variants fields
        Inet  *InetSocketAddress  `json:"-"`
        Unix  *UnixSocketAddress  `json:"-"`
        Vsock *VsockSocketAddress `json:"-"`
        Fd    *String             `json:"-"`
	Fish  *interface{}        `json:"-"`
	Food  *interface()        `json:"-"`
  }

the pointer value for 'Fish' and 'Food' fields here merely needs to
be non-NULL, it doesn't matter what the actual thing assigned is.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 5/9] qapi: golang: Generate qapi's union types in Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 4179 bytes --]

Hi,

On Thu, Sep 28, 2023 at 03:21:59PM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 27, 2023 at 01:25:40PM +0200, Victor Toso wrote:
> > This patch handles QAPI union types and generates the equivalent data
> > structures and methods in Go to handle it.
> > 
> > The QAPI union type has two types of fields: The @base and the
> > @Variants members. The @base fields can be considered common members
> > for the union while only one field maximum is set for the @Variants.
> > 
> > In the QAPI specification, it defines a @discriminator field, which is
> > an Enum type. The purpose of the  @discriminator is to identify which
> > @variant type is being used.
> > 
> > Not that @discriminator's enum might have more values than the union's
> > data struct. This is fine. The union does not need to handle all cases
> > of the enum, but it should accept them without error. For this
> > specific case, we keep the @discriminator field in every union type.
> 
> I still tend think the @discriminator field should not be
> present in the union structs. It feels like we're just trying
> to directly copy the C code in Go and so smells wrong from a
> Go POV.
> 
> For most of the unions the @discriminator field will be entirely
> redundant, becasue the commonm case is that a @variant field
> exists for every possible @discriminator value.

You are correct.

> To take one example
> 
>   type SocketAddress struct {
>         Type SocketAddressType `json:"type"`
> 
>         // Variants fields
>         Inet  *InetSocketAddress  `json:"-"`
>         Unix  *UnixSocketAddress  `json:"-"`
>         Vsock *VsockSocketAddress `json:"-"`
>         Fd    *String             `json:"-"`
>   }
> 
> If one was just writing Go code without the pre-existing knowledge
> of the QAPI C code, 'Type' is not something a Go programmer would
> be inclined add IMHO.

You don't need previous knowledge in the QAPI C code to see that
having optional field members and a discriminator field feels
very very suspicious. I wasn't too happy to add it.

> And yet you are right that we need a way to represent a
> @discriminator value that has no corresponding @variant, since
> QAPI allows for that scenario.

Thank Markus for that, really nice catch :)


> To deal with that I would suggest we just use an empty
> interface type. eg
> 
>   type SocketAddress struct {
>         Type SocketAddressType `json:"type"`
> 
>         // Variants fields
>         Inet  *InetSocketAddress  `json:"-"`
>         Unix  *UnixSocketAddress  `json:"-"`
>         Vsock *VsockSocketAddress `json:"-"`
>         Fd    *String             `json:"-"`
> 	Fish  *interface{}        `json:"-"`
> 	Food  *interface()        `json:"-"`
>   }
> 
> the pointer value for 'Fish' and 'Food' fields here merely needs to
> be non-NULL, it doesn't matter what the actual thing assigned is.

I like this idea. What happens if Fish becomes a handled in the
future?

Before:

    type SocketAddress struct {
        // Variants fields
        Inet  *InetSocketAddress  `json:"-"`
        Unix  *UnixSocketAddress  `json:"-"`
        Vsock *VsockSocketAddress `json:"-"`
        Fd    *String             `json:"-"`

        // Unhandled enum branches
        Fish  *interface{}        `json:"-"`
        Food  *interface{}        `json:"-"`
    }

to

    type SocketAddress struct {
        // Variants fields
        Inet  *InetSocketAddress  `json:"-"`
        Unix  *UnixSocketAddress  `json:"-"`
        Vsock *VsockSocketAddress `json:"-"`
        Fd    *String             `json:"-"`
        Fish  *FishSocketAddress  `json:"-"`

        // Unhandled enum branches
        Food  *interface{}        `json:"-"`
    }

An application that hat s.Fish = &something, will now error on
compile due something type not being FishSocketAddress. I think
this is acceptable. Very corner case scenario and the user
probably want to use the right struct now.

If you agree with above, I'd instead like to try a boolean
instead of *interface{}. s.Fish = true seems better and false is
simply ignored.

Cheers,
Victor

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1 5/9] qapi: golang: Generate qapi's union types in Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 4910 bytes --]

Hi,

On Fri, Sep 29, 2023 at 03:41:27PM +0200, Victor Toso wrote:
> Hi,
> 
> On Thu, Sep 28, 2023 at 03:21:59PM +0100, Daniel P. Berrangé wrote:
> > On Wed, Sep 27, 2023 at 01:25:40PM +0200, Victor Toso wrote:
> > > This patch handles QAPI union types and generates the equivalent data
> > > structures and methods in Go to handle it.
> > > 
> > > The QAPI union type has two types of fields: The @base and the
> > > @Variants members. The @base fields can be considered common members
> > > for the union while only one field maximum is set for the @Variants.
> > > 
> > > In the QAPI specification, it defines a @discriminator field, which is
> > > an Enum type. The purpose of the  @discriminator is to identify which
> > > @variant type is being used.
> > > 
> > > Not that @discriminator's enum might have more values than the union's
> > > data struct. This is fine. The union does not need to handle all cases
> > > of the enum, but it should accept them without error. For this
> > > specific case, we keep the @discriminator field in every union type.
> > 
> > I still tend think the @discriminator field should not be
> > present in the union structs. It feels like we're just trying
> > to directly copy the C code in Go and so smells wrong from a
> > Go POV.
> > 
> > For most of the unions the @discriminator field will be entirely
> > redundant, becasue the commonm case is that a @variant field
> > exists for every possible @discriminator value.
> 
> You are correct.
> 
> > To take one example
> > 
> >   type SocketAddress struct {
> >         Type SocketAddressType `json:"type"`
> > 
> >         // Variants fields
> >         Inet  *InetSocketAddress  `json:"-"`
> >         Unix  *UnixSocketAddress  `json:"-"`
> >         Vsock *VsockSocketAddress `json:"-"`
> >         Fd    *String             `json:"-"`
> >   }
> > 
> > If one was just writing Go code without the pre-existing knowledge
> > of the QAPI C code, 'Type' is not something a Go programmer would
> > be inclined add IMHO.
> 
> You don't need previous knowledge in the QAPI C code to see that
> having optional field members and a discriminator field feels
> very very suspicious. I wasn't too happy to add it.
> 
> > And yet you are right that we need a way to represent a
> > @discriminator value that has no corresponding @variant, since
> > QAPI allows for that scenario.
> 
> Thank Markus for that, really nice catch :)
> 
> 
> > To deal with that I would suggest we just use an empty
> > interface type. eg
> > 
> >   type SocketAddress struct {
> >         Type SocketAddressType `json:"type"`
> > 
> >         // Variants fields
> >         Inet  *InetSocketAddress  `json:"-"`
> >         Unix  *UnixSocketAddress  `json:"-"`
> >         Vsock *VsockSocketAddress `json:"-"`
> >         Fd    *String             `json:"-"`
> > 	Fish  *interface{}        `json:"-"`
> > 	Food  *interface()        `json:"-"`
> >   }
> > 
> > the pointer value for 'Fish' and 'Food' fields here merely needs to
> > be non-NULL, it doesn't matter what the actual thing assigned is.
> 
> I like this idea. What happens if Fish becomes a handled in the
> future?
> 
> Before:
> 
>     type SocketAddress struct {
>         // Variants fields
>         Inet  *InetSocketAddress  `json:"-"`
>         Unix  *UnixSocketAddress  `json:"-"`
>         Vsock *VsockSocketAddress `json:"-"`
>         Fd    *String             `json:"-"`
> 
>         // Unhandled enum branches
>         Fish  *interface{}        `json:"-"`
>         Food  *interface{}        `json:"-"`
>     }
> 
> to
> 
>     type SocketAddress struct {
>         // Variants fields
>         Inet  *InetSocketAddress  `json:"-"`
>         Unix  *UnixSocketAddress  `json:"-"`
>         Vsock *VsockSocketAddress `json:"-"`
>         Fd    *String             `json:"-"`
>         Fish  *FishSocketAddress  `json:"-"`
> 
>         // Unhandled enum branches
>         Food  *interface{}        `json:"-"`
>     }
> 
> An application that hat s.Fish = &something, will now error on
> compile due something type not being FishSocketAddress. I think
> this is acceptable. Very corner case scenario and the user
> probably want to use the right struct now.
> 
> If you agree with above, I'd instead like to try a boolean
> instead of *interface{}. s.Fish = true seems better and false is
> simply ignored.

I was just double checking that the Union's value for each enum
branch needs to be a Struct. So I think it is fine to use boolean
for unhandled enum branches. I'll be doing that in the next
iteration.

docs/devel/qapi-code-gen.rst: 350
    The BRANCH's value defines the branch's properties, in particular its
    type.  The type must a struct type.  The form TYPE-REF_ is shorthand
    for :code:`{ 'type': TYPE-REF }`.

Cheers,
Victor

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v1 6/9] qapi: golang: Generate qapi's event types in Go

[Victor Toso]

This patch handles QAPI event types and generates data structures in
Go that handles it.

We also define a Event interface and two helper functions MarshalEvent
and UnmarshalEvent.

Example:
qapi:
 | { 'event': 'MEMORY_DEVICE_SIZE_CHANGE',
 |   'data': { '*id': 'str', 'size': 'size', 'qom-path' : 'str'} }

go:
 | type MemoryDeviceSizeChangeEvent struct {
 |         MessageTimestamp Timestamp `json:"-"`
 |         Id               *string   `json:"id,omitempty"`
 |         Size             uint64    `json:"size"`
 |         QomPath          string    `json:"qom-path"`
 | }

usage:
 | input := `{"event":"MEMORY_DEVICE_SIZE_CHANGE",` +
 | `"timestamp":{"seconds":1588168529,"microseconds":201316},` +
 | `"data":{"id":"vm0","size":1073741824,"qom-path":"/machine/unattached/device[2]"}}`
 | e, err := UnmarshalEvent([]byte(input)
 | if err != nil {
 |     panic(err)
 | }
 | if e.GetName() == `MEMORY_DEVICE_SIZE_CHANGE` {
 |     m := e.(*MemoryDeviceSizeChangeEvent)
 |     // m.QomPath == "/machine/unattached/device[2]"
 | }

Signed-off-by: Victor Toso <victortoso@redhat.com>
---
 scripts/qapi/golang.py | 105 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 100 insertions(+), 5 deletions(-)

diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 343c9c9b95..ff3b1dd020 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -197,6 +197,55 @@
 }}
 '''
 
+TEMPLATE_EVENT = '''
+type Timestamp struct {{
+    Seconds      int64 `json:"seconds"`
+    Microseconds int64 `json:"microseconds"`
+}}
+
+type Event interface {{
+    GetName()      string
+    GetTimestamp() Timestamp
+}}
+
+func MarshalEvent(e Event) ([]byte, error) {{
+    m := make(map[string]any)
+    m["event"] = e.GetName()
+    m["timestamp"] = e.GetTimestamp()
+    if bytes, err := json.Marshal(e); err != nil {{
+        return []byte{{}}, err
+    }} else if len(bytes) > 2 {{
+        m["data"] = e
+    }}
+    return json.Marshal(m)
+}}
+
+func UnmarshalEvent(data []byte) (Event, error) {{
+    base := struct {{
+        Name             string    `json:"event"`
+        MessageTimestamp Timestamp `json:"timestamp"`
+    }}{{}}
+    if err := json.Unmarshal(data, &base); err != nil {{
+        return nil, fmt.Errorf("Failed to decode event: %s", string(data))
+    }}
+
+    switch base.Name {{
+    {cases}
+    }}
+    return nil, errors.New("Failed to recognize event")
+}}
+'''
+
+TEMPLATE_EVENT_METHODS = '''
+func (s *{type_name}) GetName() string {{
+    return "{name}"
+}}
+
+func (s *{type_name}) GetTimestamp() Timestamp {{
+    return s.MessageTimestamp
+}}
+'''
+
 
 def gen_golang(schema: QAPISchema,
                output_dir: str,
@@ -217,7 +266,8 @@ def qapi_to_field_name(name: str) -> str:
 def qapi_to_field_name_enum(name: str) -> str:
     return name.title().replace("-", "")
 
-def qapi_to_go_type_name(name: str) -> str:
+def qapi_to_go_type_name(name: str,
+                         meta: Optional[str] = None) -> str:
     if qapi_name_is_object(name):
         name = name[6:]
 
@@ -232,6 +282,11 @@ def qapi_to_go_type_name(name: str) -> str:
 
     name += ''.join(word.title() for word in words[1:])
 
+    types = ["event"]
+    if meta in types:
+        name = name[:-3] if name.endswith("Arg") else name
+        name += meta.title().replace(" ", "")
+
     return name
 
 def qapi_schema_type_to_go_type(qapitype: str) -> str:
@@ -455,7 +510,7 @@ def recursive_base(self: QAPISchemaGenGolangVisitor,
 # Helper function that is used for most of QAPI types
 def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
                           name: str,
-                          _: Optional[QAPISourceInfo],
+                          info: Optional[QAPISourceInfo],
                           __: QAPISchemaIfCond,
                           ___: List[QAPISchemaFeature],
                           base: Optional[QAPISchemaObjectType],
@@ -464,6 +519,8 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
 
 
     fields, with_nullable = recursive_base(self, base)
+    if info.defn_meta == "event":
+        fields += f'''\tMessageTimestamp Timestamp `json:"-"`\n{fields}'''
 
     if members:
         for member in members:
@@ -485,7 +542,7 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
             fields += field
             with_nullable = True if nullable else with_nullable
 
-    type_name = qapi_to_go_type_name(name)
+    type_name = qapi_to_go_type_name(name, info.defn_meta)
     content = generate_struct_type(type_name, fields)
     if with_nullable:
         content += struct_with_nullable_generate_marshal(self,
@@ -644,15 +701,34 @@ def generate_template_alternate(self: QAPISchemaGenGolangVisitor,
                                                  unmarshal_check_fields=unmarshal_check_fields[1:])
     return content
 
+def generate_template_event(events: dict[str, str]) -> str:
+    cases = ""
+    for name in sorted(events):
+        case_type = events[name]
+        cases += f'''
+case "{name}":
+    event := struct {{
+        Data {case_type} `json:"data"`
+    }}{{}}
+
+    if err := json.Unmarshal(data, &event); err != nil {{
+        return nil, fmt.Errorf("Failed to unmarshal: %s", string(data))
+    }}
+    event.Data.MessageTimestamp = base.MessageTimestamp
+    return &event.Data, nil
+'''
+    return TEMPLATE_EVENT.format(cases=cases)
+
 
 class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, _: str):
         super().__init__()
-        types = ["alternate", "enum", "helper", "struct", "union"]
+        types = ["alternate", "enum", "event", "helper", "struct", "union"]
         self.target = {name: "" for name in types}
         self.objects_seen = {}
         self.schema = None
+        self.events = {}
         self.golang_package_name = "qapi"
         self.accept_null_types = []
 
@@ -679,6 +755,7 @@ def visit_begin(self, schema):
 
     def visit_end(self):
         self.schema = None
+        self.target["event"] += generate_template_event(self.events)
 
     def visit_object_type(self: QAPISchemaGenGolangVisitor,
                           name: str,
@@ -779,7 +856,25 @@ def visit_command(self,
         pass
 
     def visit_event(self, name, info, ifcond, features, arg_type, boxed):
-        pass
+        assert name == info.defn_name
+        type_name = qapi_to_go_type_name(name, info.defn_meta)
+        self.events[name] = type_name
+
+        if isinstance(arg_type, QAPISchemaObjectType):
+            content = qapi_to_golang_struct(self,
+                                            name,
+                                            arg_type.info,
+                                            arg_type.ifcond,
+                                            arg_type.features,
+                                            arg_type.base,
+                                            arg_type.members,
+                                            arg_type.variants)
+        else:
+            args = '''MessageTimestamp Timestamp `json:"-"`'''
+            content = generate_struct_type(type_name, args)
+
+        content += TEMPLATE_EVENT_METHODS.format(name=name, type_name=type_name)
+        self.target["event"] += content
 
     def write(self, output_dir: str) -> None:
         for module_name, content in self.target.items():
-- 
2.41.0

[PATCH v1 7/9] qapi: golang: Generate qapi's command types in Go

[Victor Toso]

This patch handles QAPI command types and generates data structures in
Go that decodes from QMP JSON Object to Go data structure and vice
versa.

Similar to Event, this patch adds a Command interface and two helper
functions MarshalCommand and UnmarshalCommand.

Example:
qapi:
 | { 'command': 'set_password',
 |   'boxed': true,
 |   'data': 'SetPasswordOptions' }

go:
 | type SetPasswordCommand struct {
 |     SetPasswordOptions
 |     CommandId string `json:"-"`
 | }

usage:
 | input := `{"execute":"set_password",` +
 |          `"arguments":{"protocol":"vnc",` +
 |          `"password":"secret"}}`
 |
 | c, err := UnmarshalCommand([]byte(input))
 | if err != nil {
 |     panic(err)
 | }
 |
 | if c.GetName() == `set_password` {
 |         m := c.(*SetPasswordCommand)
 |         // m.Password == "secret"
 | }

Signed-off-by: Victor Toso <victortoso@redhat.com>
---
 scripts/qapi/golang.py | 97 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 94 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index ff3b1dd020..52a9124641 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -246,6 +246,51 @@
 }}
 '''
 
+TEMPLATE_COMMAND_METHODS = '''
+func (c *{type_name}) GetName() string {{
+    return "{name}"
+}}
+
+func (s *{type_name}) GetId() string {{
+    return s.MessageId
+}}
+'''
+
+TEMPLATE_COMMAND = '''
+type Command interface {{
+    GetId()         string
+    GetName()       string
+}}
+
+func MarshalCommand(c Command) ([]byte, error) {{
+    m := make(map[string]any)
+    m["execute"] = c.GetName()
+    if id := c.GetId(); len(id) > 0 {{
+        m["id"] = id
+    }}
+    if bytes, err := json.Marshal(c); err != nil {{
+        return []byte{{}}, err
+    }} else if len(bytes) > 2 {{
+        m["arguments"] = c
+    }}
+    return json.Marshal(m)
+}}
+
+func UnmarshalCommand(data []byte) (Command, error) {{
+    base := struct {{
+        MessageId string `json:"id,omitempty"`
+        Name      string `json:"execute"`
+    }}{{}}
+    if err := json.Unmarshal(data, &base); err != nil {{
+        return nil, fmt.Errorf("Failed to decode command: %s", string(data))
+    }}
+
+    switch base.Name {{
+    {cases}
+    }}
+    return nil, errors.New("Failed to recognize command")
+}}
+'''
 
 def gen_golang(schema: QAPISchema,
                output_dir: str,
@@ -282,7 +327,7 @@ def qapi_to_go_type_name(name: str,
 
     name += ''.join(word.title() for word in words[1:])
 
-    types = ["event"]
+    types = ["event", "command"]
     if meta in types:
         name = name[:-3] if name.endswith("Arg") else name
         name += meta.title().replace(" ", "")
@@ -521,6 +566,8 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
     fields, with_nullable = recursive_base(self, base)
     if info.defn_meta == "event":
         fields += f'''\tMessageTimestamp Timestamp `json:"-"`\n{fields}'''
+    elif info.defn_meta == "command":
+        fields += f'''\tMessageId string `json:"-"`\n{fields}'''
 
     if members:
         for member in members:
@@ -719,16 +766,36 @@ def generate_template_event(events: dict[str, str]) -> str:
 '''
     return TEMPLATE_EVENT.format(cases=cases)
 
+def generate_template_command(commands: dict[str, str]) -> str:
+    cases = ""
+    for name in sorted(commands):
+        case_type = commands[name]
+        cases += f'''
+case "{name}":
+    command := struct {{
+        Args {case_type} `json:"arguments"`
+    }}{{}}
+
+    if err := json.Unmarshal(data, &command); err != nil {{
+        return nil, fmt.Errorf("Failed to unmarshal: %s", string(data))
+    }}
+    command.Args.MessageId = base.MessageId
+    return &command.Args, nil
+'''
+    content = TEMPLATE_COMMAND.format(cases=cases)
+    return content
+
 
 class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, _: str):
         super().__init__()
-        types = ["alternate", "enum", "event", "helper", "struct", "union"]
+        types = ["alternate", "command", "enum", "event", "helper", "struct", "union"]
         self.target = {name: "" for name in types}
         self.objects_seen = {}
         self.schema = None
         self.events = {}
+        self.commands = {}
         self.golang_package_name = "qapi"
         self.accept_null_types = []
 
@@ -756,6 +823,7 @@ def visit_begin(self, schema):
     def visit_end(self):
         self.schema = None
         self.target["event"] += generate_template_event(self.events)
+        self.target["command"] += generate_template_command(self.commands)
 
     def visit_object_type(self: QAPISchemaGenGolangVisitor,
                           name: str,
@@ -853,7 +921,30 @@ def visit_command(self,
                       allow_oob: bool,
                       allow_preconfig: bool,
                       coroutine: bool) -> None:
-        pass
+        assert name == info.defn_name
+
+        type_name = qapi_to_go_type_name(name, info.defn_meta)
+        self.commands[name] = type_name
+
+        content = ""
+        if boxed or not arg_type or not qapi_name_is_object(arg_type.name):
+            args = "" if not arg_type else "\n" + arg_type.name
+            args += '''\n\tMessageId   string `json:"-"`'''
+            content = generate_struct_type(type_name, args)
+        else:
+            assert isinstance(arg_type, QAPISchemaObjectType)
+            content = qapi_to_golang_struct(self,
+                                            name,
+                                            arg_type.info,
+                                            arg_type.ifcond,
+                                            arg_type.features,
+                                            arg_type.base,
+                                            arg_type.members,
+                                            arg_type.variants)
+
+        content += TEMPLATE_COMMAND_METHODS.format(name=name,
+                                                   type_name=type_name)
+        self.target["command"] += content
 
     def visit_event(self, name, info, ifcond, features, arg_type, boxed):
         assert name == info.defn_name
-- 
2.41.0

Re: [PATCH v1 7/9] qapi: golang: Generate qapi's command types in Go

[Daniel P. Berrangé]

On Wed, Sep 27, 2023 at 01:25:42PM +0200, Victor Toso wrote:
> This patch handles QAPI command types and generates data structures in
> Go that decodes from QMP JSON Object to Go data structure and vice
> versa.
> 
> Similar to Event, this patch adds a Command interface and two helper
> functions MarshalCommand and UnmarshalCommand.
> 
> Example:
> qapi:
>  | { 'command': 'set_password',
>  |   'boxed': true,
>  |   'data': 'SetPasswordOptions' }
> 
> go:
>  | type SetPasswordCommand struct {
>  |     SetPasswordOptions
>  |     CommandId string `json:"-"`

IIUC, you renamed that to MessageId in the code now.

>  | }

Overall, I'm not entirely convinced that we will want to
have the SetPasswordCommand struct wrappers, byut it is
hard to say, as what we're missing still is the eventual
application facing API.

eg something that ultimately looks more like this:

    qemu = qemu.QMPConnection()
    qemu.Dial("/path/to/unix/socket.sock")

    qemu.VncConnectedEvent(func(ev *VncConnectedEvent) {
         fmt.Printf("VNC client %s connected\n", ev.Client.Host)
    })

    resp, err := qemu.SetPassword(SetPasswordArguments{
        protocol: "vnc",
	password: "123456",
    })

    if err != nil {
        fmt.Fprintf(os.Stderr, "Cannot set passwd: %s", err)
    }

    ..do something wit resp....   (well SetPassword has no response, but other cmmands do)

It isn't clear that the SetPasswordCommand struct will be
needed internally for the impl if QMPCommand.

> 
> usage:
>  | input := `{"execute":"set_password",` +
>  |          `"arguments":{"protocol":"vnc",` +
>  |          `"password":"secret"}}`
>  |
>  | c, err := UnmarshalCommand([]byte(input))
>  | if err != nil {
>  |     panic(err)
>  | }
>  |
>  | if c.GetName() == `set_password` {
>  |         m := c.(*SetPasswordCommand)
>  |         // m.Password == "secret"
>  | }
> 
> Signed-off-by: Victor Toso <victortoso@redhat.com>
> ---
>  scripts/qapi/golang.py | 97 ++++++++++++++++++++++++++++++++++++++++--
>  1 file changed, 94 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> index ff3b1dd020..52a9124641 100644
> --- a/scripts/qapi/golang.py
> +++ b/scripts/qapi/golang.py
> @@ -246,6 +246,51 @@
>  }}
>  '''
>  
> +TEMPLATE_COMMAND_METHODS = '''
> +func (c *{type_name}) GetName() string {{
> +    return "{name}"
> +}}
> +
> +func (s *{type_name}) GetId() string {{
> +    return s.MessageId
> +}}
> +'''
> +
> +TEMPLATE_COMMAND = '''
> +type Command interface {{
> +    GetId()         string
> +    GetName()       string
> +}}
> +
> +func MarshalCommand(c Command) ([]byte, error) {{
> +    m := make(map[string]any)
> +    m["execute"] = c.GetName()
> +    if id := c.GetId(); len(id) > 0 {{
> +        m["id"] = id
> +    }}
> +    if bytes, err := json.Marshal(c); err != nil {{
> +        return []byte{{}}, err
> +    }} else if len(bytes) > 2 {{
> +        m["arguments"] = c
> +    }}
> +    return json.Marshal(m)
> +}}
> +
> +func UnmarshalCommand(data []byte) (Command, error) {{
> +    base := struct {{
> +        MessageId string `json:"id,omitempty"`
> +        Name      string `json:"execute"`
> +    }}{{}}
> +    if err := json.Unmarshal(data, &base); err != nil {{
> +        return nil, fmt.Errorf("Failed to decode command: %s", string(data))
> +    }}
> +
> +    switch base.Name {{
> +    {cases}
> +    }}
> +    return nil, errors.New("Failed to recognize command")
> +}}
> +'''
>  
>  def gen_golang(schema: QAPISchema,
>                 output_dir: str,
> @@ -282,7 +327,7 @@ def qapi_to_go_type_name(name: str,
>  
>      name += ''.join(word.title() for word in words[1:])
>  
> -    types = ["event"]
> +    types = ["event", "command"]
>      if meta in types:
>          name = name[:-3] if name.endswith("Arg") else name
>          name += meta.title().replace(" ", "")
> @@ -521,6 +566,8 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
>      fields, with_nullable = recursive_base(self, base)
>      if info.defn_meta == "event":
>          fields += f'''\tMessageTimestamp Timestamp `json:"-"`\n{fields}'''
> +    elif info.defn_meta == "command":
> +        fields += f'''\tMessageId string `json:"-"`\n{fields}'''
>  
>      if members:
>          for member in members:
> @@ -719,16 +766,36 @@ def generate_template_event(events: dict[str, str]) -> str:
>  '''
>      return TEMPLATE_EVENT.format(cases=cases)
>  
> +def generate_template_command(commands: dict[str, str]) -> str:
> +    cases = ""
> +    for name in sorted(commands):
> +        case_type = commands[name]
> +        cases += f'''
> +case "{name}":
> +    command := struct {{
> +        Args {case_type} `json:"arguments"`
> +    }}{{}}
> +
> +    if err := json.Unmarshal(data, &command); err != nil {{
> +        return nil, fmt.Errorf("Failed to unmarshal: %s", string(data))
> +    }}
> +    command.Args.MessageId = base.MessageId
> +    return &command.Args, nil
> +'''
> +    content = TEMPLATE_COMMAND.format(cases=cases)
> +    return content
> +
>  
>  class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
>  
>      def __init__(self, _: str):
>          super().__init__()
> -        types = ["alternate", "enum", "event", "helper", "struct", "union"]
> +        types = ["alternate", "command", "enum", "event", "helper", "struct", "union"]
>          self.target = {name: "" for name in types}
>          self.objects_seen = {}
>          self.schema = None
>          self.events = {}
> +        self.commands = {}
>          self.golang_package_name = "qapi"
>          self.accept_null_types = []
>  
> @@ -756,6 +823,7 @@ def visit_begin(self, schema):
>      def visit_end(self):
>          self.schema = None
>          self.target["event"] += generate_template_event(self.events)
> +        self.target["command"] += generate_template_command(self.commands)
>  
>      def visit_object_type(self: QAPISchemaGenGolangVisitor,
>                            name: str,
> @@ -853,7 +921,30 @@ def visit_command(self,
>                        allow_oob: bool,
>                        allow_preconfig: bool,
>                        coroutine: bool) -> None:
> -        pass
> +        assert name == info.defn_name
> +
> +        type_name = qapi_to_go_type_name(name, info.defn_meta)
> +        self.commands[name] = type_name
> +
> +        content = ""
> +        if boxed or not arg_type or not qapi_name_is_object(arg_type.name):
> +            args = "" if not arg_type else "\n" + arg_type.name
> +            args += '''\n\tMessageId   string `json:"-"`'''
> +            content = generate_struct_type(type_name, args)
> +        else:
> +            assert isinstance(arg_type, QAPISchemaObjectType)
> +            content = qapi_to_golang_struct(self,
> +                                            name,
> +                                            arg_type.info,
> +                                            arg_type.ifcond,
> +                                            arg_type.features,
> +                                            arg_type.base,
> +                                            arg_type.members,
> +                                            arg_type.variants)
> +
> +        content += TEMPLATE_COMMAND_METHODS.format(name=name,
> +                                                   type_name=type_name)
> +        self.target["command"] += content
>  
>      def visit_event(self, name, info, ifcond, features, arg_type, boxed):
>          assert name == info.defn_name
> -- 
> 2.41.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 7/9] qapi: golang: Generate qapi's command types in Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 9438 bytes --]

Hi,

On Thu, Sep 28, 2023 at 03:32:54PM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 27, 2023 at 01:25:42PM +0200, Victor Toso wrote:
> > This patch handles QAPI command types and generates data structures in
> > Go that decodes from QMP JSON Object to Go data structure and vice
> > versa.
> > 
> > Similar to Event, this patch adds a Command interface and two helper
> > functions MarshalCommand and UnmarshalCommand.
> > 
> > Example:
> > qapi:
> >  | { 'command': 'set_password',
> >  |   'boxed': true,
> >  |   'data': 'SetPasswordOptions' }
> > 
> > go:
> >  | type SetPasswordCommand struct {
> >  |     SetPasswordOptions
> >  |     CommandId string `json:"-"`
> 
> IIUC, you renamed that to MessageId in the code now.

Thanks!

> 
> >  | }
> 
> Overall, I'm not entirely convinced that we will want to
> have the SetPasswordCommand struct wrappers, byut it is
> hard to say, as what we're missing still is the eventual
> application facing API.
> 
> eg something that ultimately looks more like this:
> 
>     qemu = qemu.QMPConnection()
>     qemu.Dial("/path/to/unix/socket.sock")
> 
>     qemu.VncConnectedEvent(func(ev *VncConnectedEvent) {
>          fmt.Printf("VNC client %s connected\n", ev.Client.Host)
>     })
> 
>     resp, err := qemu.SetPassword(SetPasswordArguments{
>         protocol: "vnc",
> 	password: "123456",
>     })

For the other structs, I've removed these embed struct (base).
For the commands, I wasn't sure so I left them. I think it is
worth to give another look as I do agree with you.

In the Go application is unlikely that the embed structs are
needed, the important part is to have all the fields in the
command struct.

My prior concern, if I recall correctly was:

 1. This would leave generated but unused quite a few structs.
    Should I remove them? Should I leave them?

 2. The struct might be something we might use elsewhere, so it
    would make sense to, for example:

    qemu.SetPassword.SetPasswordArguments = myArgs

    Instead of having to assign field by field, from myArgs to
    qemu.SetPassword as they would be different types.

Overall, I would prefer not having embed types so I'll give
another look to this.

Cheers,
Victor

> 
>     if err != nil {
>         fmt.Fprintf(os.Stderr, "Cannot set passwd: %s", err)
>     }
> 
>     ..do something wit resp....   (well SetPassword has no response, but other cmmands do)
> 
> It isn't clear that the SetPasswordCommand struct will be
> needed internally for the impl if QMPCommand.
> 
> > 
> > usage:
> >  | input := `{"execute":"set_password",` +
> >  |          `"arguments":{"protocol":"vnc",` +
> >  |          `"password":"secret"}}`
> >  |
> >  | c, err := UnmarshalCommand([]byte(input))
> >  | if err != nil {
> >  |     panic(err)
> >  | }
> >  |
> >  | if c.GetName() == `set_password` {
> >  |         m := c.(*SetPasswordCommand)
> >  |         // m.Password == "secret"
> >  | }
> > 
> > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > ---
> >  scripts/qapi/golang.py | 97 ++++++++++++++++++++++++++++++++++++++++--
> >  1 file changed, 94 insertions(+), 3 deletions(-)
> > 
> > diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> > index ff3b1dd020..52a9124641 100644
> > --- a/scripts/qapi/golang.py
> > +++ b/scripts/qapi/golang.py
> > @@ -246,6 +246,51 @@
> >  }}
> >  '''
> >  
> > +TEMPLATE_COMMAND_METHODS = '''
> > +func (c *{type_name}) GetName() string {{
> > +    return "{name}"
> > +}}
> > +
> > +func (s *{type_name}) GetId() string {{
> > +    return s.MessageId
> > +}}
> > +'''
> > +
> > +TEMPLATE_COMMAND = '''
> > +type Command interface {{
> > +    GetId()         string
> > +    GetName()       string
> > +}}
> > +
> > +func MarshalCommand(c Command) ([]byte, error) {{
> > +    m := make(map[string]any)
> > +    m["execute"] = c.GetName()
> > +    if id := c.GetId(); len(id) > 0 {{
> > +        m["id"] = id
> > +    }}
> > +    if bytes, err := json.Marshal(c); err != nil {{
> > +        return []byte{{}}, err
> > +    }} else if len(bytes) > 2 {{
> > +        m["arguments"] = c
> > +    }}
> > +    return json.Marshal(m)
> > +}}
> > +
> > +func UnmarshalCommand(data []byte) (Command, error) {{
> > +    base := struct {{
> > +        MessageId string `json:"id,omitempty"`
> > +        Name      string `json:"execute"`
> > +    }}{{}}
> > +    if err := json.Unmarshal(data, &base); err != nil {{
> > +        return nil, fmt.Errorf("Failed to decode command: %s", string(data))
> > +    }}
> > +
> > +    switch base.Name {{
> > +    {cases}
> > +    }}
> > +    return nil, errors.New("Failed to recognize command")
> > +}}
> > +'''
> >  
> >  def gen_golang(schema: QAPISchema,
> >                 output_dir: str,
> > @@ -282,7 +327,7 @@ def qapi_to_go_type_name(name: str,
> >  
> >      name += ''.join(word.title() for word in words[1:])
> >  
> > -    types = ["event"]
> > +    types = ["event", "command"]
> >      if meta in types:
> >          name = name[:-3] if name.endswith("Arg") else name
> >          name += meta.title().replace(" ", "")
> > @@ -521,6 +566,8 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
> >      fields, with_nullable = recursive_base(self, base)
> >      if info.defn_meta == "event":
> >          fields += f'''\tMessageTimestamp Timestamp `json:"-"`\n{fields}'''
> > +    elif info.defn_meta == "command":
> > +        fields += f'''\tMessageId string `json:"-"`\n{fields}'''
> >  
> >      if members:
> >          for member in members:
> > @@ -719,16 +766,36 @@ def generate_template_event(events: dict[str, str]) -> str:
> >  '''
> >      return TEMPLATE_EVENT.format(cases=cases)
> >  
> > +def generate_template_command(commands: dict[str, str]) -> str:
> > +    cases = ""
> > +    for name in sorted(commands):
> > +        case_type = commands[name]
> > +        cases += f'''
> > +case "{name}":
> > +    command := struct {{
> > +        Args {case_type} `json:"arguments"`
> > +    }}{{}}
> > +
> > +    if err := json.Unmarshal(data, &command); err != nil {{
> > +        return nil, fmt.Errorf("Failed to unmarshal: %s", string(data))
> > +    }}
> > +    command.Args.MessageId = base.MessageId
> > +    return &command.Args, nil
> > +'''
> > +    content = TEMPLATE_COMMAND.format(cases=cases)
> > +    return content
> > +
> >  
> >  class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> >  
> >      def __init__(self, _: str):
> >          super().__init__()
> > -        types = ["alternate", "enum", "event", "helper", "struct", "union"]
> > +        types = ["alternate", "command", "enum", "event", "helper", "struct", "union"]
> >          self.target = {name: "" for name in types}
> >          self.objects_seen = {}
> >          self.schema = None
> >          self.events = {}
> > +        self.commands = {}
> >          self.golang_package_name = "qapi"
> >          self.accept_null_types = []
> >  
> > @@ -756,6 +823,7 @@ def visit_begin(self, schema):
> >      def visit_end(self):
> >          self.schema = None
> >          self.target["event"] += generate_template_event(self.events)
> > +        self.target["command"] += generate_template_command(self.commands)
> >  
> >      def visit_object_type(self: QAPISchemaGenGolangVisitor,
> >                            name: str,
> > @@ -853,7 +921,30 @@ def visit_command(self,
> >                        allow_oob: bool,
> >                        allow_preconfig: bool,
> >                        coroutine: bool) -> None:
> > -        pass
> > +        assert name == info.defn_name
> > +
> > +        type_name = qapi_to_go_type_name(name, info.defn_meta)
> > +        self.commands[name] = type_name
> > +
> > +        content = ""
> > +        if boxed or not arg_type or not qapi_name_is_object(arg_type.name):
> > +            args = "" if not arg_type else "\n" + arg_type.name
> > +            args += '''\n\tMessageId   string `json:"-"`'''
> > +            content = generate_struct_type(type_name, args)
> > +        else:
> > +            assert isinstance(arg_type, QAPISchemaObjectType)
> > +            content = qapi_to_golang_struct(self,
> > +                                            name,
> > +                                            arg_type.info,
> > +                                            arg_type.ifcond,
> > +                                            arg_type.features,
> > +                                            arg_type.base,
> > +                                            arg_type.members,
> > +                                            arg_type.variants)
> > +
> > +        content += TEMPLATE_COMMAND_METHODS.format(name=name,
> > +                                                   type_name=type_name)
> > +        self.target["command"] += content
> >  
> >      def visit_event(self, name, info, ifcond, features, arg_type, boxed):
> >          assert name == info.defn_name
> > -- 
> > 2.41.0
> > 
> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1 7/9] qapi: golang: Generate qapi's command types in Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 9857 bytes --]

Hi,

On Thu, Sep 28, 2023 at 03:32:54PM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 27, 2023 at 01:25:42PM +0200, Victor Toso wrote:
> > This patch handles QAPI command types and generates data structures in
> > Go that decodes from QMP JSON Object to Go data structure and vice
> > versa.
> > 
> > Similar to Event, this patch adds a Command interface and two helper
> > functions MarshalCommand and UnmarshalCommand.
> > 
> > Example:
> > qapi:
> >  | { 'command': 'set_password',
> >  |   'boxed': true,
> >  |   'data': 'SetPasswordOptions' }
> > 
> > go:
> >  | type SetPasswordCommand struct {
> >  |     SetPasswordOptions
> >  |     CommandId string `json:"-"`
> 
> IIUC, you renamed that to MessageId in the code now.
> 
> >  | }
> 
> Overall, I'm not entirely convinced that we will want to
> have the SetPasswordCommand struct wrappers, byut it is
> hard to say,

I was playing with removing these embed structs now, similar to
how we did for all other base types. The extra complexity might
not be worthy to remove it, IMHO.

Actually, the SetPasswordCommand can be used as example.

No embed (proposed):
  type SetPasswordCommand struct {
      CommandId string `json:"-"`

      Password  string             `json:"password"`
      Connected *SetPasswordAction `json:"connected,omitempty"`

      Protocol  DisplayProtocol    `json:"protocol"`
      Password  string             `json:"password"`
      Connected *SetPasswordAction `json:"connected,omitempty"`

      // Variants fields
      Vnc *SetPasswordOptionsVnc `json:"-"`

      // Unbranched enum fields
      Spice bool `json:"-"`
  }

As SetPasswordOptions is a union, now we have to treat
SetPasswordCommand as union too, for Marshal/Unmarshal.

The data type could also be an Alternate, which has different
logic for Marshal/Unmarshal.

So, doing embed would add quite a bit of complexity to handling
Commands and Events too although no Event use boxed=true at the
moment in QEMU.

> as what we're missing still is the eventual application facing
> API.
> 
> eg something that ultimately looks more like this:
> 
>     qemu = qemu.QMPConnection()
>     qemu.Dial("/path/to/unix/socket.sock")
> 
>     qemu.VncConnectedEvent(func(ev *VncConnectedEvent) {
>          fmt.Printf("VNC client %s connected\n", ev.Client.Host)
>     })
> 
>     resp, err := qemu.SetPassword(SetPasswordArguments{
>         protocol: "vnc",
>         password: "123456",
>     })
> 
>     if err != nil {
>         fmt.Fprintf(os.Stderr, "Cannot set passwd: %s", err)
>     }
> 
>     ..do something wit resp....   (well SetPassword has no response, but other cmmands do)
> 
> It isn't clear that the SetPasswordCommand struct will be
> needed internally for the impl if QMPCommand.

Yes. Let's get this in so we can work on the next layer. We don't
need to declare this Go module as stable for now, till we get the
next few bits figure out.

Cheers,
Victor
 
> > usage:
> >  | input := `{"execute":"set_password",` +
> >  |          `"arguments":{"protocol":"vnc",` +
> >  |          `"password":"secret"}}`
> >  |
> >  | c, err := UnmarshalCommand([]byte(input))
> >  | if err != nil {
> >  |     panic(err)
> >  | }
> >  |
> >  | if c.GetName() == `set_password` {
> >  |         m := c.(*SetPasswordCommand)
> >  |         // m.Password == "secret"
> >  | }
> > 
> > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > ---
> >  scripts/qapi/golang.py | 97 ++++++++++++++++++++++++++++++++++++++++--
> >  1 file changed, 94 insertions(+), 3 deletions(-)
> > 
> > diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> > index ff3b1dd020..52a9124641 100644
> > --- a/scripts/qapi/golang.py
> > +++ b/scripts/qapi/golang.py
> > @@ -246,6 +246,51 @@
> >  }}
> >  '''
> >  
> > +TEMPLATE_COMMAND_METHODS = '''
> > +func (c *{type_name}) GetName() string {{
> > +    return "{name}"
> > +}}
> > +
> > +func (s *{type_name}) GetId() string {{
> > +    return s.MessageId
> > +}}
> > +'''
> > +
> > +TEMPLATE_COMMAND = '''
> > +type Command interface {{
> > +    GetId()         string
> > +    GetName()       string
> > +}}
> > +
> > +func MarshalCommand(c Command) ([]byte, error) {{
> > +    m := make(map[string]any)
> > +    m["execute"] = c.GetName()
> > +    if id := c.GetId(); len(id) > 0 {{
> > +        m["id"] = id
> > +    }}
> > +    if bytes, err := json.Marshal(c); err != nil {{
> > +        return []byte{{}}, err
> > +    }} else if len(bytes) > 2 {{
> > +        m["arguments"] = c
> > +    }}
> > +    return json.Marshal(m)
> > +}}
> > +
> > +func UnmarshalCommand(data []byte) (Command, error) {{
> > +    base := struct {{
> > +        MessageId string `json:"id,omitempty"`
> > +        Name      string `json:"execute"`
> > +    }}{{}}
> > +    if err := json.Unmarshal(data, &base); err != nil {{
> > +        return nil, fmt.Errorf("Failed to decode command: %s", string(data))
> > +    }}
> > +
> > +    switch base.Name {{
> > +    {cases}
> > +    }}
> > +    return nil, errors.New("Failed to recognize command")
> > +}}
> > +'''
> >  
> >  def gen_golang(schema: QAPISchema,
> >                 output_dir: str,
> > @@ -282,7 +327,7 @@ def qapi_to_go_type_name(name: str,
> >  
> >      name += ''.join(word.title() for word in words[1:])
> >  
> > -    types = ["event"]
> > +    types = ["event", "command"]
> >      if meta in types:
> >          name = name[:-3] if name.endswith("Arg") else name
> >          name += meta.title().replace(" ", "")
> > @@ -521,6 +566,8 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor,
> >      fields, with_nullable = recursive_base(self, base)
> >      if info.defn_meta == "event":
> >          fields += f'''\tMessageTimestamp Timestamp `json:"-"`\n{fields}'''
> > +    elif info.defn_meta == "command":
> > +        fields += f'''\tMessageId string `json:"-"`\n{fields}'''
> >  
> >      if members:
> >          for member in members:
> > @@ -719,16 +766,36 @@ def generate_template_event(events: dict[str, str]) -> str:
> >  '''
> >      return TEMPLATE_EVENT.format(cases=cases)
> >  
> > +def generate_template_command(commands: dict[str, str]) -> str:
> > +    cases = ""
> > +    for name in sorted(commands):
> > +        case_type = commands[name]
> > +        cases += f'''
> > +case "{name}":
> > +    command := struct {{
> > +        Args {case_type} `json:"arguments"`
> > +    }}{{}}
> > +
> > +    if err := json.Unmarshal(data, &command); err != nil {{
> > +        return nil, fmt.Errorf("Failed to unmarshal: %s", string(data))
> > +    }}
> > +    command.Args.MessageId = base.MessageId
> > +    return &command.Args, nil
> > +'''
> > +    content = TEMPLATE_COMMAND.format(cases=cases)
> > +    return content
> > +
> >  
> >  class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
> >  
> >      def __init__(self, _: str):
> >          super().__init__()
> > -        types = ["alternate", "enum", "event", "helper", "struct", "union"]
> > +        types = ["alternate", "command", "enum", "event", "helper", "struct", "union"]
> >          self.target = {name: "" for name in types}
> >          self.objects_seen = {}
> >          self.schema = None
> >          self.events = {}
> > +        self.commands = {}
> >          self.golang_package_name = "qapi"
> >          self.accept_null_types = []
> >  
> > @@ -756,6 +823,7 @@ def visit_begin(self, schema):
> >      def visit_end(self):
> >          self.schema = None
> >          self.target["event"] += generate_template_event(self.events)
> > +        self.target["command"] += generate_template_command(self.commands)
> >  
> >      def visit_object_type(self: QAPISchemaGenGolangVisitor,
> >                            name: str,
> > @@ -853,7 +921,30 @@ def visit_command(self,
> >                        allow_oob: bool,
> >                        allow_preconfig: bool,
> >                        coroutine: bool) -> None:
> > -        pass
> > +        assert name == info.defn_name
> > +
> > +        type_name = qapi_to_go_type_name(name, info.defn_meta)
> > +        self.commands[name] = type_name
> > +
> > +        content = ""
> > +        if boxed or not arg_type or not qapi_name_is_object(arg_type.name):
> > +            args = "" if not arg_type else "\n" + arg_type.name
> > +            args += '''\n\tMessageId   string `json:"-"`'''
> > +            content = generate_struct_type(type_name, args)
> > +        else:
> > +            assert isinstance(arg_type, QAPISchemaObjectType)
> > +            content = qapi_to_golang_struct(self,
> > +                                            name,
> > +                                            arg_type.info,
> > +                                            arg_type.ifcond,
> > +                                            arg_type.features,
> > +                                            arg_type.base,
> > +                                            arg_type.members,
> > +                                            arg_type.variants)
> > +
> > +        content += TEMPLATE_COMMAND_METHODS.format(name=name,
> > +                                                   type_name=type_name)
> > +        self.target["command"] += content
> >  
> >      def visit_event(self, name, info, ifcond, features, arg_type, boxed):
> >          assert name == info.defn_name
> > -- 
> > 2.41.0
> > 
> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v1 8/9] qapi: golang: Add CommandResult type to Go

[Victor Toso]

This patch adds a struct type in Go that will handle return values
for QAPI's command types.

The return value of a Command is, encouraged to be, QAPI's complex
types or an Array of those.

Every Command has a underlying CommandResult. The EmptyCommandReturn
is for those that don't expect any data e.g: `{ "return": {} }`.

All CommandReturn types implement the CommandResult interface.

Example:
qapi:
    | { 'command': 'query-sev', 'returns': 'SevInfo',
    |   'if': 'TARGET_I386' }

go:
    | type QuerySevCommandReturn struct {
    |     CommandId string     `json:"id,omitempty"`
    |     Result    *SevInfo   `json:"return"`
    |     Error     *QapiError `json:"error,omitempty"`
    | }

usage:
    | // One can use QuerySevCommandReturn directly or
    | // command's interface GetReturnType() instead.
    |
    | input := `{ "return": { "enabled": true, "api-major" : 0,` +
    |                        `"api-minor" : 0, "build-id" : 0,` +
    |                        `"policy" : 0, "state" : "running",` +
    |                        `"handle" : 1 } } `
    |
    | ret := QuerySevCommandReturn{}
    | err := json.Unmarshal([]byte(input), &ret)
    | if ret.Error != nil {
    |     // Handle command failure {"error": { ...}}
    | } else if ret.Result != nil {
    |     // ret.Result.Enable == true
    | }

Signed-off-by: Victor Toso <victortoso@redhat.com>
---
 scripts/qapi/golang.py | 72 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 70 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
index 52a9124641..48ca0deab0 100644
--- a/scripts/qapi/golang.py
+++ b/scripts/qapi/golang.py
@@ -40,6 +40,15 @@
 '''
 
 TEMPLATE_HELPER = '''
+type QapiError struct {
+    Class       string `json:"class"`
+    Description string `json:"desc"`
+}
+
+func (err *QapiError) Error() string {
+    return fmt.Sprintf("%s: %s", err.Class, err.Description)
+}
+
 // Creates a decoder that errors on unknown Fields
 // Returns nil if successfully decoded @from payload to @into type
 // Returns error if failed to decode @from payload to @into type
@@ -254,12 +263,17 @@
 func (s *{type_name}) GetId() string {{
     return s.MessageId
 }}
+
+func (s *{type_name}) GetReturnType() CommandReturn {{
+    return &{cmd_ret_name}{{}}
+}}
 '''
 
 TEMPLATE_COMMAND = '''
 type Command interface {{
     GetId()         string
     GetName()       string
+    GetReturnType() CommandReturn
 }}
 
 func MarshalCommand(c Command) ([]byte, error) {{
@@ -292,6 +306,45 @@
 }}
 '''
 
+TEMPLATE_COMMAND_RETURN = '''
+type CommandReturn interface {
+    GetId()          string
+    GetCommandName() string
+    GetError()       error
+}
+'''
+
+TEMPLATE_COMMAND_RETURN_METHODS = '''
+type {cmd_ret_name} struct {{
+    MessageId  string                `json:"id,omitempty"`
+    Error     *QapiError             `json:"error,omitempty"`
+{result}
+}}
+
+func (r *{cmd_ret_name}) GetCommandName() string {{
+    return "{name}"
+}}
+
+func (r *{cmd_ret_name}) GetId() string {{
+    return r.MessageId
+}}
+
+func (r *{cmd_ret_name}) GetError() error {{
+    return r.Error
+}}
+
+{marshal_empty}
+'''
+
+TEMPLATE_COMMAND_RETURN_MARSHAL_EMPTY = '''
+func (r {cmd_ret_name}) MarshalJSON() ([]byte, error) {{
+    if r.Error != nil {{
+        type Alias {cmd_ret_name}
+        return json.Marshal(Alias(r))
+    }}
+    return []byte(`{{"return":{{}}}}`), nil
+}}'''
+
 def gen_golang(schema: QAPISchema,
                output_dir: str,
                prefix: str) -> None:
@@ -327,7 +380,7 @@ def qapi_to_go_type_name(name: str,
 
     name += ''.join(word.title() for word in words[1:])
 
-    types = ["event", "command"]
+    types = ["event", "command", "command return"]
     if meta in types:
         name = name[:-3] if name.endswith("Arg") else name
         name += meta.title().replace(" ", "")
@@ -783,6 +836,7 @@ def generate_template_command(commands: dict[str, str]) -> str:
     return &command.Args, nil
 '''
     content = TEMPLATE_COMMAND.format(cases=cases)
+    content += TEMPLATE_COMMAND_RETURN
     return content
 
 
@@ -926,6 +980,15 @@ def visit_command(self,
         type_name = qapi_to_go_type_name(name, info.defn_meta)
         self.commands[name] = type_name
 
+        cmd_ret_name = qapi_to_go_type_name(name, "command return")
+        marshal_empty = TEMPLATE_COMMAND_RETURN_MARSHAL_EMPTY.format(cmd_ret_name=cmd_ret_name)
+        result = ""
+        if ret_type:
+            marshal_empty = ""
+            ret_type_name = qapi_schema_type_to_go_type(ret_type.name)
+            isptr = "*" if ret_type_name[0] not in "*[" else ""
+            result = f'''Result    {isptr}{ret_type_name} `json:"return"`'''
+
         content = ""
         if boxed or not arg_type or not qapi_name_is_object(arg_type.name):
             args = "" if not arg_type else "\n" + arg_type.name
@@ -943,7 +1006,12 @@ def visit_command(self,
                                             arg_type.variants)
 
         content += TEMPLATE_COMMAND_METHODS.format(name=name,
-                                                   type_name=type_name)
+                                                   type_name=type_name,
+                                                   cmd_ret_name=cmd_ret_name)
+        content += TEMPLATE_COMMAND_RETURN_METHODS.format(name=name,
+                                                          cmd_ret_name=cmd_ret_name,
+                                                          result=result,
+                                                          marshal_empty=marshal_empty)
         self.target["command"] += content
 
     def visit_event(self, name, info, ifcond, features, arg_type, boxed):
-- 
2.41.0

Re: [PATCH v1 8/9] qapi: golang: Add CommandResult type to Go

[Daniel P. Berrangé]

On Wed, Sep 27, 2023 at 01:25:43PM +0200, Victor Toso wrote:
> This patch adds a struct type in Go that will handle return values
> for QAPI's command types.
> 
> The return value of a Command is, encouraged to be, QAPI's complex
> types or an Array of those.
> 
> Every Command has a underlying CommandResult. The EmptyCommandReturn
> is for those that don't expect any data e.g: `{ "return": {} }`.
> 
> All CommandReturn types implement the CommandResult interface.
> 
> Example:
> qapi:
>     | { 'command': 'query-sev', 'returns': 'SevInfo',
>     |   'if': 'TARGET_I386' }
> 
> go:
>     | type QuerySevCommandReturn struct {
>     |     CommandId string     `json:"id,omitempty"`
>     |     Result    *SevInfo   `json:"return"`
>     |     Error     *QapiError `json:"error,omitempty"`
>     | }
> 
> usage:
>     | // One can use QuerySevCommandReturn directly or
>     | // command's interface GetReturnType() instead.
>     |
>     | input := `{ "return": { "enabled": true, "api-major" : 0,` +
>     |                        `"api-minor" : 0, "build-id" : 0,` +
>     |                        `"policy" : 0, "state" : "running",` +
>     |                        `"handle" : 1 } } `
>     |
>     | ret := QuerySevCommandReturn{}
>     | err := json.Unmarshal([]byte(input), &ret)
>     | if ret.Error != nil {
>     |     // Handle command failure {"error": { ...}}
>     | } else if ret.Result != nil {
>     |     // ret.Result.Enable == true
>     | }
> 
> Signed-off-by: Victor Toso <victortoso@redhat.com>
> ---
>  scripts/qapi/golang.py | 72 ++++++++++++++++++++++++++++++++++++++++--
>  1 file changed, 70 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> index 52a9124641..48ca0deab0 100644
> --- a/scripts/qapi/golang.py
> +++ b/scripts/qapi/golang.py
> @@ -40,6 +40,15 @@
>  '''
>  
>  TEMPLATE_HELPER = '''
> +type QapiError struct {

QAPIError as the name for this

> +    Class       string `json:"class"`
> +    Description string `json:"desc"`
> +}

> +
> +func (err *QapiError) Error() string {
> +    return fmt.Sprintf("%s: %s", err.Class, err.Description)
> +}

My gut feeling is that this should be just

    return err.Description

on the basis that long ago we pretty much decided that the
'Class' field was broadly a waste of time  except for a
couple of niche use cases. The error description is always
self contained and sufficient to diagnose problems, without
knowing the Class.

Keep the Class field in the struct though, as it could be
useful to check in certain cases


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 8/9] qapi: golang: Add CommandResult type to Go

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 2956 bytes --]

Hi,

On Thu, Sep 28, 2023 at 04:03:12PM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 27, 2023 at 01:25:43PM +0200, Victor Toso wrote:
> > This patch adds a struct type in Go that will handle return values
> > for QAPI's command types.
> > 
> > The return value of a Command is, encouraged to be, QAPI's complex
> > types or an Array of those.
> > 
> > Every Command has a underlying CommandResult. The EmptyCommandReturn
> > is for those that don't expect any data e.g: `{ "return": {} }`.
> > 
> > All CommandReturn types implement the CommandResult interface.
> > 
> > Example:
> > qapi:
> >     | { 'command': 'query-sev', 'returns': 'SevInfo',
> >     |   'if': 'TARGET_I386' }
> > 
> > go:
> >     | type QuerySevCommandReturn struct {
> >     |     CommandId string     `json:"id,omitempty"`
> >     |     Result    *SevInfo   `json:"return"`
> >     |     Error     *QapiError `json:"error,omitempty"`
> >     | }
> > 
> > usage:
> >     | // One can use QuerySevCommandReturn directly or
> >     | // command's interface GetReturnType() instead.
> >     |
> >     | input := `{ "return": { "enabled": true, "api-major" : 0,` +
> >     |                        `"api-minor" : 0, "build-id" : 0,` +
> >     |                        `"policy" : 0, "state" : "running",` +
> >     |                        `"handle" : 1 } } `
> >     |
> >     | ret := QuerySevCommandReturn{}
> >     | err := json.Unmarshal([]byte(input), &ret)
> >     | if ret.Error != nil {
> >     |     // Handle command failure {"error": { ...}}
> >     | } else if ret.Result != nil {
> >     |     // ret.Result.Enable == true
> >     | }
> > 
> > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > ---
> >  scripts/qapi/golang.py | 72 ++++++++++++++++++++++++++++++++++++++++--
> >  1 file changed, 70 insertions(+), 2 deletions(-)
> > 
> > diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py
> > index 52a9124641..48ca0deab0 100644
> > --- a/scripts/qapi/golang.py
> > +++ b/scripts/qapi/golang.py
> > @@ -40,6 +40,15 @@
> >  '''
> >  
> >  TEMPLATE_HELPER = '''
> > +type QapiError struct {
> 
> QAPIError as the name for this

Agreed. I'll fix it.

> > +    Class       string `json:"class"`
> > +    Description string `json:"desc"`
> > +}
> 
> > +
> > +func (err *QapiError) Error() string {
> > +    return fmt.Sprintf("%s: %s", err.Class, err.Description)
> > +}
> 
> My gut feeling is that this should be just
> 
>     return err.Description
> 
> on the basis that long ago we pretty much decided that the
> 'Class' field was broadly a waste of time  except for a
> couple of niche use cases. The error description is always
> self contained and sufficient to diagnose problems, without
> knowing the Class.
> 
> Keep the Class field in the struct though, as it could be
> useful to check in certain cases

I'll trust you on this. I'll change it.

Cheers,
Victor

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v1 9/9] docs: add notes on Golang code generator

[Victor Toso]

The goal of this patch is converge discussions into a documentation,
to make it easy and explicit design decisions, known issues and what
else might help a person interested in how the Go module is generated.

Signed-off-by: Victor Toso <victortoso@redhat.com>
---
 docs/devel/qapi-golang-code-gen.rst | 341 ++++++++++++++++++++++++++++
 1 file changed, 341 insertions(+)
 create mode 100644 docs/devel/qapi-golang-code-gen.rst

diff --git a/docs/devel/qapi-golang-code-gen.rst b/docs/devel/qapi-golang-code-gen.rst
new file mode 100644
index 0000000000..2a91f8fc60
--- /dev/null
+++ b/docs/devel/qapi-golang-code-gen.rst
@@ -0,0 +1,341 @@
+==========================
+QAPI Golang code generator
+==========================
+
+..
+   Copyright (C) 2023 Red Hat, Inc.
+
+   This work is licensed under the terms of the GNU GPL, version 2 or
+   later.  See the COPYING file in the top-level directory.
+
+
+Introduction
+============
+
+This document provides information of how the generated Go code maps
+with the QAPI specification, clarifying design decisions when needed.
+
+
+Scope of the generated Go code
+==============================
+
+The scope is limited to data structures that can interpret and be used
+to generate valid QMP messages. These data structures are generated
+from a QAPI schema and should be able to handle QMP messages from the
+same schema.
+
+The generated Go code is a Go module with data structs that uses Go
+standard library `encoding/json`, implementing its field tags and
+Marshal interface whenever needed.
+
+
+QAPI types to Go structs
+========================
+
+Enum
+----
+
+Enums are mapped as strings in Go, using a specified string type per
+Enum to help with type safety in the Go application.
+
+.. code-block:: JSON
+    { 'enum': 'HostMemPolicy',
+      'data': [ 'default', 'preferred', 'bind', 'interleave' ] }
+
+.. code-block:: go
+    type HostMemPolicy string
+
+    const (
+        HostMemPolicyDefault    HostMemPolicy = "default"
+        HostMemPolicyPreferred  HostMemPolicy = "preferred"
+        HostMemPolicyBind       HostMemPolicy = "bind"
+        HostMemPolicyInterleave HostMemPolicy = "interleave"
+    )
+
+
+Struct
+------
+
+The mapping between a QAPI struct in Go struct is very straightforward.
+ - Each member of the QAPI struct has its own field in a Go struct.
+ - Optional members are pointers type with 'omitempty' field tag set
+
+One important design decision was to _not_ embed base struct, copying
+the base members to the original struct. This reduces the complexity
+for the Go application.
+
+.. code-block:: JSON
+    { 'struct': 'BlockExportOptionsNbdBase',
+      'data': { '*name': 'str', '*description': 'str' } }
+
+    { 'struct': 'BlockExportOptionsNbd',
+      'base': 'BlockExportOptionsNbdBase',
+      'data': { '*bitmaps': ['BlockDirtyBitmapOrStr'],
+                '*allocation-depth': 'bool' } }
+
+.. code-block:: go
+    type BlockExportOptionsNbd struct {
+        Name        *string `json:"name,omitempty"`
+        Description *string `json:"description,omitempty"`
+
+        Bitmaps         []BlockDirtyBitmapOrStr `json:"bitmaps,omitempty"`
+        AllocationDepth *bool                   `json:"allocation-depth,omitempty"`
+    }
+
+
+Union
+-----
+
+Unions in QAPI are binded to a Enum type which provides all possible
+branches of the union. The most important caveat here is that the Union
+does not need to implement all possible branches for the Enum.
+Receiving a enum value of a unimplemented branch is valid. For this
+reason, we keep a discriminator field in each Union Go struct and also
+implement the Marshal interface.
+
+As each Union Go struct type has both the discriminator field and
+optional fields, it is important to note that when converting Go struct
+to JSON, we only consider the discriminator field if no optional field
+member was set. In practice, the user should use the optional fields if
+the QAPI Union type has defined them, otherwise the user can set the
+discriminator field for the unbranched enum value.
+
+.. code-block:: JSON
+    { 'union': 'ImageInfoSpecificQCow2Encryption',
+      'base': 'ImageInfoSpecificQCow2EncryptionBase',
+      'discriminator': 'format',
+      'data': { 'luks': 'QCryptoBlockInfoLUKS' } }
+
+.. code-block:: go
+    type ImageInfoSpecificQCow2Encryption struct {
+        Format BlockdevQcow2EncryptionFormat `json:"format"`
+
+        // Variants fields
+        Luks *QCryptoBlockInfoLUKS `json:"-"`
+    }
+
+    func (s ImageInfoSpecificQCow2Encryption) MarshalJSON() ([]byte, error) {
+        // Normal logic goes here
+        // ...
+
+        // Check for valid values without field members
+        if len(bytes) == 0 && err == nil &&
+            (s.Format == BlockdevQcow2EncryptionFormatAes) {
+            type Alias ImageInfoSpecificQCow2Encryption
+            bytes, err = json.Marshal(Alias(s))
+        }
+        // ...
+    }
+
+
+    func (s *ImageInfoSpecificQCow2Encryption) UnmarshalJSON(data []byte) error {
+        // Normal logic goes here
+        // ...
+
+        switch tmp.Format {
+        case BlockdevQcow2EncryptionFormatLuks:
+            // ...
+        default:
+            // Check for valid values without field members
+            if tmp.Format != BlockdevQcow2EncryptionFormatAes {
+                return fmt.Errorf(...)
+            }
+        }
+        return nil
+    }
+
+
+Alternate
+---------
+
+Like Unions, alternates can have a few branches. Unlike Unions, they
+don't have a discriminator field and each branch should be a different
+class of Type entirely (e.g: You can't have two branches of type int in
+one Alternate).
+
+While the marshalling is similar to Unions, the unmarshalling uses a
+try-and-error approach, trying to fit the data payload in one of the
+Alternate fields.
+
+The biggest caveat is handling Alternates that can take JSON Null as
+value. The issue lies on `encoding/json` library limitation where
+unmarshalling JSON Null data to a Go struct which has the 'omitempty'
+field that, it bypass the Marshal interface. The same happens when
+marshalling, if the field tag 'omitempty' is used, a nil pointer would
+never be translated to null JSON value.
+
+The problem being, we use pointer to type plus `omitempty` field to
+express a QAPI optional member.
+
+In order to handle JSON Null, the generator needs to do the following:
+  - Read the QAPI schema prior to generate any code and cache
+    all alternate types that can take JSON Null
+  - For all Go structs that should be considered optional and they type
+    are one of those alternates, do not set `omitempty` and implement
+    Marshal interface for this Go struct, to properly handle JSON Null
+  - In the Alternate, uses a boolean 'IsNull' to express a JSON Null
+    and implement the AbsentAlternate interface, to help sturcts know
+    if a given Alternate type should be considered Absent (not set) or
+    any other possible Value, including JSON Null.
+
+.. code-block:: JSON
+    { 'alternate': 'BlockdevRefOrNull',
+      'data': { 'definition': 'BlockdevOptions',
+                'reference': 'str',
+                'null': 'null' } }
+
+.. code-block:: go
+    type BlockdevRefOrNull struct {
+        Definition *BlockdevOptions
+        Reference  *string
+        IsNull     bool
+    }
+
+    func (s *BlockdevRefOrNull) ToAnyOrAbsent() (any, bool) {
+        if s != nil {
+            if s.IsNull {
+                return nil, false
+            } else if s.Definition != nil {
+                return *s.Definition, false
+            } else if s.Reference != nil {
+                return *s.Reference, false
+            }
+        }
+
+        return nil, true
+    }
+
+    func (s BlockdevRefOrNull) MarshalJSON() ([]byte, error) {
+        if s.IsNull {
+            return []byte("null"), nil
+        } else if s.Definition != nil {
+            return json.Marshal(s.Definition)
+        } else if s.Reference != nil {
+            return json.Marshal(s.Reference)
+        }
+        return []byte("{}"), nil
+    }
+
+    func (s *BlockdevRefOrNull) UnmarshalJSON(data []byte) error {
+        // Check for json-null first
+        if string(data) == "null" {
+            s.IsNull = true
+            return nil
+        }
+        // Check for BlockdevOptions
+        {
+            s.Definition = new(BlockdevOptions)
+            if err := StrictDecode(s.Definition, data); err == nil {
+                return nil
+            }
+            s.Definition = nil
+        }
+        // Check for string
+        {
+            s.Reference = new(string)
+            if err := StrictDecode(s.Reference, data); err == nil {
+                return nil
+            }
+            s.Reference = nil
+        }
+
+        return fmt.Errorf("Can't convert to BlockdevRefOrNull: %s", string(data))
+    }
+
+
+Event
+-----
+
+All events are mapped to its own struct with the additional
+MessageTimestamp field, for the over-the-wire 'timestamp' value.
+
+Marshaling and Unmarshaling happens over the Event interface, so users
+should use the MarshalEvent() and UnmarshalEvent() methods.
+
+.. code-block:: JSON
+    { 'event': 'SHUTDOWN',
+      'data': { 'guest': 'bool',
+                'reason': 'ShutdownCause' } }
+
+.. code-block:: go
+    type Event interface {
+        GetName() string
+        GetTimestamp() Timestamp
+    }
+
+    type ShutdownEvent struct {
+        MessageTimestamp Timestamp     `json:"-"`
+        Guest            bool          `json:"guest"`
+        Reason           ShutdownCause `json:"reason"`
+    }
+
+    func (s *ShutdownEvent) GetName() string {
+        return "SHUTDOWN"
+    }
+
+    func (s *ShutdownEvent) GetTimestamp() Timestamp {
+        return s.MessageTimestamp
+    }
+
+
+Command
+-------
+
+All commands are mapped to its own struct with the additional MessageId
+field for the optional 'id'. If the command has a boxed data struct,
+the option struct will be embed in the command struct.
+
+As commands do require a return value, every command has its own return
+type. The Command interface has a GetReturnType() method that returns a
+CommandReturn interface, to help Go application handling the data.
+
+Marshaling and Unmarshaling happens over the Command interface, so
+users should use the MarshalCommand() and UnmarshalCommand() methods.
+
+.. code-block:: JSON
+   { 'command': 'set_password',
+     'boxed': true,
+     'data': 'SetPasswordOptions' }
+
+.. code-block:: go
+    type Command interface {
+        GetId() string
+        GetName() string
+        GetReturnType() CommandReturn
+    }
+
+    // SetPasswordOptions is embed
+    type SetPasswordCommand struct {
+        SetPasswordOptions
+        MessageId string `json:"-"`
+    }
+
+    // This is an union
+    type SetPasswordOptions struct {
+        Protocol  DisplayProtocol    `json:"protocol"`
+        Password  string             `json:"password"`
+        Connected *SetPasswordAction `json:"connected,omitempty"`
+
+        // Variants fields
+        Vnc *SetPasswordOptionsVnc `json:"-"`
+    }
+
+Now an example of a command without boxed type.
+
+.. code-block:: JSON
+    { 'command': 'set_link',
+      'data': {'name': 'str', 'up': 'bool'} }
+
+.. code-block:: go
+    type SetLinkCommand struct {
+        MessageId string `json:"-"`
+        Name      string `json:"name"`
+        Up        bool   `json:"up"`
+    }
+
+Known issues
+============
+
+- Type names might not follow proper Go convention. Andrea suggested an
+  annotation to the QAPI schema that could solve it.
+  https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg00127.html
-- 
2.41.0

Re: [PATCH v1 9/9] docs: add notes on Golang code generator

[Daniel P. Berrangé]

On Wed, Sep 27, 2023 at 01:25:44PM +0200, Victor Toso wrote:
> The goal of this patch is converge discussions into a documentation,
> to make it easy and explicit design decisions, known issues and what
> else might help a person interested in how the Go module is generated.
> 
> Signed-off-by: Victor Toso <victortoso@redhat.com>
> ---
>  docs/devel/qapi-golang-code-gen.rst | 341 ++++++++++++++++++++++++++++

docs/devel/index.rst needs editting to reference this new doc to
prevent

  Warning, treated as error:
  /var/home/berrange/src/virt/qemu/docs/devel/qapi-golang-code-gen.rst:document isn't included in any toctree
  ninja: build stopped: subcommand failed.


>  1 file changed, 341 insertions(+)
>  create mode 100644 docs/devel/qapi-golang-code-gen.rst
> 
> diff --git a/docs/devel/qapi-golang-code-gen.rst b/docs/devel/qapi-golang-code-gen.rst
> new file mode 100644
> index 0000000000..2a91f8fc60
> --- /dev/null
> +++ b/docs/devel/qapi-golang-code-gen.rst
> @@ -0,0 +1,341 @@
> +==========================
> +QAPI Golang code generator
> +==========================
> +
> +..
> +   Copyright (C) 2023 Red Hat, Inc.
> +
> +   This work is licensed under the terms of the GNU GPL, version 2 or
> +   later.  See the COPYING file in the top-level directory.
> +
> +
> +Introduction
> +============
> +
> +This document provides information of how the generated Go code maps
> +with the QAPI specification, clarifying design decisions when needed.
> +
> +
> +Scope of the generated Go code
> +==============================
> +
> +The scope is limited to data structures that can interpret and be used
> +to generate valid QMP messages. These data structures are generated
> +from a QAPI schema and should be able to handle QMP messages from the
> +same schema.
> +
> +The generated Go code is a Go module with data structs that uses Go
> +standard library `encoding/json`, implementing its field tags and
> +Marshal interface whenever needed.


Needs to use `` instead of `

  Warning, treated as error:
  /var/home/berrange/src/virt/qemu/docs/devel/qapi-golang-code-gen.rst:27:'any' reference target not found: encoding/json
  ninja: build stopped: subcommand failed.


Repeated several other placs.

> +
> +
> +QAPI types to Go structs
> +========================
> +
> +Enum
> +----
> +
> +Enums are mapped as strings in Go, using a specified string type per
> +Enum to help with type safety in the Go application.
> +
> +.. code-block:: JSON
> +    { 'enum': 'HostMemPolicy',
> +      'data': [ 'default', 'preferred', 'bind', 'interleave' ] }

Needs a blank line after every 'code-block:: JSON' or build fails
with:

Warning, treated as error:
/var/home/berrange/src/virt/qemu/docs/devel/qapi-golang-code-gen.rst:41:Error in "code-block" directive:
maximum 1 argument(s) allowed, 12 supplied.

.. code-block:: JSON
    { 'enum': 'HostMemPolicy',
      'data': [ 'default', 'preferred', 'bind', 'interleave' ] }
ninja: build stopped: subcommand failed.


If fixing that then it still isn't happy for reasons I can't
immediately figure out.

Warning, treated as error:
/var/home/berrange/src/virt/qemu/docs/devel/qapi-golang-code-gen.rst:41:Could not lex literal_block as "JSON". Highlighting skipped.
ninja: build stopped: subcommand failed.



> +
> +.. code-block:: go
> +    type HostMemPolicy string
> +
> +    const (
> +        HostMemPolicyDefault    HostMemPolicy = "default"
> +        HostMemPolicyPreferred  HostMemPolicy = "preferred"
> +        HostMemPolicyBind       HostMemPolicy = "bind"
> +        HostMemPolicyInterleave HostMemPolicy = "interleave"
> +    )
> +
> +
> +Struct
> +------
> +
> +The mapping between a QAPI struct in Go struct is very straightforward.
> + - Each member of the QAPI struct has its own field in a Go struct.
> + - Optional members are pointers type with 'omitempty' field tag set
> +
> +One important design decision was to _not_ embed base struct, copying
> +the base members to the original struct. This reduces the complexity
> +for the Go application.
> +
> +.. code-block:: JSON
> +    { 'struct': 'BlockExportOptionsNbdBase',
> +      'data': { '*name': 'str', '*description': 'str' } }
> +
> +    { 'struct': 'BlockExportOptionsNbd',
> +      'base': 'BlockExportOptionsNbdBase',
> +      'data': { '*bitmaps': ['BlockDirtyBitmapOrStr'],
> +                '*allocation-depth': 'bool' } }
> +
> +.. code-block:: go
> +    type BlockExportOptionsNbd struct {
> +        Name        *string `json:"name,omitempty"`
> +        Description *string `json:"description,omitempty"`
> +
> +        Bitmaps         []BlockDirtyBitmapOrStr `json:"bitmaps,omitempty"`
> +        AllocationDepth *bool                   `json:"allocation-depth,omitempty"`
> +    }
> +
> +
> +Union
> +-----
> +
> +Unions in QAPI are binded to a Enum type which provides all possible
> +branches of the union. The most important caveat here is that the Union
> +does not need to implement all possible branches for the Enum.
> +Receiving a enum value of a unimplemented branch is valid. For this
> +reason, we keep a discriminator field in each Union Go struct and also
> +implement the Marshal interface.
> +
> +As each Union Go struct type has both the discriminator field and
> +optional fields, it is important to note that when converting Go struct
> +to JSON, we only consider the discriminator field if no optional field
> +member was set. In practice, the user should use the optional fields if
> +the QAPI Union type has defined them, otherwise the user can set the
> +discriminator field for the unbranched enum value.
> +
> +.. code-block:: JSON
> +    { 'union': 'ImageInfoSpecificQCow2Encryption',
> +      'base': 'ImageInfoSpecificQCow2EncryptionBase',
> +      'discriminator': 'format',
> +      'data': { 'luks': 'QCryptoBlockInfoLUKS' } }
> +
> +.. code-block:: go
> +    type ImageInfoSpecificQCow2Encryption struct {
> +        Format BlockdevQcow2EncryptionFormat `json:"format"`
> +
> +        // Variants fields
> +        Luks *QCryptoBlockInfoLUKS `json:"-"`
> +    }
> +
> +    func (s ImageInfoSpecificQCow2Encryption) MarshalJSON() ([]byte, error) {
> +        // Normal logic goes here
> +        // ...
> +
> +        // Check for valid values without field members
> +        if len(bytes) == 0 && err == nil &&
> +            (s.Format == BlockdevQcow2EncryptionFormatAes) {
> +            type Alias ImageInfoSpecificQCow2Encryption
> +            bytes, err = json.Marshal(Alias(s))
> +        }
> +        // ...
> +    }
> +
> +
> +    func (s *ImageInfoSpecificQCow2Encryption) UnmarshalJSON(data []byte) error {
> +        // Normal logic goes here
> +        // ...
> +
> +        switch tmp.Format {
> +        case BlockdevQcow2EncryptionFormatLuks:
> +            // ...
> +        default:
> +            // Check for valid values without field members
> +            if tmp.Format != BlockdevQcow2EncryptionFormatAes {
> +                return fmt.Errorf(...)
> +            }
> +        }
> +        return nil
> +    }
> +
> +
> +Alternate
> +---------
> +
> +Like Unions, alternates can have a few branches. Unlike Unions, they
> +don't have a discriminator field and each branch should be a different
> +class of Type entirely (e.g: You can't have two branches of type int in
> +one Alternate).
> +
> +While the marshalling is similar to Unions, the unmarshalling uses a
> +try-and-error approach, trying to fit the data payload in one of the
> +Alternate fields.
> +
> +The biggest caveat is handling Alternates that can take JSON Null as
> +value. The issue lies on `encoding/json` library limitation where
> +unmarshalling JSON Null data to a Go struct which has the 'omitempty'
> +field that, it bypass the Marshal interface. The same happens when
> +marshalling, if the field tag 'omitempty' is used, a nil pointer would
> +never be translated to null JSON value.
> +
> +The problem being, we use pointer to type plus `omitempty` field to
> +express a QAPI optional member.
> +
> +In order to handle JSON Null, the generator needs to do the following:
> +  - Read the QAPI schema prior to generate any code and cache
> +    all alternate types that can take JSON Null
> +  - For all Go structs that should be considered optional and they type
> +    are one of those alternates, do not set `omitempty` and implement
> +    Marshal interface for this Go struct, to properly handle JSON Null
> +  - In the Alternate, uses a boolean 'IsNull' to express a JSON Null
> +    and implement the AbsentAlternate interface, to help sturcts know
> +    if a given Alternate type should be considered Absent (not set) or
> +    any other possible Value, including JSON Null.
> +
> +.. code-block:: JSON
> +    { 'alternate': 'BlockdevRefOrNull',
> +      'data': { 'definition': 'BlockdevOptions',
> +                'reference': 'str',
> +                'null': 'null' } }
> +
> +.. code-block:: go
> +    type BlockdevRefOrNull struct {
> +        Definition *BlockdevOptions
> +        Reference  *string
> +        IsNull     bool
> +    }
> +
> +    func (s *BlockdevRefOrNull) ToAnyOrAbsent() (any, bool) {
> +        if s != nil {
> +            if s.IsNull {
> +                return nil, false
> +            } else if s.Definition != nil {
> +                return *s.Definition, false
> +            } else if s.Reference != nil {
> +                return *s.Reference, false
> +            }
> +        }
> +
> +        return nil, true
> +    }
> +
> +    func (s BlockdevRefOrNull) MarshalJSON() ([]byte, error) {
> +        if s.IsNull {
> +            return []byte("null"), nil
> +        } else if s.Definition != nil {
> +            return json.Marshal(s.Definition)
> +        } else if s.Reference != nil {
> +            return json.Marshal(s.Reference)
> +        }
> +        return []byte("{}"), nil
> +    }
> +
> +    func (s *BlockdevRefOrNull) UnmarshalJSON(data []byte) error {
> +        // Check for json-null first
> +        if string(data) == "null" {
> +            s.IsNull = true
> +            return nil
> +        }
> +        // Check for BlockdevOptions
> +        {
> +            s.Definition = new(BlockdevOptions)
> +            if err := StrictDecode(s.Definition, data); err == nil {
> +                return nil
> +            }
> +            s.Definition = nil
> +        }
> +        // Check for string
> +        {
> +            s.Reference = new(string)
> +            if err := StrictDecode(s.Reference, data); err == nil {
> +                return nil
> +            }
> +            s.Reference = nil
> +        }
> +
> +        return fmt.Errorf("Can't convert to BlockdevRefOrNull: %s", string(data))
> +    }
> +
> +
> +Event
> +-----
> +
> +All events are mapped to its own struct with the additional
> +MessageTimestamp field, for the over-the-wire 'timestamp' value.
> +
> +Marshaling and Unmarshaling happens over the Event interface, so users
> +should use the MarshalEvent() and UnmarshalEvent() methods.
> +
> +.. code-block:: JSON
> +    { 'event': 'SHUTDOWN',
> +      'data': { 'guest': 'bool',
> +                'reason': 'ShutdownCause' } }
> +
> +.. code-block:: go
> +    type Event interface {
> +        GetName() string
> +        GetTimestamp() Timestamp
> +    }
> +
> +    type ShutdownEvent struct {
> +        MessageTimestamp Timestamp     `json:"-"`
> +        Guest            bool          `json:"guest"`
> +        Reason           ShutdownCause `json:"reason"`
> +    }
> +
> +    func (s *ShutdownEvent) GetName() string {
> +        return "SHUTDOWN"
> +    }
> +
> +    func (s *ShutdownEvent) GetTimestamp() Timestamp {
> +        return s.MessageTimestamp
> +    }
> +
> +
> +Command
> +-------
> +
> +All commands are mapped to its own struct with the additional MessageId
> +field for the optional 'id'. If the command has a boxed data struct,
> +the option struct will be embed in the command struct.
> +
> +As commands do require a return value, every command has its own return
> +type. The Command interface has a GetReturnType() method that returns a
> +CommandReturn interface, to help Go application handling the data.
> +
> +Marshaling and Unmarshaling happens over the Command interface, so
> +users should use the MarshalCommand() and UnmarshalCommand() methods.
> +
> +.. code-block:: JSON
> +   { 'command': 'set_password',
> +     'boxed': true,
> +     'data': 'SetPasswordOptions' }
> +
> +.. code-block:: go
> +    type Command interface {
> +        GetId() string
> +        GetName() string
> +        GetReturnType() CommandReturn
> +    }
> +
> +    // SetPasswordOptions is embed
> +    type SetPasswordCommand struct {
> +        SetPasswordOptions
> +        MessageId string `json:"-"`
> +    }
> +
> +    // This is an union
> +    type SetPasswordOptions struct {
> +        Protocol  DisplayProtocol    `json:"protocol"`
> +        Password  string             `json:"password"`
> +        Connected *SetPasswordAction `json:"connected,omitempty"`
> +
> +        // Variants fields
> +        Vnc *SetPasswordOptionsVnc `json:"-"`
> +    }
> +
> +Now an example of a command without boxed type.
> +
> +.. code-block:: JSON
> +    { 'command': 'set_link',
> +      'data': {'name': 'str', 'up': 'bool'} }
> +
> +.. code-block:: go
> +    type SetLinkCommand struct {
> +        MessageId string `json:"-"`
> +        Name      string `json:"name"`
> +        Up        bool   `json:"up"`
> +    }
> +
> +Known issues
> +============
> +
> +- Type names might not follow proper Go convention. Andrea suggested an
> +  annotation to the QAPI schema that could solve it.
> +  https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg00127.html
> -- 
> 2.41.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 9/9] docs: add notes on Golang code generator

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 15854 bytes --]

Hi,

Sigh. I'm sorry about this one. I've added the doc after
reorganizing the patch series. I've been overly focused on python
script and the generated part and less so with checks in qemu
itself.

On Thu, Sep 28, 2023 at 02:22:06PM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 27, 2023 at 01:25:44PM +0200, Victor Toso wrote:
> > The goal of this patch is converge discussions into a documentation,
> > to make it easy and explicit design decisions, known issues and what
> > else might help a person interested in how the Go module is generated.
> > 
> > Signed-off-by: Victor Toso <victortoso@redhat.com>
> > ---
> >  docs/devel/qapi-golang-code-gen.rst | 341 ++++++++++++++++++++++++++++
> 
> docs/devel/index.rst needs editting to reference this new doc to
> prevent

I've added it in docs/devel/index-build.rst, together with
qapi-code-gen.

>   Warning, treated as error:
>   /var/home/berrange/src/virt/qemu/docs/devel/qapi-golang-code-gen.rst:document isn't included in any toctree
>   ninja: build stopped: subcommand failed.
> 
> 
> >  1 file changed, 341 insertions(+)
> >  create mode 100644 docs/devel/qapi-golang-code-gen.rst
> > 
> > diff --git a/docs/devel/qapi-golang-code-gen.rst b/docs/devel/qapi-golang-code-gen.rst
> > new file mode 100644
> > index 0000000000..2a91f8fc60
> > --- /dev/null
> > +++ b/docs/devel/qapi-golang-code-gen.rst
> > @@ -0,0 +1,341 @@
> > +==========================
> > +QAPI Golang code generator
> > +==========================
> > +
> > +..
> > +   Copyright (C) 2023 Red Hat, Inc.
> > +
> > +   This work is licensed under the terms of the GNU GPL, version 2 or
> > +   later.  See the COPYING file in the top-level directory.
> > +
> > +
> > +Introduction
> > +============
> > +
> > +This document provides information of how the generated Go code maps
> > +with the QAPI specification, clarifying design decisions when needed.
> > +
> > +
> > +Scope of the generated Go code
> > +==============================
> > +
> > +The scope is limited to data structures that can interpret and be used
> > +to generate valid QMP messages. These data structures are generated
> > +from a QAPI schema and should be able to handle QMP messages from the
> > +same schema.
> > +
> > +The generated Go code is a Go module with data structs that uses Go
> > +standard library `encoding/json`, implementing its field tags and
> > +Marshal interface whenever needed.
> 
> 
> Needs to use `` instead of `
> 
>   Warning, treated as error:
>   /var/home/berrange/src/virt/qemu/docs/devel/qapi-golang-code-gen.rst:27:'any' reference target not found: encoding/json
>   ninja: build stopped: subcommand failed.
> 
> 
> Repeated several other placs.

Fixed them all.

> > +
> > +
> > +QAPI types to Go structs
> > +========================
> > +
> > +Enum
> > +----
> > +
> > +Enums are mapped as strings in Go, using a specified string type per
> > +Enum to help with type safety in the Go application.
> > +
> > +.. code-block:: JSON
> > +    { 'enum': 'HostMemPolicy',
> > +      'data': [ 'default', 'preferred', 'bind', 'interleave' ] }
> 
> Needs a blank line after every 'code-block:: JSON' or build fails
> with:
> 
> Warning, treated as error:
> /var/home/berrange/src/virt/qemu/docs/devel/qapi-golang-code-gen.rst:41:Error in "code-block" directive:
> maximum 1 argument(s) allowed, 12 supplied.
> 
> .. code-block:: JSON
>     { 'enum': 'HostMemPolicy',
>       'data': [ 'default', 'preferred', 'bind', 'interleave' ] }
> ninja: build stopped: subcommand failed.
> 
> 
> If fixing that then it still isn't happy for reasons I can't
> immediately figure out.

Not sure either. I can reproduce. While looking around, saw Peter
had this issue in May too

    https://lists.nongnu.org/archive/html/qemu-devel/2023-05/msg04398.html

So I'm following him, changing this from code block to a literal
block instead.

After this, build works as expected.

Cheers,
Victor
 
> Warning, treated as error:
> /var/home/berrange/src/virt/qemu/docs/devel/qapi-golang-code-gen.rst:41:Could not lex literal_block as "JSON". Highlighting skipped.
> ninja: build stopped: subcommand failed.
> 
> > +
> > +.. code-block:: go
> > +    type HostMemPolicy string
> > +
> > +    const (
> > +        HostMemPolicyDefault    HostMemPolicy = "default"
> > +        HostMemPolicyPreferred  HostMemPolicy = "preferred"
> > +        HostMemPolicyBind       HostMemPolicy = "bind"
> > +        HostMemPolicyInterleave HostMemPolicy = "interleave"
> > +    )
> > +
> > +
> > +Struct
> > +------
> > +
> > +The mapping between a QAPI struct in Go struct is very straightforward.
> > + - Each member of the QAPI struct has its own field in a Go struct.
> > + - Optional members are pointers type with 'omitempty' field tag set
> > +
> > +One important design decision was to _not_ embed base struct, copying
> > +the base members to the original struct. This reduces the complexity
> > +for the Go application.
> > +
> > +.. code-block:: JSON
> > +    { 'struct': 'BlockExportOptionsNbdBase',
> > +      'data': { '*name': 'str', '*description': 'str' } }
> > +
> > +    { 'struct': 'BlockExportOptionsNbd',
> > +      'base': 'BlockExportOptionsNbdBase',
> > +      'data': { '*bitmaps': ['BlockDirtyBitmapOrStr'],
> > +                '*allocation-depth': 'bool' } }
> > +
> > +.. code-block:: go
> > +    type BlockExportOptionsNbd struct {
> > +        Name        *string `json:"name,omitempty"`
> > +        Description *string `json:"description,omitempty"`
> > +
> > +        Bitmaps         []BlockDirtyBitmapOrStr `json:"bitmaps,omitempty"`
> > +        AllocationDepth *bool                   `json:"allocation-depth,omitempty"`
> > +    }
> > +
> > +
> > +Union
> > +-----
> > +
> > +Unions in QAPI are binded to a Enum type which provides all possible
> > +branches of the union. The most important caveat here is that the Union
> > +does not need to implement all possible branches for the Enum.
> > +Receiving a enum value of a unimplemented branch is valid. For this
> > +reason, we keep a discriminator field in each Union Go struct and also
> > +implement the Marshal interface.
> > +
> > +As each Union Go struct type has both the discriminator field and
> > +optional fields, it is important to note that when converting Go struct
> > +to JSON, we only consider the discriminator field if no optional field
> > +member was set. In practice, the user should use the optional fields if
> > +the QAPI Union type has defined them, otherwise the user can set the
> > +discriminator field for the unbranched enum value.
> > +
> > +.. code-block:: JSON
> > +    { 'union': 'ImageInfoSpecificQCow2Encryption',
> > +      'base': 'ImageInfoSpecificQCow2EncryptionBase',
> > +      'discriminator': 'format',
> > +      'data': { 'luks': 'QCryptoBlockInfoLUKS' } }
> > +
> > +.. code-block:: go
> > +    type ImageInfoSpecificQCow2Encryption struct {
> > +        Format BlockdevQcow2EncryptionFormat `json:"format"`
> > +
> > +        // Variants fields
> > +        Luks *QCryptoBlockInfoLUKS `json:"-"`
> > +    }
> > +
> > +    func (s ImageInfoSpecificQCow2Encryption) MarshalJSON() ([]byte, error) {
> > +        // Normal logic goes here
> > +        // ...
> > +
> > +        // Check for valid values without field members
> > +        if len(bytes) == 0 && err == nil &&
> > +            (s.Format == BlockdevQcow2EncryptionFormatAes) {
> > +            type Alias ImageInfoSpecificQCow2Encryption
> > +            bytes, err = json.Marshal(Alias(s))
> > +        }
> > +        // ...
> > +    }
> > +
> > +
> > +    func (s *ImageInfoSpecificQCow2Encryption) UnmarshalJSON(data []byte) error {
> > +        // Normal logic goes here
> > +        // ...
> > +
> > +        switch tmp.Format {
> > +        case BlockdevQcow2EncryptionFormatLuks:
> > +            // ...
> > +        default:
> > +            // Check for valid values without field members
> > +            if tmp.Format != BlockdevQcow2EncryptionFormatAes {
> > +                return fmt.Errorf(...)
> > +            }
> > +        }
> > +        return nil
> > +    }
> > +
> > +
> > +Alternate
> > +---------
> > +
> > +Like Unions, alternates can have a few branches. Unlike Unions, they
> > +don't have a discriminator field and each branch should be a different
> > +class of Type entirely (e.g: You can't have two branches of type int in
> > +one Alternate).
> > +
> > +While the marshalling is similar to Unions, the unmarshalling uses a
> > +try-and-error approach, trying to fit the data payload in one of the
> > +Alternate fields.
> > +
> > +The biggest caveat is handling Alternates that can take JSON Null as
> > +value. The issue lies on `encoding/json` library limitation where
> > +unmarshalling JSON Null data to a Go struct which has the 'omitempty'
> > +field that, it bypass the Marshal interface. The same happens when
> > +marshalling, if the field tag 'omitempty' is used, a nil pointer would
> > +never be translated to null JSON value.
> > +
> > +The problem being, we use pointer to type plus `omitempty` field to
> > +express a QAPI optional member.
> > +
> > +In order to handle JSON Null, the generator needs to do the following:
> > +  - Read the QAPI schema prior to generate any code and cache
> > +    all alternate types that can take JSON Null
> > +  - For all Go structs that should be considered optional and they type
> > +    are one of those alternates, do not set `omitempty` and implement
> > +    Marshal interface for this Go struct, to properly handle JSON Null
> > +  - In the Alternate, uses a boolean 'IsNull' to express a JSON Null
> > +    and implement the AbsentAlternate interface, to help sturcts know
> > +    if a given Alternate type should be considered Absent (not set) or
> > +    any other possible Value, including JSON Null.
> > +
> > +.. code-block:: JSON
> > +    { 'alternate': 'BlockdevRefOrNull',
> > +      'data': { 'definition': 'BlockdevOptions',
> > +                'reference': 'str',
> > +                'null': 'null' } }
> > +
> > +.. code-block:: go
> > +    type BlockdevRefOrNull struct {
> > +        Definition *BlockdevOptions
> > +        Reference  *string
> > +        IsNull     bool
> > +    }
> > +
> > +    func (s *BlockdevRefOrNull) ToAnyOrAbsent() (any, bool) {
> > +        if s != nil {
> > +            if s.IsNull {
> > +                return nil, false
> > +            } else if s.Definition != nil {
> > +                return *s.Definition, false
> > +            } else if s.Reference != nil {
> > +                return *s.Reference, false
> > +            }
> > +        }
> > +
> > +        return nil, true
> > +    }
> > +
> > +    func (s BlockdevRefOrNull) MarshalJSON() ([]byte, error) {
> > +        if s.IsNull {
> > +            return []byte("null"), nil
> > +        } else if s.Definition != nil {
> > +            return json.Marshal(s.Definition)
> > +        } else if s.Reference != nil {
> > +            return json.Marshal(s.Reference)
> > +        }
> > +        return []byte("{}"), nil
> > +    }
> > +
> > +    func (s *BlockdevRefOrNull) UnmarshalJSON(data []byte) error {
> > +        // Check for json-null first
> > +        if string(data) == "null" {
> > +            s.IsNull = true
> > +            return nil
> > +        }
> > +        // Check for BlockdevOptions
> > +        {
> > +            s.Definition = new(BlockdevOptions)
> > +            if err := StrictDecode(s.Definition, data); err == nil {
> > +                return nil
> > +            }
> > +            s.Definition = nil
> > +        }
> > +        // Check for string
> > +        {
> > +            s.Reference = new(string)
> > +            if err := StrictDecode(s.Reference, data); err == nil {
> > +                return nil
> > +            }
> > +            s.Reference = nil
> > +        }
> > +
> > +        return fmt.Errorf("Can't convert to BlockdevRefOrNull: %s", string(data))
> > +    }
> > +
> > +
> > +Event
> > +-----
> > +
> > +All events are mapped to its own struct with the additional
> > +MessageTimestamp field, for the over-the-wire 'timestamp' value.
> > +
> > +Marshaling and Unmarshaling happens over the Event interface, so users
> > +should use the MarshalEvent() and UnmarshalEvent() methods.
> > +
> > +.. code-block:: JSON
> > +    { 'event': 'SHUTDOWN',
> > +      'data': { 'guest': 'bool',
> > +                'reason': 'ShutdownCause' } }
> > +
> > +.. code-block:: go
> > +    type Event interface {
> > +        GetName() string
> > +        GetTimestamp() Timestamp
> > +    }
> > +
> > +    type ShutdownEvent struct {
> > +        MessageTimestamp Timestamp     `json:"-"`
> > +        Guest            bool          `json:"guest"`
> > +        Reason           ShutdownCause `json:"reason"`
> > +    }
> > +
> > +    func (s *ShutdownEvent) GetName() string {
> > +        return "SHUTDOWN"
> > +    }
> > +
> > +    func (s *ShutdownEvent) GetTimestamp() Timestamp {
> > +        return s.MessageTimestamp
> > +    }
> > +
> > +
> > +Command
> > +-------
> > +
> > +All commands are mapped to its own struct with the additional MessageId
> > +field for the optional 'id'. If the command has a boxed data struct,
> > +the option struct will be embed in the command struct.
> > +
> > +As commands do require a return value, every command has its own return
> > +type. The Command interface has a GetReturnType() method that returns a
> > +CommandReturn interface, to help Go application handling the data.
> > +
> > +Marshaling and Unmarshaling happens over the Command interface, so
> > +users should use the MarshalCommand() and UnmarshalCommand() methods.
> > +
> > +.. code-block:: JSON
> > +   { 'command': 'set_password',
> > +     'boxed': true,
> > +     'data': 'SetPasswordOptions' }
> > +
> > +.. code-block:: go
> > +    type Command interface {
> > +        GetId() string
> > +        GetName() string
> > +        GetReturnType() CommandReturn
> > +    }
> > +
> > +    // SetPasswordOptions is embed
> > +    type SetPasswordCommand struct {
> > +        SetPasswordOptions
> > +        MessageId string `json:"-"`
> > +    }
> > +
> > +    // This is an union
> > +    type SetPasswordOptions struct {
> > +        Protocol  DisplayProtocol    `json:"protocol"`
> > +        Password  string             `json:"password"`
> > +        Connected *SetPasswordAction `json:"connected,omitempty"`
> > +
> > +        // Variants fields
> > +        Vnc *SetPasswordOptionsVnc `json:"-"`
> > +    }
> > +
> > +Now an example of a command without boxed type.
> > +
> > +.. code-block:: JSON
> > +    { 'command': 'set_link',
> > +      'data': {'name': 'str', 'up': 'bool'} }
> > +
> > +.. code-block:: go
> > +    type SetLinkCommand struct {
> > +        MessageId string `json:"-"`
> > +        Name      string `json:"name"`
> > +        Up        bool   `json:"up"`
> > +    }
> > +
> > +Known issues
> > +============
> > +
> > +- Type names might not follow proper Go convention. Andrea suggested an
> > +  annotation to the QAPI schema that could solve it.
> > +  https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg00127.html
> > -- 
> > 2.41.0
> > 
> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1 0/9] qapi-go: add generator for Golang interface

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 2974 bytes --]

On Wed, Sep 27, 2023 at 01:25:35PM +0200, Victor Toso wrote:
> Hi, long time no see!
> 
> This patch series intent is to introduce a generator that produces a Go
> module for Go applications to interact over QMP with QEMU.
> 
> This idea was discussed before, as RFC:
>  (RFC v1) https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00226.html
>  (RFC v2) https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00226.html

Bad copy-paste, the correct one:
    https://lists.gnu.org/archive/html/qemu-devel/2022-06/msg03105.html

> 
> The work got stuck due to changes needed around types that can take JSON
> Null as value, but that's now fixed.
> 
> I've pushed this series in my gitlab fork:
>     https://gitlab.com/victortoso/qemu/-/tree/qapi-golang-v1
> 
> I've also generated the qapi-go module over QEMU tags: v7.0.0, v7.1.0,
> v7.2.6, v8.0.0 and v8.1.1, see the commits history here:
>     https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v1-by-tags
> 
> I've also generated the qapi-go module over each commit of this series,
> see the commits history here (using previous refered qapi-golang-v1)
>     https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v1-by-patch
> 
> 
>  * Why this?
> 
> My main goal is to allow Go applications that interact with QEMU to have
> a native way of doing so.
> 
> Ideally, we can merge a new QAPI command, update qapi-go module to allow
> Go applications to consume the new command in no time (e.g: if
> development of said applications are using latest QEMU)
> 
> 
>  * Expectations
> 
> From previous discussions, there are things that are still missing. One
> simple example is Andrea's annotation suggestion to fix type names. My
> proposal is to have a qapi-go module in a formal non-stable version till
> some of those tasks get addressed or we declare it a non-problem.
> 
> I've created a docs/devel/qapi-golang-code-gen.rst to add information
> from the discussions we might have in this series. Suggestions always
> welcome.
> 
> P.S: Sorry about my broken python :)
> 
> Cheers,
> Victor
> 
> Victor Toso (9):
>   qapi: golang: Generate qapi's enum types in Go
>   qapi: golang: Generate qapi's alternate types in Go
>   qapi: golang: Generate qapi's struct types in Go
>   qapi: golang: structs: Address 'null' members
>   qapi: golang: Generate qapi's union types in Go
>   qapi: golang: Generate qapi's event types in Go
>   qapi: golang: Generate qapi's command types in Go
>   qapi: golang: Add CommandResult type to Go
>   docs: add notes on Golang code generator
> 
>  docs/devel/qapi-golang-code-gen.rst |  341 +++++++++
>  scripts/qapi/golang.py              | 1047 +++++++++++++++++++++++++++
>  scripts/qapi/main.py                |    2 +
>  3 files changed, 1390 insertions(+)
>  create mode 100644 docs/devel/qapi-golang-code-gen.rst
>  create mode 100644 scripts/qapi/golang.py
> 
> -- 
> 2.41.0
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1 0/9] qapi-go: add generator for Golang interface

[Daniel P. Berrangé]

On Wed, Sep 27, 2023 at 01:25:35PM +0200, Victor Toso wrote:
> Hi, long time no see!
> 
> This patch series intent is to introduce a generator that produces a Go
> module for Go applications to interact over QMP with QEMU.

snip
 
> Victor Toso (9):
>   qapi: golang: Generate qapi's enum types in Go
>   qapi: golang: Generate qapi's alternate types in Go
>   qapi: golang: Generate qapi's struct types in Go
>   qapi: golang: structs: Address 'null' members
>   qapi: golang: Generate qapi's union types in Go
>   qapi: golang: Generate qapi's event types in Go
>   qapi: golang: Generate qapi's command types in Go
>   qapi: golang: Add CommandResult type to Go
>   docs: add notes on Golang code generator
> 
>  docs/devel/qapi-golang-code-gen.rst |  341 +++++++++
>  scripts/qapi/golang.py              | 1047 +++++++++++++++++++++++++++
>  scripts/qapi/main.py                |    2 +
>  3 files changed, 1390 insertions(+)
>  create mode 100644 docs/devel/qapi-golang-code-gen.rst
>  create mode 100644 scripts/qapi/golang.py

So the formatting of the code is kinda all over the place eg

func (s *StrOrNull) ToAnyOrAbsent() (any, bool) {
    if s != nil {
        if s.IsNull {
            return nil, false
    } else if s.S != nil {
        return *s.S, false
        }
    }

    return nil, true
}


ought to be

func (s *StrOrNull) ToAnyOrAbsent() (any, bool) {
        if s != nil {
                if s.IsNull {
                        return nil, false
                } else if s.S != nil {
                        return *s.S, false
                }
        }

        return nil, true
}

I'd say we should add a 'make check-go' target, wired into 'make check'
that runs 'go fmt' on the generated code to validate that we generated
correct code. Then fix the generator to actually emit the reqired
format.


Having said that, this brings up the question of how we expect apps to
consume the Go code. Generally an app would expect to just add the
module to their go.mod file, and have the toolchain download it on
the fly during build.

If we assume a go namespace under qemu.org, we'll want one or more
modules. Either we have one module, containing APIs for all of QEMU,
QGA, and QSD, or we have separate go modules for each. I'd probably
tend towards the latter, since it means we can version them separately
which might be useful if we're willing to break API in one of them,
but not the others.

IOW, integrating this directly into qemu.git as a build time output
is not desirable in this conext though, as 'go build' can't consume
that.

IOW, it would push towards

   go-qemu.git
   go-qga.git
   go-qsd.git

and at time of each QEMU release, we would need to invoke the code
generator, and store its output in the respective git modules.

This would also need the generator application to be a standalone
tool, separate from the C QAPI generator.

Finally Go apps would want to do

   import (
       qemu.org/go/qemu
       qemu.org/go/qga
       qemu.org/go/gsd
   )

and would need us to create the "https://qemu.org/go/qemu" page
containing dummy HTML content with 

    <meta name="go-import" content="qemu.org/go/qemu git https://gitlab.com/qemu-project/go-qemu.git@/>

and likewise for the other modules.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 0/9] qapi-go: add generator for Golang interface

[Daniel P. Berrangé]

On Thu, Sep 28, 2023 at 02:40:27PM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 27, 2023 at 01:25:35PM +0200, Victor Toso wrote:
> > Hi, long time no see!
> > 
> > This patch series intent is to introduce a generator that produces a Go
> > module for Go applications to interact over QMP with QEMU.
> 
> snip
>  
> > Victor Toso (9):
> >   qapi: golang: Generate qapi's enum types in Go
> >   qapi: golang: Generate qapi's alternate types in Go
> >   qapi: golang: Generate qapi's struct types in Go
> >   qapi: golang: structs: Address 'null' members
> >   qapi: golang: Generate qapi's union types in Go
> >   qapi: golang: Generate qapi's event types in Go
> >   qapi: golang: Generate qapi's command types in Go
> >   qapi: golang: Add CommandResult type to Go
> >   docs: add notes on Golang code generator
> > 
> >  docs/devel/qapi-golang-code-gen.rst |  341 +++++++++
> >  scripts/qapi/golang.py              | 1047 +++++++++++++++++++++++++++
> >  scripts/qapi/main.py                |    2 +
> >  3 files changed, 1390 insertions(+)
> >  create mode 100644 docs/devel/qapi-golang-code-gen.rst
> >  create mode 100644 scripts/qapi/golang.py
> 
> So the formatting of the code is kinda all over the place eg
> 
> func (s *StrOrNull) ToAnyOrAbsent() (any, bool) {
>     if s != nil {
>         if s.IsNull {
>             return nil, false
>     } else if s.S != nil {
>         return *s.S, false
>         }
>     }
> 
>     return nil, true
> }
> 
> 
> ought to be
> 
> func (s *StrOrNull) ToAnyOrAbsent() (any, bool) {
>         if s != nil {
>                 if s.IsNull {
>                         return nil, false
>                 } else if s.S != nil {
>                         return *s.S, false
>                 }
>         }
> 
>         return nil, true
> }
> 
> I'd say we should add a 'make check-go' target, wired into 'make check'
> that runs 'go fmt' on the generated code to validate that we generated
> correct code. Then fix the generator to actually emit the reqired
> format.
> 
> 
> Having said that, this brings up the question of how we expect apps to
> consume the Go code. Generally an app would expect to just add the
> module to their go.mod file, and have the toolchain download it on
> the fly during build.
> 
> If we assume a go namespace under qemu.org, we'll want one or more
> modules. Either we have one module, containing APIs for all of QEMU,
> QGA, and QSD, or we have separate go modules for each. I'd probably
> tend towards the latter, since it means we can version them separately
> which might be useful if we're willing to break API in one of them,
> but not the others.
> 
> IOW, integrating this directly into qemu.git as a build time output
> is not desirable in this conext though, as 'go build' can't consume
> that.
> 
> IOW, it would push towards
> 
>    go-qemu.git
>    go-qga.git
>    go-qsd.git
> 
> and at time of each QEMU release, we would need to invoke the code
> generator, and store its output in the respective git modules.
> 
> This would also need the generator application to be a standalone
> tool, separate from the C QAPI generator.

Oh, and we need to assume that all CONFIG conditionals in the QAPI
files are true, as we want the Go API to be feature complete such
that it can be used with any QEMU build, regardless of which CONFIG
conditions are turned on/off. We also don't want applications to
suddenly fail to compile because some API was stopped being generated
by a disabled CONFIG condition - it needs to be a runtime error
that apps can catch and handle as they desire.

> 
> Finally Go apps would want to do
> 
>    import (
>        qemu.org/go/qemu
>        qemu.org/go/qga
>        qemu.org/go/gsd
>    )
> 
> and would need us to create the "https://qemu.org/go/qemu" page
> containing dummy HTML content with 
> 
>     <meta name="go-import" content="qemu.org/go/qemu git https://gitlab.com/qemu-project/go-qemu.git@/>
> 
> and likewise for the other modules.
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 0/9] qapi-go: add generator for Golang interface

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 1232 bytes --]

Hi,

On Thu, Sep 28, 2023 at 02:54:10PM +0100, Daniel P. Berrangé wrote:
> On Thu, Sep 28, 2023 at 02:40:27PM +0100, Daniel P. Berrangé wrote:
> > On Wed, Sep 27, 2023 at 01:25:35PM +0200, Victor Toso wrote:
> > IOW, it would push towards
> > 
> >    go-qemu.git
> >    go-qga.git
> >    go-qsd.git
> > 
> > and at time of each QEMU release, we would need to invoke the code
> > generator, and store its output in the respective git modules.
> > 
> > This would also need the generator application to be a standalone
> > tool, separate from the C QAPI generator.
> 
> Oh, and we need to assume that all CONFIG conditionals in the QAPI
> files are true, as we want the Go API to be feature complete such
> that it can be used with any QEMU build, regardless of which CONFIG
> conditions are turned on/off. We also don't want applications to
> suddenly fail to compile because some API was stopped being generated
> by a disabled CONFIG condition - it needs to be a runtime error
> that apps can catch and handle as they desire.

I haven't tested if what is provided to scripts/qapi/golang.py
relies on enabled CONFIG only, I think not. But yes, the
generated module should have it all.

Cheers,
Victor

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v1 0/9] qapi-go: add generator for Golang interface

[Victor Toso]

[-- Attachment #1: Type: text/plain, Size: 4470 bytes --]

Hi,

On Thu, Sep 28, 2023 at 02:40:27PM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 27, 2023 at 01:25:35PM +0200, Victor Toso wrote:
> > Hi, long time no see!
> > 
> > This patch series intent is to introduce a generator that produces a Go
> > module for Go applications to interact over QMP with QEMU.
> 
> snip
>  
> > Victor Toso (9):
> >   qapi: golang: Generate qapi's enum types in Go
> >   qapi: golang: Generate qapi's alternate types in Go
> >   qapi: golang: Generate qapi's struct types in Go
> >   qapi: golang: structs: Address 'null' members
> >   qapi: golang: Generate qapi's union types in Go
> >   qapi: golang: Generate qapi's event types in Go
> >   qapi: golang: Generate qapi's command types in Go
> >   qapi: golang: Add CommandResult type to Go
> >   docs: add notes on Golang code generator
> > 
> >  docs/devel/qapi-golang-code-gen.rst |  341 +++++++++
> >  scripts/qapi/golang.py              | 1047 +++++++++++++++++++++++++++
> >  scripts/qapi/main.py                |    2 +
> >  3 files changed, 1390 insertions(+)
> >  create mode 100644 docs/devel/qapi-golang-code-gen.rst
> >  create mode 100644 scripts/qapi/golang.py
> 
> So the formatting of the code is kinda all over the place eg
> 
> func (s *StrOrNull) ToAnyOrAbsent() (any, bool) {
>     if s != nil {
>         if s.IsNull {
>             return nil, false
>     } else if s.S != nil {
>         return *s.S, false
>         }
>     }
> 
>     return nil, true
> }
> 
> 
> ought to be
> 
> func (s *StrOrNull) ToAnyOrAbsent() (any, bool) {
>         if s != nil {
>                 if s.IsNull {
>                         return nil, false
>                 } else if s.S != nil {
>                         return *s.S, false
>                 }
>         }
> 
>         return nil, true
> }
> 
> I'd say we should add a 'make check-go' target, wired into 'make check'
> that runs 'go fmt' on the generated code to validate that we generated
> correct code. Then fix the generator to actually emit the reqired
> format.

As mentioned in another thread, my main concern with this is
requiring go binaries in the make script. Might be fine if the
scope is only when a release is done, but shouldn't be a default
requirement.

> Having said that, this brings up the question of how we expect apps to
> consume the Go code. Generally an app would expect to just add the
> module to their go.mod file, and have the toolchain download it on
> the fly during build.
> 
> If we assume a go namespace under qemu.org, we'll want one or more
> modules. Either we have one module, containing APIs for all of QEMU,
> QGA, and QSD, or we have separate go modules for each. I'd probably
> tend towards the latter, since it means we can version them separately
> which might be useful if we're willing to break API in one of them,
> but not the others.
> 
> IOW, integrating this directly into qemu.git as a build time output
> is not desirable in this conext though, as 'go build' can't consume
> that.
> 
> IOW, it would push towards
> 
>    go-qemu.git
>    go-qga.git
>    go-qsd.git
> 
> and at time of each QEMU release, we would need to invoke the code
> generator, and store its output in the respective git modules.

In which point, I think it is fair to run the gofmt and goimports.
Still, if you think it isn't a problem to add such make check-go
target with tooling specific to go code in them, I'll add that to
next iteration.

> This would also need the generator application to be a
> standalone tool, separate from the C QAPI generator.

It is. I mean, both run together now but that can be improved.

> Finally Go apps would want to do
> 
>    import (
>        qemu.org/go/qemu
>        qemu.org/go/qga
>        qemu.org/go/gsd
>    )
> 
> and would need us to create the "https://qemu.org/go/qemu" page
> containing dummy HTML content with 
> 
>     <meta name="go-import" content="qemu.org/go/qemu git https://gitlab.com/qemu-project/go-qemu.git@/>

Neat. I didn't know this. Yes, we want that, but with different
name for the git [0]. Perhaps just another folder:

    https://gitlab.com/qemu-project/go/qemu.git
    https://gitlab.com/qemu-project/go/qga.git
    https://gitlab.com/qemu-project/go/gsd.git

> and likewise for the other modules.

[0] https://github.com/digitalocean/go-qemu

Thanks again for the reviews!

Cheers,
Victor

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]