Golang design session follow-up

Golang design session follow-up

[Nick Rosbrook]

Hello,

Here are the notes from the golang bindings design session at Xen Summit
earlier this month:

  Possible improvements: 
    - Expanding the IDL to have information about device add / remove functions
    
    Ian: It's important that the IDL generates the C prototypes as well.

    We could generate some man pages as well.
    
    "AO" functions need to be a separate thing, not just listed as arguments.
    
    How to distinguish between "easy to wrap" things vs "this is internal infrastructure"
  
    Idea: "Classes" in IDL: "Device add/remove function", &c

    Rust: Probably want two versions of the bindings; sync and async. Same thing for python.
  
  # Notification of events

    We don't want a golang wrapper for the sigchld callback

    Helper function to "do the domain shutdown thing"
  
  # Long-term home of the package
  
    Ian: Autogenerated stuff is becoming more annoying.

    Delete all the libxl auto-generated stuff from staging & master, and have "output branch".

    The reason we have these in-tree is that otherwise you can't build *from git* if you don't 
    have new enough versions of the right tools.

    Distribution: Make a repo on xenbits!

Our main focus for "improvements" was on expanding the libxl IDL to support
function prototypes so that wrappers could be easily generated. I
volunteered to work on this front, and have started some patches that I
will send in an RFC state soon (likely after 4.14 is released).

On "notification of events", I think we were just discussing what needs
to be done for the xenlight package to support domain
destruction/reaping. This is also something I will work on.

Finally, we came to the decision that the golang bindings should be
pushed to their own repo and tagged independently of xen.git. I think
tasks still need to be divvied up for this one.

If I missed anything important or made any mistakes, please let me know.

Thanks,
NR

Re: Golang design session follow-up

[George Dunlap]

> On Jul 21, 2020, at 1:35 AM, Nick Rosbrook <rosbrookn@gmail.com> wrote:
> 
>  # Long-term home of the package
> 
>    Ian: Autogenerated stuff is becoming more annoying.
> 
>    Delete all the libxl auto-generated stuff from staging & master, and have "output branch".
> 
>    The reason we have these in-tree is that otherwise you can't build *from git* if you don't 
>    have new enough versions of the right tools.
> 
>    Distribution: Make a repo on xenbits!

So thinking about this: 

The first plan I had was to have a script in tools/golang/xenlight (and/or the Makefile), which would be handed a directory, and would then:

1. Sync static files from tools/golang/xenlight into that directory

2. Run gengotypes.py, having the resulting generated files put into that directory

3. Run `git diff` in the target directory; if there are any changes, then automatically run `git commit` to check in the changes.

That way you could just set up a cron job to sync things over on a regular basis.

Thinking about GPL considerations, however, you’d also want to include libxl_types.idl and idl.py.  And then of course, you should also include a way to build the generated code from those two.

At which point… would it make sense to just move the package out to its separate repo entirely?  I.e., have actual development happen in the repo which ends up being cloned in the end?

Obviously there are nice things about having the code in the same repo; but there’s also something satisfying about being a full downstream.

I was actually thinking it might make sense to put the repo at https://gitlab.com/xen-project/go-xenlight , to try out that as a development model.

Any thoughts?

 -George

Re: Golang design session follow-up

[George Dunlap]

> On Aug 28, 2020, at 4:57 PM, George Dunlap <george.dunlap@citrix.com> wrote:
> 
> 
> 
>> On Jul 21, 2020, at 1:35 AM, Nick Rosbrook <rosbrookn@gmail.com> wrote:
>> 
>> # Long-term home of the package
>> 
>>   Ian: Autogenerated stuff is becoming more annoying.
>> 
>>   Delete all the libxl auto-generated stuff from staging & master, and have "output branch".
>> 
>>   The reason we have these in-tree is that otherwise you can't build *from git* if you don't 
>>   have new enough versions of the right tools.
>> 
>>   Distribution: Make a repo on xenbits!
> 
> So thinking about this: 
> 
> The first plan I had was to have a script in tools/golang/xenlight (and/or the Makefile), which would be handed a directory, and would then:
> 
> 1. Sync static files from tools/golang/xenlight into that directory
> 
> 2. Run gengotypes.py, having the resulting generated files put into that directory
> 
> 3. Run `git diff` in the target directory; if there are any changes, then automatically run `git commit` to check in the changes.
> 
> That way you could just set up a cron job to sync things over on a regular basis.
> 
> Thinking about GPL considerations, however, you’d also want to include libxl_types.idl and idl.py.  And then of course, you should also include a way to build the generated code from those two.
> 
> At which point… would it make sense to just move the package out to its separate repo entirely?  I.e., have actual development happen in the repo which ends up being cloned in the end?
> 
> Obviously there are nice things about having the code in the same repo; but there’s also something satisfying about being a full downstream.
> 
> I was actually thinking it might make sense to put the repo at https://gitlab.com/xen-project/go-xenlight , to try out that as a development model.

I’ve put a sort of draft module up at https://gitlab.com/martyros/go-xen ; you can test it by adding the "gitlab.com/xen-project/go-xen/xenlight” package, but adding the following line to the go.mod of the test program:

replace gitlab.com/xen-project/go-xen => gitlab.com/martyros/go-xen master

This builds but for me unfortunately crashes during domain creation with the following error:

libxl: error: libxl_create.c:90:libxl__domain_build_info_setdefault: Unable to copy deprecated fields
libxl: error: libxl_create.c:1182:libxl__domain_config_setdefault: Unable to set domain build info defaults

I’ll have to dig a bit more into it next week.

 -George

Re: Golang design session follow-up

[Nick Rosbrook]

On Fri, Aug 28, 2020 at 07:05:08PM +0000, George Dunlap wrote:
> 
> 
> > On Aug 28, 2020, at 4:57 PM, George Dunlap <george.dunlap@citrix.com> wrote:
> > 
> > 
> > 
> >> On Jul 21, 2020, at 1:35 AM, Nick Rosbrook <rosbrookn@gmail.com> wrote:
> >> 
> >> # Long-term home of the package
> >> 
> >>   Ian: Autogenerated stuff is becoming more annoying.
> >> 
> >>   Delete all the libxl auto-generated stuff from staging & master, and have "output branch".
> >> 
> >>   The reason we have these in-tree is that otherwise you can't build *from git* if you don't 
> >>   have new enough versions of the right tools.
> >> 
> >>   Distribution: Make a repo on xenbits!
> > 
> > So thinking about this: 
> > 
> > The first plan I had was to have a script in tools/golang/xenlight (and/or the Makefile), which would be handed a directory, and would then:
> > 
> > 1. Sync static files from tools/golang/xenlight into that directory
> > 
> > 2. Run gengotypes.py, having the resulting generated files put into that directory
> > 
> > 3. Run `git diff` in the target directory; if there are any changes, then automatically run `git commit` to check in the changes.
> > 
> > That way you could just set up a cron job to sync things over on a regular basis.
> > 
> > Thinking about GPL considerations, however, you’d also want to include libxl_types.idl and idl.py.  And then of course, you should also include a way to build the generated code from those two.
> > 
> > At which point… would it make sense to just move the package out to its separate repo entirely?  I.e., have actual development happen in the repo which ends up being cloned in the end?
> > 
> > Obviously there are nice things about having the code in the same repo; but there’s also something satisfying about being a full downstream.
> > 
> > I was actually thinking it might make sense to put the repo at https://gitlab.com/xen-project/go-xenlight , to try out that as a development model.
Would that mean completely moving off of xen-devel for development? I can't 
think of a huge reason why we wouldn't be able to do this if we wanted.
> 
> I’ve put a sort of draft module up at https://gitlab.com/martyros/go-xen ; you can test it by adding the "gitlab.com/xen-project/go-xen/xenlight” package, but adding the following line to the go.mod of the test program:
I have a couple of patches I was going to send out on xen-devel today. I 
could PR them to this repo instead (or in addition) if you want to try out 
the gitlab workflow. 

-NR

Re: Golang design session follow-up

[George Dunlap]

> On Aug 31, 2020, at 2:55 PM, Nick Rosbrook <rosbrookn@gmail.com> wrote:
> 
> On Fri, Aug 28, 2020 at 07:05:08PM +0000, George Dunlap wrote:
>> 
>> 
>>> On Aug 28, 2020, at 4:57 PM, George Dunlap <george.dunlap@citrix.com> wrote:
>>> 
>>> 
>>> 
>>>> On Jul 21, 2020, at 1:35 AM, Nick Rosbrook <rosbrookn@gmail.com> wrote:
>>>> 
>>>> # Long-term home of the package
>>>> 
>>>>  Ian: Autogenerated stuff is becoming more annoying.
>>>> 
>>>>  Delete all the libxl auto-generated stuff from staging & master, and have "output branch".
>>>> 
>>>>  The reason we have these in-tree is that otherwise you can't build *from git* if you don't 
>>>>  have new enough versions of the right tools.
>>>> 
>>>>  Distribution: Make a repo on xenbits!
>>> 
>>> So thinking about this: 
>>> 
>>> The first plan I had was to have a script in tools/golang/xenlight (and/or the Makefile), which would be handed a directory, and would then:
>>> 
>>> 1. Sync static files from tools/golang/xenlight into that directory
>>> 
>>> 2. Run gengotypes.py, having the resulting generated files put into that directory
>>> 
>>> 3. Run `git diff` in the target directory; if there are any changes, then automatically run `git commit` to check in the changes.
>>> 
>>> That way you could just set up a cron job to sync things over on a regular basis.
>>> 
>>> Thinking about GPL considerations, however, you’d also want to include libxl_types.idl and idl.py.  And then of course, you should also include a way to build the generated code from those two.
>>> 
>>> At which point… would it make sense to just move the package out to its separate repo entirely?  I.e., have actual development happen in the repo which ends up being cloned in the end?
>>> 
>>> Obviously there are nice things about having the code in the same repo; but there’s also something satisfying about being a full downstream.
>>> 
>>> I was actually thinking it might make sense to put the repo at https://gitlab.com/xen-project/go-xenlight , to try out that as a development model.
> Would that mean completely moving off of xen-devel for development? I can't 
> think of a huge reason why we wouldn't be able to do this if we wanted.

I mean obviously the changes to libxl_types.idl and idl.py would have to happen on xen-devel; but yeah, changes to the external repo would happen within gitlab.

>> 
>> I’ve put a sort of draft module up at https://gitlab.com/martyros/go-xen ; you can test it by adding the "gitlab.com/xen-project/go-xen/xenlight” package, but adding the following line to the go.mod of the test program:
> I have a couple of patches I was going to send out on xen-devel today. I 
> could PR them to this repo instead (or in addition) if you want to try out 
> the gitlab workflow. 

Yeah, we could give that a try.

 -George