Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Paolo Bonzini]

I have a few comments on the series.  I like it in general, but I
think it's way too incomplete to be a first incremental step, and it's
not clear what your plans are about this.  I think I could boil it
down to two requirements before committing this.

1) The biggest comment is that you have a huge difference between your
original proposal and this one: in the full proposal as I remember it,
link properties work on any QOM object; right now, however, buses are
not QOM-ified.

This is a problem because a qdev bus mediates between the parent and
children's interfaces.  You have the ingredients here for doing this
(the bus can be any device with a link to the parent and child
properties for the children; children also have backwards link), but
the bus in the middle right now is not QOM-ified and so the mediation
cannot be represented in the prototype.

Also, it is not clear how the bus will get hold of this interface from
the parent.  For the children it is a bit simpler, since the bus has
access to the DeviceInfo; I'm not sure however what the plan is and
whether you still plan on keeping the DeviceInfo descendents.

So, my #1 requirement for this to be committed is: Have all
relationships in the device tree expressed as properties.  Make sure
that all required abstractions can be implemented in terms of a QOM
composition tree.

2) Related to this, you have a lot of nice infrastructure, but (unlike
what you did with QAPI) you haven't provided yet a clear plan for how
to get rid of the old code.  You also have only very limited uses of
the infrastructure (see above).

And it's not clear how the infrastructure will evolve.  For example,
how do you plan to implement the realization phase that you had in the
original model?

(Regarding this, I'm partially guilty of the same with my SCSI
refactoring.  In the end almost everything came nicely out of the
refactoring, but I did have to backtrack a couple of times.  But the
damage I could do, and the churn caused by the backtracking, was much
more limited than with something like QOM!)

3) I still don't understand why the vtable need to be per-property and
not per-class.  I see no reason not to have DevicePropertyInfo
(statically defined) + DeviceProperty (a pointer to DevicePropertyInfo
+ the queue entry + name/type/whatever).  Please do make an attempt at
using composition to derive property types, as this is in general what
_you_ gave in the past as the direction for QEMU (see Notifiers).
It's better for various reasons--type safety and ease of use--even if
it costs some boilerplate.  For example the child property should be
as simple as

+struct ChildDeviceProperty {
+     DeviceProperty prop;
+     DeviceState *child;
+}
+
+struct DevicePropertyInfo child_device_property_info = {
+    .size = sizeof(ChildDeviceProperty);
+    .get = qdev_get_child_property,
+};
+
 void qdev_property_add_child(DeviceState *dev, const char *name,
                              DeviceState *child, Error **errp)
 {
     gchar *type;

     type = g_strdup_printf("child<%s>", child->info->name);

-    qdev_property_add(dev, name, type, qdev_get_child_property,
-                      NULL, NULL, child, errp);
+    prop = (ChildDeviceProperty *)
+        qdev_property_add(&child_device_property_info,
+                      dev, name, type, errp);
+
+    /* TODO: check errp, if it is NULL -> return immediately.  */
+    prop->child = child;
+
+     /* Shouldn't this ref the child instead??  Also, where is the matching
+       unref??  Needs a release method? */
     qdev_ref(dev);

     g_free(type);
}

I think also that the type should not be part of DeviceProperty.
Instead it should be defined by subclasses, with the
DevicePropertyInfo providing a function to format the type to a
string.

Also in favor of containment and a property hierarchy, there is no
reason why Property cannot become a subclass of DeviceProperty.  The
DeviceInfo/BusInfo would contain a prototype that you can clone to the
heap when building the device's dynamic properties.  If instead you
want the declarative nature of qdev properties to be replaced by
imperative descriptions, that's fine, but please provide a proof of
concept of how to do the conversion (a perl script would be fine, even
if it works only for 2-3 device models).

For artificial properties you would need one-off classes; but you can
still provide a helper that creates a specialized
DeviceProperty+DevicePropertyInfo from the functions.  I'm thinking of
how people implement prototype-based OO on top of class-based OO, but
it might be just a macro.

I wouldn't make this a requirement, but you really need a stronger
justification that what you gave so far.

4) I really hate naming something as "legacy", particularly when we
have hundreds of examples and no plan for converting them.  In fact,
there's nothing legacy in them except the string-based parsing.  Even
if you convert them away from the Property model to the visitor model,
they do not disappear, not at all.  They're simply not dynamic.  If
you call them legacy, I might be worried that I have to replace all

     dev->num_queues

with

    ((int) (intptr_t) dev->num_queues_prop->opaque)

So, my #2 requirement is to make it clear what the fate will be for
"legacy" properties, and possibly make enough conversion work that
they can just be called "static".  Do consider converting the
parse/print functions in Property to visitor-style (it's trivial).
Then provide a "legacy" bridge in the _reverse_ direction: there are
hundreds of qdev properties, and just a handful of users of them.

Going to bed now...

Paolo

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Anthony Liguori]

On 12/02/2011 06:56 PM, Paolo Bonzini wrote:
> I have a few comments on the series.  I like it in general, but I
> think it's way too incomplete to be a first incremental step, and it's
> not clear what your plans are about this.  I think I could boil it
> down to two requirements before committing this.
>
> 1) The biggest comment is that you have a huge difference between your
> original proposal and this one:

It's actually exactly the same.  It's just an incremental conversion of qdev 
into QOM.  I can't think of a way to maintain compatibility (and sanity) without 
morphing qdev incrementally into QOM.

> in the full proposal as I remember it,
> link properties work on any QOM object;

That is still true.  The next step, inheritance, will pull the properties into a 
base class.  That base class can be used elsewhere outside of the device model.

But this is already a 20 patch series.  If you want all of that in one series, 
it's going to be 100 patches that are not terribly easy to review at once.

> right now, however, buses are
> not QOM-ified.
>
> This is a problem because a qdev bus mediates between the parent and
> children's interfaces.  You have the ingredients here for doing this
> (the bus can be any device with a link to the parent and child
> properties for the children; children also have backwards link), but
> the bus in the middle right now is not QOM-ified and so the mediation
> cannot be represented in the prototype.
>
> Also, it is not clear how the bus will get hold of this interface from
> the parent.  For the children it is a bit simpler, since the bus has
> access to the DeviceInfo; I'm not sure however what the plan is and
> whether you still plan on keeping the DeviceInfo descendents.
>
> So, my #1 requirement for this to be committed is: Have all
> relationships in the device tree expressed as properties.

This series is about one thing: introducing stable paths.  The fact that link 
properties are being added is just because it's easy.  I'm perfect happy pulling 
out link properties and waiting until we have enough of the object model to 
introduce links.

But I want to get started filling out the composition tree throughout QEMU. 
There's a lot of value in this.  Most notably, we can switch to referencing 
devices based on their canonical path during live migration which means all of 
the non-sense around instance ids can disappear.

> Make sure
> that all required abstractions can be implemented in terms of a QOM
> composition tree.

Not composition tree, you mean via the link graph.  But you're getting ahead of 
yourself.

>
> 2) Related to this, you have a lot of nice infrastructure, but (unlike
> what you did with QAPI) you haven't provided yet a clear plan for how
> to get rid of the old code.  You also have only very limited uses of
> the infrastructure (see above).

Old style properties go away as they're converted to new style properties.

> And it's not clear how the infrastructure will evolve.  For example,
> how do you plan to implement the realization phase that you had in the
> original model?

That's step three.  realize is a property but it's best implemented as a virtual 
function that is dispatched.  We can't do this until we have virtual method 
dispatch.

If you want to see how this all is going to look, look at the old tree :-) 
That's where we're going to end up.

> 3) I still don't understand why the vtable need to be per-property and
> not per-class.  I see no reason not to have DevicePropertyInfo
> (statically defined) + DeviceProperty (a pointer to DevicePropertyInfo
> + the queue entry + name/type/whatever).  Please do make an attempt at
> using composition to derive property types, as this is in general what
> _you_ gave in the past as the direction for QEMU (see Notifiers).
> It's better for various reasons--type safety and ease of use--even if
> it costs some boilerplate.  For example the child property should be
> as simple as
>
> +struct ChildDeviceProperty {
> +     DeviceProperty prop;
> +     DeviceState *child;
> +}
> +
> +struct DevicePropertyInfo child_device_property_info = {
> +    .size = sizeof(ChildDeviceProperty);
> +    .get = qdev_get_child_property,
> +};
> +
>   void qdev_property_add_child(DeviceState *dev, const char *name,
>                                DeviceState *child, Error **errp)
>   {
>       gchar *type;
>
>       type = g_strdup_printf("child<%s>", child->info->name);
>
> -    qdev_property_add(dev, name, type, qdev_get_child_property,
> -                      NULL, NULL, child, errp);
> +    prop = (ChildDeviceProperty *)
> +        qdev_property_add(&child_device_property_info,
> +                      dev, name, type, errp);
> +
> +    /* TODO: check errp, if it is NULL ->  return immediately.  */
> +    prop->child = child;

This seems quite a bit uglier to me.

> +
> +     /* Shouldn't this ref the child instead??  Also, where is the matching
> +       unref??  Needs a release method? */
>       qdev_ref(dev);

Yup, that's a bug.  Thanks.

In terms of unref, we need to tie life cycles.  That's probably something that 
should go in this patch (and this is how it works in my qom branch btw).

>
>       g_free(type);
> }
>
> I think also that the type should not be part of DeviceProperty.
> Instead it should be defined by subclasses, with the
> DevicePropertyInfo providing a function to format the type to a
> string.

I don't really know what you mean by this.

>
> Also in favor of containment and a property hierarchy, there is no
> reason why Property cannot become a subclass of DeviceProperty.  The
> DeviceInfo/BusInfo would contain a prototype that you can clone to the
> heap when building the device's dynamic properties.  If instead you
> want the declarative nature of qdev properties to be replaced by
> imperative descriptions, that's fine, but please provide a proof of
> concept of how to do the conversion (a perl script would be fine, even
> if it works only for 2-3 device models).

I think you see just as well as I do that it's not that complicated.  Working 
out the details of how properties are implemented are more important IMHO than 
starting to do conversions.  We jump too quickly at doing conversions without 
thinking through the underlying interface.

You may find it odd to hear me say this, but I grow weary of adding too much 
class hierarchy and inheritance with properties.  Yes, we could make it very 
sophisticated but in practice it doesn't have to be.  Properties should be one 
of two things: primitive types or link/childs.

This is where qdev goes very wrong.  It mixes up user interface details (how to 
format an integer on the command line) with implementation details.  There's no 
reason we should be indicating whether a property should be display in hex or 
decimal in the device itself.

> For artificial properties you would need one-off classes; but you can
> still provide a helper that creates a specialized
> DeviceProperty+DevicePropertyInfo from the functions.  I'm thinking of
> how people implement prototype-based OO on top of class-based OO, but
> it might be just a macro.

I think you're over thinking the problem.  There are going to be maybe a dozen 
property types and that's it.  They all with correspond exactly to C types with 
the exception of links/children.

> I wouldn't make this a requirement, but you really need a stronger
> justification that what you gave so far.
>
> 4) I really hate naming something as "legacy", particularly when we
> have hundreds of examples and no plan for converting them.  In fact,
> there's nothing legacy in them except the string-based parsing.  Even
> if you convert them away from the Property model to the visitor model,
> they do not disappear, not at all.  They're simply not dynamic.  If
> you call them legacy, I might be worried that I have to replace all
>
>       dev->num_queues
>
> with
>
>      ((int) (intptr_t) dev->num_queues_prop->opaque)
>
> So, my #2 requirement is to make it clear what the fate will be for
> "legacy" properties, and possibly make enough conversion work that
> they can just be called "static".  Do consider converting the
> parse/print functions in Property to visitor-style (it's trivial).
> Then provide a "legacy" bridge in the _reverse_ direction: there are
> hundreds of qdev properties, and just a handful of users of them.

The trouble with the legacy properties are how they mix stringification with the 
properties themselves.  In order to preserve compatibility, we're probably going 
to have to do something like mark all legacy properties deprecated, replace them 
with different named properties, and declare a flag day where we remove all 
legacy properties.

Yes, it sucks, but it's part of the ABI that we cannot now change.  We have to 
treat the input to pcidevice.address as a hex integer in a string.

Regards,

Anthony Liguori

> Going to bed now...
>
> Paolo
>

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Paolo Bonzini]

On 12/03/2011 03:40 AM, Anthony Liguori wrote:
> That is still true. The next step, inheritance, will pull the properties
> into a base class. That base class can be used elsewhere outside of the
> device model.
>
> But this is already a 20 patch series. If you want all of that in one
> series, it's going to be 100 patches that are not terribly easy to
> review at once.

Without a design document and a roadmap, however, it's impossible to try 
to understand how the pieces will be together.  100 patches may require 
some time to digest, but 20 patches require a crystal ball to figure out 
what's ahead.

>> Make sure that all required abstractions can be implemented in
>> terms of a QOM composition tree.
>
> Not composition tree, you mean via the link graph.

I mean both, but the link graph is already understandable (1-to-1 is 
easy).  Right now I can hardly understand how the composition tree will 
work, for example: how do you constrain children to be subclasses of 
some class (or to implement some interface)?

>> 2) Related to this, you have a lot of nice infrastructure, but (unlike
>> what you did with QAPI) you haven't provided yet a clear plan for how
>> to get rid of the old code. You also have only very limited uses of
>> the infrastructure (see above).
>
> Old style properties go away as they're converted to new style properties.

And how do they?  How much code churn does that entail, in the devices 
and in general?  In fact, why do they need conversion at all?  Static 
properties are special enough to warrant something special.

> If you want to see how this all is going to look, look at the old tree
> :-) That's where we're going to end up.

Let's write documentation on that already.

>> It's better for various reasons--type safety and ease of use--even if
>> it costs some boilerplate. For example the child property should be
>> as simple as
>>
>> +struct ChildDeviceProperty {
>> + DeviceProperty prop;
>> + DeviceState *child;
>> +}
>> +
>> +struct DevicePropertyInfo child_device_property_info = {
>> + .size = sizeof(ChildDeviceProperty);
>> + .get = qdev_get_child_property,
>> +};
>> +
>> void qdev_property_add_child(DeviceState *dev, const char *name,
>> DeviceState *child, Error **errp)
>> {
>> gchar *type;
>>
>> type = g_strdup_printf("child<%s>", child->info->name);
>>
>> - qdev_property_add(dev, name, type, qdev_get_child_property,
>> - NULL, NULL, child, errp);
>> + prop = (ChildDeviceProperty *)
>> + qdev_property_add(&child_device_property_info,
>> + dev, name, type, errp);
>> +
>> + /* TODO: check errp, if it is NULL -> return immediately. */
>> + prop->child = child;
>
> This seems quite a bit uglier to me.

I did say it costs some boilerplate, but it's once per type and you said 
there's a dozen of those.  The type safety and more flexibility (an 
arbitrary struct for subclasses rather than an opaque pointer) is worth 
the ugliness.

>> I think also that the type should not be part of DeviceProperty.
>> Instead it should be defined by subclasses, with the
>> DevicePropertyInfo providing a function to format the type to a
>> string.
>
> I don't really know what you mean by this.

Once the type is written as child<RTCState>, you've lost all info on it. 
  Each property type should have its own representation for types (it 
can be implicit, or it can be an enum, or it can be a DeviceInfo), with 
a method to pretty-print it.

> We jump too quickly at
> doing conversions without thinking through the underlying interface.

I hoped you had thought it through. ;)

> You may find it odd to hear me say this, but I grow weary of adding too
> much class hierarchy and inheritance with properties. Yes, we could make
> it very sophisticated but in practice it doesn't have to be. Properties
> should be one of two things: primitive types or link/childs.

... and interfaces.  Accessing them by name as a property should work well.

> This is where qdev goes very wrong. It mixes up user interface details
> (how to format an integer on the command line) with implementation
> details. There's no reason we should be indicating whether a property
> should be display in hex or decimal in the device itself.

That's totally true.  But there's no reason why qdev properties cannot 
be split in two parts, a sane one and a legacy one.  If you don't do 
this, you take the unmodifiable ABI and force its use as an API 
throughout all QOM.  We can do better.

>> For artificial properties you would need one-off classes; but you can
>> still provide a helper that creates a specialized
>> DeviceProperty+DevicePropertyInfo from the functions. I'm thinking of
>> how people implement prototype-based OO on top of class-based OO, but
>> it might be just a macro.
>
> I think you're over thinking the problem. There are going to be maybe a
> dozen property types and that's it. They all with correspond exactly to
> C types with the exception of links/children.

I was thinking of one-off types such as rtc's struct tm.  Also, besides 
C primitive types there will be property types for structs: for example 
virtio or SCSI requests, S/G lists, and so on.

Paolo

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Anthony Liguori]

On 12/03/2011 08:24 AM, Paolo Bonzini wrote:
> On 12/03/2011 03:40 AM, Anthony Liguori wrote:
>> That is still true. The next step, inheritance, will pull the properties
>> into a base class. That base class can be used elsewhere outside of the
>> device model.
>>
>> But this is already a 20 patch series. If you want all of that in one
>> series, it's going to be 100 patches that are not terribly easy to
>> review at once.
>
> Without a design document and a roadmap, however, it's impossible to try to
> understand how the pieces will be together. 100 patches may require some time to
> digest, but 20 patches require a crystal ball to figure out what's ahead.

You can see a bit further by looking at:

https://github.com/aliguori/qemu/commits/qom-next

That fills out the composition tree pretty well for the pc.  The next step is 
aggressive refactoring such that the qdev objects reflect the composition.  IOW, 
we should create the rtc from within the piix3 initialization function.

But before we can do that, we need to split construction and introduce realize 
which requires inheritance.

So I'd like to fill out the composition tree next (HEAD~2 in that branch, but 
split up nicely).  Having a flushed out composition tree really helps figure out 
what the code should look like and my hope is that other platforms can start 
refactoring too.

Inheritance is not terribly hard.  First step is adding a type pointer to 
DeviceState *, and then introducing some dynamic_cast-like macros and then 
touching every file to use them.

Second step is QOM-style methods and polymorphism.  That's not terribly hard either.

Once we have that, we can do realize.

Note that I haven't really talked about busses.  I do want to get rid of qdev 
busses but I'm in no rush at all.  We can do 95% of the necessary refactoring 
without touching busses and hot plug and I think that's the right strategy.

>
>>> Make sure that all required abstractions can be implemented in
>>> terms of a QOM composition tree.
>>
>> Not composition tree, you mean via the link graph.
>
> I mean both, but the link graph is already understandable (1-to-1 is easy).
> Right now I can hardly understand how the composition tree will work, for
> example: how do you constrain children to be subclasses of some class (or to
> implement some interface)?

Composition never involves subclasses.  The tree I point you to above is about 
as complete as it can be right now without doing more qdev conversions.  It 
should answer all of your questions re: composition.

>>> 2) Related to this, you have a lot of nice infrastructure, but (unlike
>>> what you did with QAPI) you haven't provided yet a clear plan for how
>>> to get rid of the old code. You also have only very limited uses of
>>> the infrastructure (see above).
>>
>> Old style properties go away as they're converted to new style properties.
>
> And how do they? How much code churn does that entail, in the devices and in
> general? In fact, why do they need conversion at all? Static properties are
> special enough to warrant something special.

They can stay around forever (or until someone needs non-static properties) but 
I strongly suspect that we'll get rid of most of them as part of refactoring.

An awful lot of properties deal with connecting to back ends and/or bus 
addressing.  None of that will be needed anymore.

>> If you want to see how this all is going to look, look at the old tree
>> :-) That's where we're going to end up.
>
> Let's write documentation on that already.

I have written lots of documentation.  Just take a look at the wiki.  Nothing 
speaks better than code.

>>> It's better for various reasons--type safety and ease of use--even if
>>> it costs some boilerplate. For example the child property should be
>>> as simple as
>>>
>>> +struct ChildDeviceProperty {
>>> + DeviceProperty prop;
>>> + DeviceState *child;
>>> +}
>>> +
>>> +struct DevicePropertyInfo child_device_property_info = {
>>> + .size = sizeof(ChildDeviceProperty);
>>> + .get = qdev_get_child_property,
>>> +};
>>> +
>>> void qdev_property_add_child(DeviceState *dev, const char *name,
>>> DeviceState *child, Error **errp)
>>> {
>>> gchar *type;
>>>
>>> type = g_strdup_printf("child<%s>", child->info->name);
>>>
>>> - qdev_property_add(dev, name, type, qdev_get_child_property,
>>> - NULL, NULL, child, errp);
>>> + prop = (ChildDeviceProperty *)
>>> + qdev_property_add(&child_device_property_info,
>>> + dev, name, type, errp);
>>> +
>>> + /* TODO: check errp, if it is NULL -> return immediately. */
>>> + prop->child = child;
>>
>> This seems quite a bit uglier to me.
>
> I did say it costs some boilerplate, but it's once per type and you said there's
> a dozen of those. The type safety and more flexibility (an arbitrary struct for
> subclasses rather than an opaque pointer) is worth the ugliness.

I disagree.  Take a look at how string properties work in the above tree.  It's 
strongly type safe.  My old qom tree builds properties based on code generation 
too so the whole discussion on safety is somewhat moot.

>
>>> I think also that the type should not be part of DeviceProperty.
>>> Instead it should be defined by subclasses, with the
>>> DevicePropertyInfo providing a function to format the type to a
>>> string.
>>
>> I don't really know what you mean by this.
>
> Once the type is written as child<RTCState>, you've lost all info on it. Each
> property type should have its own representation for types (it can be implicit,
> or it can be an enum, or it can be a DeviceInfo), with a method to pretty-print it.

I don't know what you mean by "lost all info on it" but I'm strongly opposed to 
a "pretty-print" method.  Having the pretty printing information in the type is 
almost never what you want.

>> We jump too quickly at
>> doing conversions without thinking through the underlying interface.
>
> I hoped you had thought it through. ;)

I have :-)  But I take your position as arguing that we should convert half the 
tree before merging anything which I disagree with.

>> You may find it odd to hear me say this, but I grow weary of adding too
>> much class hierarchy and inheritance with properties. Yes, we could make
>> it very sophisticated but in practice it doesn't have to be. Properties
>> should be one of two things: primitive types or link/childs.
>
> ... and interfaces. Accessing them by name as a property should work well.

No, not interfaces.  The object *is-a* interface.  It doesn't has-a interface.

We've gone through this debate multiple times.  As I said above, we really don't 
need to even worry about this until most of the work is done so let's avoid this 
debate for now.  The property infrastructure is flexible enough to support both 
models.

>> This is where qdev goes very wrong. It mixes up user interface details
>> (how to format an integer on the command line) with implementation
>> details. There's no reason we should be indicating whether a property
>> should be display in hex or decimal in the device itself.
>
> That's totally true. But there's no reason why qdev properties cannot be split
> in two parts, a sane one and a legacy one.

Bingo!  Hence, the 'legacy<>' namespace.  If you want to do a declare, struct 
member based syntax that encodes/decodes as primitive types to a Visitor, be my 
guest :-)

I don't think it's usually what we want, but I'm not opposed to it.

>>> For artificial properties you would need one-off classes; but you can
>>> still provide a helper that creates a specialized
>>> DeviceProperty+DevicePropertyInfo from the functions. I'm thinking of
>>> how people implement prototype-based OO on top of class-based OO, but
>>> it might be just a macro.
>>
>> I think you're over thinking the problem. There are going to be maybe a
>> dozen property types and that's it. They all with correspond exactly to
>> C types with the exception of links/children.
>
> I was thinking of one-off types such as rtc's struct tm.

I'd really like to get to a point where these type visitors were being generated.

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Anthony Liguori]

On 12/03/2011 03:34 PM, Anthony Liguori wrote:
> On 12/03/2011 08:24 AM, Paolo Bonzini wrote:
>> On 12/03/2011 03:40 AM, Anthony Liguori wrote:
>>> That is still true. The next step, inheritance, will pull the properties
>>> into a base class. That base class can be used elsewhere outside of the
>>> device model.
>>>
>>> But this is already a 20 patch series. If you want all of that in one
>>> series, it's going to be 100 patches that are not terribly easy to
>>> review at once.
>>
>> Without a design document and a roadmap, however, it's impossible to try to
>> understand how the pieces will be together. 100 patches may require some time to
>> digest, but 20 patches require a crystal ball to figure out what's ahead.
>
> You can see a bit further by looking at:
>
> https://github.com/aliguori/qemu/commits/qom-next
>
> That fills out the composition tree pretty well for the pc. The next step is
> aggressive refactoring such that the qdev objects reflect the composition. IOW,
> we should create the rtc from within the piix3 initialization function.

I've begun the work of introducing proper inheritance.  There's a lot going on 
but the basic idea is:

1) introduce QOM base type (Object), make qdev inherit from it

2) create a dynamic typeinfo based DeviceInfo, make device class point to deviceinfo

3) model qdev hierarchy in QOM

4) starting from the bottom of the hierarchy, remove DeviceInfo subclass and 
push that functionality into QOM classes

5) once (4) is complete, remove DeviceInfo

6) refactor any use of multiple child busses into separate devices with one bus

7) refactor busstate as an interface

8) refactor device model to make more aggressive use of composition

9) refactor life cycle events into virtual methods

The tree I've posted is on step (4).

Regards,

Anthony Liguori

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Paolo Bonzini]

On 12/04/2011 10:01 PM, Anthony Liguori wrote:
>>
>
> I've begun the work of introducing proper inheritance.  There's a
> lot going on but the basic idea is:
>
> 1) introduce QOM base type (Object), make qdev inherit from it
>
> 2) create a dynamic typeinfo based DeviceInfo, make device class
> point to deviceinfo
>
> 3) model qdev hierarchy in QOM
>
> 4) starting from the bottom of the hierarchy, remove DeviceInfo
> subclass and push that functionality into QOM classes

Ok, now we're talking.  I looked at the SCSI bits and it looks
sane---though I still don't understand your loathe for static
initializers (which also causes a lot of code churn).

> I do want to get rid of qdev busses but I'm in no rush at all.  We
> can do 95% of the necessary refactoring without touching busses and
> hot plug and I think that's the right strategy.

Yes, as long as you have a plan for them.  Some of the buses have data
so they will have to become objects of their own, with a bidirectional
link between them and the parent device.  (Because I'm not going to
write code like this for each HBA:

     void lsi_set_unit_attention(Device *dev, SCSISense sense)
     {
          LSIDevice *lsi = LSI_DEVICE(dev);
          lsi->unit_attention = sense;
     }

No, I'm not.  Over my dead body).

>> Right now I can hardly understand how the composition tree will
>> work, for example: how do you constrain children to be subclasses
>> of some class (or to implement some interface)?
>
> Composition never involves subclasses.  The tree I point you to above
> is about as complete as it can be right now without doing more qdev
> conversions.  It should answer all of your questions re: composition.

All except one: why can't child (and link too IIRC) properties be 
created with a NULL value?

>> And how do they? How much code churn does that entail, in the
>> devices and in general? In fact, why do they need conversion at
>> all? Static properties are special enough to warrant something
>> special.
>
> They can stay around forever (or until someone needs non-static
> properties) but I strongly suspect that we'll get rid of most of them
> as part of refactoring.
>
> An awful lot of properties deal with connecting to back ends and/or
> bus addressing.  None of that will be needed anymore.

True for PCI, but what about ISA or MMIO devices?  They do their own
address/IRQ decoding.  You don't see those properties in many cases
because they're hidden beneath sysbus magic, but there are hundreds.

>> Let's write documentation on that already.
>
> I have written lots of documentation.  Just take a look at the wiki.

It's down. :(

>> Once the type is written as child<RTCState>, you've lost all info
>> on it. Each property type should have its own representation for
>> types (it can be implicit, or it can be an enum, or it can be a
>> DeviceInfo), with a method to pretty-print it.
>
> I don't know what you mean by "lost all info on it" but I'm strongly
> opposed to a "pretty-print" method.  Having the pretty printing
> information in the type is almost never what you want.

I agree---I was talking about pretty-printing *the type itself*.  The
type falls outside the visitor model, which is only concerned about the
contents.

>>> You may find it odd to hear me say this, but I grow weary of
>>> adding too much class hierarchy and inheritance with properties.
>>>  Yes, we could make it very sophisticated but in practice it
>>> doesn't have to be. Properties should be one of two things:
>>> primitive types or link/childs.
>>
>> ... and interfaces. Accessing them by name as a property should
>> work well.
>
> No, not interfaces.  The object *is-a* interface.  It doesn't has-a
> interface.>
>
> We've gone through this debate multiple times.

You misunderstood (and I wasn't clear enough).

The is-a/has-a debate is indeed settled for things such as PCI-ness
where we'll keep the current three-level class hierarchy (two abstract,
one concrete):

     Device
         PCIDevice
             ... concrete PCI devices ...
         ISADevice
             ... concrete ISA devices ...

The problem is that, in addition to the is-a class hierarchy, you have
the interface hierarchy.  Here, C-level struct "inheritance" does not
help you because you do not have a fixed place for the vtable.  So, for
example, instead of

     struct SCSIBusInfo {
         .command_complete = lsi_command_complete,
         ...
     };

I'll have something like this:

     Interface *interface;
     SCSIBusIface *sbi;
     interface = qdev_add_interface(&dev->qdev, TYPE_SCSI_BUS_INTERFACE);
     sbi = SCSI_BUS_INTERFACE(interface);
     sbi->command_complete = lsi_command_complete;
     ...

Right?  Then I create the SCSIBus object with something as simple as:

     struct LSIDevice {
         ...
         SCSIBus *bus;
     }

     dev->bus = scsi_bus_new(&dev->qdev);

and scsi_bus_new will do something like

     Interface *interface;
     interface = qdev_get_interface(dev, TYPE_SCSI_BUS_INTERFACE);
     scsi_bus->sbi = SCSI_BUS_INTERFACE(interface);

Perhaps hidden with some macro that lets me just write
SCSI_BUS_INTERFACE(dev), but that's the idea; such a lookup function is
pretty much what all object models do.  GObject has
G_TYPE_INSTANCE_GET_INTERFACE, COM/XPCOM has QueryInterface, etc.

If I understood everything so far, then here is my question.  Are
interfaces properties?  I'm starting to understand now that the answer
is probably "no because interfaces have nothing to do with visitors",
and that's fine.  Then I suppose you'll have another list to lookup
besides properties, no problem with that.

>>> This is where qdev goes very wrong. It mixes up user interface
>>> details (how to format an integer on the command line) with
>>> implementation details. There's no reason we should be indicating
>>> whether a property should be display in hex or decimal in the
>>> device itself.
>>
>> That's totally true. But there's no reason why qdev properties
>> cannot be split in two parts, a sane one and a legacy one.
>
> Bingo!  Hence, the 'legacy<>' namespace.  If you want to do a
> declare, struct member based syntax that encodes/decodes as primitive
> types to a Visitor, be my guest

That's not what I meant.  The legacy<> namespace splits the set of QOM
properties in two parts, sane ones and legacy ones.  That's wrong,
because the old broken interface remains there.  Worse, it remains
there as user-visible API in the JSON etc., and it will remain forever 
since we cannot get rid of -device overnight.

What I suggested is to provide two implementations for each old-style
property: an old string-based one (used for -device etc.) and a modern
visitor-based one (used for qom_*).  In other words, old-style
properties would expose both a print/parse legacy interface, and a sane
get/set visitor interface.  No need for a legacy<> namespace, because
new-style consumers would not see the old-style string ABI.

>> I was thinking of one-off types such as rtc's struct tm.
>
> I'd really like to get to a point where these type visitors were
> being generated.

Yeah, long term that'd be great.

Paolo

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Anthony Liguori]

On 12/05/2011 03:52 AM, Paolo Bonzini wrote:
> On 12/04/2011 10:01 PM, Anthony Liguori wrote:
>>>
>>
>> I've begun the work of introducing proper inheritance. There's a
>> lot going on but the basic idea is:
>>
>> 1) introduce QOM base type (Object), make qdev inherit from it
>>
>> 2) create a dynamic typeinfo based DeviceInfo, make device class
>> point to deviceinfo
>>
>> 3) model qdev hierarchy in QOM
>>
>> 4) starting from the bottom of the hierarchy, remove DeviceInfo
>> subclass and push that functionality into QOM classes
>
> Ok, now we're talking. I looked at the SCSI bits and it looks
> sane---though I still don't understand your loathe for static
> initializers (which also causes a lot of code churn).

It's to support method inheritance.  In qdev, the various DeviceInfo structures 
correspond roughly to the class of the object.  When you create an ISADeviceInfo 
(the ISA subclass), you declare it statically.

Any methods you aren't overriding are going to be initialized to zero.  You want 
those methods to inherit their values from the base class.  To do this in qdev, 
you have to introduce a base-class specific registration function 
(isa_qdev_register).

There's not a lot of discipline in how these functions are implemented and 
generally makes type registration more complicated as you have to understand 
what methods get overridden.

What QOM does (GObject does this too) is use a function to override methods.  I 
agree that it's not as pretty, but it means you can just memcpy() the parent 
class' contents into the derived class.  This makes inheritance much cleaner and 
also means you can register all types through a single interface.

You can see this refactoring happening in that branch.  I'm slowly getting rid 
of all of the DeviceInfo derivatives.  Once everything is using DeviceInfo 
directly, then I'll replace DeviceInfo with TypeInfo.  This is going to be the 
biggest single change in the QOM conversion.  It's going to have to be done by a 
script.

Getting rid of SysBusInfo will probably also require a script.

>> I do want to get rid of qdev busses but I'm in no rush at all. We
>> can do 95% of the necessary refactoring without touching busses and
>> hot plug and I think that's the right strategy.
>
> Yes, as long as you have a plan for them. Some of the buses have data
> so they will have to become objects of their own, with a bidirectional
> link between them and the parent device. (Because I'm not going to
> write code like this for each HBA:
>
> void lsi_set_unit_attention(Device *dev, SCSISense sense)
> {
> LSIDevice *lsi = LSI_DEVICE(dev);
> lsi->unit_attention = sense;
> }
>
> No, I'm not. Over my dead body).

Yes, my current plan is to make buses interfaces which means you would need to 
have getters/setters for state.  Yes, I know that can get ugly quickly.

But this comes down to refactoring the bus interfaces I think.  I honestly can't 
tell you how I would handle this cleaner without sitting down and thinking about 
the SCSI bus interface.

Generally speaking, accessors are a very good thing.  It makes code much easier 
to refactor and you'll notice that a lot of my refactoring patches start by 
adding accessors/wrapper functions that probably should have been there since 
day one.

But I agree, we don't want to end up with something where you add 10 lines 
instead of one line.

>>> Right now I can hardly understand how the composition tree will
>>> work, for example: how do you constrain children to be subclasses
>>> of some class (or to implement some interface)?
>>
>> Composition never involves subclasses. The tree I point you to above
>> is about as complete as it can be right now without doing more qdev
>> conversions. It should answer all of your questions re: composition.
>
> All except one: why can't child (and link too IIRC) properties be created with a
> NULL value?

links are nullable and usually start out as NULL.

childs are not nullable. I can't really think of a reason why they should be 
nullable.  What are you thinking here?

>>> And how do they? How much code churn does that entail, in the
>>> devices and in general? In fact, why do they need conversion at
>>> all? Static properties are special enough to warrant something
>>> special.
>>
>> They can stay around forever (or until someone needs non-static
>> properties) but I strongly suspect that we'll get rid of most of them
>> as part of refactoring.
>>
>> An awful lot of properties deal with connecting to back ends and/or
>> bus addressing. None of that will be needed anymore.
>
> True for PCI, but what about ISA or MMIO devices? They do their own
> address/IRQ decoding. You don't see those properties in many cases
> because they're hidden beneath sysbus magic, but there are hundreds.

I've thought a lot about bus properties.  I've looked at a lot of code at this 
point and for the most part, I think that the reason they even exist is because 
we can't inherit a default set of properties.

SCSI is a good example.  The bus properties really make more sense as SCSIDevice 
properties that are inherited.

I dislike these properties in the first place, but I'd like to find a way to 
convert everything to the QOM type system before we start rearchitecting how 
hotplug works.

>>> Let's write documentation on that already.
>>
>> I have written lots of documentation. Just take a look at the wiki.
>
> It's down. :(

It's up now.

>
>>> Once the type is written as child<RTCState>, you've lost all info
>>> on it. Each property type should have its own representation for
>>> types (it can be implicit, or it can be an enum, or it can be a
>>> DeviceInfo), with a method to pretty-print it.
>>
>> I don't know what you mean by "lost all info on it" but I'm strongly
>> opposed to a "pretty-print" method. Having the pretty printing
>> information in the type is almost never what you want.
>
> I agree---I was talking about pretty-printing *the type itself*. The
> type falls outside the visitor model, which is only concerned about the
> contents.
>
>>>> You may find it odd to hear me say this, but I grow weary of
>>>> adding too much class hierarchy and inheritance with properties.
>>>> Yes, we could make it very sophisticated but in practice it
>>>> doesn't have to be. Properties should be one of two things:
>>>> primitive types or link/childs.
>>>
>>> ... and interfaces. Accessing them by name as a property should
>>> work well.
>>
>> No, not interfaces. The object *is-a* interface. It doesn't has-a
>> interface.>
>>
>> We've gone through this debate multiple times.
>
> You misunderstood (and I wasn't clear enough).
>
> The is-a/has-a debate is indeed settled for things such as PCI-ness
> where we'll keep the current three-level class hierarchy (two abstract,
> one concrete):
>
> Device
> PCIDevice
> ... concrete PCI devices ...
> ISADevice
> ... concrete ISA devices ...
>
> The problem is that, in addition to the is-a class hierarchy, you have
> the interface hierarchy. Here, C-level struct "inheritance" does not
> help you because you do not have a fixed place for the vtable. So, for
> example, instead of
>
> struct SCSIBusInfo {
> .command_complete = lsi_command_complete,
> ...
> };
>
> I'll have something like this:
>
> Interface *interface;
> SCSIBusIface *sbi;
> interface = qdev_add_interface(&dev->qdev, TYPE_SCSI_BUS_INTERFACE);
> sbi = SCSI_BUS_INTERFACE(interface);
> sbi->command_complete = lsi_command_complete;
> ...
>
> Right? Then I create the SCSIBus object with something as simple as:
>
> struct LSIDevice {
> ...
> SCSIBus *bus;
> }
>
> dev->bus = scsi_bus_new(&dev->qdev);
>
> and scsi_bus_new will do something like
>
> Interface *interface;
> interface = qdev_get_interface(dev, TYPE_SCSI_BUS_INTERFACE);
> scsi_bus->sbi = SCSI_BUS_INTERFACE(interface);


No, you would do:

struct SCSIBus {
    Interface parent;
    void (*command_complete)(SCSIBus *bus, SCSIRequest *req);
};

TypeInfo scsi_bus_info = {
    .name = TYPE_SCSI_BUS,
    .parent = TYPE_INTERFACE,
};

type_register_static(&scsi_bus_info);

--------

struct LSIDevice {
     PCIDevice parent;
};

static void lsi_command_complete(SCSIBus *bus, SCSIRequest *req)
{
    LSIDevice *dev = LSI_DEVICE(bus);
    ...
}

static void lsi_scsi_bus_initfn(Interface *iface)
{
    SCSIBus *bus = SCSI_BUS(iface);

    bus->command_complete = lsi_command_complete;
}

TypeInfo lsi_device_info = {
   .name = TYPE_LSI,
   .parent = TYPE_PCI_DEVICE,
   .interfaces = (Interface[]){
      {
         .name = TYPE_SCSI_BUS,
         .interface_initfn = lsi_scsi_bus_initfn,
      }, {
      }
   },
};

type_register_static(&lsi_device_info);

>
> Perhaps hidden with some macro that lets me just write
> SCSI_BUS_INTERFACE(dev), but that's the idea; such a lookup function is
> pretty much what all object models do. GObject has
> G_TYPE_INSTANCE_GET_INTERFACE, COM/XPCOM has QueryInterface, etc.
>
> If I understood everything so far, then here is my question. Are
> interfaces properties?

No.  A device is-a interface.  Hopefully the above example will make it more clear.

> I'm starting to understand now that the answer
> is probably "no because interfaces have nothing to do with visitors",
> and that's fine. Then I suppose you'll have another list to lookup
> besides properties, no problem with that.

Yes, there's a list of interfaces embedded in the Object type.

>>>> This is where qdev goes very wrong. It mixes up user interface
>>>> details (how to format an integer on the command line) with
>>>> implementation details. There's no reason we should be indicating
>>>> whether a property should be display in hex or decimal in the
>>>> device itself.
>>>
>>> That's totally true. But there's no reason why qdev properties
>>> cannot be split in two parts, a sane one and a legacy one.
>>
>> Bingo! Hence, the 'legacy<>' namespace. If you want to do a
>> declare, struct member based syntax that encodes/decodes as primitive
>> types to a Visitor, be my guest
>
> That's not what I meant. The legacy<> namespace splits the set of QOM
> properties in two parts, sane ones and legacy ones. That's wrong,
> because the old broken interface remains there. Worse, it remains
> there as user-visible API in the JSON etc., and it will remain forever since we
> cannot get rid of -device overnight.
>
> What I suggested is to provide two implementations for each old-style
> property: an old string-based one (used for -device etc.) and a modern
> visitor-based one (used for qom_*). In other words, old-style
> properties would expose both a print/parse legacy interface, and a sane
> get/set visitor interface. No need for a legacy<> namespace, because
> new-style consumers would not see the old-style string ABI.

Yeah, I'd like to do something like this but I'm in no rush.  I agree that when 
we declare QOM as a supported interface, we should have replacements for 
anything that's in the legacy<> space.  That may be from some magic Property 
tricks where we introduce Visitor to parse/print or because we introduce new and 
improved properties.

Maybe now is the right time to rename the legacy properties to all be prefixed 
with qdev-?  That way we don't need to introduce two different types for a 
single property.

Regards,

Anthony Liguori

>
>>> I was thinking of one-off types such as rtc's struct tm.
>>
>> I'd really like to get to a point where these type visitors were
>> being generated.
>
> Yeah, long term that'd be great.
>
> Paolo
>

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Peter Maydell]

On 5 December 2011 14:36, Anthony Liguori <anthony@codemonkey.ws> wrote:
> On 12/05/2011 03:52 AM, Paolo Bonzini wrote:
> struct SCSIBus {
>   Interface parent;
>   void (*command_complete)(SCSIBus *bus, SCSIRequest *req);
> };
>
> TypeInfo scsi_bus_info = {
>   .name = TYPE_SCSI_BUS,
>   .parent = TYPE_INTERFACE,
> };
>
> type_register_static(&scsi_bus_info);
>
> --------
>
> struct LSIDevice {
>    PCIDevice parent;
> };
>
> static void lsi_command_complete(SCSIBus *bus, SCSIRequest *req)
> {
>   LSIDevice *dev = LSI_DEVICE(bus);
>   ...
> }

What is the LSI_DEVICE macro actually doing here? I assume
it's not just a cast...

> static void lsi_scsi_bus_initfn(Interface *iface)
> {
>   SCSIBus *bus = SCSI_BUS(iface);
>
>   bus->command_complete = lsi_command_complete;
> }
>
> TypeInfo lsi_device_info = {
>  .name = TYPE_LSI,
>  .parent = TYPE_PCI_DEVICE,
>  .interfaces = (Interface[]){
>     {
>        .name = TYPE_SCSI_BUS,
>        .interface_initfn = lsi_scsi_bus_initfn,
>     }, {
>     }
>  },
> };
>
> type_register_static(&lsi_device_info);
>
>
>>
>> Perhaps hidden with some macro that lets me just write
>> SCSI_BUS_INTERFACE(dev), but that's the idea; such a lookup function is
>> pretty much what all object models do. GObject has
>> G_TYPE_INSTANCE_GET_INTERFACE, COM/XPCOM has QueryInterface, etc.
>>
>> If I understood everything so far, then here is my question. Are
>> interfaces properties?
>
>
> No.  A device is-a interface.  Hopefully the above example will make it more
> clear.

Saying a device is-a interface doesn't match reality. Devices
have multiple interfaces with the rest of the world. (This is
one of the major reasons why SysBus exists: it provides a suboptimal
but usuable model of this for the two most common kinds of interface,
MMIO regions and random gpio.)

-- PMM

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Anthony Liguori]

On 12/05/2011 08:50 AM, Peter Maydell wrote:
> On 5 December 2011 14:36, Anthony Liguori<anthony@codemonkey.ws>  wrote:
>> struct LSIDevice {
>>     PCIDevice parent;
>> };
>>
>> static void lsi_command_complete(SCSIBus *bus, SCSIRequest *req)
>> {
>>    LSIDevice *dev = LSI_DEVICE(bus);
>>    ...
>> }
>
> What is the LSI_DEVICE macro actually doing here? I assume
> it's not just a cast...

https://github.com/aliguori/qemu/blob/qom-next/hw/object.c#L376

It's quite literally dynamic_cast<LSIDevice>(bus) in C++.

>> static void lsi_scsi_bus_initfn(Interface *iface)
>> {
>>    SCSIBus *bus = SCSI_BUS(iface);
>>
>>    bus->command_complete = lsi_command_complete;
>> }
>>
>> TypeInfo lsi_device_info = {
>>   .name = TYPE_LSI,
>>   .parent = TYPE_PCI_DEVICE,
>>   .interfaces = (Interface[]){
>>      {
>>         .name = TYPE_SCSI_BUS,
>>         .interface_initfn = lsi_scsi_bus_initfn,
>>      }, {
>>      }
>>   },
>> };
>>
>> type_register_static(&lsi_device_info);
>>
>>
>>>
>>> Perhaps hidden with some macro that lets me just write
>>> SCSI_BUS_INTERFACE(dev), but that's the idea; such a lookup function is
>>> pretty much what all object models do. GObject has
>>> G_TYPE_INSTANCE_GET_INTERFACE, COM/XPCOM has QueryInterface, etc.
>>>
>>> If I understood everything so far, then here is my question. Are
>>> interfaces properties?
>>
>>
>> No.  A device is-a interface.  Hopefully the above example will make it more
>> clear.
>
> Saying a device is-a interface doesn't match reality. Devices
> have multiple interfaces with the rest of the world.

There are two ways a device can interact with the rest of the world.  It can 
expose a portion of it's functionality (such as an IRQ) via a child object. 
This is how it would expose MemoryRegions too.

You can take a subset of the exposed children (and perhaps some mapping logic), 
and for an ad-hoc interface.

But sometimes, you want the entire device to act like a specific thing.  In this 
case, you want the LSIDevice to act like SCSIBus.  Interfaces are just a more 
formal form of what would otherwise be an ad-hoc interface.

> (This is
> one of the major reasons why SysBus exists: it provides a suboptimal
> but usuable model of this for the two most common kinds of interface,
> MMIO regions and random gpio.)

My expectation is that most things that use SysBus today would not implement any 
interfaces.  They would just expose child properties.

Regards,

Anthony Liguori


> -- PMM
>

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Peter Maydell]

On 5 December 2011 15:04, Anthony Liguori <anthony@codemonkey.ws> wrote:
> On 12/05/2011 08:50 AM, Peter Maydell wrote:
>>
>> On 5 December 2011 14:36, Anthony Liguori<anthony@codemonkey.ws>  wrote:
>>>
>>> struct LSIDevice {
>>>
>>>    PCIDevice parent;
>>> };
>>>
>>> static void lsi_command_complete(SCSIBus *bus, SCSIRequest *req)
>>> {
>>>   LSIDevice *dev = LSI_DEVICE(bus);
>>>   ...
>>> }
>>
>>
>> What is the LSI_DEVICE macro actually doing here? I assume
>> it's not just a cast...
>
>
> https://github.com/aliguori/qemu/blob/qom-next/hw/object.c#L376
>
> It's quite literally dynamic_cast<LSIDevice>(bus) in C++.

This is confusing "interface I expose to the world" with
Device. Since "interface I expose to the world shouldn't
be a subtype of Device [they are fundamentally different kinds
of object and having exposed interfaces be kinds-of Devices
is extremely confusing] I'm not sure how this would work.

>> Saying a device is-a interface doesn't match reality. Devices
>> have multiple interfaces with the rest of the world.
>
>
> There are two ways a device can interact with the rest of the world.  It can
> expose a portion of it's functionality (such as an IRQ) via a child object.
> This is how it would expose MemoryRegions too.

Perhaps this is terminology confusion again. To me that is 'exposing
an interface to the rest of the world' -- it's a named point where
we expose a specific API. (For an MMIO region probably that's just
"return a MemoryRegion*".)

The other point of confusion here is that "child objects" are doing
two things:
 * composition, ie "my model of the Foo SoC has a child object FooUART
   which implements its serial ports" (but that is an internal
   implementation detail and might change in future)
 * providing connections to the rest of the world, ie "my Foo SoC has
   a child object Pin named 'my_signal' which is part of its API
   contract with the outside world"

Obviously it's useful (and neat) if the implementation of these things
is broadly similar, but I think that because they're conceptually
totally different things we should strive to pick terminology (and
type/class names) that maintain and make clear that difference.

> You can take a subset of the exposed children (and perhaps some mapping
> logic), and for an ad-hoc interface.

This is composition-of-interfaces, right? That would also be useful
(but probably not needed to start with).

> But sometimes, you want the entire device to act like a specific thing.  In
> this case, you want the LSIDevice to act like SCSIBus.  Interfaces are just
> a more formal form of what would otherwise be an ad-hoc interface.

This is something different again -- it's not just composition-of-interfaces
to provide a SCSI-bus-connector, it's claiming the whole device is-a
SCSI-bus-connector.

I think we should have one way of doing connections between devices,
and we should make sure it does what we want. If we immediately say
"oh, SCSI and PCI connect in a special way" then we've removed a lot
of the impetus to make the standard connection method usefully flexible
and convenient.

>> (This is
>> one of the major reasons why SysBus exists: it provides a suboptimal
>> but usuable model of this for the two most common kinds of interface,
>> MMIO regions and random gpio.)
>
> My expectation is that most things that use SysBus today would not implement
> any interfaces.  They would just expose child properties.

Terminology confusion again. They definitely need to be able to expose
the equivalent of gpio in and out pins, and memory regions, by name.

-- PMM

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Anthony Liguori]

On 12/05/2011 09:33 AM, Peter Maydell wrote:
> On 5 December 2011 15:04, Anthony Liguori<anthony@codemonkey.ws>  wrote:
>> On 12/05/2011 08:50 AM, Peter Maydell wrote:
>>>
>>> On 5 December 2011 14:36, Anthony Liguori<anthony@codemonkey.ws>    wrote:
>>>>
>>>> struct LSIDevice {
>>>>
>>>>     PCIDevice parent;
>>>> };
>>>>
>>>> static void lsi_command_complete(SCSIBus *bus, SCSIRequest *req)
>>>> {
>>>>    LSIDevice *dev = LSI_DEVICE(bus);
>>>>    ...
>>>> }
>>>
>>>
>>> What is the LSI_DEVICE macro actually doing here? I assume
>>> it's not just a cast...
>>
>>
>> https://github.com/aliguori/qemu/blob/qom-next/hw/object.c#L376
>>
>> It's quite literally dynamic_cast<LSIDevice>(bus) in C++.
>
> This is confusing "interface I expose to the world" with
> Device. Since "interface I expose to the world shouldn't
> be a subtype of Device [they are fundamentally different kinds
> of object and having exposed interfaces be kinds-of Devices
> is extremely confusing] I'm not sure how this would work.

No, an interface is not a subclass of DeviceState.  SCSIBus is a subclass of 
Interface.

I think this will become more obvious once we start converting BusState's to 
interfaces.  That's still a ways off so I'd suggest we defer this discussion 
until then.

>>> Saying a device is-a interface doesn't match reality. Devices
>>> have multiple interfaces with the rest of the world.
>>
>>
>> There are two ways a device can interact with the rest of the world.  It can
>> expose a portion of it's functionality (such as an IRQ) via a child object.
>> This is how it would expose MemoryRegions too.
>
> Perhaps this is terminology confusion again. To me that is 'exposing
> an interface to the rest of the world' -- it's a named point where
> we expose a specific API. (For an MMIO region probably that's just
> "return a MemoryRegion*".)

Yes, what you describe here is possible and will be very common.  What I'm 
talking about as "Interfaces" is a higher level concept.

> The other point of confusion here is that "child objects" are doing
> two things:
>   * composition, ie "my model of the Foo SoC has a child object FooUART
>     which implements its serial ports" (but that is an internal
>     implementation detail and might change in future)

Yup.  Does you see this as a problem?

>   * providing connections to the rest of the world, ie "my Foo SoC has
>     a child object Pin named 'my_signal' which is part of its API
>     contract with the outside world"

Providing an ABI and providing full backwards compatibility are two different 
things.

An ABI means (or API really) that if you ask QEMU to does something, if it 
doesn't fail in a predictable fashion, then what it ends up doing will always be 
the same.

IOW, if I say, -device virtio-blk-pci, it should create a virtio block device 
using the PCI virtio transport.  If it caused QEMU to print device information 
about that device, that's an ABI breakage.

Backwards compatibility is stronger than this.  It says that if you invoke QEMU 
a certain way, it will behave in an indistinguishable way from an older version.

I want QOM to be ABI compatible but not backwards compatible.  I want to give us 
the flexibility to change device names, break up devices, or add devices.  When 
things are removed, it should result in errors.  It should also be possible to 
do enough introspection to understand what is supported without relying on 
version numbers.

We should provide a backwards compatible interface, but it should be much higher 
level.  The way I think about it, qom-* is the non-backwards compatible API 
within QMP, and then we'll introduce higher level interfaces (much like 
query-pci) that will remain long term backwards compatible.

> Obviously it's useful (and neat) if the implementation of these things
> is broadly similar, but I think that because they're conceptually
> totally different things we should strive to pick terminology (and
> type/class names) that maintain and make clear that difference.
>
>> You can take a subset of the exposed children (and perhaps some mapping
>> logic), and for an ad-hoc interface.
>
> This is composition-of-interfaces, right? That would also be useful
> (but probably not needed to start with).
>
>> But sometimes, you want the entire device to act like a specific thing.  In
>> this case, you want the LSIDevice to act like SCSIBus.  Interfaces are just
>> a more formal form of what would otherwise be an ad-hoc interface.
>
> This is something different again -- it's not just composition-of-interfaces
> to provide a SCSI-bus-connector, it's claiming the whole device is-a
> SCSI-bus-connector.
>
> I think we should have one way of doing connections between devices,
> and we should make sure it does what we want. If we immediately say
> "oh, SCSI and PCI connect in a special way" then we've removed a lot
> of the impetus to make the standard connection method usefully flexible
> and convenient.
>
>>> (This is
>>> one of the major reasons why SysBus exists: it provides a suboptimal
>>> but usuable model of this for the two most common kinds of interface,
>>> MMIO regions and random gpio.)
>>
>> My expectation is that most things that use SysBus today would not implement
>> any interfaces.  They would just expose child properties.
>
> Terminology confusion again. They definitely need to be able to expose
> the equivalent of gpio in and out pins, and memory regions, by name.

Yes, child properties.  Pins and MemoryRegions need to be Objects (not 
necessarily DeviceStates).

Regards,

Anthony Liguori

> -- PMM
>

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Paolo Bonzini]

On 12/05/2011 03:36 PM, Anthony Liguori wrote:
> It's to support method inheritance. In qdev, the various DeviceInfo
> structures correspond roughly to the class of the object. When you
> create an ISADeviceInfo (the ISA subclass), you declare it statically.
>
> Any methods you aren't overriding are going to be initialized to zero.
> You want those methods to inherit their values from the base class. To
> do this in qdev, you have to introduce a base-class specific
> registration function (isa_qdev_register).
>
> There's not a lot of discipline in how these functions are implemented
> and generally makes type registration more complicated as you have to
> understand what methods get overridden.

Yeah, that's true.  I think in general our class hierarchy is shallow 
enough that we could live with that, but I appreciate that dynamic 
initialization has advantages.

> links are nullable and usually start out as NULL.
>
> childs are not nullable. I can't really think of a reason why they
> should be nullable. What are you thinking here?

Ok, I understand now better what children are.

> I've thought a lot about bus properties. I've looked at a lot of code at
> this point and for the most part, I think that the reason they even
> exist is because we can't inherit a default set of properties.
>
> SCSI is a good example. The bus properties really make more sense as
> SCSIDevice properties that are inherited.

Yeah, bus properties *are* most of the time properties that you add to 
the abstract class, so...

> I dislike these properties in the first place, but I'd like to find a
> way to convert everything to the QOM type system before we start
> rearchitecting how hotplug works.

... just change them to properties on the abstract class.

>> Perhaps hidden with some macro that lets me just write
>> SCSI_BUS_INTERFACE(dev), but that's the idea; such a lookup function is
>> pretty much what all object models do. GObject has
>> G_TYPE_INSTANCE_GET_INTERFACE, COM/XPCOM has QueryInterface, etc.
>>
>> If I understood everything so far, then here is my question. Are
>> interfaces properties?
>
> No. A device is-a interface. Hopefully the above example will make it
> more clear.

No, but I'm confident that there will be a sane way to access the list 
of interfaces that you embed in the Object type. :)

>> That's not what I meant. The legacy<> namespace splits the set of QOM
>> properties in two parts, sane ones and legacy ones. That's wrong,
>> because the old broken interface remains there. Worse, it remains
>> there as user-visible API in the JSON etc., and it will remain forever
>> since we
>> cannot get rid of -device overnight.
>>
>> What I suggested is to provide two implementations for each old-style
>> property: an old string-based one (used for -device etc.) and a modern
>> visitor-based one (used for qom_*). In other words, old-style
>> properties would expose both a print/parse legacy interface, and a sane
>> get/set visitor interface. No need for a legacy<> namespace, because
>> new-style consumers would not see the old-style string ABI.
>
> Yeah, I'd like to do something like this but I'm in no rush. I agree
> that when we declare QOM as a supported interface, we should have
> replacements for anything that's in the legacy<> space. That may be from
> some magic Property tricks where we introduce Visitor to parse/print or
> because we introduce new and improved properties.

Yeah, extending Property looks like a feasible plan.  The get/set pair 
is an adaptor between JSON/Visitor-type data and C struct fields, the 
parse/print pair is an adaptor between strings and C struct fields.

> Maybe now is the right time to rename the legacy properties to all be
> prefixed with qdev-? That way we don't need to introduce two different
> types for a single property.

Why do you need such a prefix?

Paolo

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Anthony Liguori]

On 12/05/2011 09:47 AM, Paolo Bonzini wrote:
> On 12/05/2011 03:36 PM, Anthony Liguori wrote:
>> No. A device is-a interface. Hopefully the above example will make it
>> more clear.
>
> No, but I'm confident that there will be a sane way to access the list of
> interfaces that you embed in the Object type. :)

We definitely need to think through introspection.  There are a few things we 
need to address with introspection:

0) What are all of the classes available

1) What are all of the classes/interfaces implemented by a type

2) What are all of the properties available in a class

(0) and (1) just need QMP interfaces.  It's very straight forward.

(2) is a bit more tricky.  You can instantiate a dummy object and introspect on 
the live object.  Dynamic properties make that a bit challenging through.

>> Maybe now is the right time to rename the legacy properties to all be
>> prefixed with qdev-? That way we don't need to introduce two different
>> types for a single property.
>
> Why do you need such a prefix?

To avoid an all-at-once conversion.  I don't want to break -device and 
maintaining string properties make it quite a bit easier to support -device.

I'd like to leave the string properties in place until we have a good way to 
create objects via QMP.

My rough thinking is that we keep -device around until 2.0.

Regards,

Anthony Liguori

>
> Paolo
>

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Paolo Bonzini]

On 12/05/2011 05:13 PM, Anthony Liguori wrote:
>
>>> Maybe now is the right time to rename the legacy properties to all be
>>> prefixed with qdev-? That way we don't need to introduce two different
>>> types for a single property.
>>
>> Why do you need such a prefix?
>
> To avoid an all-at-once conversion.  I don't want to break -device and
> maintaining string properties make it quite a bit easier to support
> -device.

-device would use parse/print, not get/set.  So it would still use the 
strings.

Paolo

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Anthony Liguori]

On 12/05/2011 10:29 AM, Paolo Bonzini wrote:
> On 12/05/2011 05:13 PM, Anthony Liguori wrote:
>>
>>>> Maybe now is the right time to rename the legacy properties to all be
>>>> prefixed with qdev-? That way we don't need to introduce two different
>>>> types for a single property.
>>>
>>> Why do you need such a prefix?
>>
>> To avoid an all-at-once conversion. I don't want to break -device and
>> maintaining string properties make it quite a bit easier to support
>> -device.
>
> -device would use parse/print, not get/set. So it would still use the strings.

I don't want -device to have special knowledge of properties.  I want it to set 
the properties are strings.  The property implementation is private to Object.

Regards,

Anthony Liguori

>
> Paolo

Re: [Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Paolo Bonzini]

On 12/05/2011 05:38 PM, Anthony Liguori wrote:
>>>
>>
>> -device would use parse/print, not get/set. So it would still use the
>> strings.
>
> I don't want -device to have special knowledge of properties.  I want it
> to set the properties are strings.  The property implementation is
> private to Object.

Ok, I see where you're coming from with legacy properties too.  Yeah, 
then it makes sense to rename properties (to legacy<name> perhaps) as 
long as the equivalent "name" property is available that uses visitors. 
  I.e. each qdev property would be registered twice, with two different 
names, one for QOM and one for -device.

Paolo

[Qemu-devel] [PATCH v2 00/18] qom: dynamic properties and composition tree (v2)

[Anthony Liguori]

This is a follow up to my previous series to get us started in the QOM
direction.  A few things are different this time around.  Most notably:

 1) Devices no longer have names.  Instead, path names are always used to
    identify devices.

 2) In order to support (1), dynamic properties are now supported.

 3) The concept of a "root device" has been introduced.  The root device is
    roughly modelling the motherboard of a machine.  This type is a container
    type and it's best to think of it as something like a PCB board I guess.

To try it out, here's an example session:

Launch:

anthony@titi:~/build/qemu$ x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img -snapshot -device virtio-balloon-pci,id=foo -qmp unix:/tmp/server.sock,server,nowait

Explore the object model:

anthony@titi:~/git/qemu/QMP$ ./qom-list /
peripheral/
i440fx/
anthony@titi:~/git/qemu/QMP$ ./qom-list /i440fx/
piix3/
anthony@titi:~/git/qemu/QMP$ ./qom-list /i440fx/piix3
rtc/
anthony@titi:~/git/qemu/QMP$ ./qom-list /i440fx/piix3/rtc
date
base_year
anthony@titi:~/git/qemu/QMP$ ./qom-get /i440fx/piix3/rtc.date
tm_sec: 33
tm_hour: 21
tm_mday: 30
tm_year: 111
tm_mon: 10
tm_min: 2
anthony@titi:~/git/qemu/QMP$ ./qom-get rtc.date
tm_sec: 38
tm_hour: 21
tm_mday: 30
tm_year: 111
tm_mon: 10
tm_min: 2
anthony@titi:~/git/qemu/QMP$ ./qom-list /peripheral
foo/
anthony@titi:~/git/qemu/QMP$ ./qom-list /peripheral/foo
event_idx
indirect_desc

Anthony Liguori (18):
  qom: add a reference count to qdev objects
  qom: add new dynamic property infrastructure based on Visitors (v2)
  qom: register legacy properties as new style properties (v2)
  qom: introduce root device
  qdev: provide an interface to return canonical path from root (v2)
  qdev: provide a path resolution (v2)
  qom: add child properties (composition) (v2)
  qom: add link properties (v2)
  qapi: allow a 'gen' key to suppress code generation
  qmp: add qom-list command
  qom: qom_{get,set} monitor commands (v2)
  qdev: add explicitly named devices to the root complex
  dev: add an anonymous peripheral container
  rtc: make piix3 set the rtc as a child (v2)
  rtc: add a dynamic property for retrieving the date
  qom: optimize qdev_get_canonical_path using a parent link
  qmp: make qmp.py easier to use
  qom: add test tools (v2)

 Makefile.objs            |    2 +-
 QMP/qmp.py               |    6 +
 QMP/qom-get              |   26 +++
 QMP/qom-list             |   30 +++
 QMP/qom-set              |   21 ++
 hw/container.c           |   20 ++
 hw/mc146818rtc.c         |   27 +++
 hw/pc_piix.c             |   11 +
 hw/piix_pci.c            |    3 +
 hw/qdev.c                |  489 +++++++++++++++++++++++++++++++++++++++++++++-
 hw/qdev.h                |  245 +++++++++++++++++++++++
 monitor.h                |    4 +
 qapi-schema.json         |  107 ++++++++++
 qerror.c                 |    4 +
 qerror.h                 |    3 +
 qmp-commands.hx          |   18 ++
 qmp.c                    |   93 +++++++++
 scripts/qapi-commands.py |    1 +
 scripts/qapi-types.py    |    1 +
 19 files changed, 1109 insertions(+), 2 deletions(-)
 create mode 100755 QMP/qom-get
 create mode 100755 QMP/qom-list
 create mode 100755 QMP/qom-set
 create mode 100644 hw/container.c

-- 
1.7.4.1