Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori <anthony@codemonkey.ws>]

On 04/23/2010 09:24 AM, Avi Kivity wrote:
> On 04/23/2010 04:48 PM, Anthony Liguori wrote:
>> On 04/23/2010 07:48 AM, Avi Kivity wrote:
>>> On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>>>>> real API. Say, adding a device libvirt doesn't know about or 
>>>>> stopping the VM
>>>>> while libvirt thinks it's still running or anything like that.
>>>>   Another problem is issuing Monitor commands that could confuse 
>>>> libvirt's
>>>>
>>>> We need to make libvirt and qemu smarter.
>>>>
>>>> We already face this problem today with multiple libvirt users.  
>>>> This is why sophisticated management mechanisms (like LDAP) have 
>>>> mechanisms to do transactions or at least a series of atomic 
>>>> operations.
>>>
>>> And people said qmp/json was overengineered...
>>>
>>> But seriously, transactions won't help anything.  qemu maintains 
>>> state, and when you have two updaters touching a shared variable not 
>>> excepting each other to, things break, no matter how much locking 
>>> there is.
>>
>> Let's consider some concrete examples.  I'm using libvirt and QMP and 
>> in QMP, I want to hot unplug a device.
>>
>> Today, I do this by listing the pci devices, and issuing a pci_del 
>> that takes a PCI address.  This is intrinsically racy though because 
>> in the worst case scenario, in between when I enumerate pci devices 
>> and do the pci_del in QMP, in libvirt, I've done a pci_del and then a 
>> pci_add within libvirt of a completely different device.
>
> Obviously you should do the pci_del through libvirt.  Once libvirt 
> supports an API, use it.

It was just an example...

>>
>> There are a few ways to solve this, the simplest being that we give 
>> devices unique ids that are never reused and instead of pci_del 
>> taking a pci bus address, it takes a device id.  That would address 
>> this race.
>>
>> You can get very far by just being clever about unique ids and 
>> notifications.  There are some cases where a true RMW may be required 
>> but I can't really think of one off hand.  The way LDAP addresses 
>> this is that it has a batched operation and a simple set of boolean 
>> comparison operations.  This lets you execute a batched operation 
>> that will do a RMW.
>
> I'm sure we can be very clever, but I'd rather direct this cleverness 
> to qemu core issues, not to the QMP (which in turn requires that users 
> be clever to use it correctly).  QMP is a low bandwidth protocol, so 
> races will never show up in testing.  We're laying mines here for 
> users to step on that we will never encounter ourselves.
>
>>
>>>   The only way that separate monitors could work is if they touch 
>>> completely separate state, which is difficult to ensure if you 
>>> upgrade your libvirt.
>>>
>>
>> I don't think this is as difficult of a problem as you think it is.  
>> If you look at Active Directory and the whole set of management tools 
>> based on it, they certainly allow concurrent management 
>> applications.  You can certainly get into trouble still but with just 
>> some careful considerations, you can make two management applications 
>> work together 90% of the time without much fuss on the applications 
>> part.
>
> Maybe.  We'll still have issues.  For example, sVirt: if a QMP command 
> names a labeled resource, the non-libvirt user will have no way of 
> knowing how to label it.

This is orthogonal to QMP and has to do strictly with how libvirt 
prepares a resource for qemu.

> Much better to exact a commitment from libvirt to track all QMP (and 
> command line) capabilities.  Instead of adding cleverness to QMP, add 
> APIs to libvirt.
>

Let's step back for a minute because I think we're missing the forest 
through the trees.

We're trying to address a few distinct problems:

1) Allow libvirt users to access features of qemu that are not exposed 
through libvirt

2) Provide a means for non-libvirt users to interact with qemu

3) Provide a unified and interoperable view of the world for non-libvirt 
and libvirt users

For (1), we all agree that the best case scenario would be for libvirt 
to support every qemu feature.  I think we can also all agree though 
that this is not really practical and certainly not practical for 
developers since there is a development cost associated with libvirt 
support (to model an API appropriately).

The new API proposed addresses (1) by allowing a user to drill down to 
the QMP context.  It's a good solution IMHO and I think we all agree 
that there's an inherent risk to this that users will have to evaluate 
on a case-by-case basis.  It's a good stop-gap though.

(2) is largely addressed by QMP and a config file.  I'd like to see a 
nice C library, but I think a lot of other folks are happy with JSON 
support in higher level languages.

(3) is the place where there are still potential challenges.  I think at 
the very least, our goal should be to enable conversion from (2) and (1) 
to be as easy as possible.  That's why I have proposed implementing a C 
library for the JSON transport because we could plumb that through the 
new libvirt API.  This would allow a user to very quickly port an 
application from QMP to libvirt.  In order to do this, we need the 
libvirt API to expose a dedicated monitor because we'll need to be able 
to manipulate events and negotiate features.

Beyond simple porting, there's a secondary question of having 
non-libvirt apps co-exist with libvirt apps.  I think it's a good long 
term goal, but I don't think we should worry too much about it now.

Regards,

Anthony Liguori