Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity <avi@redhat.com>]

On 04/23/2010 04:48 PM, Anthony Liguori wrote:
> On 04/23/2010 07:48 AM, Avi Kivity wrote:
>> On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>>>> real API. Say, adding a device libvirt doesn't know about or 
>>>> stopping the VM
>>>> while libvirt thinks it's still running or anything like that.
>>>   Another problem is issuing Monitor commands that could confuse 
>>> libvirt's
>>>
>>> We need to make libvirt and qemu smarter.
>>>
>>> We already face this problem today with multiple libvirt users.  
>>> This is why sophisticated management mechanisms (like LDAP) have 
>>> mechanisms to do transactions or at least a series of atomic 
>>> operations.
>>
>> And people said qmp/json was overengineered...
>>
>> But seriously, transactions won't help anything.  qemu maintains 
>> state, and when you have two updaters touching a shared variable not 
>> excepting each other to, things break, no matter how much locking 
>> there is.
>
> Let's consider some concrete examples.  I'm using libvirt and QMP and 
> in QMP, I want to hot unplug a device.
>
> Today, I do this by listing the pci devices, and issuing a pci_del 
> that takes a PCI address.  This is intrinsically racy though because 
> in the worst case scenario, in between when I enumerate pci devices 
> and do the pci_del in QMP, in libvirt, I've done a pci_del and then a 
> pci_add within libvirt of a completely different device.

Obviously you should do the pci_del through libvirt.  Once libvirt 
supports an API, use it.

>
> There are a few ways to solve this, the simplest being that we give 
> devices unique ids that are never reused and instead of pci_del taking 
> a pci bus address, it takes a device id.  That would address this race.
>
> You can get very far by just being clever about unique ids and 
> notifications.  There are some cases where a true RMW may be required 
> but I can't really think of one off hand.  The way LDAP addresses this 
> is that it has a batched operation and a simple set of boolean 
> comparison operations.  This lets you execute a batched operation that 
> will do a RMW.

I'm sure we can be very clever, but I'd rather direct this cleverness to 
qemu core issues, not to the QMP (which in turn requires that users be 
clever to use it correctly).  QMP is a low bandwidth protocol, so races 
will never show up in testing.  We're laying mines here for users to 
step on that we will never encounter ourselves.

>
>>   The only way that separate monitors could work is if they touch 
>> completely separate state, which is difficult to ensure if you 
>> upgrade your libvirt.
>>
>
> I don't think this is as difficult of a problem as you think it is.  
> If you look at Active Directory and the whole set of management tools 
> based on it, they certainly allow concurrent management applications.  
> You can certainly get into trouble still but with just some careful 
> considerations, you can make two management applications work together 
> 90% of the time without much fuss on the applications part.

Maybe.  We'll still have issues.  For example, sVirt: if a QMP command 
names a labeled resource, the non-libvirt user will have no way of 
knowing how to label it.

Much better to exact a commitment from libvirt to track all QMP (and 
command line) capabilities.  Instead of adding cleverness to QMP, add 
APIs to libvirt.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.