Re: [Qemu-devel] Re: [patch 2/3] Add support for live block copy

[Anthony Liguori <anthony@codemonkey.ws>]

On 02/24/2011 09:22 AM, Avi Kivity wrote:
> On 02/24/2011 05:00 PM, Anthony Liguori wrote:
>> On 02/24/2011 02:54 AM, Avi Kivity wrote:
>>> On 02/23/2011 10:18 PM, Anthony Liguori wrote:
>>>>> Then the management stack has to worry about yet another way of 
>>>>> interacting via qemu.
>>>>
>>>>
>>>> { 'StateItem': { 'key': 'str', 'value': 'str' } }
>>>> { 'StateSection': { 'kind': 'str', 'name': 'str', 'items': [ 
>>>> 'StateItem' ] } }
>>>> { 'StateInfo': { 'sections': [ 'StateSection' ] } }
>>>>
>>>> { 'query-state', {}, {}, 'StateInfo' }
>>>>
>>>> A management tool never need to worry about anything other than 
>>>> this command if it so chooses.  If we have the pre-machine init 
>>>> mode for 0.16, then this can even be used to inspect state without 
>>>> running a guest.
>>>
>>> So we have yet another information tree.  If we store the cd-rom 
>>> eject state here, then we need to make an association between the 
>>> device path of the cd-rom, and the StateItem key.
>>
>> And this linkage is key.
>>
>> Let's say I launch QEMU with:
>>
>> qemu -cdrom ~/foo.img
>>
>> And then in the monitor, I do:
>>
>> (qemu) eject ide1-cd0
>>
>> The question is, what command can I now use to launch the same qemu 
>> instance?
>>
>> When I think of stateful config, what I really think of is a way to 
>> spit out a command line that essentially becomes, "this is how you 
>> now launch QEMU".
>>
>> In this case, it would be:
>>
>> qemu -cdrom ~/foo.img -device ide-disk,id=ide1-cd0,drive=
>>
>> Or, we could think of this in terms of:
>>
>> qemu -cdrom ~/foo.img -readconfig foo.cfg
>>
>> Where foo.cfg contained:
>>
>> [device "ide1-cd0"]
>> driver="ide-disk"
>> drive=""
>>
>> So what I'm really suggesting is that we generate foo.cfg whenever 
>> monitor commands do things that change the command line and introduce 
>> a new option to reflect this, IOW:
>>
>> qemu -cdrom ~/foo.img -config foo.cfg
>
> If you move the cdrom to a different IDE channel, you have to update 
> the stateful non-config file.
>
> Whereas if you do
>
>    $ qemu-img create -f cd-tray -b ~/foo.img ~/foo-media-tray.img
>    $ qemu -cdrom ~/foo-media-tray.img
>
> the cd-rom tray state will be tracked in the image file.

Yeah, but how do you move it?  If you do a remove/add through QMP, then 
the config file will reflect things just fine.

If you want to do it outside of QEMU, then you can just ignore the 
config file and manage all of the state yourself.  But it's never going 
to work as well (it will be racy) and you're pushing a tremendous amount 
of knowledge that ultimately belongs in QEMU (what state needs to 
persist) to something that isn't QEMU which means it's probably not 
going to be done correctly.

I know you're a big fan of the omnipotent management tool but my 
experience has been that we need to help the management tooling folks 
more by expecting less from them.

>>> Far better to store it in the device itself.  For example, we could 
>>> make a layered block format driver that stores the eject state and a 
>>> "backing file" containing the actual media.  Eject and media change 
>>> would be recorded in the block format driver's state.  You could 
>>> then hot-unplug a USB cd-writer and hot-plug it back into a 
>>> different guest, implementing a virtual sneakernet.
>>
>> I think you're far too hung up on "store it in the device itself".  
>> The recipe to create the device model is not intrinsic to the device 
>> model.  It's an independent thing that's a combination of the command 
>> line arguments and any executed monitor commands.
>>
>> Maybe a better way to think about the stateful config file is a 
>> mechanism to replay the monitor history.
>
> Again the question is who is the authoritative source of the 
> configuration.  Is it the management tool or is it qemu?

QEMU.   No question about it.  At any point in time, we are the 
authoritative source of what the guest's configuration is.  There's no 
doubt about it.  A management tool can try to keep up with us, but 
ultimately we are the only ones that know for sure.

We have all of this information internally.  Just persisting it is not a 
major architectural change.  It's something we should have been doing 
(arguably) from the very beginning.

> The management tool already has to keep track of (the optional parts 
> of) the guest device tree.  It cannot start reading the stateful 
> non-config file at random points in time.  So all that is left is the 
> guest controlled portions of the device tree, which are pretty rare, 
> and random events like live-copy migration.  I think that introducing 
> a new authoritative source of information will create a lot of problems.

QEMU has always been the authoritative source.  Nothing new has been 
introduced.  We never persisted the machine's configuration which meant 
management tools had to try to aggressively keep up with us which is 
intrinsically error prone.  Fixing this will only improve existing 
management tools.

>
>>>>
>>>> The fact that the state is visible in the filesystem is an 
>>>> implementation detail.
>>>
>>> A detail that has to be catered for by the management stack - it has 
>>> to provide a safe place for it, back it up, etc.
>>
>> If it cares for QEMU to preserve state.  Today, this all gets thrown 
>> away.
>
> Right, but we should make it easy, not hard.

Yeah, I fail to see how this makes it hard.  We conveniently are saying, 
hey, this is all the state that needs to be persisted.  We'll persist it 
for you if you want, otherwise, we'll expose it in a central location.

If the tool wants to ignore it and guess based on various combinations 
of other commands, more power to it.

>>
>>>> It doesn't work for eject unless you interpose an acknowledged 
>>>> event.  Ultimately, this is a simple problem.  If you want 
>>>> reliability, we either need symmetric RPCs so that the device model 
>>>> can call (and wait) to the management layer to acknowledge a change 
>>>> or QEMU can post an event to the management layer, and maintain the 
>>>> state in a reliable fashion.
>>>
>>> I don't see why it doesn't work.  Please explain.
>>
>> 1) guest eject
>> 2) qemu posts eject event
>> 3) qemu acknowledges eject to the guest
>> 4) management tool sees eject event and updates guest config
>>
>> There's a race between 3 & 4.  It can only be addressed by 
>> interposing 4 between 2 and 3 OR making qemu persist this state 
>> between 2 and 3 such that the management tool can reliably query it.
>
> If "it" is my cd-rom tray block format driver, it works.  It's really 
> the same in action as the stateful non-config, except it's part of the 
> device/image, not a central location.

Because you've introduced a one-off.  Having a bunch of one-offs 
(especially being a bunch of new block formats!) is not going to make 
things simpler for management tools.

>>> I disagree.  Storing NVRAM as a disk image is a simple extension of 
>>> existing management tools.  Block live-copy and cd-rom eject state 
>>> also make sense as per-image state if you take hotunplug and hotplug 
>>> into account.
>>
>> Everything can be stored in a block driver but when the data is 
>> highly structured, isn't it nice to expose it in a structured, human 
>> readable way?  I know I'd personally prefer a text representation of 
>> CMOS than a binary blob.
>
> Have a tool expose it.  Part of the range is unspecified anyway.

I guess we need to agree to disagree then.

> Using a block format driver means that we don't have to care about a 
> crash during a write, that we can snapshot it, etc.

Why?  We always need to care about a crash during write.  What I've been 
thinking for a config file is the class approach of using a ~ and .# 
file to make sure that we write out the new file and then atomically 
rename it to get the new contents.  Yeah, it's a bit heavy weight but 
this shouldn't be a very common thing to update.

>>> Device settings should be stored with the devices, not with qemu.
>>>
>>> Suppose we take the cold-plug on startup via the monitor approach.  
>>> So we start with a bare machine, cold plug stuff into it.  Now qemu 
>>> has to reconcile the stateful non-config file with the hardware.  
>>> What if something has changed?  A device moved into a different slot?
>>
>> Sorry, I'm confused.  Is there anything in the stateful config file 
>> when we start up?  If so, the act of starting up will add a bunch of 
>> hardware.
>
> Suppose it has information about ide1-cd0's media tray.  Now we 
> restart qemu and cold-plug the cdrom into ide0-cd0.  What happens to 
> the information?

Whether media is present is not a property of a blockdev, it's a 
property of a device.  What does it even mean to use your media-tray 
format with something like a CMOS device?

>> Technically, mac address is stored on eeprom and we store that as a 
>> device property today.  We can't persist device properties even 
>> though you can change the mac address of a network card and it does 
>> persist across reboots.  Are you advocating that we introduce an 
>> eeprom for every network card (all in a slightly different format) 
>> and have special tools to manipulate the eeprom to store and view the 
>> mac address?
>
>
> Yes -- if we really want to support it.  Obviously we have to store 
> the entire eeprom, not just the portion containing the MAC address, so 
> it's not just a key/value store.  A card may even have its firmware in 
> flash.

I think that's overengineering.  I think we can go very far by just 
persisting small amounts of information in a central location.  We're 
not building a cycle-accurate simulator here afterall.

Regards,

Anthony Liguori