[Qemu-devel] sending pci information over the wire

[Qemu-devel] sending pci information over the wire

[Glauber Costa]

Hi folks,

I have some trial code here for a proposal, which I'd like to hear your
opinions about (Well, I _had_, because I'm totally stupid and git reset'd --hard
the wrong location, that happened to contain part of the code for it)

Let me start by explaining what I'm trying to accomplish, and say that I myself
am not sure this is the best approach, it is just a crazy idea that poped up.

Right now, migration of pci devices work by a bit of luck. This is because the
other side of the wire can enumerate the devices in a different order, causing
them to end up at different addresses. Markus approach of pci_addr= patches
do help that fact. 

However, theoretically, there can be a case in which we:
 * start receiving guest, with parameters determined by pci_addr=
 * start live migration
 * add a device.

The receiving guest won't know about that device, and migration then fail.
This is not a problem _today_, as mgmt tools disallow addition of hardware
during migration. But this is a restriction that comes exactly from the lack
of robustness of migration! Furthermore, mgmt tools might want to change
that behaviour in the future...

That said, my proposal is as following:

in the savevm part of the pci bus, list properties of all present devices.
on the load part of the pci bus, scan the bus looking for devices that
should be present and are not, and create them, if needed.

A new machine type gets added, pc_migr, that does not put pci devices in the bus.
All pci devices will be missing, and them them all get created.

To do that, we also have to save/load some qemu internal state. For example,
nd_table has to be transferred, to guarantee that proper configuration will
exist on the other side. Ultimately, that can be used to transfer _all_ qemu
internal state. A side effect, is that a qemu instance receiving migration
can be shot with : qemu-system-x86_64 -M pc_migr -incoming addr -vnc :x.
It is not that important, if you use mgmt apps, but a nice side effect, otherwise.

I tested this with some network cards, and it kinda works. So, let me know what
is the general feeling about that. If there is a compelling case for it,
I can go back and hack it more. Otherwise, it was fun anyway.


----- End forwarded message -----

Re: [Qemu-devel] sending pci information over the wire

[Anthony Liguori]

Glauber Costa wrote:
> Hi folks,
>
> I have some trial code here for a proposal, which I'd like to hear your
> opinions about (Well, I _had_, because I'm totally stupid and git reset'd --hard
> the wrong location, that happened to contain part of the code for it)
>
> Let me start by explaining what I'm trying to accomplish, and say that I myself
> am not sure this is the best approach, it is just a crazy idea that poped up.
>
> Right now, migration of pci devices work by a bit of luck. This is because the
> other side of the wire can enumerate the devices in a different order, causing
> them to end up at different addresses. Markus approach of pci_addr= patches
> do help that fact. 
>
> However, theoretically, there can be a case in which we:
>  * start receiving guest, with parameters determined by pci_addr=
>  * start live migration
>  * add a device.
>   

Once we have a machine config (and please, let's avoid the 2017 jokes 
please), I imagine that the machine config will be transferred as a 
non-live section.

That will solve this problem quite nicely.

Regards,

Anthony Liguori

Re: [Qemu-devel] sending pci information over the wire

[Glauber Costa]

On Thu, Jun 18, 2009 at 03:03:21PM -0500, Anthony Liguori wrote:
> Glauber Costa wrote:
>> Hi folks,
>>
>> I have some trial code here for a proposal, which I'd like to hear your
>> opinions about (Well, I _had_, because I'm totally stupid and git reset'd --hard
>> the wrong location, that happened to contain part of the code for it)
>>
>> Let me start by explaining what I'm trying to accomplish, and say that I myself
>> am not sure this is the best approach, it is just a crazy idea that poped up.
>>
>> Right now, migration of pci devices work by a bit of luck. This is because the
>> other side of the wire can enumerate the devices in a different order, causing
>> them to end up at different addresses. Markus approach of pci_addr= patches
>> do help that fact. 
>>
>> However, theoretically, there can be a case in which we:
>>  * start receiving guest, with parameters determined by pci_addr=
>>  * start live migration
>>  * add a device.
>>   
>
> Once we have a machine config (and please, let's avoid the 2017 jokes  
> please), I imagine that the machine config will be transferred as a  
> non-live section.
>
> That will solve this problem quite nicely.
Sure.

Question is, whether or not there is any value in doing this now.
I may argue that this could raise issues early, and the rework needed
to make it work can pave the way for a machine config, making things
easier for it.

Or maybe not.

Re: [Qemu-devel] sending pci information over the wire

[Anthony Liguori]

Glauber Costa wrote:
> Sure.
>
> Question is, whether or not there is any value in doing this now.
>   

Doing what you proposed would be difficult to support long term.  I 
don't think it qualifies as an interim solution.  You would have to pass 
host configuration information in the migration protocol and that's a 
fundamental change to the way migration works in QEMU today.

If you assume the management application knows about the devices added, 
then it also knows about what the new arguments should be for QEMU.  
Maybe we should think about ways to delay the launching of QEMU (or at 
least the argument parsing) until after migration has completed.

Then, we have the ability to transfer the command line arguments 
(through a separate channel) similar to how ssh migration used to work.

We're really close to a machine config for pc though, I'm not sure that 
that's not a quicker solution.

Regards,

Anthony Liguori

Re: [Qemu-devel] sending pci information over the wire

[Gerd Hoffmann]

On 06/18/09 22:38, Anthony Liguori wrote:
> Glauber Costa wrote:
>> Sure.
>>
>> Question is, whether or not there is any value in doing this now.

Prototyping, playing with it, figure any roadblocks: yes.
Merging: no.

I think we should go for the "big" solution, i.e. machine-config for 
*everything*, not just PCI.  The move from the current model to the 
machine-config based one will be difficuilt enougth, we should not make 
it even harder to support by merging a half-done interim solution.

> We're really close to a machine config for pc though, I'm not sure that
> that's not a quicker solution.

Well, we are not *that* close.

Paul Brooks fdt works fine for some simple embedded boards.  To make it 
fly with pc a noticable amount of work is needed (I'm working on that 
;).  Partly that is a simple conversion of drivers to qdev.  Partly that 
involves reorganization and cleanup of drivers and subsystems and the 
dependencies they have.  Which certainly is a good thing because it 
cleans up the code.  But it needs to be done before machine-config can 
work for pc.

Speaking of qdev:  You seem to be reluctant committing qdev patches 
(except obviously correct one-liners), which is quite understandable as 
qdev is done by Paul.  Problem is that there are lots of qdev patches on 
the list from me and others which are largely ignored.  Some of my 
patches got committed.  Fine.  Some got comments from Paul.  Fine too, 
so I know why they are not committed and can work on the issues.  Most 
of my patches are ignored.  Hmm.  How to go forward here?  I suspect 
reposting over and over wouldn't help here.

Likewise devtree:  Paul posted the patches, got a number of reviews 
comments.  No discussion of the concerns happened though.  There is also 
no activity in Pauls git tree.  There are some quite fundamental design 
issues which need to be decided:

   (1) How will we handle fdt and hotplug?  We can keep the qdev tree
       and the fdt in sync all the time.  Or we generate a fresh fdt
       from the qdev tree when needed (i.e. for savevm, migration and
       maybe the ppc kernel).
   (2) Depending on (1): How will we handle attributes?  Read/write
       directly from/to the fdt?  Keep them in qdev data structures?

I really like to see some progress here, otherwise 2017 wouldn't be just 
a running gag.

cheers,
   Gerd

Re: [Qemu-devel] sending pci information over the wire

[Anthony Liguori]

Gerd Hoffmann wrote:
> On 06/18/09 22:38, Anthony Liguori wrote:
>> Glauber Costa wrote:
>>> Sure.
>>>
>>> Question is, whether or not there is any value in doing this now.
>
> Prototyping, playing with it, figure any roadblocks: yes.
> Merging: no.
>
> I think we should go for the "big" solution, i.e. machine-config for 
> *everything*, not just PCI.  The move from the current model to the 
> machine-config based one will be difficuilt enougth, we should not 
> make it even harder to support by merging a half-done interim solution.
>
>> We're really close to a machine config for pc though, I'm not sure that
>> that's not a quicker solution.
>
> Well, we are not *that* close.
>
> Paul Brooks fdt works fine for some simple embedded boards.  To make 
> it fly with pc a noticable amount of work is needed (I'm working on 
> that ;).  Partly that is a simple conversion of drivers to qdev.  
> Partly that involves reorganization and cleanup of drivers and 
> subsystems and the dependencies they have.  Which certainly is a good 
> thing because it cleans up the code.  But it needs to be done before 
> machine-config can work for pc.

I've been considering queue all patches on the list to keep track of 
which ones don't get applied.  The problem is, something like your qdev 
series really needs to be applied quickly because it conflicts with 
quite a lot of other patches.

My plan for your qdev series is to give a little more time for feedback, 
then ask you to rebase/resubmit so they can be applied.  Give it a 
couple more days to give people a chance to review.

> Likewise devtree:  Paul posted the patches, got a number of reviews 
> comments.  No discussion of the concerns happened though.  There is 
> also no activity in Pauls git tree.

I think we're all just waiting for Paul to commit something.

>   There are some quite fundamental design issues which need to be 
> decided:
>
>   (1) How will we handle fdt and hotplug?  We can keep the qdev tree
>       and the fdt in sync all the time.  Or we generate a fresh fdt
>       from the qdev tree when needed (i.e. for savevm, migration and
>       maybe the ppc kernel).

I'd lean toward the later.

>   (2) Depending on (1): How will we handle attributes?  Read/write
>       directly from/to the fdt?  Keep them in qdev data structures?

I think qdev is the master tree and fdt just becomes a conversion format.

> I really like to see some progress here, otherwise 2017 wouldn't be 
> just a running gag.

I agree.  Not every patch is reasonable to commit in 24 hours though.  
Some require giving people appropriate time to review.  I think 
something we could do better is avoiding losing track of patches 
though.  I'm working on that.

Regards,

Anthony Liguori

> cheers,
>   Gerd

Re: [Qemu-devel] sending pci information over the wire

[Gerd Hoffmann]

On 06/19/09 15:10, Anthony Liguori wrote:
> My plan for your qdev series is to give a little more time for feedback,
> then ask you to rebase/resubmit so they can be applied. Give it a couple
> more days to give people a chance to review.

Good.

>> Likewise devtree: Paul posted the patches, got a number of reviews
>> comments. No discussion of the concerns happened though. There is also
>> no activity in Pauls git tree.
>
> I think we're all just waiting for Paul to commit something.

Looking forward to that.

>> (1) How will we handle fdt and hotplug? We can keep the qdev tree
>> and the fdt in sync all the time. Or we generate a fresh fdt
>> from the qdev tree when needed (i.e. for savevm, migration and
>> maybe the ppc kernel).
>
> I'd lean toward the later.

It is probably easier to handle that way.

>> I really like to see some progress here, otherwise 2017 wouldn't be
>> just a running gag.
>
> I agree. Not every patch is reasonable to commit in 24 hours though.

Sure.  Especially large, non-trivial patches need some more time to 
review.  The oldest patches of the qdev series are out for more than a 
week now though.

The first one ("update pci device registration") is quite invasive and 
probably conflicts with everybody doing pci-related qdev work.  It also 
fixes the fundamental issue that the code in master right now can't 
handle pci devices which need config space access handlers.

> I think something
> we could do better is avoiding losing track of patches though. I'm
> working on that.

Related issues:  Looks like the qemu-commits list is lossy.  I havn't 
seen messages for my two most recent vnc patches.  I figured while 
rebasing the vnc patches because git surprised me with "nothing to do". 
  Also it seems for merges only a message for the merge commit goes to 
the list, not for the individual patches.

cheers,
   Gerd

Re: [Qemu-devel] sending pci information over the wire

[Paul Brook]

>   Also it seems for merges only a message for the merge commit goes to
> the list, not for the individual patches.

This depends how the merge is done. A few of the recent merges did not rebase 
the changes before comitting. This leads to nonlinear revision history, as can 
easily be seen with e.g. .gitk e510e05b

Paul

Re: [Qemu-devel] sending pci information over the wire

[Anthony Liguori]

Gerd Hoffmann wrote:
>> I think something
>> we could do better is avoiding losing track of patches though. I'm
>> working on that.
>
> Related issues:  Looks like the qemu-commits list is lossy.  I havn't 
> seen messages for my two most recent vnc patches.  I figured while 
> rebasing the vnc patches because git surprised me with "nothing to 
> do".  Also it seems for merges only a message for the merge commit 
> goes to the list, not for the individual patches.

With respect to merge commits, it wouldn't surprise me.

There is a certain amount of lossy-ness to qemu-commits because our 
internal SMTP servers frequently like to go down.  It's a git commit 
hook so having a mechanism to retry later would be overly complicated.

Regards,

Anthony Liguori

> cheers,
>   Gerd

Re: [Qemu-devel] sending pci information over the wire

[Gerd Hoffmann]

On 06/19/09 17:16, Anthony Liguori wrote:
> There is a certain amount of lossy-ness to qemu-commits because our
> internal SMTP servers frequently like to go down. It's a git commit hook
> so having a mechanism to retry later would be overly complicated.

Just an idea:  Punt the issue to an MTA?

Install local postfix, setup internal smtp as smarthost for delivery, 
point the commit hook to localhost.  postfix should take care that 
nothing gets lost then (unless the outage is very long ...).

cheers,
   Gerd

Re: [Qemu-devel] sending pci information over the wire

[Anthony Liguori]

Gerd Hoffmann wrote:
> On 06/19/09 17:16, Anthony Liguori wrote:
>> There is a certain amount of lossy-ness to qemu-commits because our
>> internal SMTP servers frequently like to go down. It's a git commit hook
>> so having a mechanism to retry later would be overly complicated.
>
> Just an idea:  Punt the issue to an MTA?
>
> Install local postfix, setup internal smtp as smarthost for delivery, 
> point the commit hook to localhost.  postfix should take care that 
> nothing gets lost then (unless the outage is very long ...).

It'll migrate to qemu.org very soon.  I just need to spend some time 
setting up the server.  One qemu.org, it'll run a local mta.

Regards,

Anthony Liguori

> cheers,
>   Gerd

Re: [Qemu-devel] sending pci information over the wire

[Avi Kivity]

On 06/18/2009 11:03 PM, Anthony Liguori wrote:
> Once we have a machine config (and please, let's avoid the 2017 jokes 
> please), I imagine that the machine config will be transferred as a 
> non-live section.

What exactly do you propose to send?  the machine config file, or an 
internal representation thereof?

We already have the internal representation in the form of the qdev tree.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.