qemu-kvm.git now live

qemu-kvm.git now live

[Avi Kivity]

After a lengthy testing phase, qemu-kvm.git has replaced 
kvm-userspace.git as the source repository for kvm userspace development.

Differences from kvm-userspace.git are as follows:
- everything under qemu/ has been moved to the top-level directory
- everything not under qemu/ has been moved under a new top-level
directory, kvm/
- all qemu subversion commits have been rewritten to be compatible with
what will become the master qemu git repository
- all branches and tags have been converted
- qemu-kvm.git builds standalone (does not require kvm.git)

The repository is compatible with upstream qemu.git; merges and 
cherry-picking will work fine in either direction

Still missing:
- I have not tested powerpc or ia64.  Patches welcome!
- testsuite (kvm/user/ directory) does not build

git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git


-- 
error compiling committee.c: too many arguments to function

Re: qemu-kvm.git now live

[Jan Kiszka]

Avi Kivity wrote:
> After a lengthy testing phase, qemu-kvm.git has replaced
> kvm-userspace.git as the source repository for kvm userspace development.
> 
> Differences from kvm-userspace.git are as follows:
> - everything under qemu/ has been moved to the top-level directory
> - everything not under qemu/ has been moved under a new top-level
> directory, kvm/
> - all qemu subversion commits have been rewritten to be compatible with
> what will become the master qemu git repository
> - all branches and tags have been converted
> - qemu-kvm.git builds standalone (does not require kvm.git)
> 
> The repository is compatible with upstream qemu.git; merges and
> cherry-picking will work fine in either direction
> 
> Still missing:
> - I have not tested powerpc or ia64.  Patches welcome!
> - testsuite (kvm/user/ directory) does not build

- proper guest memory setup

During merge with upstream, the call to qemu_alloc_physram dropped under
the table (due to removal of phys_ram_base from qemu). And that means we
no longer call kvm_setup_guest_memory (fork will cause troubles, deja
vu...) and -mempath is broken. Sorry, I'm not deep enough into all parts
to provide a quick-fix.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: qemu-kvm.git now live

[Avi Kivity]

Jan Kiszka wrote:
> Avi Kivity wrote:
>   
>> After a lengthy testing phase, qemu-kvm.git has replaced
>> kvm-userspace.git as the source repository for kvm userspace development.
>>
>> Differences from kvm-userspace.git are as follows:
>> - everything under qemu/ has been moved to the top-level directory
>> - everything not under qemu/ has been moved under a new top-level
>> directory, kvm/
>> - all qemu subversion commits have been rewritten to be compatible with
>> what will become the master qemu git repository
>> - all branches and tags have been converted
>> - qemu-kvm.git builds standalone (does not require kvm.git)
>>
>> The repository is compatible with upstream qemu.git; merges and
>> cherry-picking will work fine in either direction
>>
>> Still missing:
>> - I have not tested powerpc or ia64.  Patches welcome!
>> - testsuite (kvm/user/ directory) does not build
>>     
>
> - proper guest memory setup
>
> During merge with upstream, the call to qemu_alloc_physram dropped under
> the table (due to removal of phys_ram_base from qemu). And that means we
> no longer call kvm_setup_guest_memory (fork will cause troubles, deja
> vu...) and -mempath is broken. Sorry, I'm not deep enough into all parts
> to provide a quick-fix.
>   

That's unrelated to the repository conversion.

I'm unhappy with both qemu.git and qemu-kvm.git memory slot management; 
qemu-kvm.git is clumsy, and qemu.git is too simplistic (for example, it 
ignores the fact that dirty logging is a global resource with multiple 
users).

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: qemu-kvm.git now live

[Jan Kiszka]

Avi Kivity wrote:
> Jan Kiszka wrote:
>> Avi Kivity wrote:
>>  
>>> After a lengthy testing phase, qemu-kvm.git has replaced
>>> kvm-userspace.git as the source repository for kvm userspace
>>> development.
>>>
>>> Differences from kvm-userspace.git are as follows:
>>> - everything under qemu/ has been moved to the top-level directory
>>> - everything not under qemu/ has been moved under a new top-level
>>> directory, kvm/
>>> - all qemu subversion commits have been rewritten to be compatible with
>>> what will become the master qemu git repository
>>> - all branches and tags have been converted
>>> - qemu-kvm.git builds standalone (does not require kvm.git)
>>>
>>> The repository is compatible with upstream qemu.git; merges and
>>> cherry-picking will work fine in either direction
>>>
>>> Still missing:
>>> - I have not tested powerpc or ia64.  Patches welcome!
>>> - testsuite (kvm/user/ directory) does not build
>>>     
>>
>> - proper guest memory setup
>>
>> During merge with upstream, the call to qemu_alloc_physram dropped under
>> the table (due to removal of phys_ram_base from qemu). And that means we
>> no longer call kvm_setup_guest_memory (fork will cause troubles, deja
>> vu...) and -mempath is broken. Sorry, I'm not deep enough into all parts
>> to provide a quick-fix.
>>   
> 
> That's unrelated to the repository conversion.

Then it's a merge artifact, but it's a problematic regression compared
to still-working kvm-userspace.

> 
> I'm unhappy with both qemu.git and qemu-kvm.git memory slot management;
> qemu-kvm.git is clumsy, and qemu.git is too simplistic (for example, it
> ignores the fact that dirty logging is a global resource with multiple
> users).

Don't get your point yet. Can you name a concrete scenario that is
problematic in upstream? I'd really like to get slot management right
there before we drop the old version of qemu-kvm.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: qemu-kvm.git now live

[Avi Kivity]

Jan Kiszka wrote:
>> I'm unhappy with both qemu.git and qemu-kvm.git memory slot management;
>> qemu-kvm.git is clumsy, and qemu.git is too simplistic (for example, it
>> ignores the fact that dirty logging is a global resource with multiple
>> users).
>>     
>
> Don't get your point yet. Can you name a concrete scenario that is
> problematic in upstream? I'd really like to get slot management right
> there before we drop the old version of qemu-kvm.
>   

If migration disables dirty memory logging, it must keep the vga logging 
enabled, and vice versa.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: qemu-kvm.git now live

[Jan Kiszka]

Avi Kivity wrote:
> Jan Kiszka wrote:
>>> I'm unhappy with both qemu.git and qemu-kvm.git memory slot management;
>>> qemu-kvm.git is clumsy, and qemu.git is too simplistic (for example, it
>>> ignores the fact that dirty logging is a global resource with multiple
>>> users).
>>>     
>>
>> Don't get your point yet. Can you name a concrete scenario that is
>> problematic in upstream? I'd really like to get slot management right
>> there before we drop the old version of qemu-kvm.
>>   
> 
> If migration disables dirty memory logging, it must keep the vga logging
> enabled, and vice versa.

So we need some notifier callbacks on slot changes so that all users can
re-enable dirty logging after the update as required. Where/how does the
migration code disable dirty logging?

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: qemu-kvm.git now live

[Avi Kivity]

Jan Kiszka wrote:
>> If migration disables dirty memory logging, it must keep the vga logging
>> enabled, and vice versa.
>>     
>
> So we need some notifier callbacks on slot changes so that all users can
> re-enable dirty logging after the update as required. 

Simpler to do reference counting.  When a user enables logging for a 
memory range, the refcount for all slots containing that range gets 
bumped.  When a user enables logging for all of memory, a global 
refcount is bumped.  For a given slot, dirty logging is enabled if 
either the slot logging refcount or the global logging refcount is nonzero.

That's sort of what's implemented in qemu-kvm.git.  In qemu.git vga 
logging does not get disabled, which is really broken.  It prevents 
optimizations like disabling logging when the screen is not displayed to 
a human.

> Where/how does the
> migration code disable dirty logging?
>   

Should be phase 3 of ram_save_live().

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: qemu-kvm.git now live

[Jan Kiszka]

Avi Kivity wrote:
> Jan Kiszka wrote:
>>> If migration disables dirty memory logging, it must keep the vga logging
>>> enabled, and vice versa.
>>>     
>>
>> So we need some notifier callbacks on slot changes so that all users can
>> re-enable dirty logging after the update as required. 
> 
> Simpler to do reference counting.  When a user enables logging for a
> memory range, the refcount for all slots containing that range gets
> bumped.  When a user enables logging for all of memory, a global
> refcount is bumped.  For a given slot, dirty logging is enabled if
> either the slot logging refcount or the global logging refcount is nonzero.

OK, but then we just need to check for that global flag after
registering/modifying slots and enable logging unconditionally. That's
indeed a simple add-on.

> 
> That's sort of what's implemented in qemu-kvm.git.  In qemu.git vga
> logging does not get disabled, which is really broken.  It prevents
> optimizations like disabling logging when the screen is not displayed to
> a human.

Is there a channel that tells vga "nothing will be displayed"? I may
have missed it while removing all those disable-logging-as-it-may-
confuse-slot-management hooks.

> 
>> Where/how does the
>> migration code disable dirty logging?
>>   
> 
> Should be phase 3 of ram_save_live().

But only in qemu-kvm. What is the plan about pushing it upstream? Then
we could discuss how to extend the exiting support best.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: qemu-kvm.git now live

[Avi Kivity]

Jan Kiszka wrote:
>> That's sort of what's implemented in qemu-kvm.git.  In qemu.git vga
>> logging does not get disabled, which is really broken.  It prevents
>> optimizations like disabling logging when the screen is not displayed to
>> a human.
>>     
>
> Is there a channel that tells vga "nothing will be displayed"? I may
> have missed it while removing all those disable-logging-as-it-may-
> confuse-slot-management hooks.
>   

I think currently qemu simply stops calling vga_draw_graphic().  This 
makes sense for tcg since it needs to track dirty memory regardless (so 
it can invalidate TBs).  But for kvm we'll want to add an explicit channel.

Note that it isn't likely to make a huge difference: if you don't 
actively read-and-reset the dirty bitmap, kvm will keep the shadow ptes 
with write permission and you won't see any performance hit.  The only 
difference is whether large pages can be used or not.

>>> Where/how does the
>>> migration code disable dirty logging?
>>>   
>>>       
>> Should be phase 3 of ram_save_live().
>>     
>
> But only in qemu-kvm. What is the plan about pushing it upstream? Then
> we could discuss how to extend the exiting support best.
>   

Pushing things upstream is quite difficult because of the very different 
infrastructure.  It's unfortunate that upstream rewrote everything 
instead of changing things incrementally.  Rewrites are almost always a 
mistake since they throw away accumulated knowledge.

-- 
error compiling committee.c: too many arguments to function

Re: qemu-kvm.git now live

[Jan Kiszka]

Avi Kivity wrote:
>>>> Where/how does the
>>>> migration code disable dirty logging?
>>>>         
>>> Should be phase 3 of ram_save_live().
>>>     
>>
>> But only in qemu-kvm. What is the plan about pushing it upstream? Then
>> we could discuss how to extend the exiting support best.
>>   
> 
> Pushing things upstream is quite difficult because of the very different
> infrastructure.

Isn't the midterm goal to get rid of most of these differences (namely
libkvm)?

> It's unfortunate that upstream rewrote everything
> instead of changing things incrementally.  Rewrites are almost always a
> mistake since they throw away accumulated knowledge.

I disagree, at least in this particular case. Upstream already diverged
from qemu-kvm, and the latter provided no comparable alternative for
slot management and dirty logging. And I still don't see that we lost
anything that could not easily be re-integrated into upstream (ie.
global dirty logging), finally leading to a cleaner and more complete
result.

So, what bits are missing to make KVM migration work in upstream?

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: qemu-kvm.git now live

[Avi Kivity]

Jan Kiszka wrote:
> Avi Kivity wrote:
>   
>>>>> Where/how does the
>>>>> migration code disable dirty logging?
>>>>>         
>>>>>           
>>>> Should be phase 3 of ram_save_live().
>>>>     
>>>>         
>>> But only in qemu-kvm. What is the plan about pushing it upstream? Then
>>> we could discuss how to extend the exiting support best.
>>>   
>>>       
>> Pushing things upstream is quite difficult because of the very different
>> infrastructure.
>>     
>
> Isn't the midterm goal to get rid of most of these differences (namely
> libkvm)?
>   

Yes, but not by removing existing functionality.

>   
>> It's unfortunate that upstream rewrote everything
>> instead of changing things incrementally.  Rewrites are almost always a
>> mistake since they throw away accumulated knowledge.
>>     
>
> I disagree, at least in this particular case. Upstream already diverged
> from qemu-kvm, and the latter provided no comparable alternative for
> slot management and dirty logging. And I still don't see that we lost
> anything that could not easily be re-integrated into upstream (ie.
> global dirty logging), finally leading to a cleaner and more complete
> result.
>   

It could have been done differently, by morphing the existing support 
into something mergable, and merging that.  In this way, we'd ensure no 
needed functionality is lost.

As is, we're adding something simple, then discovering it's 
insufficient.  We're throwing away information, that's not a good way to 
make progress.

> So, what bits are missing to make KVM migration work in upstream?
>   

I don't know of anything beyond dirty logging.


-- 
error compiling committee.c: too many arguments to function

Re: qemu-kvm.git now live

[Jan Kiszka]

Avi Kivity wrote:
> Jan Kiszka wrote:
>> Avi Kivity wrote:
>>  
>>>>>> Where/how does the
>>>>>> migration code disable dirty logging?
>>>>>>                   
>>>>> Should be phase 3 of ram_save_live().
>>>>>             
>>>> But only in qemu-kvm. What is the plan about pushing it upstream? Then
>>>> we could discuss how to extend the exiting support best.
>>>>         
>>> Pushing things upstream is quite difficult because of the very different
>>> infrastructure.
>>>     
>>
>> Isn't the midterm goal to get rid of most of these differences (namely
>> libkvm)?
>>   
> 
> Yes, but not by removing existing functionality.

No one said this.

> 
>>  
>>> It's unfortunate that upstream rewrote everything
>>> instead of changing things incrementally.  Rewrites are almost always a
>>> mistake since they throw away accumulated knowledge.
>>>     
>>
>> I disagree, at least in this particular case. Upstream already diverged
>> from qemu-kvm, and the latter provided no comparable alternative for
>> slot management and dirty logging. And I still don't see that we lost
>> anything that could not easily be re-integrated into upstream (ie.
>> global dirty logging), finally leading to a cleaner and more complete
>> result.
>>   
> 
> It could have been done differently, by morphing the existing support
> into something mergable, and merging that.  In this way, we'd ensure no
> needed functionality is lost.

The existing support lacked features upstream already had and instead
required additional hacks to make qemu-kvm work.

> 
> As is, we're adding something simple, then discovering it's
> insufficient.  We're throwing away information, that's not a good way to
> make progress.

I doubt this applies here.

> 
>> So, what bits are missing to make KVM migration work in upstream?
>>   
> 
> I don't know of anything beyond dirty logging.
> 

OK, then I will pick this up and have a look at something comparable to
cpu_physical_memory_set_dirty_tracking() for upstream.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: qemu-kvm.git now live

[Avi Kivity]

Jan Kiszka wrote:
>>>>>         
>>>>>           
>>>> Pushing things upstream is quite difficult because of the very different
>>>> infrastructure.
>>>>     
>>>>         
>>> Isn't the midterm goal to get rid of most of these differences (namely
>>> libkvm)?
>>>   
>>>       
>> Yes, but not by removing existing functionality.
>>     
>
> No one said this.
>   

I'm sure no one meant it either, but that is what's happening.

This is the flow:
* qemu.git has very limited kvm support
* more kvm support is added to qemu.git
* because it's a rewrite, not all the knowledge that has gone into 
qemu-kvm.git is added
* when I merge qemu.git into qemu-kvm.git, the two implementation 
methods clash, and things break

This has happened on most clean rewrites I've seen.  The new code is 
clean, but the old code works better.  It's much better to morph the old 
code into shape (though more time consuming and a lot less fun).

>> As is, we're adding something simple, then discovering it's
>> insufficient.  We're throwing away information, that's not a good way to
>> make progress.
>>     
>
> I doubt this applies here.
>   

In fact it's happened.  qemu-kvm.git knows that the dirty logging flag 
is a shared resource.  qemu-kvm.git also handles older kernels (though I 
think that was intentional).

-- 
error compiling committee.c: too many arguments to function