[Qemu-devel] KVM call agenda for Mar 16

[Qemu-devel] KVM call agenda for Mar 16

[Chris Wright]

Please send in any agenda items you are interested in covering.

thanks,
-chris

[Qemu-devel] Re: KVM call agenda for Mar 16

[Juan Quintela]

Chris Wright <chrisw@redhat.com> wrote:
> Please send in any agenda items you are interested in covering.

Migration:
- flexible migration:  I hope to sent an RFC patch on time for the
  call.  idea is to use subsections.

- callbacks.  block migration introduced several callbacks:
  * cancel()
  * get_status()
  * release()
  in spice we need now another to callbacks: on_start() and on_end().
   * on_start(): tells spice that migration has started (it will then
     manage certificates, passwords, ... itself)
   * on_end(): it is called when migration ends.  spice use it to
     transparently connect to the new host and user don't have to "reconnect"

- what to do on migration error:
  - target side:  libvirt folks want the program to print a message if
    it fails.  Current code spent 100% cpu time doing select on a closed
    fd.  (patches already on the list to make it wait without using
    cpu).
  - source side: current behaviour if migration fails is to stop the
    vm.  We have requests to make it continue (remember that this is
    live migration).  what to do?  adding a paramenter like the block
    layer:
      migration_error=[stop|continue]
    any better ideas.

- block migration: it added the set_params() callback, investigating
  why.  I think that it can be done with inside block_save_live(), but I
  haven't investigated it fully yet.

Later, Juan.



> thanks,
> -chris

[Qemu-devel] Re: KVM call agenda for Mar 16

[Daniel P. Berrange]

On Tue, Mar 16, 2010 at 10:18:03AM +0100, Juan Quintela wrote:
> Chris Wright <chrisw@redhat.com> wrote:
> > Please send in any agenda items you are interested in covering.
> 
> Migration:
> - flexible migration:  I hope to sent an RFC patch on time for the
>   call.  idea is to use subsections.
> 
> - callbacks.  block migration introduced several callbacks:
>   * cancel()
>   * get_status()
>   * release()
>   in spice we need now another to callbacks: on_start() and on_end().
>    * on_start(): tells spice that migration has started (it will then
>      manage certificates, passwords, ... itself)
>    * on_end(): it is called when migration ends.  spice use it to
>      transparently connect to the new host and user don't have to "reconnect"
> 
> - what to do on migration error:
>   - target side:  libvirt folks want the program to print a message if
>     it fails.  Current code spent 100% cpu time doing select on a closed
>     fd.  (patches already on the list to make it wait without using
>     cpu).

No, that is not correct. We want QEMU to exit when incoming migration
fails. Printing to stderr is just something that will end up in the
logs for admin to further diagnose the problem if required. There is 
nothing to be gained by leaving QEMU running, and everything to loose
since the failed migration may have left it in a dangerous state from 
which you do not want to attempt incoming migration again.

If we really want to leave it running when migration fails, then we're
going to have to add yet another QMP event to inform libvirt when
migration has finished/failed, and/or make 'query_migrate' work on
the destination too.

>   - source side: current behaviour if migration fails is to stop the
>     vm.  We have requests to make it continue (remember that this is
>     live migration).  what to do?  adding a paramenter like the block
>     layer:
>       migration_error=[stop|continue]
>     any better ideas.

A parameter to the 'migrate' monitor command would be the logical
place if we needed this configurable.

Incidentally I have a feeling we might need to introduce a migration
event in QMP. Currently libvirt polls on the 'query_migrate' command
to get the ongoing migration status. This means there can be a delay
in detecting completion as long as the polling interval - for this
reason we just dropped libvirt's polling time from 1/2 sec to 50ms
to ensure prompt detection.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: KVM call agenda for Mar 16

[Avi Kivity]

On 03/16/2010 11:29 AM, Daniel P. Berrange wrote:
> On Tue, Mar 16, 2010 at 10:18:03AM +0100, Juan Quintela wrote:
>    
>> Chris Wright<chrisw@redhat.com>  wrote:
>>      
>>> Please send in any agenda items you are interested in covering.
>>>        
>> Migration:
>> - flexible migration:  I hope to sent an RFC patch on time for the
>>    call.  idea is to use subsections.
>>
>> - callbacks.  block migration introduced several callbacks:
>>    * cancel()
>>    * get_status()
>>    * release()
>>    in spice we need now another to callbacks: on_start() and on_end().
>>     * on_start(): tells spice that migration has started (it will then
>>       manage certificates, passwords, ... itself)
>>     * on_end(): it is called when migration ends.  spice use it to
>>       transparently connect to the new host and user don't have to "reconnect"
>>
>> - what to do on migration error:
>>    - target side:  libvirt folks want the program to print a message if
>>      it fails.  Current code spent 100% cpu time doing select on a closed
>>      fd.  (patches already on the list to make it wait without using
>>      cpu).
>>      
> No, that is not correct. We want QEMU to exit when incoming migration
> fails. Printing to stderr is just something that will end up in the
> logs for admin to further diagnose the problem if required. There is
> nothing to be gained by leaving QEMU running, and everything to loose
> since the failed migration may have left it in a dangerous state from
> which you do not want to attempt incoming migration again.
>
> If we really want to leave it running when migration fails, then we're
> going to have to add yet another QMP event to inform libvirt when
> migration has finished/failed, and/or make 'query_migrate' work on
> the destination too.
>    

A qmp event seems the logical thing to do?  Exiting can happen for many 
reasons, a qmp event is unambiguous.

>>    - source side: current behaviour if migration fails is to stop the
>>      vm.  We have requests to make it continue (remember that this is
>>      live migration).  what to do?  adding a paramenter like the block
>>      layer:
>>        migration_error=[stop|continue]
>>      any better ideas.
>>      
> A parameter to the 'migrate' monitor command would be the logical
> place if we needed this configurable.
>
> Incidentally I have a feeling we might need to introduce a migration
> event in QMP. Currently libvirt polls on the 'query_migrate' command
> to get the ongoing migration status. This means there can be a delay
> in detecting completion as long as the polling interval - for this
> reason we just dropped libvirt's polling time from 1/2 sec to 50ms
> to ensure prompt detection.
>    

Whenever you implement a polling loop, can you send an event to qemu-devel@?

Polling loops are an indication that something is wrong.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: KVM call agenda for Mar 16

[Daniel P. Berrange]

On Tue, Mar 16, 2010 at 11:43:48AM +0200, Avi Kivity wrote:
> On 03/16/2010 11:29 AM, Daniel P. Berrange wrote:
> >On Tue, Mar 16, 2010 at 10:18:03AM +0100, Juan Quintela wrote:
> >   
> >>Chris Wright<chrisw@redhat.com>  wrote:
> >>     
> >>>Please send in any agenda items you are interested in covering.
> >>>       
> >>Migration:
> >>- flexible migration:  I hope to sent an RFC patch on time for the
> >>   call.  idea is to use subsections.
> >>
> >>- callbacks.  block migration introduced several callbacks:
> >>   * cancel()
> >>   * get_status()
> >>   * release()
> >>   in spice we need now another to callbacks: on_start() and on_end().
> >>    * on_start(): tells spice that migration has started (it will then
> >>      manage certificates, passwords, ... itself)
> >>    * on_end(): it is called when migration ends.  spice use it to
> >>      transparently connect to the new host and user don't have to 
> >>      "reconnect"
> >>
> >>- what to do on migration error:
> >>   - target side:  libvirt folks want the program to print a message if
> >>     it fails.  Current code spent 100% cpu time doing select on a closed
> >>     fd.  (patches already on the list to make it wait without using
> >>     cpu).
> >>     
> >No, that is not correct. We want QEMU to exit when incoming migration
> >fails. Printing to stderr is just something that will end up in the
> >logs for admin to further diagnose the problem if required. There is
> >nothing to be gained by leaving QEMU running, and everything to loose
> >since the failed migration may have left it in a dangerous state from
> >which you do not want to attempt incoming migration again.
> >
> >If we really want to leave it running when migration fails, then we're
> >going to have to add yet another QMP event to inform libvirt when
> >migration has finished/failed, and/or make 'query_migrate' work on
> >the destination too.
> >   
> 
> A qmp event seems the logical thing to do?  Exiting can happen for many 
> reasons, a qmp event is unambiguous.

Yes, for the QEMU upstream adding an event is more flexible. I had
originally suggested exiting in the context of the Fedora bug report
which was for QEMU 0.10.x which has no events capability.

> >
> >Incidentally I have a feeling we might need to introduce a migration
> >event in QMP. Currently libvirt polls on the 'query_migrate' command
> >to get the ongoing migration status. This means there can be a delay
> >in detecting completion as long as the polling interval - for this
> >reason we just dropped libvirt's polling time from 1/2 sec to 50ms
> >to ensure prompt detection.
> >   
> 
> Whenever you implement a polling loop, can you send an event to qemu-devel@?

Yep, sure thing. This is the only polling loop that isn't related to I/O
stats collection.

> 
> Polling loops are an indication that something is wrong.

Except when people suggest they are the right answer, qcow high 
watermark ;-P

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: KVM call agenda for Mar 16

[Avi Kivity]

On 03/16/2010 12:31 PM, Daniel P. Berrange wrote:
>> Polling loops are an indication that something is wrong.
>>      
> Except when people suggest they are the right answer, qcow high
> watermark ;-P
>    

I liked Anthony's suggestion of an lvm2 block format driver.  No polling.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: KVM call agenda for Mar 16

[Christoph Hellwig]

On Tue, Mar 16, 2010 at 12:38:02PM +0200, Avi Kivity wrote:
> On 03/16/2010 12:31 PM, Daniel P. Berrange wrote:
>>> Polling loops are an indication that something is wrong.
>>>      
>> Except when people suggest they are the right answer, qcow high
>> watermark ;-P
>>    
>
> I liked Anthony's suggestion of an lvm2 block format driver.  No polling.

I have done some work on linking the new lvm library to qemu to control
snapshotting.  But introducing a whole new block format seems like a lot
of duplication to me.

[Qemu-devel] Re: KVM call agenda for Mar 16

[Daniel P. Berrange]

On Tue, Mar 16, 2010 at 12:38:02PM +0200, Avi Kivity wrote:
> On 03/16/2010 12:31 PM, Daniel P. Berrange wrote:
> >>Polling loops are an indication that something is wrong.
> >>     
> >Except when people suggest they are the right answer, qcow high
> >watermark ;-P
> >   
> 
> I liked Anthony's suggestion of an lvm2 block format driver.  No polling.

Doesn't that require giving QEMU privileges to perform LVM operations which
implies QEMU having CAP_SYS_ADMIN  ?


Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: KVM call agenda for Mar 16

[Avi Kivity]

On 03/16/2010 12:45 PM, Daniel P. Berrange wrote:
> On Tue, Mar 16, 2010 at 12:38:02PM +0200, Avi Kivity wrote:
>    
>> On 03/16/2010 12:31 PM, Daniel P. Berrange wrote:
>>      
>>>> Polling loops are an indication that something is wrong.
>>>>
>>>>          
>>> Except when people suggest they are the right answer, qcow high
>>> watermark ;-P
>>>
>>>        
>> I liked Anthony's suggestion of an lvm2 block format driver.  No polling.
>>      
> Doesn't that require giving QEMU privileges to perform LVM operations which
> implies QEMU having CAP_SYS_ADMIN  ?
>    

Ouch.  I expect fd permissions on the volume are insufficient, and fd 
permissions on the group are excessive.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: KVM call agenda for Mar 16

[Anthony Liguori]

On 03/16/2010 05:45 AM, Daniel P. Berrange wrote:
> On Tue, Mar 16, 2010 at 12:38:02PM +0200, Avi Kivity wrote:
>    
>> On 03/16/2010 12:31 PM, Daniel P. Berrange wrote:
>>      
>>>> Polling loops are an indication that something is wrong.
>>>>
>>>>          
>>> Except when people suggest they are the right answer, qcow high
>>> watermark ;-P
>>>
>>>        
>> I liked Anthony's suggestion of an lvm2 block format driver.  No polling.
>>      
> Doesn't that require giving QEMU privileges to perform LVM operations which
> implies QEMU having CAP_SYS_ADMIN  ?
>    

If QEMU is able to resize an LVM partition, it needs to carry privileges.

I'm not sure how this can be done safely in a lesser privileged 
environment.  Presumably, you're over committing storage and there's not 
much you can do if the guests exhaust their storage all at once.

Regards,

Anthony Liguori

> Daniel
>    

[Qemu-devel] Re: KVM call agenda for Mar 16

[Daniel P. Berrange]

On Tue, Mar 16, 2010 at 10:05:40AM -0500, Anthony Liguori wrote:
> On 03/16/2010 05:45 AM, Daniel P. Berrange wrote:
> >On Tue, Mar 16, 2010 at 12:38:02PM +0200, Avi Kivity wrote:
> >   
> >>On 03/16/2010 12:31 PM, Daniel P. Berrange wrote:
> >>     
> >>>>Polling loops are an indication that something is wrong.
> >>>>
> >>>>         
> >>>Except when people suggest they are the right answer, qcow high
> >>>watermark ;-P
> >>>
> >>>       
> >>I liked Anthony's suggestion of an lvm2 block format driver.  No polling.
> >>     
> >Doesn't that require giving QEMU privileges to perform LVM operations which
> >implies QEMU having CAP_SYS_ADMIN  ?
> >   
> 
> If QEMU is able to resize an LVM partition, it needs to carry privileges.
> 
> I'm not sure how this can be done safely in a lesser privileged 
> environment.  Presumably, you're over committing storage and there's not 
> much you can do if the guests exhaust their storage all at once.

In the context of the RHEV management application, iSCSI/SCSI Fibre are
providing the raw storage, with LVM VGs on top and the carving LVs for
the guests. In the common case the admin/app would monitor VG usage & LV
rate of increase to ensure extra space was available in the VG ahead of
it being needed. eg if the VG comes close to exhaustion then further LUNS 
can be obtained and added as PVs to the LVM volume group. So you can't
guarentee that a VM won't stop on ENOSPC, but it is very unlikely if the
system is operating correctly.

As an added complication, since cluster-LVM isn't used, all LVM operations
have to be performed on a dedicated/exclusive storage host and then metadata
refreshed/propagated to other hosts running VMs. This last issue implies that 
letting QEMU resize its LV would never be possible, even if it were not for
the permissions problem.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: KVM call agenda for Mar 16

[Anthony Liguori]

On 03/16/2010 10:23 AM, Daniel P. Berrange wrote:
> In the context of the RHEV management application, iSCSI/SCSI Fibre are
> providing the raw storage, with LVM VGs on top and the carving LVs for
> the guests. In the common case the admin/app would monitor VG usage&  LV
> rate of increase to ensure extra space was available in the VG ahead of
> it being needed. eg if the VG comes close to exhaustion then further LUNS
> can be obtained and added as PVs to the LVM volume group. So you can't
> guarentee that a VM won't stop on ENOSPC, but it is very unlikely if the
> system is operating correctly.
>
> As an added complication, since cluster-LVM isn't used, all LVM operations
> have to be performed on a dedicated/exclusive storage host and then metadata
> refreshed/propagated to other hosts running VMs. This last issue implies that
> letting QEMU resize its LV would never be possible, even if it were not for
> the permissions problem.
>    

Sounds like a good argument for polling :-)

Regards,

Anthony Liguori

> Regards,
> Daniel
>    

[Qemu-devel] Re: KVM call agenda for Mar 16

[Daniel P. Berrange]

On Tue, Mar 16, 2010 at 09:29:44AM +0000, Daniel P. Berrange wrote:
> On Tue, Mar 16, 2010 at 10:18:03AM +0100, Juan Quintela wrote:
> > Chris Wright <chrisw@redhat.com> wrote:
> > > Please send in any agenda items you are interested in covering.
> > 
> > Migration:
> > - flexible migration:  I hope to sent an RFC patch on time for the
> >   call.  idea is to use subsections.
> > 
> > - callbacks.  block migration introduced several callbacks:
> >   * cancel()
> >   * get_status()
> >   * release()
> >   in spice we need now another to callbacks: on_start() and on_end().
> >    * on_start(): tells spice that migration has started (it will then
> >      manage certificates, passwords, ... itself)
> >    * on_end(): it is called when migration ends.  spice use it to
> >      transparently connect to the new host and user don't have to "reconnect"
> > 
> > - what to do on migration error:
> >   - target side:  libvirt folks want the program to print a message if
> >     it fails.  Current code spent 100% cpu time doing select on a closed
> >     fd.  (patches already on the list to make it wait without using
> >     cpu).
> 
> No, that is not correct. We want QEMU to exit when incoming migration
> fails. Printing to stderr is just something that will end up in the
> logs for admin to further diagnose the problem if required. There is 
> nothing to be gained by leaving QEMU running, and everything to loose
> since the failed migration may have left it in a dangerous state from 
> which you do not want to attempt incoming migration again.

Sorry, I forgot to include the original BZ report about this problem from
Fedora. In essence, we just truncated the original save state image and
then tried to restore from it to check handling in the event of corrupted
save image.

  https://bugzilla.redhat.com/show_bug.cgi?id=518032


Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|