[Qemu-devel] migrating instances from qemu-kvm to qemu

[Qemu-devel] migrating instances from qemu-kvm to qemu

[Serge E. Hallyn]

Hi,

at https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321
it was found that migrating running vms from a machine with
qemu-kvm to one with qemu, migration fails due to some mismatches.
The first one we usually hit is

Length mismatch: vga.vram: 1000000 in != 800000

while the second one is network card (and I have not gone beyond
that).  The vga one can be handled on the command line by
specifying the -global cirrus-vga.vrammem_mb=8.  However that
doesn't help with a libvirt migration.

I guess this happens at ram_load() unfortunately - is there any
good way that this could be detected at incoming migration time
and the virtual hardware modified as needed for migration to
continue?

thanks,
-serge

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Paolo Bonzini]

Il 19/03/2014 16:42, Serge E. Hallyn ha scritto:
> Hi,
>
> at https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321
> it was found that migrating running vms from a machine with
> qemu-kvm to one with qemu, migration fails due to some mismatches.
> The first one we usually hit is
>
> Length mismatch: vga.vram: 1000000 in != 800000
>
> while the second one is network card (and I have not gone beyond
> that).

This is just a different default.  Make sure you specify the right model 
on the command line.

> The vga one can be handled on the command line by
> specifying the -global cirrus-vga.vrammem_mb=8.  However that
> doesn't help with a libvirt migration.

You can patch Ubuntu's QEMU to detect old machine types (pc-1.2 and 
earlier) and give cirrus-vga 16MB memory by default.  Migration only 
works with versioned machine types (or between same-version QEMU), so 
it's okay to only do it there.

> I guess this happens at ram_load() unfortunately - is there any
> good way that this could be detected at incoming migration time
> and the virtual hardware modified as needed for migration to
> continue?

It cannot, but the destination can be patched to operate correctly for 
the old machine types, on the assumption that migration from the old 
machine types is always from Ubuntu's qemu-kvm.

Paolo

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Serge E. Hallyn]

Quoting Paolo Bonzini (pbonzini@redhat.com):
> Il 19/03/2014 16:42, Serge E. Hallyn ha scritto:
> >Hi,
> >
> >at https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321
> >it was found that migrating running vms from a machine with
> >qemu-kvm to one with qemu, migration fails due to some mismatches.
> >The first one we usually hit is
> >
> >Length mismatch: vga.vram: 1000000 in != 800000
> >
> >while the second one is network card (and I have not gone beyond
> >that).
> 
> This is just a different default.  Make sure you specify the right
> model on the command line.
> 
> >The vga one can be handled on the command line by
> >specifying the -global cirrus-vga.vrammem_mb=8.  However that
> >doesn't help with a libvirt migration.
> 
> You can patch Ubuntu's QEMU to detect old machine types (pc-1.2 and
> earlier) and give cirrus-vga 16MB memory by default.  Migration only
> works with versioned machine types (or between same-version QEMU),
> so it's okay to only do it there.
> 
> >I guess this happens at ram_load() unfortunately - is there any
> >good way that this could be detected at incoming migration time
> >and the virtual hardware modified as needed for migration to
> >continue?
> 
> It cannot, but the destination can be patched to operate correctly
> for the old machine types, on the assumption that migration from the
> old machine types is always from Ubuntu's qemu-kvm.
> 
> Paolo

Ah, thank you, that assumption should be safe and sounds like a
great idea.

thanks,
-serge

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Serge E. Hallyn]

Quoting Serge E. Hallyn (serge@hallyn.com):
> Quoting Paolo Bonzini (pbonzini@redhat.com):
> > Il 19/03/2014 16:42, Serge E. Hallyn ha scritto:
> > >Hi,
> > >
> > >at https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321
> > >it was found that migrating running vms from a machine with
> > >qemu-kvm to one with qemu, migration fails due to some mismatches.
> > >The first one we usually hit is
> > >
> > >Length mismatch: vga.vram: 1000000 in != 800000
> > >
> > >while the second one is network card (and I have not gone beyond
> > >that).
> > 
> > This is just a different default.  Make sure you specify the right
> > model on the command line.
> > 
> > >The vga one can be handled on the command line by
> > >specifying the -global cirrus-vga.vrammem_mb=8.  However that
> > >doesn't help with a libvirt migration.
> > 
> > You can patch Ubuntu's QEMU to detect old machine types (pc-1.2 and
> > earlier) and give cirrus-vga 16MB memory by default.  Migration only
> > works with versioned machine types (or between same-version QEMU),
> > so it's okay to only do it there.
> > 
> > >I guess this happens at ram_load() unfortunately - is there any
> > >good way that this could be detected at incoming migration time
> > >and the virtual hardware modified as needed for migration to
> > >continue?
> > 
> > It cannot, but the destination can be patched to operate correctly
> > for the old machine types, on the assumption that migration from the
> > old machine types is always from Ubuntu's qemu-kvm.
> > 
> > Paolo
> 
> Ah, thank you, that assumption should be safe and sounds like a
> great idea.

Although, some people are using newer qemu with '-M pc-1.0'.  So we'd be
stopping thing from migrating to support the people coming from qemu-kvm.
Hmm.

-serge

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Paolo Bonzini]

Il 19/03/2014 17:16, Serge E. Hallyn ha scritto:
>> >
>> > Ah, thank you, that assumption should be safe and sounds like a
>> > great idea.
> Although, some people are using newer qemu with '-M pc-1.0'.  So we'd be
> stopping thing from migrating to support the people coming from qemu-kvm.
> Hmm.

Yes, you need to document this somehow.

Paolo

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Dr. David Alan Gilbert]

* Serge E. Hallyn (serge@hallyn.com) wrote:

> Although, some people are using newer qemu with '-M pc-1.0'.  So we'd be
> stopping thing from migrating to support the people coming from qemu-kvm.
> Hmm.

What sometimes happens is that you create a VM (lets say with virt-manager)
on an old installation, and it has the XML file for that VM which I think
includes the machine type.   You upgrade your host or copy your VMs over
and so you still use the old machine definition, and hence still have the
-M pc-1.0

Some people have snapshots rather than just disk images taken with particular
versions and they expect those to be reloadable into newer versions.

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Serge E. Hallyn]

Quoting Dr. David Alan Gilbert (dgilbert@redhat.com):
> * Serge E. Hallyn (serge@hallyn.com) wrote:
> 
> > Although, some people are using newer qemu with '-M pc-1.0'.  So we'd be
> > stopping thing from migrating to support the people coming from qemu-kvm.
> > Hmm.
> 
> What sometimes happens is that you create a VM (lets say with virt-manager)
> on an old installation, and it has the XML file for that VM which I think
> includes the machine type.   You upgrade your host or copy your VMs over
> and so you still use the old machine definition, and hence still have the
> -M pc-1.0

Right, but so long as it is a shutdown vm image, it shouldn't matter to
this case whether qemu 1.7 uses the old qemu-kvm cirrus vga ram size, or
the qemu ram size, right?

> Some people have snapshots rather than just disk images taken with particular
> versions and they expect those to be reloadable into newer versions.

Disk snapshots should be fine right?  Live snapshots no, but then
I didn't they were actually very well supported anyway.  In any case,
they point would remain that if x% of people have snapshots from qemu-kvm,
and the rest have snapshots created using qemu but older machine types,
then whether or not we change the values, either way the other x% or
100-x % will not be able to restore...

So still it seems the only sure way to handle this would be for qemu
to say "old vram size != new vram size;  so let me resize the vram".

-serge

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Dr. David Alan Gilbert]

* Serge E. Hallyn (serge@hallyn.com) wrote:
> Quoting Dr. David Alan Gilbert (dgilbert@redhat.com):
> > * Serge E. Hallyn (serge@hallyn.com) wrote:
> > 
> > > Although, some people are using newer qemu with '-M pc-1.0'.  So we'd be
> > > stopping thing from migrating to support the people coming from qemu-kvm.
> > > Hmm.
> > 
> > What sometimes happens is that you create a VM (lets say with virt-manager)
> > on an old installation, and it has the XML file for that VM which I think
> > includes the machine type.   You upgrade your host or copy your VMs over
> > and so you still use the old machine definition, and hence still have the
> > -M pc-1.0
> 
> Right, but so long as it is a shutdown vm image, it shouldn't matter to
> this case whether qemu 1.7 uses the old qemu-kvm cirrus vga ram size, or
> the qemu ram size, right?

Right - but they might have kept it as -M pc-1.0 on purpose, so that they
can migrate it back to another machine with an older qemu.

> > Some people have snapshots rather than just disk images taken with particular
> > versions and they expect those to be reloadable into newer versions.
> 
> Disk snapshots should be fine right?  Live snapshots no, but then
> I didn't they were actually very well supported anyway.  In any case,
> they point would remain that if x% of people have snapshots from qemu-kvm,
> and the rest have snapshots created using qemu but older machine types,
> then whether or not we change the values, either way the other x% or
> 100-x % will not be able to restore...
> 
> So still it seems the only sure way to handle this would be for qemu
> to say "old vram size != new vram size;  so let me resize the vram".

I'd say if the machine type and other config matches they must stay the same,
if the machine type or appropriate config is different you can change stuff.

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Paolo Bonzini]

Il 19/03/2014 18:46, Dr. David Alan Gilbert ha scritto:
> * Serge E. Hallyn (serge@hallyn.com) wrote:
>> Quoting Dr. David Alan Gilbert (dgilbert@redhat.com):
>>> * Serge E. Hallyn (serge@hallyn.com) wrote:
>>>
>>>> Although, some people are using newer qemu with '-M pc-1.0'.  So we'd be
>>>> stopping thing from migrating to support the people coming from qemu-kvm.
>>>> Hmm.
>>>
>>> What sometimes happens is that you create a VM (lets say with virt-manager)
>>> on an old installation, and it has the XML file for that VM which I think
>>> includes the machine type.   You upgrade your host or copy your VMs over
>>> and so you still use the old machine definition, and hence still have the
>>> -M pc-1.0
>>
>> Right, but so long as it is a shutdown vm image, it shouldn't matter to
>> this case whether qemu 1.7 uses the old qemu-kvm cirrus vga ram size, or
>> the qemu ram size, right?
>
> Right - but they might have kept it as -M pc-1.0 on purpose, so that they
> can migrate it back to another machine with an older qemu.

No, this is not supported upstream.  Upstream only supports forward 
migration.

>>> Some people have snapshots rather than just disk images taken with particular
>>> versions and they expect those to be reloadable into newer versions.
>>
>> Disk snapshots should be fine right?  Live snapshots no, but then
>> I didn't they were actually very well supported anyway.  In any case,
>> they point would remain that if x% of people have snapshots from qemu-kvm,
>> and the rest have snapshots created using qemu but older machine types,
>> then whether or not we change the values, either way the other x% or
>> 100-x % will not be able to restore...

This is correct.

Paolo

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Cole Robinson]

On 03/19/2014 12:16 PM, Serge E. Hallyn wrote:
> Quoting Serge E. Hallyn (serge@hallyn.com):
>> Quoting Paolo Bonzini (pbonzini@redhat.com):
>>> Il 19/03/2014 16:42, Serge E. Hallyn ha scritto:
>>>> Hi,
>>>>
>>>> at https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321
>>>> it was found that migrating running vms from a machine with
>>>> qemu-kvm to one with qemu, migration fails due to some mismatches.
>>>> The first one we usually hit is
>>>>
>>>> Length mismatch: vga.vram: 1000000 in != 800000
>>>>
>>>> while the second one is network card (and I have not gone beyond
>>>> that).
>>>
>>> This is just a different default.  Make sure you specify the right
>>> model on the command line.
>>>
>>>> The vga one can be handled on the command line by
>>>> specifying the -global cirrus-vga.vrammem_mb=8.  However that
>>>> doesn't help with a libvirt migration.
>>>
>>> You can patch Ubuntu's QEMU to detect old machine types (pc-1.2 and
>>> earlier) and give cirrus-vga 16MB memory by default.  Migration only
>>> works with versioned machine types (or between same-version QEMU),
>>> so it's okay to only do it there.
>>>
>>>> I guess this happens at ram_load() unfortunately - is there any
>>>> good way that this could be detected at incoming migration time
>>>> and the virtual hardware modified as needed for migration to
>>>> continue?
>>>
>>> It cannot, but the destination can be patched to operate correctly
>>> for the old machine types, on the assumption that migration from the
>>> old machine types is always from Ubuntu's qemu-kvm.
>>>
>>> Paolo
>>
>> Ah, thank you, that assumption should be safe and sounds like a
>> great idea.
> 
> Although, some people are using newer qemu with '-M pc-1.0'.  So we'd be
> stopping thing from migrating to support the people coming from qemu-kvm.
> Hmm.
> 

FWIW, we've carried a patch like this in Fedora 18, 19, and 20:

http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20

As you say, it breaks incoming migration from stock qemu.git. But we never
shipped qemu.git prior to F18 (well, not in any timeframe that matters here),
so back compat wasn't a real concern.

For Fedora 21 we are dropping this patch, since there's been enough releases
for the people to transition:

http://blog.wikichoon.com/2013/12/kvm-migration-from-fedora-17-to-fedora.html

- Cole

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Serge E. Hallyn]

Quoting Cole Robinson (crobinso@redhat.com):
> On 03/19/2014 12:16 PM, Serge E. Hallyn wrote:
> > Quoting Serge E. Hallyn (serge@hallyn.com):
> >> Quoting Paolo Bonzini (pbonzini@redhat.com):
> >>> Il 19/03/2014 16:42, Serge E. Hallyn ha scritto:
> >>>> Hi,
> >>>>
> >>>> at https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321
> >>>> it was found that migrating running vms from a machine with
> >>>> qemu-kvm to one with qemu, migration fails due to some mismatches.
> >>>> The first one we usually hit is
> >>>>
> >>>> Length mismatch: vga.vram: 1000000 in != 800000
> >>>>
> >>>> while the second one is network card (and I have not gone beyond
> >>>> that).
> >>>
> >>> This is just a different default.  Make sure you specify the right
> >>> model on the command line.
> >>>
> >>>> The vga one can be handled on the command line by
> >>>> specifying the -global cirrus-vga.vrammem_mb=8.  However that
> >>>> doesn't help with a libvirt migration.
> >>>
> >>> You can patch Ubuntu's QEMU to detect old machine types (pc-1.2 and
> >>> earlier) and give cirrus-vga 16MB memory by default.  Migration only
> >>> works with versioned machine types (or between same-version QEMU),
> >>> so it's okay to only do it there.
> >>>
> >>>> I guess this happens at ram_load() unfortunately - is there any
> >>>> good way that this could be detected at incoming migration time
> >>>> and the virtual hardware modified as needed for migration to
> >>>> continue?
> >>>
> >>> It cannot, but the destination can be patched to operate correctly
> >>> for the old machine types, on the assumption that migration from the
> >>> old machine types is always from Ubuntu's qemu-kvm.
> >>>
> >>> Paolo
> >>
> >> Ah, thank you, that assumption should be safe and sounds like a
> >> great idea.
> > 
> > Although, some people are using newer qemu with '-M pc-1.0'.  So we'd be
> > stopping thing from migrating to support the people coming from qemu-kvm.
> > Hmm.
> > 
> 
> FWIW, we've carried a patch like this in Fedora 18, 19, and 20:
> 
> http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20
> 
> As you say, it breaks incoming migration from stock qemu.git. But we never
> shipped qemu.git prior to F18 (well, not in any timeframe that matters here),
> so back compat wasn't a real concern.
> 
> For Fedora 21 we are dropping this patch, since there's been enough releases
> for the people to transition:
> 
> http://blog.wikichoon.com/2013/12/kvm-migration-from-fedora-17-to-fedora.html

Thanks, Cole.  I notice it doesn't do anything for -vga cirrus - is that
just bc in general fedora users tend to use -vga qxl?

-serge

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Cole Robinson]

On 03/21/2014 01:00 PM, Serge E. Hallyn wrote:
> Quoting Cole Robinson (crobinso@redhat.com):
>> On 03/19/2014 12:16 PM, Serge E. Hallyn wrote:
>>> Quoting Serge E. Hallyn (serge@hallyn.com):
>>>> Quoting Paolo Bonzini (pbonzini@redhat.com):
>>>>> Il 19/03/2014 16:42, Serge E. Hallyn ha scritto:
>>>>>> Hi,
>>>>>>
>>>>>> at https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321
>>>>>> it was found that migrating running vms from a machine with
>>>>>> qemu-kvm to one with qemu, migration fails due to some mismatches.
>>>>>> The first one we usually hit is
>>>>>>
>>>>>> Length mismatch: vga.vram: 1000000 in != 800000
>>>>>>
>>>>>> while the second one is network card (and I have not gone beyond
>>>>>> that).
>>>>>
>>>>> This is just a different default.  Make sure you specify the right
>>>>> model on the command line.
>>>>>
>>>>>> The vga one can be handled on the command line by
>>>>>> specifying the -global cirrus-vga.vrammem_mb=8.  However that
>>>>>> doesn't help with a libvirt migration.
>>>>>
>>>>> You can patch Ubuntu's QEMU to detect old machine types (pc-1.2 and
>>>>> earlier) and give cirrus-vga 16MB memory by default.  Migration only
>>>>> works with versioned machine types (or between same-version QEMU),
>>>>> so it's okay to only do it there.
>>>>>
>>>>>> I guess this happens at ram_load() unfortunately - is there any
>>>>>> good way that this could be detected at incoming migration time
>>>>>> and the virtual hardware modified as needed for migration to
>>>>>> continue?
>>>>>
>>>>> It cannot, but the destination can be patched to operate correctly
>>>>> for the old machine types, on the assumption that migration from the
>>>>> old machine types is always from Ubuntu's qemu-kvm.
>>>>>
>>>>> Paolo
>>>>
>>>> Ah, thank you, that assumption should be safe and sounds like a
>>>> great idea.
>>>
>>> Although, some people are using newer qemu with '-M pc-1.0'.  So we'd be
>>> stopping thing from migrating to support the people coming from qemu-kvm.
>>> Hmm.
>>>
>>
>> FWIW, we've carried a patch like this in Fedora 18, 19, and 20:
>>
>> http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20
>>
>> As you say, it breaks incoming migration from stock qemu.git. But we never
>> shipped qemu.git prior to F18 (well, not in any timeframe that matters here),
>> so back compat wasn't a real concern.
>>
>> For Fedora 21 we are dropping this patch, since there's been enough releases
>> for the people to transition:
>>
>> http://blog.wikichoon.com/2013/12/kvm-migration-from-fedora-17-to-fedora.html
> 
> Thanks, Cole.  I notice it doesn't do anything for -vga cirrus - is that
> just bc in general fedora users tend to use -vga qxl?
> 

Line 149 of the patch changes the cirrus-vga default memory for old machine
types, seems to fix things in my testing.

- Cole

Re: [Qemu-devel] migrating instances from qemu-kvm to qemu

[Serge E. Hallyn]

Quoting Cole Robinson (crobinso@redhat.com):
> On 03/21/2014 01:00 PM, Serge E. Hallyn wrote:
> > Quoting Cole Robinson (crobinso@redhat.com):
> >> On 03/19/2014 12:16 PM, Serge E. Hallyn wrote:
> >>> Quoting Serge E. Hallyn (serge@hallyn.com):
> >>>> Quoting Paolo Bonzini (pbonzini@redhat.com):
> >>>>> Il 19/03/2014 16:42, Serge E. Hallyn ha scritto:
> >>>>>> Hi,
> >>>>>>
> >>>>>> at https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321
> >>>>>> it was found that migrating running vms from a machine with
> >>>>>> qemu-kvm to one with qemu, migration fails due to some mismatches.
> >>>>>> The first one we usually hit is
> >>>>>>
> >>>>>> Length mismatch: vga.vram: 1000000 in != 800000
> >>>>>>
> >>>>>> while the second one is network card (and I have not gone beyond
> >>>>>> that).
> >>>>>
> >>>>> This is just a different default.  Make sure you specify the right
> >>>>> model on the command line.
> >>>>>
> >>>>>> The vga one can be handled on the command line by
> >>>>>> specifying the -global cirrus-vga.vrammem_mb=8.  However that
> >>>>>> doesn't help with a libvirt migration.
> >>>>>
> >>>>> You can patch Ubuntu's QEMU to detect old machine types (pc-1.2 and
> >>>>> earlier) and give cirrus-vga 16MB memory by default.  Migration only
> >>>>> works with versioned machine types (or between same-version QEMU),
> >>>>> so it's okay to only do it there.
> >>>>>
> >>>>>> I guess this happens at ram_load() unfortunately - is there any
> >>>>>> good way that this could be detected at incoming migration time
> >>>>>> and the virtual hardware modified as needed for migration to
> >>>>>> continue?
> >>>>>
> >>>>> It cannot, but the destination can be patched to operate correctly
> >>>>> for the old machine types, on the assumption that migration from the
> >>>>> old machine types is always from Ubuntu's qemu-kvm.
> >>>>>
> >>>>> Paolo
> >>>>
> >>>> Ah, thank you, that assumption should be safe and sounds like a
> >>>> great idea.
> >>>
> >>> Although, some people are using newer qemu with '-M pc-1.0'.  So we'd be
> >>> stopping thing from migrating to support the people coming from qemu-kvm.
> >>> Hmm.
> >>>
> >>
> >> FWIW, we've carried a patch like this in Fedora 18, 19, and 20:
> >>
> >> http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20
> >>
> >> As you say, it breaks incoming migration from stock qemu.git. But we never
> >> shipped qemu.git prior to F18 (well, not in any timeframe that matters here),
> >> so back compat wasn't a real concern.
> >>
> >> For Fedora 21 we are dropping this patch, since there's been enough releases
> >> for the people to transition:
> >>
> >> http://blog.wikichoon.com/2013/12/kvm-migration-from-fedora-17-to-fedora.html
> > 
> > Thanks, Cole.  I notice it doesn't do anything for -vga cirrus - is that
> > just bc in general fedora users tend to use -vga qxl?
> > 
> 
> Line 149 of the patch changes the cirrus-vga default memory for old machine
> types, seems to fix things in my testing.

Hh, hm, how did i miss that.

thanks,
-serge