[Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

I'm trying to migrate between a pc-1.0 machine on qemu 1.0 and a pc-1.0 machine on qemu-2.0. This appears not to work.

It looks to me as if at some point along the line the definition of pc-1.0 has changed (eek).

The machine was booted with the command line [1] below, then migrated (for the sake of ease including a full block device dump) to a file. I'm happy to provide this (it's a blank cirros image but on a 20GB disk so quite large).

Restoring the machine with the command line [2] (merely adding -machine pc-1.0 which is not the default for qemu-2.0) produces the error:
Length mismatch: vga.vram: 1000000 in != 800000
qemu: warning: error while loading state for instance 0x0 of device 'ram'
load of migration failed

This can be fixed using command line [3] (adding an additional -global cirrus-vga.vgamem_mb=10) in which case it gets a little further.
Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000
qemu: warning: error while loading state for instance 0x0 of device 'ram'
load of migration failed

I don't know how to get around that on the command line.

This Ubuntu bug:
   https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321

suggests that the root cause is a change in the definition of pc-1.0 between qemu 1.0 and 1.7. That's annoying, but obviously we can't change the definition of pc-1.0 now because users of later versions of qemu may be using the (newer) pc-1.0.

Assuming I can't find this on the command line (which I would be ok with I suppose), I propose to define a new machine type (e.g. pc-1.0-qemu-1.0) which would look like the original qemu 1.0 pc-1.0 machine type. I think I can see how to do this for the video ram, but I'm puzzled about how to do this for the virtio-net-pci.rom (and indeed anything else that I might come across).

Is this a profitable route of investigation, or should I just give up now and assume live migration between 1.0 and 2.0 is impossible?

-- 
Alex Bligh



[1] Use to start the VM on qemu 1.0

/usr/bin/kvm -name vm -uuid 33372d76-6d65-6e74-3132-333435363738 -m 512 -smp 1,cores=1,threads=1,sockets=1 -vga cirrus -usbdevice tablet -k en-gb -vnc 0.0.0.0:6313,password -monitor telnet:127.0.0.1:12345,server,nowait -drive file=/storage/img.qcow2,if=virtio,index=0,format=qcow2,media=disk,cache=writeback -boot order=cd -net nic,macaddr=00:16:3e:28:04:51,model=virtio,vlan=0 -net tap,ifname=tap13.0,script=no,downscript=no,vlan=0 -qmp unix:/var/run/vm.monitor,server,nowait -pidfile /var/run/vm.pid

HMP then used to save the device out with migrate -b.

[2] First attempt to start the VM on qemu 2.0 - just add -machine pc-1.0

/usr/bin/kvm -name testimport -machine pc-1.0 -uuid 33372d76-6d65-6e74-3132-333435363738 -m 512 -smp 1,cores=1,threads=1,sockets=1 -vga cirrus -usbdevice tablet -k en-gb -vnc 0.0.0.0:6314,password -monitor telnet:127.0.0.1:12345,server,nowait -drive file=/storage/img2.qcow2,if=virtio,index=0,format=qcow2,media=disk,cache=writeback -boot order=cd -net nic,macaddr=00:16:3e:28:04:51,model=virtio,vlan=0 -net socket,mcast=230.0.0.1:1234,vlan=0 -qmp unix:/var/run/testimport.monitor,server,nowait -pidfile /var/run/testimport.pid -incoming fd:</storage/vm.out

Result:

Length mismatch: vga.vram: 1000000 in != 800000
qemu: warning: error while loading state for instance 0x0 of device 'ram'
load of migration failed

[3] Second attempt to start the VM on qemu 2.0 - also add -global cirrus-vga.vgamem_mb=10

/usr/bin/kvm -name testimport -machine pc-1.0 -global cirrus-vga.vgamem_mb=10 -uuid 33372d76-6d65-6e74-3132-333435363738 -m 512 -smp 1,cores=1,threads=1,sockets=1 -vga cirrus -usbdevice tablet -k en-gb -vnc 0.0.0.0:6314,password -monitor telnet:127.0.0.1:12345,server,nowait -drive file=/storage/img2.qcow2,if=virtio,index=0,format=qcow2,media=disk,cache=writeback -boot order=cd -net nic,macaddr=00:16:3e:28:04:51,model=virtio,vlan=0 -net socket,mcast=230.0.0.1:1234,vlan=0 -qmp unix:/var/run/testimport.monitor,server,nowait -pidfile /var/run/testimport.pid -incoming fd:</storage/vm.out

Result:

Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000
qemu: warning: error while loading state for instance 0x0 of device 'ram'
load of migration failed

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Paolo Bonzini]

Il 19/07/2014 01:33, Alex Bligh ha scritto:
> I'm trying to migrate between a pc-1.0 machine on qemu 1.0 and a pc-1.0 machine on qemu-2.0. This appears not to work.
> 
> It looks to me as if at some point along the line the definition of pc-1.0 has changed (eek).
> 
> The machine was booted with the command line [1] below, then migrated (for the sake of ease including a full block device dump) to a file. I'm happy to provide this (it's a blank cirros image but on a 20GB disk so quite large).
> 
> Restoring the machine with the command line [2] (merely adding -machine pc-1.0 which is not the default for qemu-2.0) produces the error:
> Length mismatch: vga.vram: 1000000 in != 800000
> qemu: warning: error while loading state for instance 0x0 of device 'ram'
> load of migration failed

This is unfortunately a known difference between qemu-kvm and qemu.

> This can be fixed using command line [3] (adding an additional -global cirrus-vga.vgamem_mb=10) in which case it gets a little further.
> Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000
> qemu: warning: error while loading state for instance 0x0 of device 'ram'
> load of migration failed

Looks like your source and destination machines have different iPXE
ROMs.  This could be a packaging problem in your distro.

Paolo

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

Paolo,

On 19 Jul 2014, at 06:51, Paolo Bonzini <pbonzini@redhat.com> wrote:

>> Restoring the machine with the command line [2] (merely adding -machine pc-1.0 which is not the default for qemu-2.0) produces the error:
>> Length mismatch: vga.vram: 1000000 in != 800000
>> qemu: warning: error while loading state for instance 0x0 of device 'ram'
>> load of migration failed
> 
> This is unfortunately a known difference between qemu-kvm and qemu.

Indeed. So this one I can work around.

>> This can be fixed using command line [3] (adding an additional -global cirrus-vga.vgamem_mb=10) in which case it gets a little further.
>> Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000
>> qemu: warning: error while loading state for instance 0x0 of device 'ram'
>> load of migration failed
> 
> Looks like your source and destination machines have different iPXE
> ROMs.  This could be a packaging problem in your distro.

This is Ubuntu 12.04 to Ubuntu 14.04. I think the 14.04 one is correct, and don't want to run with a non-standard qemu there.

Can I use a different ROM (e.g. the old one) with a command line option just for the migrated machines? Obviously when they reboot, they'l get a hardware change (which is fine). Obviously the iPXE ROM makes no difference when the machine is running, it's purely the size which is causing an issue.

-- 
Alex Bligh

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Paolo Bonzini]

Il 19/07/2014 09:10, Alex Bligh ha scritto:
>>> Looks like your source and destination machines have different
>>> iPXE ROMs.  This could be a packaging problem in your distro.
> 
> This is Ubuntu 12.04 to Ubuntu 14.04. I think the 14.04 one is
> correct, and don't want to run with a non-standard qemu there.

No, the old one is _always_ the correct one.

The pxe-* ROMs must be 128k, the efi-* ROMs must be 256k.  If distros do
not do that, they have a bug (and as you noticed it's not one that they
can easily work around, since now you have different releases with
different ROM sizes for the same machine types).

> Can I use a different ROM (e.g. the old one) with a command line
> option just for the migrated machines? Obviously when they reboot,
> they'l get a hardware change (which is fine).

Yes, "-global foo.romfile" on the destination QEMU.  You can just pass
an empty 128k file to the destination since ROM contents are migrated
properly, and hardware will only change when QEMU restarts.

Migration is extremely complex because you have to strike the right
balance between compatibility and maintainability.  Whether it works, it
depends on a lot of factors that vary per distro, and the only thing
that can keep it working is a lot of testing.

I don't know of any distro that actively tests migration except RHEL,
SLES.  I'm not sure about Debian.  Since there is no free-beer SLES,
you'd better switch to CentOS 7 than keeping Ubuntu 14.04.

Paolo

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

Paolo,

On 19 Jul 2014, at 08:30, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 19/07/2014 09:10, Alex Bligh ha scritto:
>>>> Looks like your source and destination machines have different
>>>> iPXE ROMs.  This could be a packaging problem in your distro.
>> 
>> This is Ubuntu 12.04 to Ubuntu 14.04. I think the 14.04 one is
>> correct, and don't want to run with a non-standard qemu there.
> 
> No, the old one is _always_ the correct one.
> 
> The pxe-* ROMs must be 128k, the efi-* ROMs must be 256k.

On 12.04
# ls -la /usr/share/qemu/pxe-virtio.rom
-rw-r--r-- 1 root root 63488 Jun 12 23:23 /usr/share/qemu/pxe-virtio.rom

On 14.04 (after resolving a pile of symlinks)
# ls -la /usr/lib/ipxe/qemu/pxe-virtio.rom
-rw-r--r-- 1 root root 83968 Jan  6  2014 /usr/lib/ipxe/qemu/pxe-virtio.rom

I'm guessing the ROM size gets rounded up to the next power of 2, hence
the ROM size on 12.04 is 64k, and on 14.04 is 128k, right?

So it would appear that if the length is actually coming from the
length of the file on disk, that on 12.04 the pxe-ROM is not 128k
but 64k. So if 'pxe-* ROMs must be 128k' is correct, I'm not sure
that's incompatible with  'the old one is _always_ the correct
one'.

Whilst (per below) your workaround gets past the issue for this ROM,
I'm confused about the numbers in the error message, as 64k and
128k do not equal 10,000 or 20,000:
 Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000


>  If distros do
> not do that, they have a bug (and as you noticed it's not one that they
> can easily work around, since now you have different releases with
> different ROM sizes for the same machine types).

Thanks. So I am discovering.

>> Can I use a different ROM (e.g. the old one) with a command line
>> option just for the migrated machines? Obviously when they reboot,
>> they'l get a hardware change (which is fine).
> 
> Yes, "-global foo.romfile" on the destination QEMU.  You can just pass
> an empty 128k file to the destination since ROM contents are migrated
> properly, and hardware will only change when QEMU restarts.

Thanks.

So, copying the old ROM file across (out of an abundance of caution)
plus the magic incantation:

-machine pc-1.0 -global cirrus-vga.vgamem_mb=10 -global virtio-net-pci.romfile=pxe-virtio.rom.12.04

would appear to be sufficient to permit the migration to get past the
initial stages and commence the block migration.

However, after the block migration is finished, I now see:

Receiving block device images
Completed 100 %
Unknown savevm section type 255
load of migration failed

This would appear to be similar to the issue you reported here
  http://comments.gmane.org/gmane.comp.emulators.qemu/49702
(though given the dates that particular issue should be fixed
in both versions), which presumably means there is some binary
format incompatibility between sender and receiver.

My guess is this might be the bogus inclusion of PITCommonState
per the last hunk of
 http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20

Is there a technique to debug this sort of thing?

> Migration is extremely complex because you have to strike the right
> balance between compatibility and maintainability.  Whether it works, it
> depends on a lot of factors that vary per distro, and the only thing
> that can keep it working is a lot of testing.

Indeed, which is what I now appear to be spending my weekend doing ...

> I don't know of any distro that actively tests migration except RHEL,
> SLES.  I'm not sure about Debian.  Since there is no free-beer SLES,
> you'd better switch to CentOS 7 than keeping Ubuntu 14.04.

Switching is not an option here; we have way too much other stuff dependent
on Ubuntu.

Ubuntu is about to become tested (and a patch or workaround generated
at least for migrations from 12.04 - the previous LTS version);
whether or not it is upstreamed won't be up to me though.

To be fair, I think the issue is that they are supporting migration from
stock qemu.git pc-1.0 (which is what I think they shipped with the
subsequent non-LTS releases, which means they can't (easily) also
support migration from qemu-kvm pc-1.0. That's the bit I want to fix.

Fortunately we have a relatively limited number of qemu VM configs out
there in the wild.

-- 
Alex Bligh

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Peter Maydell]

On 19 July 2014 09:43, Alex Bligh <alex@alex.org.uk> wrote:
> Whilst (per below) your workaround gets past the issue for this ROM,
> I'm confused about the numbers in the error message, as 64k and
> 128k do not equal 10,000 or 20,000:
>  Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000

0x10000 hex == 65536 decimal == 64K
0x20000 hex == 131072 decimal == 128K

The error message should probably have "0x" prefixes added.

-- PMM

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

On 19 Jul 2014, at 09:54, Peter Maydell <peter.maydell@linaro.org> wrote:

> On 19 July 2014 09:43, Alex Bligh <alex@alex.org.uk> wrote:
>> Whilst (per below) your workaround gets past the issue for this ROM,
>> I'm confused about the numbers in the error message, as 64k and
>> 128k do not equal 10,000 or 20,000:
>> Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000
> 
> 0x10000 hex == 65536 decimal == 64K
> 0x20000 hex == 131072 decimal == 128K
> 
> The error message should probably have "0x" prefixes added.

Doh! Patch coming up.

-- 
Alex Bligh

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Paolo Bonzini]

Il 19/07/2014 10:43, Alex Bligh ha scritto:
> Paolo,
> 
> On 19 Jul 2014, at 08:30, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
>> Il 19/07/2014 09:10, Alex Bligh ha scritto:
>>>>> Looks like your source and destination machines have different
>>>>> iPXE ROMs.  This could be a packaging problem in your distro.
>>>
>>> This is Ubuntu 12.04 to Ubuntu 14.04. I think the 14.04 one is
>>> correct, and don't want to run with a non-standard qemu there.
>>
>> No, the old one is _always_ the correct one.
>>
>> The pxe-* ROMs must be 128k, the efi-* ROMs must be 256k.
> 
> On 12.04
> # ls -la /usr/share/qemu/pxe-virtio.rom
> -rw-r--r-- 1 root root 63488 Jun 12 23:23 /usr/share/qemu/pxe-virtio.rom
> 
> On 14.04 (after resolving a pile of symlinks)
> # ls -la /usr/lib/ipxe/qemu/pxe-virtio.rom
> -rw-r--r-- 1 root root 83968 Jan  6  2014 /usr/lib/ipxe/qemu/pxe-virtio.rom
> 
> I'm guessing the ROM size gets rounded up to the next power of 2, hence
> the ROM size on 12.04 is 64k, and on 14.04 is 128k, right?
> 
> So it would appear that if the length is actually coming from the
> length of the file on disk, that on 12.04 the pxe-ROM is not 128k
> but 64k. So if 'pxe-* ROMs must be 128k' is correct, I'm not sure
> that's incompatible with  'the old one is _always_ the correct
> one'.

Sorry, pxe-virtio.rom must be 64k.  It's pxe-e1000.rom that is 128k
because it exceeds 64k by a little:

-rw-rw-r-- 1 pbonzini users   67072  2 dic  2013 pxe-e1000.rom
-rw-rw-r-- 1 pbonzini users   61440  2 dic  2013 pxe-eepro100.rom
-rw-rw-r-- 1 pbonzini users   61440  2 dic  2013 pxe-ne2k_pci.rom
-rw-rw-r-- 1 pbonzini users   61440  2 dic  2013 pxe-pcnet.rom
-rw-rw-r-- 1 pbonzini users   61440  2 dic  2013 pxe-rtl8139.rom
-rw-rw-r-- 1 pbonzini users   60416  2 dic  2013 pxe-virtio.rom
-rw-r--r-- 1 pbonzini users  194560 16 lug 13.16 efi-e1000.rom
-rw-r--r-- 1 pbonzini users  196096 16 lug 13.16 efi-eepro100.rom
-rw-r--r-- 1 pbonzini users  194560 16 lug 13.16 efi-ne2k_pci.rom
-rw-r--r-- 1 pbonzini users  194560 16 lug 13.16 efi-pcnet.rom
-rw-r--r-- 1 pbonzini users  198144 16 lug 13.16 efi-rtl8139.rom
-rw-r--r-- 1 pbonzini users  192000 16 lug 13.16 efi-virtio.rom

> Whilst (per below) your workaround gets past the issue for this ROM,
> I'm confused about the numbers in the error message, as 64k and
> 128k do not equal 10,000 or 20,000:
>  Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000

These are hexadecimal.

>>> Can I use a different ROM (e.g. the old one) with a command line
>>> option just for the migrated machines? Obviously when they reboot,
>>> they'l get a hardware change (which is fine).
>>
>> Yes, "-global foo.romfile" on the destination QEMU.  You can just pass
>> an empty 128k file to the destination since ROM contents are migrated
>> properly, and hardware will only change when QEMU restarts.
> 
> Thanks.
> 
> So, copying the old ROM file across (out of an abundance of caution)
> plus the magic incantation:
> 
> -machine pc-1.0 -global cirrus-vga.vgamem_mb=10 -global virtio-net-pci.romfile=pxe-virtio.rom.12.04

It's really 16 (again hex).

> would appear to be sufficient to permit the migration to get past the
> initial stages and commence the block migration.

Ok, at least the memory map is now the same.

> However, after the block migration is finished, I now see:
> 
> Receiving block device images
> Completed 100 %
> Unknown savevm section type 255
> load of migration failed

This could be another incompatibility between qemu-kvm and qemu.  I know
we had a couple in Fedora.

> My guess is this might be the bogus inclusion of PITCommonState
> per the last hunk of
>  http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20
> 
> Is there a technique to debug this sort of thing?

You can try using Amit's vmstate checker.

>> I don't know of any distro that actively tests migration except RHEL,
>> SLES.  I'm not sure about Debian.  Since there is no free-beer SLES,
>> you'd better switch to CentOS 7 than keeping Ubuntu 14.04.
> 
> Switching is not an option here; we have way too much other stuff dependent
> on Ubuntu.

You could try rebuilding the patched QEMU sources from CentOS.  CentOS
7's rhel6.1.0 or rhel6.2.0 machine types are comparable to pc-1.0, with
some luck they might even be compatible.

> Ubuntu is about to become tested (and a patch or workaround generated
> at least for migrations from 12.04 - the previous LTS version);
> whether or not it is upstreamed won't be up to me though.

Yes, because Ubuntu hardly introduces any new feature during the LTS
support period.  But in 2 years you will have to endure roughly the same
pain.

> To be fair, I think the issue is that they are supporting migration from
> stock qemu.git pc-1.0 (which is what I think they shipped with the
> subsequent non-LTS releases, which means they can't (easily) also
> support migration from qemu-kvm pc-1.0. That's the bit I want to fix.

Yes, Ubuntu didn't bother testing migration from 12.04 to non-LTS
releases or 14.04.  They don't have the manpower to do so anyway.

Paolo

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

Paolo,

On 19 Jul 2014, at 11:53, Paolo Bonzini <pbonzini@redhat.com> wrote:

>>> No, the old one is _always_ the correct one.
>>> 
>>> The pxe-* ROMs must be 128k, the efi-* ROMs must be 256k.
>> 
>> On 12.04
>> # ls -la /usr/share/qemu/pxe-virtio.rom
>> -rw-r--r-- 1 root root 63488 Jun 12 23:23 /usr/share/qemu/pxe-virtio.rom
>> 
>> On 14.04 (after resolving a pile of symlinks)
>> # ls -la /usr/lib/ipxe/qemu/pxe-virtio.rom
>> -rw-r--r-- 1 root root 83968 Jan  6  2014 /usr/lib/ipxe/qemu/pxe-virtio.rom
>> 
>> I'm guessing the ROM size gets rounded up to the next power of 2, hence
>> the ROM size on 12.04 is 64k, and on 14.04 is 128k, right?
>> 
>> So it would appear that if the length is actually coming from the
>> length of the file on disk, that on 12.04 the pxe-ROM is not 128k
>> but 64k. So if 'pxe-* ROMs must be 128k' is correct, I'm not sure
>> that's incompatible with  'the old one is _always_ the correct
>> one'.
> 
> Sorry, pxe-virtio.rom must be 64k.  It's pxe-e1000.rom that is 128k
> because it exceeds 64k by a little:

OK, well it has naughtily grown quite a bit on Ubuntu 14.04 as
above. That's easily fixable.

>> Whilst (per below) your workaround gets past the issue for this ROM,
>> I'm confused about the numbers in the error message, as 64k and
>> 128k do not equal 10,000 or 20,000:
>> Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000
> 
> These are hexadecimal.

Have sent a patch to add '0x'

>> However, after the block migration is finished, I now see:
>> 
>> Receiving block device images
>> Completed 100 %
>> Unknown savevm section type 255
>> load of migration failed
> 
> This could be another incompatibility between qemu-kvm and qemu.  I know
> we had a couple in Fedora.
> 
>> My guess is this might be the bogus inclusion of PITCommonState
>> per the last hunk of
>> http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20
>> 
>> Is there a technique to debug this sort of thing?
> 
> You can try using Amit's vmstate checker.

I think this requires a json output in the format given by -dump-vmstate
(unless we're at cross purposes). qemu 1.0 does not produce such a JSON
output. I only have the vmstate to go on (same problem exists if
-S is used throughout so the VM never starts and without block migration).

Currently my idea is 'gdb' :-)

> You could try rebuilding the patched QEMU sources from CentOS.  CentOS
> 7's rhel6.1.0 or rhel6.2.0 machine types are comparable to pc-1.0, with
> some luck they might even be compatible.

I think the issue is that there are at least two versions of pc-1.0 (or
how the state is serialised); see the memory layout issues. I suspect
both qemu-git and qemu-kvm had slightly different pc-1.0. I actually
need it to be compatible with Ubuntu's qemu-kvm pc-1.0 which might
of course be different again :-/

>> Ubuntu is about to become tested (and a patch or workaround generated
>> at least for migrations from 12.04 - the previous LTS version);
>> whether or not it is upstreamed won't be up to me though.
> 
> Yes, because Ubuntu hardly introduces any new feature during the LTS
> support period.

My guess is they will take a non-intrusive SRU which provides the
ability for live migrates to work. However, if not we will just
maintain it out of tree (like we were doing for various other
qemu bits).

> But in 2 years you will have to endure roughly the same
> pain.

I am aware of this.

-- 
Alex Bligh

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Paolo Bonzini]

Il 19/07/2014 13:37, Alex Bligh ha scritto:
>>> Whilst (per below) your workaround gets past the issue for this ROM,
>>> I'm confused about the numbers in the error message, as 64k and
>>> 128k do not equal 10,000 or 20,000:
>>> Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000
>>
>> These are hexadecimal.
> 
> Have sent a patch to add '0x'

Thanks.

>>> My guess is this might be the bogus inclusion of PITCommonState
>>> per the last hunk of
>>> http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20
>>>
>>> Is there a technique to debug this sort of thing?
>>
>> You can try using Amit's vmstate checker.
> 
> I think this requires a json output in the format given by -dump-vmstate
> (unless we're at cross purposes). qemu 1.0 does not produce such a JSON
> output. I only have the vmstate to go on (same problem exists if
> -S is used throughout so the VM never starts and without block migration).

You'd have to backport it, yes.

> Currently my idea is 'gdb' :-)

ghex too.  Add a printf at the end of every successful section load.

>> You could try rebuilding the patched QEMU sources from CentOS.  CentOS
>> 7's rhel6.1.0 or rhel6.2.0 machine types are comparable to pc-1.0, with
>> some luck they might even be compatible.
> 
> I think the issue is that there are at least two versions of pc-1.0 (or
> how the state is serialised); see the memory layout issues. I suspect
> both qemu-git and qemu-kvm had slightly different pc-1.0.

qemu 1.0 in theory has the same pc-1.0 as qemu 1.3 and newer.

qemu-kvm 1.0 is different.

> I actually
> need it to be compatible with Ubuntu's qemu-kvm pc-1.0 which might
> of course be different again :-/

Unlikely. :)

>>> Ubuntu is about to become tested (and a patch or workaround generated
>>> at least for migrations from 12.04 - the previous LTS version);
>>> whether or not it is upstreamed won't be up to me though.
>>
>> Yes, because Ubuntu hardly introduces any new feature during the LTS
>> support period.
> 
> My guess is they will take a non-intrusive SRU which provides the
> ability for live migrates to work. However, if not we will just
> maintain it out of tree (like we were doing for various other
> qemu bits).

Quite frankly: good luck.

Ubuntu has 1-2 people looking after QEMU (2-3 if you add Debian manpower).

RHEL sometimes has 5 people or more looking at migration bugs, and QE
tests machine types backwords compatibility periodically every six
months, so we never get into ugly situations or at least we can buy
ourselves a month to fix them.

Also, Ubuntu will have to choose between making 12.04->14.04 work and
making 14.04 (no updates) -> 14.04 (updated) work.

I'm not saying that Ubuntu sucks.  I'm just saying that there's only so
much that poor Serge can do.

Paolo

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

Paolo,

>> Currently my idea is 'gdb' :-)
> 
> ghex too.  Add a printf at the end of every successful section load.

I did this and indeed got it to work (see below).

>>> You could try rebuilding the patched QEMU sources from CentOS.  CentOS
>>> 7's rhel6.1.0 or rhel6.2.0 machine types are comparable to pc-1.0, with
>>> some luck they might even be compatible.
>> 
>> I think the issue is that there are at least two versions of pc-1.0 (or
>> how the state is serialised); see the memory layout issues. I suspect
>> both qemu-git and qemu-kvm had slightly different pc-1.0.
> 
> qemu 1.0 in theory has the same pc-1.0 as qemu 1.3 and newer.
> 
> qemu-kvm 1.0 is different.

Yep, that's the issue.

For my use case I need (effectively) the PIT and piix4 hunks of this:
  http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20

I was going to put these together into a machine type (e.g. pc-1.0-qemu-kvm)
but I need to get my head around whether that's possible to do (a sort of
non-linear chain of machine types).

Any interest in taking that upstream if I do?

>> My guess is they will take a non-intrusive SRU which provides the
>> ability for live migrates to work. However, if not we will just
>> maintain it out of tree (like we were doing for various other
>> qemu bits).
> 
> Quite frankly: good luck.

Your best wishes are appreciated :-)

-- 
Alex Bligh

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Paolo Bonzini]

Il 21/07/2014 15:59, Alex Bligh ha scritto:
>>> >> My guess is they will take a non-intrusive SRU which provides the
>>> >> ability for live migrates to work. However, if not we will just
>>> >> maintain it out of tree (like we were doing for various other
>>> >> qemu bits).
>> > 
>> > Quite frankly: good luck.
> Your best wishes are appreciated :-)

Good luck was referring to Ubuntu fixing it

Maintaining out of tree for you should not be hard indeed.

Please test reboot after migration.  It broke for us, and we couldn't
upstream the patches exactly because it depended on the source version
and not on the machine type.

Paolo

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

Paolo,

On 21 Jul 2014, at 15:11, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 21/07/2014 15:59, Alex Bligh ha scritto:
>>>>>> My guess is they will take a non-intrusive SRU which provides the
>>>>>> ability for live migrates to work. However, if not we will just
>>>>>> maintain it out of tree (like we were doing for various other
>>>>>> qemu bits).
>>>> 
>>>> Quite frankly: good luck.
>> Your best wishes are appreciated :-)
> 
> Good luck was referring to Ubuntu fixing it
> 
> Maintaining out of tree for you should not be hard indeed.
> 
> Please test reboot after migration.  It broke for us, and we couldn't
> upstream the patches exactly because it depended on the source version
> and not on the machine type.

Actually for me (and possibly for others) reboot isn't an issue,
because we'd be quite content if on reboot the machine type was
changed to pc-1.0 or perhaps even the current type (despite
the fact this probably gives Windows its normal apoplectic
licensing fit).

The reason is because in our environment (and possibly others)
the VMs are run by a large number of end customers who don't
want to have to reboot their machines exactly when our customer
(that runs the host) upgrades their host; hence live migrate
is essential. But saying "we've upgraded stuff and hence
your virtual hardware will change slightly when the machine
next reboots" is acceptable; indeed it's desirable given we
want to minimise the number of virtual hardware configurations
running. Then in 2 years when we have to deal with a 14.04->16.04
upgrade, we only need worry about one source migration type
(assuming the machines are rebooted at least once every two years).

But yes I will test it anyway.

-- 
Alex Bligh

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Paolo Bonzini]

Il 21/07/2014 15:59, Alex Bligh ha scritto:
> 
> For my use case I need (effectively) the PIT and piix4 hunks of this:
>   http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20
> 
> I was going to put these together into a machine type (e.g. pc-1.0-qemu-kvm)
> but I need to get my head around whether that's possible to do (a sort of
> non-linear chain of machine types).
> 
> Any interest in taking that upstream if I do?

Sure.

Paolo

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Amit Shah]

On (Sat) 19 Jul 2014 [12:37:31], Alex Bligh wrote:
> Paolo,
> 
> On 19 Jul 2014, at 11:53, Paolo Bonzini <pbonzini@redhat.com> wrote:

<snip>

> >> However, after the block migration is finished, I now see:
> >> 
> >> Receiving block device images
> >> Completed 100 %
> >> Unknown savevm section type 255
> >> load of migration failed
> > 
> > This could be another incompatibility between qemu-kvm and qemu.  I know
> > we had a couple in Fedora.
> > 
> >> My guess is this might be the bogus inclusion of PITCommonState
> >> per the last hunk of
> >> http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20
> >> 
> >> Is there a technique to debug this sort of thing?
> > 
> > You can try using Amit's vmstate checker.
> 
> I think this requires a json output in the format given by -dump-vmstate
> (unless we're at cross purposes). qemu 1.0 does not produce such a JSON
> output. I only have the vmstate to go on (same problem exists if
> -S is used throughout so the VM never starts and without block migration).

My long-term plan is to put all the json dumps in the git tree, so
comparing these jsons at release-time becomes easy.

I've backported the -dump-vmstate patches to qemu-1.0, the backport is
here:

http://paste.fedoraproject.org/119724/14060126/

(I'll be publishing these to my git repo soon.)

and the json dump is here:

http://paste.fedoraproject.org/119725/01268914/

Comparing qemu-1.0 to qemu-2.1 -M pc-1.0 gives a few errors:

$ ./scripts/vmstate-static-checker.py -s v1.0-1.0.json -d upstream-20140722-v1.0.json 
Section "xio3130-downstream" Description "xio3130-express-downstream-port": version error: 2 > 0
Section "xio3130-downstream": Description "PCIDevice" missing, got "PCIEDevice" instead; skipping
Section "usb-ccid", Description "usb-ccid": expected field "abProtocolDataStructure", got "abProtocolDataStructure.data"; skipping rest
Section "ich9-ahci": Description "ahci" missing, got "ich9_ahci" instead; skipping
Section "usb-host" Section "usb-host" Description "usb-host": minimum version error: 0 < 1
Section "ich9-usb-ehci1" Section "ich9-usb-ehci1" Description "ehci": minimum version error: 0 < 1
Section "x3130-upstream" Description "xio3130-express-upstream-port": version error: 2 > 0
Section "x3130-upstream": Description "PCIDevice" missing, got "PCIEDevice" instead; skipping
Section "vmware-svga", Description "vmware_vga": expected field "card", got "parent_obj"; skipping rest
Section "usb-ehci" Section "usb-ehci" Description "ehci": minimum version error: 0 < 1
Section "apic", Description "apic": expected field "timer", got "timer_expiry"; skipping rest
Section "ioh3420" Description "ioh-3240-express-root-port": version error: 2 > 0
Section "ioh3420": Description "PCIDevice" missing, got "PCIEDevice" instead; skipping
Section "isa-pit", Description "i8254": expected field "channels", got "channels[0].irq_disabled"; skipping rest
Section "PIIX4_PM" Section "PIIX4_PM" Description "piix4_pm": minimum version error: 2 < 3
Section "PIIX4_PM", Description "piix4_pm": expected field "pm1a.sts", got "ar.pm1.evt.sts"; skipping rest
Section "lsi53c895a", Description "lsiscsi": expected field "dev", got "parent_obj"; skipping rest
Section "mc146818rtc", Description "mc146818rtc": expected field "current_tm.tm_sec", got "unused"; skipping rest

Of these, section renames can be handled by whitelisting them in the
script.  The version and minimum_version errors will have to be looked
at in detail.

Hopefully you can use these results and share your findings.


		Amit

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Amit Shah]

On (Tue) 22 Jul 2014 [12:41:30], Amit Shah wrote:
> On (Sat) 19 Jul 2014 [12:37:31], Alex Bligh wrote:
> > Paolo,
> > 
> > On 19 Jul 2014, at 11:53, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> <snip>
> 
> > >> However, after the block migration is finished, I now see:
> > >> 
> > >> Receiving block device images
> > >> Completed 100 %
> > >> Unknown savevm section type 255
> > >> load of migration failed
> > > 
> > > This could be another incompatibility between qemu-kvm and qemu.  I know
> > > we had a couple in Fedora.
> > > 
> > >> My guess is this might be the bogus inclusion of PITCommonState
> > >> per the last hunk of
> > >> http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20
> > >> 
> > >> Is there a technique to debug this sort of thing?
> > > 
> > > You can try using Amit's vmstate checker.
> > 
> > I think this requires a json output in the format given by -dump-vmstate
> > (unless we're at cross purposes). qemu 1.0 does not produce such a JSON
> > output. I only have the vmstate to go on (same problem exists if
> > -S is used throughout so the VM never starts and without block migration).
> 
> My long-term plan is to put all the json dumps in the git tree, so
> comparing these jsons at release-time becomes easy.
> 
> I've backported the -dump-vmstate patches to qemu-1.0, the backport is
> here:
> 
> http://paste.fedoraproject.org/119724/14060126/
> 
> (I'll be publishing these to my git repo soon.)
> 
> and the json dump is here:
> 
> http://paste.fedoraproject.org/119725/01268914/
> 
> Comparing qemu-1.0 to qemu-2.1 -M pc-1.0 gives a few errors:
> 
> $ ./scripts/vmstate-static-checker.py -s v1.0-1.0.json -d upstream-20140722-v1.0.json 
> Section "xio3130-downstream" Description "xio3130-express-downstream-port": version error: 2 > 0
> Section "xio3130-downstream": Description "PCIDevice" missing, got "PCIEDevice" instead; skipping
> Section "usb-ccid", Description "usb-ccid": expected field "abProtocolDataStructure", got "abProtocolDataStructure.data"; skipping rest
> Section "ich9-ahci": Description "ahci" missing, got "ich9_ahci" instead; skipping
> Section "usb-host" Section "usb-host" Description "usb-host": minimum version error: 0 < 1
> Section "ich9-usb-ehci1" Section "ich9-usb-ehci1" Description "ehci": minimum version error: 0 < 1
> Section "x3130-upstream" Description "xio3130-express-upstream-port": version error: 2 > 0
> Section "x3130-upstream": Description "PCIDevice" missing, got "PCIEDevice" instead; skipping
> Section "vmware-svga", Description "vmware_vga": expected field "card", got "parent_obj"; skipping rest
> Section "usb-ehci" Section "usb-ehci" Description "ehci": minimum version error: 0 < 1
> Section "apic", Description "apic": expected field "timer", got "timer_expiry"; skipping rest
> Section "ioh3420" Description "ioh-3240-express-root-port": version error: 2 > 0
> Section "ioh3420": Description "PCIDevice" missing, got "PCIEDevice" instead; skipping
> Section "isa-pit", Description "i8254": expected field "channels", got "channels[0].irq_disabled"; skipping rest
> Section "PIIX4_PM" Section "PIIX4_PM" Description "piix4_pm": minimum version error: 2 < 3
> Section "PIIX4_PM", Description "piix4_pm": expected field "pm1a.sts", got "ar.pm1.evt.sts"; skipping rest
> Section "lsi53c895a", Description "lsiscsi": expected field "dev", got "parent_obj"; skipping rest
> Section "mc146818rtc", Description "mc146818rtc": expected field "current_tm.tm_sec", got "unused"; skipping rest
> 
> Of these, section renames can be handled by whitelisting them in the
> script.  The version and minimum_version errors will have to be looked
> at in detail.

These are the ones that remain after fixing up the rest:

Section "xio3130-downstream" Description "xio3130-express-downstream-port": version error: 2 > 0
Section "usb-host" Section "usb-host" Description "usb-host": minimum version error: 0 < 1
Section "ich9-usb-ehci1" Section "ich9-usb-ehci1" Description "ehci": minimum version error: 0 < 1
Section "x3130-upstream" Description "xio3130-express-upstream-port": version error: 2 > 0
Section "usb-ehci" Section "usb-ehci" Description "ehci": minimum version error: 0 < 1
Section "ioh3420" Description "ioh-3240-express-root-port": version error: 2 > 0
Section "isa-pit", Description "i8254": expected field "channels", got "channels[0].irq_disabled"; skipping rest
Section "PIIX4_PM" Section "PIIX4_PM" Description "piix4_pm": minimum version error: 2 < 3


		Amit

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Paolo Bonzini]

Il 22/07/2014 11:50, Amit Shah ha scritto:
> These are the ones that remain after fixing up the rest:
> 
> Section "xio3130-downstream" Description "xio3130-express-downstream-port": version error: 2 > 0
> Section "usb-host" Section "usb-host" Description "usb-host": minimum version error: 0 < 1
> Section "ich9-usb-ehci1" Section "ich9-usb-ehci1" Description "ehci": minimum version error: 0 < 1
> Section "x3130-upstream" Description "xio3130-express-upstream-port": version error: 2 > 0
> Section "usb-ehci" Section "usb-ehci" Description "ehci": minimum version error: 0 < 1
> Section "ioh3420" Description "ioh-3240-express-root-port": version error: 2 > 0
> Section "isa-pit", Description "i8254": expected field "channels", got "channels[0].irq_disabled"; skipping rest
> Section "PIIX4_PM" Section "PIIX4_PM" Description "piix4_pm": minimum version error: 2 < 3

What about qemu-kvm 1.0 vs qemu 2.1?

Paolo

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Amit Shah]

On (Tue) 22 Jul 2014 [11:55:29], Paolo Bonzini wrote:
> Il 22/07/2014 11:50, Amit Shah ha scritto:
> > These are the ones that remain after fixing up the rest:
> > 
> > Section "xio3130-downstream" Description "xio3130-express-downstream-port": version error: 2 > 0
> > Section "usb-host" Section "usb-host" Description "usb-host": minimum version error: 0 < 1
> > Section "ich9-usb-ehci1" Section "ich9-usb-ehci1" Description "ehci": minimum version error: 0 < 1
> > Section "x3130-upstream" Description "xio3130-express-upstream-port": version error: 2 > 0
> > Section "usb-ehci" Section "usb-ehci" Description "ehci": minimum version error: 0 < 1
> > Section "ioh3420" Description "ioh-3240-express-root-port": version error: 2 > 0
> > Section "isa-pit", Description "i8254": expected field "channels", got "channels[0].irq_disabled"; skipping rest
> > Section "PIIX4_PM" Section "PIIX4_PM" Description "piix4_pm": minimum version error: 2 < 3
> 
> What about qemu-kvm 1.0 vs qemu 2.1?

Just one extra failure:

Section "pci-assign" does not exist in dest

Also interesting is qemu 1.0 -> qemu-kvm-1.0:

Section "i6300esb" Section "i6300esb" Description "i6300esb_wdt": minimum version error: 1336 < 1368
Section "isa-pit", Description "i8254": expected field "channels", got "flags"; skipping rest

And qemu-kvm-1.0 to qemu-1.0:

Section "kvmclock" does not exist in dest
Section "i6300esb" version error: 1368 > 1336
Section "i6300esb" Description "i6300esb_wdt": version error: 1368 > 1336
Section "isa-pit", Description "i8254": expected field "flags", got "channels"; skipping rest
Section "pci-assign" does not exist in dest



		Amit

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

On 22 Jul 2014, at 11:22, Amit Shah wrote:

> Also interesting is qemu 1.0 -> qemu-kvm-1.0:
> 
> Section "isa-pit", Description "i8254": expected field "channels", got "flags"; skipping rest

This one, together with the PIIX4 one (which for some reason doesn't
show up) where the two I hit, after manually fixing the rom sizes
stuff on the command line.

Apparently "flags" and "channels" are pseudonyms. The piix4_m thing
is that qemu-kvm-1.0 is actually using version 3 but says it's
using version 2, which is perhaps why it doesn't show up in your
test.

-- 
Alex Bligh

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Amit Shah]

On (Tue) 22 Jul 2014 [11:32:41], Alex Bligh wrote:
> 
> On 22 Jul 2014, at 11:22, Amit Shah wrote:
> 
> > Also interesting is qemu 1.0 -> qemu-kvm-1.0:
> > 
> > Section "isa-pit", Description "i8254": expected field "channels", got "flags"; skipping rest
> 
> This one, together with the PIIX4 one (which for some reason doesn't
> show up) where the two I hit, after manually fixing the rom sizes
> stuff on the command line.
> 
> Apparently "flags" and "channels" are pseudonyms.

No, they're not; flags is an extra 4-byte param in qemu-kvm-1.0, which
doesn't exist in qemu-1.0.

In qemu-2.1 -M 1.0, "flags" got renamed to "channels[0].irq_disabled",
which is also flagged.

> The piix4_m thing
> is that qemu-kvm-1.0 is actually using version 3 but says it's
> using version 2, which is perhaps why it doesn't show up in your
> test.

You're looking at the wrong output.  You should see one message back,
where I compare qemu-1.0 with qemu-2.1 -M pc-1.0.  That has the
piix4_pm version mismatch flagged.


		Amit

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

On 22 Jul 2014, at 11:54, Amit Shah <amit.shah@redhat.com> wrote:
>> 
>> This one, together with the PIIX4 one (which for some reason doesn't
>> show up) where the two I hit, after manually fixing the rom sizes
>> stuff on the command line.
>> 
>> Apparently "flags" and "channels" are pseudonyms.
> 
> No, they're not; flags is an extra 4-byte param in qemu-kvm-1.0, which
> doesn't exist in qemu-1.0.
> 
> In qemu-2.1 -M 1.0, "flags" got renamed to "channels[0].irq_disabled",
> which is also flagged.

I was basing that on the comment in Cole Robinson's patch here:
 http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20

--- a/hw/timer/i8254_common.c
+++ b/hw/timer/i8254_common.c
@@ -267,7 +267,12 @@ static const VMStateDescription vmstate_pit_common = {
     .pre_save = pit_dispatch_pre_save,
     .post_load = pit_dispatch_post_load,
     .fields = (VMStateField[]) {
-        VMSTATE_UINT32_V(channels[0].irq_disabled, PITCommonState, 3),
+        /* qemu-kvm version_id=2 had 'flags' here which is equivalent
+         * This fixes incoming migration from qemu-kvm 1.0, but breaks
+         * incoming migration from qemu < 1.1
+         */
+        //VMSTATE_UINT32_V(channels[0].irq_disabled, PITCommonState, 3),
+        VMSTATE_UINT32(channels[0].irq_disabled, PITCommonState),
         VMSTATE_STRUCT_ARRAY(channels, PITCommonState, 3, 2,
                              vmstate_pit_channel, PITChannelState),
         VMSTATE_INT64(channels[0].next_transition_time,

I think what this means is that flags is equivalent to channels[0].irq_disabled,
yes?

>> The piix4_m thing
>> is that qemu-kvm-1.0 is actually using version 3 but says it's
>> using version 2, which is perhaps why it doesn't show up in your
>> test.
> 
> You're looking at the wrong output.  You should see one message back,
> where I compare qemu-1.0 with qemu-2.1 -M pc-1.0.  That has the
> piix4_pm version mismatch flagged.

Don't think I am

a) because I'm using qemu-kvm-1.0, not qemu-1.0. You looked at qemu-1.0
   in the previous message; that's not directly relevant for what I'm
   looking at as my source is qemu-kvm-1.0 (qemu vs qemu-kvm).

b) because I wasn't actually using your Json, but testing what made the
   thing load in real life!

-- 
Alex Bligh

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Amit Shah]

On (Tue) 22 Jul 2014 [12:38:14], Alex Bligh wrote:
> 
> On 22 Jul 2014, at 11:54, Amit Shah <amit.shah@redhat.com> wrote:
> >> 
> >> This one, together with the PIIX4 one (which for some reason doesn't
> >> show up) where the two I hit, after manually fixing the rom sizes
> >> stuff on the command line.
> >> 
> >> Apparently "flags" and "channels" are pseudonyms.
> > 
> > No, they're not; flags is an extra 4-byte param in qemu-kvm-1.0, which
> > doesn't exist in qemu-1.0.
> > 
> > In qemu-2.1 -M 1.0, "flags" got renamed to "channels[0].irq_disabled",
> > which is also flagged.
> 
> I was basing that on the comment in Cole Robinson's patch here:
>  http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20
> 
> --- a/hw/timer/i8254_common.c
> +++ b/hw/timer/i8254_common.c
> @@ -267,7 +267,12 @@ static const VMStateDescription vmstate_pit_common = {
>      .pre_save = pit_dispatch_pre_save,
>      .post_load = pit_dispatch_post_load,
>      .fields = (VMStateField[]) {
> -        VMSTATE_UINT32_V(channels[0].irq_disabled, PITCommonState, 3),
> +        /* qemu-kvm version_id=2 had 'flags' here which is equivalent
> +         * This fixes incoming migration from qemu-kvm 1.0, but breaks
> +         * incoming migration from qemu < 1.1
> +         */
> +        //VMSTATE_UINT32_V(channels[0].irq_disabled, PITCommonState, 3),
> +        VMSTATE_UINT32(channels[0].irq_disabled, PITCommonState),
>          VMSTATE_STRUCT_ARRAY(channels, PITCommonState, 3, 2,
>                               vmstate_pit_channel, PITChannelState),
>          VMSTATE_INT64(channels[0].next_transition_time,
> 
> I think what this means is that flags is equivalent to channels[0].irq_disabled,
> yes?

Yes, that's what I said above.

> >> The piix4_m thing
> >> is that qemu-kvm-1.0 is actually using version 3 but says it's
> >> using version 2, which is perhaps why it doesn't show up in your
> >> test.
> > 
> > You're looking at the wrong output.  You should see one message back,
> > where I compare qemu-1.0 with qemu-2.1 -M pc-1.0.  That has the
> > piix4_pm version mismatch flagged.
> 
> Don't think I am
> 
> a) because I'm using qemu-kvm-1.0, not qemu-1.0. You looked at qemu-1.0
>    in the previous message; that's not directly relevant for what I'm
>    looking at as my source is qemu-kvm-1.0 (qemu vs qemu-kvm).

Sigh; please read both my replies.

The only difference between qemu-kvm-1.0 and qemu-1.0 is the presence
of the pci-assign section in qemu-kvm-1.0.  All other output from
qemu-1.0 -> qemu-2.1 is equally applicable to qemu-kvm-1.0 ->
qemu-2.1.

> b) because I wasn't actually using your Json, but testing what made the
>    thing load in real life!

I know that; but you brought up the comparison of your experiments
with the output of the checker.  I just pointed out the checker
flagged both those things you saw (and a few others too).


		Amit

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Paolo Bonzini]

Il 22/07/2014 13:54, Amit Shah ha scritto:
>> > a) because I'm using qemu-kvm-1.0, not qemu-1.0. You looked at qemu-1.0
>> >    in the previous message; that's not directly relevant for what I'm
>> >    looking at as my source is qemu-kvm-1.0 (qemu vs qemu-kvm).
> Sigh; please read both my replies.
> 
> The only difference between qemu-kvm-1.0 and qemu-1.0 is the presence
> of the pci-assign section in qemu-kvm-1.0.  All other output from
> qemu-1.0 -> qemu-2.1 is equally applicable to qemu-kvm-1.0 ->
> qemu-2.1.

Not really true.

qemu 1.0 didn't have neither channels[0].irq_disabled nor flags and is
version 2.

qemu-2.1 has channels[0].irq_disabled and is version 3.

qemu-kvm-1.0 had flags, which is now called channels[0].irq_disabled,
and is version 2.

qemu-2.1 loads version 2 as something that doesn't have
channels[0].irq_disabled.  So it breaks if you feed it with qemu-kvm-1.0
data.

There's something similar going on with PIIX4_PM but I don't remember
the details.

Paolo

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

On 22 Jul 2014, at 13:12, Paolo Bonzini <pbonzini@redhat.com> wrote:

> There's something similar going on with PIIX4_PM but I don't remember
> the details.

From memory:

* qemu-1.0 uses the v2 format
* qemu-kvm-1.0 uses the v3 format but advertises itself as v2
* qemu-2.1 uses the v3 format

I don't quite understand whether Amit's checker is designed to pick
up all three situations as different, or whether it (quite reasonably)
expects people to play ball with version numbers.

-- 
Alex Bligh

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Amit Shah]

On (Tue) 22 Jul 2014 [13:19:43], Alex Bligh wrote:
> 
> On 22 Jul 2014, at 13:12, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> > There's something similar going on with PIIX4_PM but I don't remember
> > the details.
> 
> From memory:
> 
> * qemu-1.0 uses the v2 format
> * qemu-kvm-1.0 uses the v3 format but advertises itself as v2
> * qemu-2.1 uses the v3 format

Yea; I vaguely recall that happening.  Quite unfortunate.

> I don't quite understand whether Amit's checker is designed to pick
> up all three situations as different, or whether it (quite reasonably)
> expects people to play ball with version numbers.

This is what it says:

Section "PIIX4_PM" Section "PIIX4_PM" Description "piix4_pm": minimum version error: 2 < 3


Which means qemu-2.1 expects a minimum version of 3, but qemu-kvm-1.0
is sending it version 2.  Now even though they may be compatible,
version-number-wise they're not, as the dest qemu will also complain
about.

		Amit

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Amit Shah]

On (Tue) 22 Jul 2014 [14:12:00], Paolo Bonzini wrote:
> Il 22/07/2014 13:54, Amit Shah ha scritto:
> >> > a) because I'm using qemu-kvm-1.0, not qemu-1.0. You looked at qemu-1.0
> >> >    in the previous message; that's not directly relevant for what I'm
> >> >    looking at as my source is qemu-kvm-1.0 (qemu vs qemu-kvm).
> > Sigh; please read both my replies.
> > 
> > The only difference between qemu-kvm-1.0 and qemu-1.0 is the presence
> > of the pci-assign section in qemu-kvm-1.0.  All other output from
> > qemu-1.0 -> qemu-2.1 is equally applicable to qemu-kvm-1.0 ->
> > qemu-2.1.
> 
> Not really true.

Here's the complete output:

Section "xio3130-downstream" Description "xio3130-express-downstream-port": version error: 2 > 0
Section "usb-host" Section "usb-host" Description "usb-host": minimum version error: 0 < 1
Section "ich9-usb-ehci1" Section "ich9-usb-ehci1" Description "ehci": minimum version error: 0 < 1
Section "x3130-upstream" Description "xio3130-express-upstream-port": version error: 2 > 0
Section "usb-ehci" Section "usb-ehci" Description "ehci": minimum version error: 0 < 1
Section "ioh3420" Description "ioh-3240-express-root-port": version error: 2 > 0
Section "pci-assign" does not exist in dest
Section "PIIX4_PM" Section "PIIX4_PM" Description "piix4_pm": minimum version error: 2 < 3

> qemu 1.0 didn't have neither channels[0].irq_disabled nor flags and is
> version 2.
>
> qemu-2.1 has channels[0].irq_disabled and is version 3.
> 
> qemu-kvm-1.0 had flags, which is now called channels[0].irq_disabled,
> and is version 2.

Right - 'flags' here is renamed, and doesn't appear in that output;
but the src version was 0 according to the json output.

> qemu-2.1 loads version 2 as something that doesn't have
> channels[0].irq_disabled.  So it breaks if you feed it with qemu-kvm-1.0
> data.

OK - that's not something the static checker can flag, though..

> There's something similar going on with PIIX4_PM but I don't remember
> the details.

According to the output above, minimum version that qemu-2.1 accepts
changed to 3, and qemu-kvm-1.0 sends version 2.


		Amit

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

On 22 Jul 2014, at 12:54, Amit Shah <amit.shah@redhat.com> wrote:

> Sigh; please read both my replies.
> 
> The only difference between qemu-kvm-1.0 and qemu-1.0 is the presence
> of the pci-assign section in qemu-kvm-1.0.  All other output from
> qemu-1.0 -> qemu-2.1 is equally applicable to qemu-kvm-1.0 ->
> qemu-2.1.
...
> I know that; but you brought up the comparison of your experiments
> with the output of the checker.  I just pointed out the checker
> flagged both those things you saw (and a few others too).

Apologies for my naivety here as it's the first time I've looked
at this stuff. So sorry if I'm getting the wrong end of the stick.

However, am I correct in assuming your JSON checker is only
checking for the field structure, and not for things like
changed defaults? For instance the cirrus vga video ram size
appears to be different between qemu-kvm-1.0 and qemu-1.0,
but this isn't something brought up in your email highlighting
the difference between a 2.1->qemu-1.0 comparison and a
2.1->qemu-kvm-1.0 comparison. I know you haven't directly
compared qemu-1.0 and qemu-1.0-kvm, but if this difference
does exist (which I think it does) I'd expect to see it
in the list of differences (i.e. your second email), if your
tool looks at that too.

-- 
Alex Bligh

Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Amit Shah]

On (Tue) 22 Jul 2014 [13:15:41], Alex Bligh wrote:
> 
> On 22 Jul 2014, at 12:54, Amit Shah <amit.shah@redhat.com> wrote:
> 
> > Sigh; please read both my replies.
> > 
> > The only difference between qemu-kvm-1.0 and qemu-1.0 is the presence
> > of the pci-assign section in qemu-kvm-1.0.  All other output from
> > qemu-1.0 -> qemu-2.1 is equally applicable to qemu-kvm-1.0 ->
> > qemu-2.1.
> ...
> > I know that; but you brought up the comparison of your experiments
> > with the output of the checker.  I just pointed out the checker
> > flagged both those things you saw (and a few others too).
> 
> Apologies for my naivety here as it's the first time I've looked
> at this stuff. So sorry if I'm getting the wrong end of the stick.
> 
> However, am I correct in assuming your JSON checker is only
> checking for the field structure, and not for things like
> changed defaults?

It only looks at the vmstate structs.

> For instance the cirrus vga video ram size
> appears to be different between qemu-kvm-1.0 and qemu-1.0,

Right, this won't get flagged.

> but this isn't something brought up in your email highlighting
> the difference between a 2.1->qemu-1.0 comparison and a
> 2.1->qemu-kvm-1.0 comparison. I know you haven't directly
> compared qemu-1.0 and qemu-1.0-kvm, but if this difference
> does exist (which I think it does) I'd expect to see it
> in the list of differences (i.e. your second email), if your
> tool looks at that too.

I did compare qemu-1.0 and qemu-kvm-1.0, but this difference won't pop
up.

The other differences that the checker spotted may not be relevant to
you, as your VMs don't have those devices enabled.

		Amit