1.1.1 -> 1.1.2 migrate /managedsave issue

1.1.1 -> 1.1.2 migrate /managedsave issue

[Doug Goldstein]

I'm using libvirt 0.10.2 and I had qemu-kvm 1.1.1 running all my VMs.
I used libvirt's managedsave command to pause all the VMs and write
them to disk and when I brought up the machine again I had upgraded to
qemu-kvm 1.1.2 and attempted to resume the VMs from their state. It
unfortunately fails. During the life of the VM I did not attempt to
adjust the amount of memory it had via the balloon device unless of
course libvirt did behind the scenes on me. Below is the command line
invocation and the error:

LC_ALL=C PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin
HOME=/root USER=root QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -name expo
-S -M pc-1.0 -cpu
Penryn,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid
19034754-aa3f-9671-d247-1bc53134e3f0 -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/expo.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/lib/libvirt/images/expo.img,if=none,id=drive-ide0-0-0,format=raw,cache=none
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-netdev tap,fd=23,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:0b:29:d9,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-spice port=5901,addr=127.0.0.1,disable-ticketing -vga qxl -global
qxl-vga.vram_size=67108864 -incoming fd:20 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/7
qemu: warning: error while loading state for instance 0x0 of device 'ram'
load of migration failed

Let me know what specifics I can provide to make this easier to debug.

Thanks.
-- 
Doug Goldstein

Re: 1.1.1 -> 1.1.2 migrate /managedsave issue

[Philipp Hahn]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

Hello Doug,

On Saturday 20 October 2012 00:46:43 Doug Goldstein wrote:
> I'm using libvirt 0.10.2 and I had qemu-kvm 1.1.1 running all my VMs.
...
> I had upgraded to qemu-kvm 1.1.2
... 
> qemu: warning: error while loading state for instance 0x0 of device 'ram'
> load of migration failed

That error can be from many things. For me it was that the PXE-ROM images for 
the network cards were updated as well. Their size changed over the next 
power-of-two size, so kvm needed to allocate less/more memory and changed 
some PCI configuration registers, where the size of the ROM region is stored.
On loading the saved state those sizes were compared and failed to validate. 
KVM then aborts loading the saved state with that little helpful message.

So you might want to check, if your case is similar to mine.

I diagnosed that using gdb to single step kvm until I found 
hw/pci.c#get_pci_config_device() returning -EINVAL.

Hope that helps.

Sincerely
Philipp
-- 
Philipp Hahn           Open Source Software Engineer      hahn@univention.de
Univention GmbH        be open.                       fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen                 fax: +49 421 22 232-99
                                                   http://www.univention.de/

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: 1.1.1 -> 1.1.2 migrate /managedsave issue

[Avi Kivity]

On 10/22/2012 09:04 AM, Philipp Hahn wrote:
> Hello Doug,
> 
> On Saturday 20 October 2012 00:46:43 Doug Goldstein wrote:
>> I'm using libvirt 0.10.2 and I had qemu-kvm 1.1.1 running all my VMs.
> ...
>> I had upgraded to qemu-kvm 1.1.2
> ... 
>> qemu: warning: error while loading state for instance 0x0 of device 'ram'
>> load of migration failed
> 
> That error can be from many things. For me it was that the PXE-ROM images for 
> the network cards were updated as well. Their size changed over the next 
> power-of-two size, so kvm needed to allocate less/more memory and changed 
> some PCI configuration registers, where the size of the ROM region is stored.
> On loading the saved state those sizes were compared and failed to validate. 
> KVM then aborts loading the saved state with that little helpful message.
> 
> So you might want to check, if your case is similar to mine.
> 
> I diagnosed that using gdb to single step kvm until I found 
> hw/pci.c#get_pci_config_device() returning -EINVAL.
> 

Seems reasonable.  Doug, please verify to see if it's the same issue or
another one.

Juan, how can we fix this?  It's clear that the option ROM size has to
be fixed and not change whenever the blob is updated.  This will fix it
for future releases.  But what to do about the ones in the field?

-- 
error compiling committee.c: too many arguments to function

Re: 1.1.1 -> 1.1.2 migrate /managedsave issue

[Doug Goldstein]

On Mon, Oct 22, 2012 at 6:23 AM, Avi Kivity <avi@redhat.com> wrote:
> On 10/22/2012 09:04 AM, Philipp Hahn wrote:
>> Hello Doug,
>>
>> On Saturday 20 October 2012 00:46:43 Doug Goldstein wrote:
>>> I'm using libvirt 0.10.2 and I had qemu-kvm 1.1.1 running all my VMs.
>> ...
>>> I had upgraded to qemu-kvm 1.1.2
>> ...
>>> qemu: warning: error while loading state for instance 0x0 of device 'ram'
>>> load of migration failed
>>
>> That error can be from many things. For me it was that the PXE-ROM images for
>> the network cards were updated as well. Their size changed over the next
>> power-of-two size, so kvm needed to allocate less/more memory and changed
>> some PCI configuration registers, where the size of the ROM region is stored.
>> On loading the saved state those sizes were compared and failed to validate.
>> KVM then aborts loading the saved state with that little helpful message.
>>
>> So you might want to check, if your case is similar to mine.
>>
>> I diagnosed that using gdb to single step kvm until I found
>> hw/pci.c#get_pci_config_device() returning -EINVAL.
>>
>
> Seems reasonable.  Doug, please verify to see if it's the same issue or
> another one.

Sorry it took a little bit to juggle the break points with libvirt and
qemu to get gdb attached correctly. But yes I can confirm that
vmstate_load_state() which is calling field->info->get() which is
calling get_pci_config_device() is returning -EINVAL.

>
> Juan, how can we fix this?  It's clear that the option ROM size has to
> be fixed and not change whenever the blob is updated.  This will fix it
> for future releases.  But what to do about the ones in the field?
>

Any recommendations to fix this? Or do I need to kill the saved state
and start over?

Thanks.
-- 
Doug Goldstein

Re: 1.1.1 -> 1.1.2 migrate /managedsave issue

[Philipp Hahn]

[-- Attachment #1.1: Type: text/plain, Size: 2323 bytes --]

Hello Doug,

On Tuesday 23 October 2012 22:38:47 Doug Goldstein wrote:
> >>> qemu: warning: error while loading state for instance 0x0 of device
> >>> 'ram' load of migration failed
...
> >> I diagnosed that using gdb to single step kvm until I found
> >> hw/pci.c#get_pci_config_device() returning -EINVAL.
...
> But yes I can confirm that
> vmstate_load_state() which is calling field->info->get() which is
> calling get_pci_config_device() is returning -EINVAL.
...
> Any recommendations to fix this? Or do I need to kill the saved state
> and start over?

For start try to re-get the old PXE ROM files, that is for Debian re-install 
the old etherboot-qemu or an older Version of the ipxe-qemu package. If that 
then works again, you can go further. Otherwise you need to get the exact 
index "i" where get_pci_config_device() returns -EINVAL and match that with 
the PCI spec. For short see hw/pci_regs.h.


For our Debian based UCS distribution I patched qemu(-kvm) to use the old 
image files (which are still available and installed in parallel to the new 
files) if the pc-level is 0.14 or lower (which is the version we currently 
ship; only our next release will switch to 1.1.2).

I've attached my patch FYI. That is only a work around until QEMU provides a 
real solution. If you've used an e1000, that is not enougth because some more 
PCI configuration bits (PCI_STATUS_CAP_LIST) were changed in an incompatible 
way.

If I remember correctly there was a different bug report, where changed PCI 
bits were also responsible for breaking saved states: Some PCI devices were 
changed to multi-function devices (or the other way around), which also 
breaks loading of previous saved stated. I think it was a bug in libvirt 
which started adding an explicit "multifunction=on", while the save was done 
using an implicit "multifinction=off". 
(<https://forge.univention.org/bugzilla/show_bug.cgi?id=22877#c6> in our 
German BZ)

Sincerely
Philipp Hahn
-- 
Philipp Hahn           Open Source Software Engineer      hahn@univention.de
Univention GmbH        be open.                       fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen                 fax: +49 421 22 232-99
                                                   http://www.univention.de/

[-- Attachment #1.2: 0002-Bug-24702-Rom-file-compatibility.patch --]
[-- Type: text/x-diff, Size: 2942 bytes --]

diff -urN qemu-kvm-1.1.2+dfsg.orig/debian/patches/romfile-compatibility.diff qemu-kvm-1.1.2+dfsg/debian/patches/romfile-compatibility.diff
--- qemu-kvm-1.1.2+dfsg.orig/debian/patches/romfile-compatibility.diff	1970-01-01 01:00:00.000000000 +0100
+++ qemu-kvm-1.1.2+dfsg/debian/patches/romfile-compatibility.diff	2012-10-08 14:25:09.225656404 +0200
@@ -0,0 +1,39 @@
+Bug #24702: Fix PXE ROM size mismatch
+
+For pc-0.14 and older use the PXE ROMs from the deprecated etherboot-qemu
+package, which have different sizes. Without the patch the virtual PC gets
+build with the current ROM sizes. Loading then fails because the stored PCI ROM
+BAR size mismatches the configured size.
+--- a/hw/pc_piix.c
++++ b/hw/pc_piix.c
+@@ -433,6 +433,30 @@ static QEMUMachine pc_machine_v0_15 = {
+             .driver   = "virtio-balloon-pci",\
+             .property = "event_idx",\
+             .value    = "off",\
++        },{\
++            .driver   = "ne2000",\
++            .property = "romfile",\
++            .value   = "/usr/lib/etherboot/rtl8029.rom",\
++        },{\
++            .driver   = "rtl8139",\
++            .property = "romfile",\
++            .value   = "/usr/lib/etherboot/rtl8139.rom",\
++        },{\
++            .driver   = "e1000",\
++            .property = "romfile",\
++            .value   = "/usr/lib/etherboot/e1000-82540em.rom",\
++        },{\
++            .driver   = "pcnet",\
++            .property = "romfile",\
++            .value   = "/usr/lib/etherboot/pcnet32.rom",\
++        },{\
++            .driver   = "ne2k-isa",\
++            .property = "romfile",\
++            .value   = "/usr/lib/etherboot/ne.rom",\
++        },{\
++            .driver   = "virtio-net-pci",\
++            .property = "romfile",\
++            .value   = "/usr/lib/etherboot/virtio-net.rom",\
+         }
+ 
+ static QEMUMachine pc_machine_v0_14 = {
diff -urN qemu-kvm-1.1.2+dfsg.orig/debian/patches/series qemu-kvm-1.1.2+dfsg/debian/patches/series
--- qemu-kvm-1.1.2+dfsg.orig/debian/patches/series	2012-09-10 12:14:05.000000000 +0200
+++ qemu-kvm-1.1.2+dfsg/debian/patches/series	2012-10-08 14:24:51.497155189 +0200
@@ -8,3 +8,4 @@
 
 net-add--netdev-options-to-man-page.patch
 revert-serial-fix-retry-logic.patch
+romfile-compatibility.diff
diff -urN qemu-kvm-1.1.2+dfsg.orig/debian/control qemu-kvm-1.1.2+dfsg/debian/control
--- qemu-kvm-1.1.2+dfsg.orig/debian/control	2012-09-11 08:29:43.000000000 +0200
+++ qemu-kvm-1.1.2+dfsg/debian/control	2012-10-08 15:06:39.201155102 +0200
@@ -37,7 +37,8 @@
  seabios (>> 1.7.0~), vgabios (>= 0.6c-3~),
  qemu-keymaps, qemu-utils (>> 0.14.0),
  ipxe-qemu | ipxe (<< 1.0.0+git-20120202.f6840ba-2)
-Recommends: bridge-utils, iproute
+Recommends: bridge-utils, iproute,
+ etherboot-qemu
 Suggests: debootstrap, vde2, samba
 Provides: kvm
 Conflicts: kvm-source (<= 18-1), kvm-data (<= 66+dfsg-1.1), kvm (<< 1:0)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: 1.1.1 -> 1.1.2 migrate /managedsave issue

[Doug Goldstein]

On Mon, Oct 22, 2012 at 6:23 AM, Avi Kivity <avi@redhat.com> wrote:
> On 10/22/2012 09:04 AM, Philipp Hahn wrote:
>> Hello Doug,
>>
>> On Saturday 20 October 2012 00:46:43 Doug Goldstein wrote:
>>> I'm using libvirt 0.10.2 and I had qemu-kvm 1.1.1 running all my VMs.
>> ...
>>> I had upgraded to qemu-kvm 1.1.2
>> ...
>>> qemu: warning: error while loading state for instance 0x0 of device 'ram'
>>> load of migration failed
>>
>> That error can be from many things. For me it was that the PXE-ROM images for
>> the network cards were updated as well. Their size changed over the next
>> power-of-two size, so kvm needed to allocate less/more memory and changed
>> some PCI configuration registers, where the size of the ROM region is stored.
>> On loading the saved state those sizes were compared and failed to validate.
>> KVM then aborts loading the saved state with that little helpful message.
>>
>> So you might want to check, if your case is similar to mine.
>>
>> I diagnosed that using gdb to single step kvm until I found
>> hw/pci.c#get_pci_config_device() returning -EINVAL.
>>
>
> Seems reasonable.  Doug, please verify to see if it's the same issue or
> another one.
>
> Juan, how can we fix this?  It's clear that the option ROM size has to
> be fixed and not change whenever the blob is updated.  This will fix it
> for future releases.  But what to do about the ones in the field?
>
> --
> error compiling committee.c: too many arguments to function

Avi,

Please consider the following patch based off qemu master:

http://article.gmane.org/gmane.comp.emulators.kvm.devel/100231

It should hopefully help users with this issue in the future.

-- 
Doug Goldstein

Re: 1.1.1 -> 1.1.2 migrate /managedsave issue

[Anthony Liguori]

Avi Kivity <avi@redhat.com> writes:

> On 10/22/2012 09:04 AM, Philipp Hahn wrote:
>> Hello Doug,
>> 
>> On Saturday 20 October 2012 00:46:43 Doug Goldstein wrote:
>>> I'm using libvirt 0.10.2 and I had qemu-kvm 1.1.1 running all my VMs.
>> ...
>>> I had upgraded to qemu-kvm 1.1.2
>> ... 
>>> qemu: warning: error while loading state for instance 0x0 of device 'ram'
>>> load of migration failed
>> 
>> That error can be from many things. For me it was that the PXE-ROM images for 
>> the network cards were updated as well. Their size changed over the next 
>> power-of-two size, so kvm needed to allocate less/more memory and changed 
>> some PCI configuration registers, where the size of the ROM region is stored.
>> On loading the saved state those sizes were compared and failed to validate. 
>> KVM then aborts loading the saved state with that little helpful message.
>> 
>> So you might want to check, if your case is similar to mine.
>> 
>> I diagnosed that using gdb to single step kvm until I found 
>> hw/pci.c#get_pci_config_device() returning -EINVAL.
>> 
>
> Seems reasonable.  Doug, please verify to see if it's the same issue or
> another one.
>
> Juan, how can we fix this?  It's clear that the option ROM size has to
> be fixed and not change whenever the blob is updated.  This will fix it
> for future releases.  But what to do about the ones in the field?

This is not a problem upstream because we don't alter the ROMs.  If we
did, we would keep the old ROMs around and set the romfile property in
the compatible machine.

This is what distros that are shipping ROMs outside of QEMU ought to
do.  It's a bug to unconditionally change the ROMs (in a guest visible
way) without adding compatibility support.

Regards,

Anthony Liguori

>
> -- 
> error compiling committee.c: too many arguments to function
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 1.1.1 -> 1.1.2 migrate /managedsave issue

[Doug Goldstein]

On Sun, Nov 4, 2012 at 3:51 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> Avi Kivity <avi@redhat.com> writes:
>
>> On 10/22/2012 09:04 AM, Philipp Hahn wrote:
>>> Hello Doug,
>>>
>>> On Saturday 20 October 2012 00:46:43 Doug Goldstein wrote:
>>>> I'm using libvirt 0.10.2 and I had qemu-kvm 1.1.1 running all my VMs.
>>> ...
>>>> I had upgraded to qemu-kvm 1.1.2
>>> ...
>>>> qemu: warning: error while loading state for instance 0x0 of device 'ram'
>>>> load of migration failed
>>>
>>> That error can be from many things. For me it was that the PXE-ROM images for
>>> the network cards were updated as well. Their size changed over the next
>>> power-of-two size, so kvm needed to allocate less/more memory and changed
>>> some PCI configuration registers, where the size of the ROM region is stored.
>>> On loading the saved state those sizes were compared and failed to validate.
>>> KVM then aborts loading the saved state with that little helpful message.
>>>
>>> So you might want to check, if your case is similar to mine.
>>>
>>> I diagnosed that using gdb to single step kvm until I found
>>> hw/pci.c#get_pci_config_device() returning -EINVAL.
>>>
>>
>> Seems reasonable.  Doug, please verify to see if it's the same issue or
>> another one.
>>
>> Juan, how can we fix this?  It's clear that the option ROM size has to
>> be fixed and not change whenever the blob is updated.  This will fix it
>> for future releases.  But what to do about the ones in the field?
>
> This is not a problem upstream because we don't alter the ROMs.  If we
> did, we would keep the old ROMs around and set the romfile property in
> the compatible machine.
>
> This is what distros that are shipping ROMs outside of QEMU ought to
> do.  It's a bug to unconditionally change the ROMs (in a guest visible
> way) without adding compatibility support.
>
> Regards,
>
> Anthony Liguori
>

Anthony,

Gerd updated seabios on August 7th and before that on April 17. The
default VGA ROM size also changed in recent releases. There are no old
versions of the ROMs included once these updates are performed so a
user building a new version from source will hit this problem. Juan
Quintela even mentioned that he has been bit by this issue and had to
use gdb to track it down as did Philipp that responded earlier in the
thread. The patch is a simple fprintf() which would have saved at
least 3 users the effort of tracking down an issue with gdb. So I urge
you to reconsider.

Thanks.
-- 
Doug Goldstein

Re: 1.1.1 -> 1.1.2 migrate /managedsave issue

[Juan Quintela]

Doug Goldstein <cardoe@gentoo.org> wrote:
> On Sun, Nov 4, 2012 at 3:51 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
>>>
>>> Seems reasonable.  Doug, please verify to see if it's the same issue or
>>> another one.
>>>
>>> Juan, how can we fix this?  It's clear that the option ROM size has to
>>> be fixed and not change whenever the blob is updated.  This will fix it
>>> for future releases.  But what to do about the ones in the field?
>>
>> This is not a problem upstream because we don't alter the ROMs.  If we
>> did, we would keep the old ROMs around and set the romfile property in
>> the compatible machine.
>>
>> This is what distros that are shipping ROMs outside of QEMU ought to
>> do.  It's a bug to unconditionally change the ROMs (in a guest visible
>> way) without adding compatibility support.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>
> Anthony,
>
> Gerd updated seabios on August 7th and before that on April 17. The
> default VGA ROM size also changed in recent releases. There are no old
> versions of the ROMs included once these updates are performed so a
> user building a new version from source will hit this problem. Juan
> Quintela even mentioned that he has been bit by this issue and had to
> use gdb to track it down as did Philipp that responded earlier in the
> thread. The patch is a simple fprintf() which would have saved at
> least 3 users the effort of tracking down an issue with gdb. So I urge
> you to reconsider.

I hit this problem.  But it was a bug, the problem was to detect it.

The problem was doing migration to an old version, we now "round" the
RAM amount to a multiple of 8k.  If your old ram memory was mulitple of
4k, you get this prolbem with migration.

And it only prints "migration failed".  Printing a message telling that:
memory size of %foo is %d and expected %d would have make error trivial
to found.

Later, Juan.

PD.  Problem was really a bit more complex than this, this is a
     simplification.