* [Qemu-devel] Live migration bug, possible missing ram in migration?
@ 2014-12-19 13:15 Frerot, Jean-Sébastien
2014-12-19 13:18 ` Frerot, Jean-Sébastien
2014-12-19 14:40 ` Paolo Bonzini
0 siblings, 2 replies; 7+ messages in thread
From: Frerot, Jean-Sébastien @ 2014-12-19 13:15 UTC (permalink / raw)
To: qemu-devel; +Cc: josh.durgin, uintela
[-- Attachment #1: Type: text/plain, Size: 9985 bytes --]
Hi, I've been using kvm for some time now using live migration as well with
ceph backend. Recently I started running into an issue with only one of my
VM, which happens to be a windows server (2012). When I migrate this
particular VM it seems that not all the ram is transferred. So when the
migration completes, the vm that has been migrated simply hangs and I have
to force a shutdown. Notice that not long ago it was working fine, however
I didn't notice when it started to have the issue.
Notice that my diagnostic of the problem may be wrong, I mean the behavior
i'm having is that my VM hangs after live migration, and I don't see the
problem on other VMs. But I noticed the difference in RAM usage by the qemu
process which leads me to believe that the ram is not fully transferred.
before migration on node2:
15225 libvirt+ 20 0 7128048 4.200g 13592 S 15.0 27.3 1:14.18
qemu-system-x86
libvirt+ 15225 52.7 27.3 7128048 4404244 ? Sl 12:57 1:14
qemu-system-x86_64
after migration on node1:
16507 libvirt+ 20 0 6571864 1.610g 13152 R 7.6 10.5 0:07.63
qemu-system-x86
libvirt+ 16507 15.7 10.4 6571864 1688392 ? Sl 13:00 0:08
qemu-system-x86_64
libvirtd.log on node02:
2014-12-19 13:01:06.654+0000: 6845: warning :
qemuMigrationCancelDriveMirror:1421 : Unable to stop block job on
drive-virtio-disk0
2014-12-19 13:01:06.656+0000: 6845: warning :
qemuMigrationCancelDriveMirror:1421 : Unable to stop block job on
drive-virtio-disk1
libvirtd.log on node01:
2014-12-19 13:00:52.346+0000: 7258: warning :
qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2014-12-19 13:00:52.436+0000: 7258: warning :
qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2014-12-19 13:00:52.437+0000: 7258: warning :
qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2014-12-19 13:00:52.441+0000: 7258: warning :
qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2014-12-19 13:00:52.480+0000: 7258: warning :
qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2014-12-19 13:00:52.480+0000: 7258: warning :
qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2014-12-19 13:00:52.481+0000: 7258: warning :
qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
ceph version 0.87-1trusty
ubuntu 14.04 LTS
Linux compute02 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux (on both nodes)
virsh cmd to migrate:
virsh migrate --live win12 qemu+ssh://192.x.x.x/system
echo $?
0
qemu cmd line
before on node2:
30308 ? Sl 449:03 qemu-system-x86_64 -enable-kvm -name win12 -S
-machine pc-i440fx-trusty,accel=kvm,usb=off -cpu Nehalem -m 4096 -realtime
mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid
be2c1183-de3b-4994-a20b-2b89a9c4b073 -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/win12.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot menu=off,strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
file=rbd:libvirt-pool/win12:id=libvirt:key=AQCKS4ZTQMYDKBAA1Q8zuys/l+OJ/n9GJjlk9g==:auth_supported=cephx\;none:mon_host=compute01\:6789\;compute02\:6789\;mgmt01\:6789,if=none,id=drive-virtio-disk0,cache=writeback
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=rbd:live_migration/win2012_backups:id=live_migration:key=AQCyRsRRaI6IAxAAPW74dBKlAVUJvkaXadaecw==:auth_supported=cephx\;none:mon_host=compute01\:6789\;compute02\:6789\;mgmt01\:6789,if=none,id=drive-virtio-disk1
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk1,id=virtio-disk1
-drive if=none,id=drive-ide0-0-1,readonly=on,format=raw -device
ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev
tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:fa:aa:c6,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -vnc 0.0.0.0:0 -device
qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
after on node1:
1489 ? Sl 0:47 qemu-system-x86_64 -enable-kvm -name win12 -S
-machine pc-i440fx-trusty,accel=kvm,usb=off -cpu Nehalem -m 4096 -realtime
mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid
be2c1183-de3b-4994-a20b-2b89a9c4b073 -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/win12.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot menu=off,strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
file=rbd:libvirt-pool/win12:id=libvirt:key=AQCKS4ZTQMYDKBAA1Q8zuys/l+OJ/n9GJjlk9g==:auth_supported=cephx\;none:mon_host=compute01\:6789\;compute02\:6789\;mgmt01\:6789,if=none,id=drive-virtio-disk0,cache=writeback
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=rbd:live_migration/win2012_backups:id=live_migration:key=AQCyRsRRaI6IAxAAPW74dBKlAVUJvkaXadaecw==:auth_supported=cephx\;none:mon_host=compute01\:6789\;compute02\:6789\;mgmt01\:6789,if=none,id=drive-virtio-disk1
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk1,id=virtio-disk1
-drive if=none,id=drive-ide0-0-1,readonly=on,format=raw -device
ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev
tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:fa:aa:c6,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -vnc 0.0.0.0:1 -device
qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -incoming tcp:[::]:49152 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
I also noticed that when I stopped the VM (after it hangs) on the migrated
node (1) the vm definition isn't saved... But it still exists on node2
[root@compute01 ~]# virsh list
Id Name State
----------------------------------------------------
2 vpn01 running
3 win12 running
[root@compute01 ~]# virsh destroy win12
Domain win12 destroyed
[root@compute01 ~]# virsh list --all
Id Name State
----------------------------------------------------
2 vpn01 running
- mon01 shut off
- tools01 shut off
- web01 shut off
[root@compute02 ~]# virsh list --all
Id Name State
----------------------------------------------------
- mon01 shut off
- testpc shut off
- tools01 shut off
- web01 shut off
- win12 shut off
--
Jean-Sébastien Frerot
jsfrerot@egliseespoir.com
[-- Attachment #2: Type: text/html, Size: 12059 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] Live migration bug, possible missing ram in migration?
2014-12-19 13:15 [Qemu-devel] Live migration bug, possible missing ram in migration? Frerot, Jean-Sébastien
@ 2014-12-19 13:18 ` Frerot, Jean-Sébastien
2014-12-19 14:40 ` Paolo Bonzini
1 sibling, 0 replies; 7+ messages in thread
From: Frerot, Jean-Sébastien @ 2014-12-19 13:18 UTC (permalink / raw)
To: qemu-devel; +Cc: josh.durgin, quintela
[-- Attachment #1: Type: text/plain, Size: 10529 bytes --]
fixing quintela@redhat.com email.
--
Jean-Sébastien Frerot
jsfrerot@egliseespoir.com
2014-12-19 8:15 GMT-05:00 Frerot, Jean-Sébastien <jsfrerot@egliseespoir.com>
:
>
> Hi, I've been using kvm for some time now using live migration as well
> with ceph backend. Recently I started running into an issue with only one
> of my VM, which happens to be a windows server (2012). When I migrate this
> particular VM it seems that not all the ram is transferred. So when the
> migration completes, the vm that has been migrated simply hangs and I have
> to force a shutdown. Notice that not long ago it was working fine, however
> I didn't notice when it started to have the issue.
>
> Notice that my diagnostic of the problem may be wrong, I mean the behavior
> i'm having is that my VM hangs after live migration, and I don't see the
> problem on other VMs. But I noticed the difference in RAM usage by the qemu
> process which leads me to believe that the ram is not fully transferred.
>
> before migration on node2:
> 15225 libvirt+ 20 0 7128048 4.200g 13592 S 15.0 27.3 1:14.18
> qemu-system-x86
> libvirt+ 15225 52.7 27.3 7128048 4404244 ? Sl 12:57 1:14
> qemu-system-x86_64
>
> after migration on node1:
> 16507 libvirt+ 20 0 6571864 1.610g 13152 R 7.6 10.5 0:07.63
> qemu-system-x86
> libvirt+ 16507 15.7 10.4 6571864 1688392 ? Sl 13:00 0:08
> qemu-system-x86_64
>
> libvirtd.log on node02:
> 2014-12-19 13:01:06.654+0000: 6845: warning :
> qemuMigrationCancelDriveMirror:1421 : Unable to stop block job on
> drive-virtio-disk0
> 2014-12-19 13:01:06.656+0000: 6845: warning :
> qemuMigrationCancelDriveMirror:1421 : Unable to stop block job on
> drive-virtio-disk1
>
> libvirtd.log on node01:
> 2014-12-19 13:00:52.346+0000: 7258: warning :
> qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2014-12-19 13:00:52.436+0000: 7258: warning :
> qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2014-12-19 13:00:52.437+0000: 7258: warning :
> qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2014-12-19 13:00:52.441+0000: 7258: warning :
> qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2014-12-19 13:00:52.480+0000: 7258: warning :
> qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2014-12-19 13:00:52.480+0000: 7258: warning :
> qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2014-12-19 13:00:52.481+0000: 7258: warning :
> qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
>
>
> ceph version 0.87-1trusty
> ubuntu 14.04 LTS
> Linux compute02 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC
> 2014 x86_64 x86_64 x86_64 GNU/Linux (on both nodes)
>
> virsh cmd to migrate:
> virsh migrate --live win12 qemu+ssh://192.x.x.x/system
> echo $?
> 0
>
> qemu cmd line
>
> before on node2:
> 30308 ? Sl 449:03 qemu-system-x86_64 -enable-kvm -name win12 -S
> -machine pc-i440fx-trusty,accel=kvm,usb=off -cpu Nehalem -m 4096 -realtime
> mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid
> be2c1183-de3b-4994-a20b-2b89a9c4b073 -no-user-config -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/win12.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc
> base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
> PIIX4_PM.disable_s4=1 -boot menu=off,strict=on -device
> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device
> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5
> -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1
> -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2
> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
> file=rbd:libvirt-pool/win12:id=libvirt:key=AQCKS4ZTQMYDKBAA1Q8zuys/l+OJ/n9GJjlk9g==:auth_supported=cephx\;none:mon_host=compute01\:6789\;compute02\:6789\;mgmt01\:6789,if=none,id=drive-virtio-disk0,cache=writeback
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> -drive
> file=rbd:live_migration/win2012_backups:id=live_migration:key=AQCyRsRRaI6IAxAAPW74dBKlAVUJvkaXadaecw==:auth_supported=cephx\;none:mon_host=compute01\:6789\;compute02\:6789\;mgmt01\:6789,if=none,id=drive-virtio-disk1
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk1,id=virtio-disk1
> -drive if=none,id=drive-ide0-0-1,readonly=on,format=raw -device
> ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev
> tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:fa:aa:c6,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -chardev
> spicevmc,id=charchannel0,name=vdagent -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
> -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -device
> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2
> -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
> spicevmc,id=charredir0,name=usbredir -device
> usb-redir,chardev=charredir0,id=redir0 -chardev
> spicevmc,id=charredir1,name=usbredir -device
> usb-redir,chardev=charredir1,id=redir1 -chardev
> spicevmc,id=charredir2,name=usbredir -device
> usb-redir,chardev=charredir2,id=redir2 -chardev
> spicevmc,id=charredir3,name=usbredir -device
> usb-redir,chardev=charredir3,id=redir3 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
>
> after on node1:
> 1489 ? Sl 0:47 qemu-system-x86_64 -enable-kvm -name win12 -S
> -machine pc-i440fx-trusty,accel=kvm,usb=off -cpu Nehalem -m 4096 -realtime
> mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid
> be2c1183-de3b-4994-a20b-2b89a9c4b073 -no-user-config -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/win12.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc
> base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
> PIIX4_PM.disable_s4=1 -boot menu=off,strict=on -device
> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device
> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5
> -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1
> -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2
> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
> file=rbd:libvirt-pool/win12:id=libvirt:key=AQCKS4ZTQMYDKBAA1Q8zuys/l+OJ/n9GJjlk9g==:auth_supported=cephx\;none:mon_host=compute01\:6789\;compute02\:6789\;mgmt01\:6789,if=none,id=drive-virtio-disk0,cache=writeback
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> -drive
> file=rbd:live_migration/win2012_backups:id=live_migration:key=AQCyRsRRaI6IAxAAPW74dBKlAVUJvkaXadaecw==:auth_supported=cephx\;none:mon_host=compute01\:6789\;compute02\:6789\;mgmt01\:6789,if=none,id=drive-virtio-disk1
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk1,id=virtio-disk1
> -drive if=none,id=drive-ide0-0-1,readonly=on,format=raw -device
> ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev
> tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:fa:aa:c6,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -chardev
> spicevmc,id=charchannel0,name=vdagent -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
> -device usb-tablet,id=input0 -vnc 0.0.0.0:1 -device
> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2
> -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
> spicevmc,id=charredir0,name=usbredir -device
> usb-redir,chardev=charredir0,id=redir0 -chardev
> spicevmc,id=charredir1,name=usbredir -device
> usb-redir,chardev=charredir1,id=redir1 -chardev
> spicevmc,id=charredir2,name=usbredir -device
> usb-redir,chardev=charredir2,id=redir2 -chardev
> spicevmc,id=charredir3,name=usbredir -device
> usb-redir,chardev=charredir3,id=redir3 -incoming tcp:[::]:49152 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
>
> I also noticed that when I stopped the VM (after it hangs) on the migrated
> node (1) the vm definition isn't saved... But it still exists on node2
> [root@compute01 ~]# virsh list
> Id Name State
> ----------------------------------------------------
> 2 vpn01 running
> 3 win12 running
>
> [root@compute01 ~]# virsh destroy win12
> Domain win12 destroyed
>
> [root@compute01 ~]# virsh list --all
> Id Name State
> ----------------------------------------------------
> 2 vpn01 running
> - mon01 shut off
> - tools01 shut off
> - web01 shut off
>
> [root@compute02 ~]# virsh list --all
> Id Name State
> ----------------------------------------------------
> - mon01 shut off
> - testpc shut off
> - tools01 shut off
> - web01 shut off
> - win12 shut off
>
> --
> Jean-Sébastien Frerot
> jsfrerot@egliseespoir.com
>
[-- Attachment #2: Type: text/html, Size: 12272 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] Live migration bug, possible missing ram in migration?
2014-12-19 13:15 [Qemu-devel] Live migration bug, possible missing ram in migration? Frerot, Jean-Sébastien
2014-12-19 13:18 ` Frerot, Jean-Sébastien
@ 2014-12-19 14:40 ` Paolo Bonzini
2014-12-19 14:41 ` Frerot, Jean-Sébastien
1 sibling, 1 reply; 7+ messages in thread
From: Paolo Bonzini @ 2014-12-19 14:40 UTC (permalink / raw)
To: "Frerot, Jean-Sébastien", qemu-devel; +Cc: josh.durgin, uintela
On 19/12/2014 14:15, Frerot, Jean-Sébastien wrote:
> Hi, I've been using kvm for some time now using live migration as well
> with ceph backend. Recently I started running into an issue with only
> one of my VM, which happens to be a windows server (2012). When I
> migrate this particular VM it seems that not all the ram is transferred.
> So when the migration completes, the vm that has been migrated simply
> hangs and I have to force a shutdown. Notice that not long ago it was
> working fine, however I didn't notice when it started to have the issue.
>
> Notice that my diagnostic of the problem may be wrong, I mean the
> behavior i'm having is that my VM hangs after live migration, and I
> don't see the problem on other VMs. But I noticed the difference in RAM
> usage by the qemu process which leads me to believe that the ram is not
> fully transferred.
>
> before migration on node2:
> 15225 libvirt+ 20 0 7128048 4.200g 13592 S 15.0 27.3 1:14.18
> qemu-system-x86
> libvirt+ 15225 52.7 27.3 7128048 4404244 ? Sl 12:57 1:14
> qemu-system-x86_64
>
> after migration on node1:
> 16507 libvirt+ 20 0 6571864 1.610g 13152 R 7.6 10.5 0:07.63
> qemu-system-x86
> libvirt+ 16507 15.7 10.4 6571864 1688392 ? Sl 13:00 0:08
> qemu-system-x86_64
>
Windows VMs often have a lot of zero pages, because Windows zeroes pages
in the background. These are migrated, but they are left unallocated on
the destination machine. This is the reason why you are seeing smaller
memory usage on the destination.
Paolo
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] Live migration bug, possible missing ram in migration?
2014-12-19 14:40 ` Paolo Bonzini
@ 2014-12-19 14:41 ` Frerot, Jean-Sébastien
2014-12-19 14:42 ` Frerot, Jean-Sébastien
2014-12-19 17:28 ` Dr. David Alan Gilbert
0 siblings, 2 replies; 7+ messages in thread
From: Frerot, Jean-Sébastien @ 2014-12-19 14:41 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: josh.durgin, uintela, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 1878 bytes --]
Thanks for the info, how should I debug this then? Cause I can't live
migrate my windows VM now.
--
Jean-Sébastien Frerot
jsfrerot@egliseespoir.com
2014-12-19 9:40 GMT-05:00 Paolo Bonzini <pbonzini@redhat.com>:
>
>
>
> On 19/12/2014 14:15, Frerot, Jean-Sébastien wrote:
> > Hi, I've been using kvm for some time now using live migration as well
> > with ceph backend. Recently I started running into an issue with only
> > one of my VM, which happens to be a windows server (2012). When I
> > migrate this particular VM it seems that not all the ram is transferred.
> > So when the migration completes, the vm that has been migrated simply
> > hangs and I have to force a shutdown. Notice that not long ago it was
> > working fine, however I didn't notice when it started to have the issue.
> >
> > Notice that my diagnostic of the problem may be wrong, I mean the
> > behavior i'm having is that my VM hangs after live migration, and I
> > don't see the problem on other VMs. But I noticed the difference in RAM
> > usage by the qemu process which leads me to believe that the ram is not
> > fully transferred.
> >
> > before migration on node2:
> > 15225 libvirt+ 20 0 7128048 4.200g 13592 S 15.0 27.3 1:14.18
> > qemu-system-x86
> > libvirt+ 15225 52.7 27.3 7128048 4404244 ? Sl 12:57 1:14
> > qemu-system-x86_64
> >
> > after migration on node1:
> > 16507 libvirt+ 20 0 6571864 1.610g 13152 R 7.6 10.5 0:07.63
> > qemu-system-x86
> > libvirt+ 16507 15.7 10.4 6571864 1688392 ? Sl 13:00 0:08
> > qemu-system-x86_64
> >
>
> Windows VMs often have a lot of zero pages, because Windows zeroes pages
> in the background. These are migrated, but they are left unallocated on
> the destination machine. This is the reason why you are seeing smaller
> memory usage on the destination.
>
> Paolo
>
[-- Attachment #2: Type: text/html, Size: 2552 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] Live migration bug, possible missing ram in migration?
2014-12-19 14:41 ` Frerot, Jean-Sébastien
@ 2014-12-19 14:42 ` Frerot, Jean-Sébastien
2014-12-19 17:28 ` Dr. David Alan Gilbert
1 sibling, 0 replies; 7+ messages in thread
From: Frerot, Jean-Sébastien @ 2014-12-19 14:42 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: josh.durgin, qemu-devel, quintela
[-- Attachment #1: Type: text/plain, Size: 2123 bytes --]
re-adding right email: quintela@redhat.com
--
Jean-Sébastien Frerot
jsfrerot@egliseespoir.com
2014-12-19 9:41 GMT-05:00 Frerot, Jean-Sébastien <jsfrerot@egliseespoir.com>
:
>
> Thanks for the info, how should I debug this then? Cause I can't live
> migrate my windows VM now.
>
>
> --
> Jean-Sébastien Frerot
> jsfrerot@egliseespoir.com
>
> 2014-12-19 9:40 GMT-05:00 Paolo Bonzini <pbonzini@redhat.com>:
>
>>
>>
>> On 19/12/2014 14:15, Frerot, Jean-Sébastien wrote:
>> > Hi, I've been using kvm for some time now using live migration as well
>> > with ceph backend. Recently I started running into an issue with only
>> > one of my VM, which happens to be a windows server (2012). When I
>> > migrate this particular VM it seems that not all the ram is transferred.
>> > So when the migration completes, the vm that has been migrated simply
>> > hangs and I have to force a shutdown. Notice that not long ago it was
>> > working fine, however I didn't notice when it started to have the issue.
>> >
>> > Notice that my diagnostic of the problem may be wrong, I mean the
>> > behavior i'm having is that my VM hangs after live migration, and I
>> > don't see the problem on other VMs. But I noticed the difference in RAM
>> > usage by the qemu process which leads me to believe that the ram is not
>> > fully transferred.
>> >
>> > before migration on node2:
>> > 15225 libvirt+ 20 0 7128048 4.200g 13592 S 15.0 27.3 1:14.18
>> > qemu-system-x86
>> > libvirt+ 15225 52.7 27.3 7128048 4404244 ? Sl 12:57 1:14
>> > qemu-system-x86_64
>> >
>> > after migration on node1:
>> > 16507 libvirt+ 20 0 6571864 1.610g 13152 R 7.6 10.5 0:07.63
>> > qemu-system-x86
>> > libvirt+ 16507 15.7 10.4 6571864 1688392 ? Sl 13:00 0:08
>> > qemu-system-x86_64
>> >
>>
>> Windows VMs often have a lot of zero pages, because Windows zeroes pages
>> in the background. These are migrated, but they are left unallocated on
>> the destination machine. This is the reason why you are seeing smaller
>> memory usage on the destination.
>>
>> Paolo
>>
>
[-- Attachment #2: Type: text/html, Size: 3183 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] Live migration bug, possible missing ram in migration?
2014-12-19 14:41 ` Frerot, Jean-Sébastien
2014-12-19 14:42 ` Frerot, Jean-Sébastien
@ 2014-12-19 17:28 ` Dr. David Alan Gilbert
2014-12-19 18:11 ` Frerot, Jean-Sébastien
1 sibling, 1 reply; 7+ messages in thread
From: Dr. David Alan Gilbert @ 2014-12-19 17:28 UTC (permalink / raw)
To: Frerot, Jean-S?bastien; +Cc: Paolo Bonzini, uintela, qemu-devel, josh.durgin
* Frerot, Jean-S?bastien (jsfrerot@egliseespoir.com) wrote:
> Thanks for the info, how should I debug this then? Cause I can't live
> migrate my windows VM now.
Are the host clocks on the servers synchronised with ntp? Having a clock
mismatch with migration really confuses stuff (especially windows guests)
Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] Live migration bug, possible missing ram in migration?
2014-12-19 17:28 ` Dr. David Alan Gilbert
@ 2014-12-19 18:11 ` Frerot, Jean-Sébastien
0 siblings, 0 replies; 7+ messages in thread
From: Frerot, Jean-Sébastien @ 2014-12-19 18:11 UTC (permalink / raw)
To: Dr. David Alan Gilbert; +Cc: Paolo Bonzini, uintela, qemu-devel, josh.durgin
[-- Attachment #1: Type: text/plain, Size: 1590 bytes --]
yes it is.
[jsfrerot@compute02 ~]$ ntpq -pn
remote refid st t when poll reach delay offset
jitter
==============================================================================
+208.80.96.70 142.3.100.2 2 u 148 1024 377 19.689 0.037
1.219
*192.95.25.79 213.251.128.249 2 u 372 1024 377 7.999 0.248
0.797
+199.182.221.110 30.114.5.31 2 u 178 1024 377 65.084 -2.775
1.255
-174.137.63.116 206.108.0.131 2 u 367 1024 377 21.245 -14.059
38.755
[jsfrerot@compute01 ~]$ ntpq -pn
remote refid st t when poll reach delay offset
jitter
==============================================================================
+208.80.96.70 142.3.100.2 2 u 709 1024 377 22.092 1.155
1.440
+192.95.25.79 213.251.128.249 2 u 683 1024 377 8.517 0.696
1.364
-199.182.221.110 30.114.5.31 2 u 939 1024 377 64.795 -2.720
1.456
*206.108.0.131 .PPS. 1 u 969 1024 377 13.182 -1.305
2.076
--
Jean-Sébastien Frerot
jsfrerot@egliseespoir.com
2014-12-19 12:28 GMT-05:00 Dr. David Alan Gilbert <dgilbert@redhat.com>:
>
> * Frerot, Jean-S?bastien (jsfrerot@egliseespoir.com) wrote:
> > Thanks for the info, how should I debug this then? Cause I can't live
> > migrate my windows VM now.
>
> Are the host clocks on the servers synchronised with ntp? Having a clock
> mismatch with migration really confuses stuff (especially windows guests)
>
> Dave
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
[-- Attachment #2: Type: text/html, Size: 2408 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2014-12-19 18:12 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-12-19 13:15 [Qemu-devel] Live migration bug, possible missing ram in migration? Frerot, Jean-Sébastien
2014-12-19 13:18 ` Frerot, Jean-Sébastien
2014-12-19 14:40 ` Paolo Bonzini
2014-12-19 14:41 ` Frerot, Jean-Sébastien
2014-12-19 14:42 ` Frerot, Jean-Sébastien
2014-12-19 17:28 ` Dr. David Alan Gilbert
2014-12-19 18:11 ` Frerot, Jean-Sébastien
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.