Re: I/O errors in guest OS after repeated migration

[Brian Jackson <iggy@theiggy.com>]

On Tuesday, October 16, 2012 11:33:44 AM Guido Winkelmann wrote:
> Hi,
> 
> I'm experiencing I/O errors in a guest machine after migrating it from one
> host to another, and then back to the original host. After doing this, I
> find the following in the dmesg output of the guest machine:
> 
> [  345.390543] end_request: I/O error, dev vda, sector 273871
> [  345.391125] end_request: I/O error, dev vda, sector 273871
> [  345.391705] end_request: I/O error, dev vda, sector 273871
> [  345.394796] end_request: I/O error, dev vda, sector 1745983
> [  345.396005] end_request: I/O error, dev vda, sector 1745983
> [  346.083160] end_request: I/O error, dev vdb, sector 54528008
> [  346.083179] Buffer I/O error on device dm-0, logical block 6815745
> [  346.083181] lost page write due to I/O error on dm-0
> [  346.083193] end_request: I/O error, dev vdb, sector 54528264
> [  346.083195] Buffer I/O error on device dm-0, logical block 6815777
> [  346.083197] lost page write due to I/O error on dm-0
> [  346.083201] end_request: I/O error, dev vdb, sector 2056
> [  346.083204] Buffer I/O error on device dm-0, logical block 1
> [  346.083206] lost page write due to I/O error on dm-0
> [  346.083209] Buffer I/O error on device dm-0, logical block 2
> [  346.083211] lost page write due to I/O error on dm-0
> [  346.083215] end_request: I/O error, dev vdb, sector 10248
> [  346.083217] Buffer I/O error on device dm-0, logical block 1025
> [  346.083219] lost page write due to I/O error on dm-0
> [  346.091499] end_request: I/O error, dev vdb, sector 76240
> [  346.091506] Buffer I/O error on device dm-0, logical block 9274
> [  346.091508] lost page write due to I/O error on dm-0
> [  346.091572] JBD2: Detected IO errors while flushing file data on dm-0-8
> [  346.091915] end_request: I/O error, dev vdb, sector 38017360
> [  346.091956] Aborting journal on device dm-0-8.
> [  346.092557] end_request: I/O error, dev vdb, sector 38012928
> [  346.092566] Buffer I/O error on device dm-0, logical block 4751360
> [  346.092569] lost page write due to I/O error on dm-0
> [  346.092624] JBD2: I/O error detected when updating journal superblock
> for dm-0-8.
> [  346.100940] end_request: I/O error, dev vdb, sector 2048
> [  346.100948] Buffer I/O error on device dm-0, logical block 0
> [  346.100952] lost page write due to I/O error on dm-0
> [  346.101027] EXT4-fs error (device dm-0): ext4_journal_start_sb:327:
> Detected aborted journal
> [  346.101038] EXT4-fs (dm-0): Remounting filesystem read-only
> [  346.101051] EXT4-fs (dm-0): previous I/O error to superblock detected
> [  346.101836] end_request: I/O error, dev vdb, sector 2048
> [  346.101845] Buffer I/O error on device dm-0, logical block 0
> [  346.101849] lost page write due to I/O error on dm-0
> [  373.006680] end_request: I/O error, dev vda, sector 624319
> [  373.007543] end_request: I/O error, dev vda, sector 624319
> [  373.008327] end_request: I/O error, dev vda, sector 624319
> [  374.886674] end_request: I/O error, dev vda, sector 624319
> [  374.887563] end_request: I/O error, dev vda, sector 624319
> 
> The hosts are both running Fedora 17 with qemu-kvm-1.0.1-1.fc17.x86_64. The
> guest machine has been started and migrated using libvirt (0.9.11). Kernel
> version is 3.5.6-1.fc17.x86_64 on the first host and 3.5.5-2.fc17.x86_64 on
> the second.
> The guest machine is on Kernel 3.3.8 and uses ext4 on its disks.
> 
> The commandline, as generated by libvirtd, looks like this:
> 
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
> QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc-0.15 -enable-kvm -m 1024
> -smp 1,sockets=1,cores=1,threads=1 -name migratetest2 -uuid
> ddbf11e9-387e-902b-4849-8c3067dc42a2 -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/migratetest2.monitor,serve
> r,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-reboot -no- shutdown -device
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/data/migratetest2_system,if=none,id=drive-virtio-
> disk0,format=qcow2,cache=none -device virtio-blk-
> pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-
> disk0,bootindex=1 -drive file=/data/migratetest2_data-1,if=none,id=drive-
> virtio-disk1,format=qcow2,cache=none -device virtio-blk-
> pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -
> netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-
> pci,netdev=hostnet0,id=net0,mac=02:00:00:00:00:0c,bus=pci.0,addr=0x3 -vnc
> 127.0.0.1:2,password -k de -vga cirrus -incoming tcp:0.0.0.0:49153 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6


I see qcow2 in there. Live migration of qcow2 was a new feature in 1.0. Have 
you tried other formats or different qemu/kvm versions?


> 
> The second host has an ext4 filesystem mounted under /data, which it
> exports using NFSv3 over TCP to the first host, which also mounts it under
> /data.
> 
> So far, the problem seems reproducible: When I start another guest machine
> and do the same thing with it, the same problem happens.
> 
> Can anybody help me with this problem?
> 
> 	Guido
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html