I/O errors after migration - why?

I/O errors after migration - why?

[Tomasz Chmielewski]

I'm trying to perform live migration by following the instructions on 
http://www.linux-kvm.org/page/Migration.
Unfortunately, it doesn't work very well - guest is migrated, but looses 
access to its disk.

On the destination host, I'm starting the guest with exactly the same 
options as on the source host, with "-incoming tcp:0:4444".
On the source host, I start the migration with "migrate -d tcp:B:4444".

Both hosts use the same iSCSI device and can access it.

Looks like the destination host can't really access the iSCSI device 
after all? No - after I reboot the guest (echo b > /proc/sysrq-trigger), 
it boots just fine from its disk. Also lsof on the host shows that the 
kvm process accesses the correct /dev/sdX device.

Both hosts use kvm-84.


This is what kernel says on the guest after migration:

sd 0:0:0:0: ABORT operation started. 
 

sd 0:0:0:0: ABORT operation timed-out. 
 

sd 0:0:0:0: ABORT operation started. 
 

sd 0:0:0:0: ABORT operation timed-out. 
 

sd 0:0:0:0: ABORT operation started. 
 

sd 0:0:0:0: ABORT operation timed-out. 
 

sd 0:0:0:0: ABORT operation started. 
 

sd 0:0:0:0: ABORT operation timed-out. 
 

sd 0:0:0:0: ABORT operation started. 
 

sd 0:0:0:0: ABORT operation timed-out. 
 

sd 0:0:0:0: DEVICE RESET operation started. 
 

sd 0:0:0:0: DEVICE RESET operation timed-out. 
 

sd 0:0:0:0: BUS RESET operation started. 
 

sym0: suspicious SCSI data while resetting the BUS. 
 

sym0: dp1,d15-8,dp0,d7-0,rst,req,ack,bsy,sel,atn,msg,c/d,i/o = 0x0, 
expecting 0x100 

sd 0:0:0:0: BUS RESET operation timed-out. 
 

sd 0:0:0:0: HOST RESET operation started. 
 

sym0: suspicious SCSI data while resetting the BUS. 
 

sym0: dp1,d15-8,dp0,d7-0,rst,req,ack,bsy,sel,atn,msg,c/d,i/o = 0x0, 
expecting 0x100 

sym0: the chip cannot lock the frequency 
 

sym0: SCSI BUS has been reset. 
 

sd 0:0:0:0: HOST RESET operation timed-out. 
 

sd 0:0:0:0: scsi: Device offlined - not ready after error recovery
(...)
Buffer I/O error on device sda1, logical block 1 
 

lost page write due to I/O error on sda1 
 

sd 0:0:0:0: rejecting I/O to offline device


-- 
Tomasz Chmielewski
http://wpkg.org

Re: I/O errors after migration - why?

[Tomasz Chmielewski]

Tomasz Chmielewski schrieb:
> I'm trying to perform live migration by following the instructions on 
> http://www.linux-kvm.org/page/Migration.
> Unfortunately, it doesn't work very well - guest is migrated, but looses 
> access to its disk.
> 
> On the destination host, I'm starting the guest with exactly the same 
> options as on the source host, with "-incoming tcp:0:4444".
> On the source host, I start the migration with "migrate -d tcp:B:4444".
> 
> Both hosts use the same iSCSI device and can access it.
> 
> Looks like the destination host can't really access the iSCSI device 
> after all? No - after I reboot the guest (echo b > /proc/sysrq-trigger), 
> it boots just fine from its disk. Also lsof on the host shows that the 
> kvm process accesses the correct /dev/sdX device.

Similar symptoms with virtio_blk (i.e., when guest is booted off a live 
CD and tries to access the disk after migration).

The only difference between SCSI and virtio_blk is that SCSI signals 
errors and aborts, and virtio_blk waits forever and doesn't give a clue.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: I/O errors after migration - why?

[Tomasz Chmielewski]

Tomasz Chmielewski schrieb:
> Tomasz Chmielewski schrieb:
>> I'm trying to perform live migration by following the instructions on 
>> http://www.linux-kvm.org/page/Migration.
>> Unfortunately, it doesn't work very well - guest is migrated, but 
>> looses access to its disk.
>>
>> On the destination host, I'm starting the guest with exactly the same 
>> options as on the source host, with "-incoming tcp:0:4444".
>> On the source host, I start the migration with "migrate -d tcp:B:4444".
>>
>> Both hosts use the same iSCSI device and can access it.
>>
>> Looks like the destination host can't really access the iSCSI device 
>> after all? No - after I reboot the guest (echo b > 
>> /proc/sysrq-trigger), it boots just fine from its disk. Also lsof on 
>> the host shows that the kvm process accesses the correct /dev/sdX device.
> 
> Similar symptoms with virtio_blk (i.e., when guest is booted off a live 
> CD and tries to access the disk after migration).
> 
> The only difference between SCSI and virtio_blk is that SCSI signals 
> errors and aborts, and virtio_blk waits forever and doesn't give a clue.

I get this kernel BUG when I remove virtio_blk after migration
(virtio block device was not mounted or used during migration).


------------[ cut here ]------------
kernel BUG at drivers/virtio/virtio.c:140!
invalid opcode: 0000 [#1] SMP
Modules linked in: virtio_blk(-) ipv6 video output ac battery button e1000 ppdev parport_pc i2c_piix4 i2c_core btrfs libcrc32c raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 dm_snapshot dm_mirror dm_log dm_mod sbp2 ohci1394 ieee1394 sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd osst sym53c8xx atp870u hptiop ses enclosure aic79xx aic7xxx aic94xx ppa raid_class sym53c500_cs qlogic_cs qlogicfas408 aacraid imm parport mvsas libsas 3w_xxxx initio gdth arcmsr stex tmscsim dc395x iscsi_tcp 3w_9xxx a100u2w BusLogic sr_mod cdrom libsrp libiscsi st ch scsi_transport_srp scsi_transport_spi qla4xxx scsi_transport_iscsi qla2xxx lpfc scsi_transport_fc scsi_transport_sas qla1280 megaraid_sas megaraid ata_piix pdc_adma ahci sata_vsc sata_via sata_uli sata_sx4 sata_svw sata_sis sata_sil 
 sata_sil24 sata_qstor sata_promise sata_nv sata_mv sata_inic162x scsi_wait_scan pata_via pata_triflex pata_sl82c105 pata_sis pata_sil680 pata_serverworks pata_sch pata_pdc202xx_old pata_pdc2
027x pata_pcmcia pata_opti pata_optidma pata_oldpiix pata_ns87415 pata_ns87410 pata_ninja32 pata_netcell pata_mpiix pata_marvell pata_jmicron pata_it821x pata_it8213 pata_hpt3x3 pata_hpt3x2n pata_hpt37x pata_hpt366 pata_efar pata_cypress pata_cs5530 pata_cs5520 pata_cmd64x pata_cmd640 pata_atiixp pata_artop pata_amd pata_ali pata_acpi libata

Pid: 6496, comm: rmmod Not tainted (2.6.27.19-std117 #1)
EIP: 0060:[<c0779f51>] EFLAGS: 00010286 CPU: 0
EIP is at virtio_dev_remove+0x21/0x36
EAX: 000000ff EBX: d8a15c00 ECX: c132653c EDX: 0000c092
ESI: d9dcfd44 EDI: d9dcfd44 EBP: d757cef8 ESP: d757cef4
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmmod (pid: 6496, ti=d757c000 task=d63998e0 task.ti=d757c000)
Stack: d8a15c04 d757cf08 c07068f5 d8a15c04 d8a15cc0 d757cf1c c0706c7b d9dcfd44
       00000000 c09c2cf0 d757cf30 c0705f64 00000000 d9dcfd44 00000880 d757cf40
       c0706ce9 d9dcfe00 00000000 d757cf48 c077a00a d757cf50 d9dcf674 d757cfb0
Call Trace:
 [<c07068f5>] ? __device_release_driver+0x5b/0x78
 [<c0706c7b>] ? driver_detach+0x72/0x97
 [<c0705f64>] ? bus_remove_driver+0x63/0x7f
 [<c0706ce9>] ? driver_unregister+0x2a/0x2e
 [<c077a00a>] ? unregister_virtio_driver+0x8/0xa
 [<d9dcf674>] ? cleanup_module+0x1c/0x1e [virtio_blk]
 [<c044807a>] ? sys_delete_module+0x182/0x1d0
 [<c043bc74>] ? up_read+0x8/0xa
 [<c0811e64>] ? do_page_fault+0x36e/0x672
 [<c0403f02>] ? syscall_call+0x7/0xb
 =======================
Code: 94 c0 51 e8 0b 80 f2 ff c9 c3 55 89 e5 53 8d 58 fc 8b 93 d4 00 00 00 89 d8 ff 52 40 8b 93 40 01 00 00 89 d8 ff 52 08 84 c0 74 04 <0f> 0b eb fe 89 d8 ba 01 00 00 00 e8 f2 fe ff ff 31 c0 5b 5d c3
EIP: [<c0779f51>] virtio_dev_remove+0x21/0x36 SS:ESP 0068:d757cef4
---[ end trace 1d9e100e68f9d27e ]---



-- 
Tomasz Chmielewski
http://wpkg.org

Re: I/O errors after migration - why?

[Tomasz Chmielewski]

Tomasz Chmielewski schrieb:
> Tomasz Chmielewski schrieb:
>> Tomasz Chmielewski schrieb:
>>> I'm trying to perform live migration by following the instructions on 
>>> http://www.linux-kvm.org/page/Migration.
>>> Unfortunately, it doesn't work very well - guest is migrated, but 
>>> looses access to its disk.
>>>
>>> On the destination host, I'm starting the guest with exactly the same 
>>> options as on the source host, with "-incoming tcp:0:4444".
>>> On the source host, I start the migration with "migrate -d tcp:B:4444".
>>>
>>> Both hosts use the same iSCSI device and can access it.
>>>
>>> Looks like the destination host can't really access the iSCSI device 
>>> after all? No - after I reboot the guest (echo b > 
>>> /proc/sysrq-trigger), it boots just fine from its disk. Also lsof on 
>>> the host shows that the kvm process accesses the correct /dev/sdX 
>>> device.
>>
>> Similar symptoms with virtio_blk (i.e., when guest is booted off a 
>> live CD and tries to access the disk after migration).
>>
>> The only difference between SCSI and virtio_blk is that SCSI signals 
>> errors and aborts, and virtio_blk waits forever and doesn't give a clue.

That is interesting (or not?).

In monitor, after migration, "info block" says:

scsi0-hd0: type=hd ...

Before migration, it was:

virtio0: type=hd ...

?

On both sides the guest was started with the same option (delta 
"-incoming...").

-- 
Tomasz Chmielewski
http://wpkg.org

Re: I/O errors after migration - why?

[Anthony Liguori]

Tomasz Chmielewski wrote:
> Tomasz Chmielewski schrieb:
>> Tomasz Chmielewski schrieb:
>>> Tomasz Chmielewski schrieb:
>>>> I'm trying to perform live migration by following the instructions 
>>>> on http://www.linux-kvm.org/page/Migration.
>>>> Unfortunately, it doesn't work very well - guest is migrated, but 
>>>> looses access to its disk.
>>>>
>>>> On the destination host, I'm starting the guest with exactly the 
>>>> same options as on the source host, with "-incoming tcp:0:4444".
>>>> On the source host, I start the migration with "migrate -d 
>>>> tcp:B:4444".
>>>>
>>>> Both hosts use the same iSCSI device and can access it.
>>>>
>>>> Looks like the destination host can't really access the iSCSI 
>>>> device after all? No - after I reboot the guest (echo b > 
>>>> /proc/sysrq-trigger), it boots just fine from its disk. Also lsof 
>>>> on the host shows that the kvm process accesses the correct 
>>>> /dev/sdX device.
>>>
>>> Similar symptoms with virtio_blk (i.e., when guest is booted off a 
>>> live CD and tries to access the disk after migration).
>>>
>>> The only difference between SCSI and virtio_blk is that SCSI signals 
>>> errors and aborts, and virtio_blk waits forever and doesn't give a 
>>> clue.
>
> That is interesting (or not?).
>
> In monitor, after migration, "info block" says:
>
> scsi0-hd0: type=hd ...
>
> Before migration, it was:
>
> virtio0: type=hd ...
>
> ?
>
> On both sides the guest was started with the same option (delta 
> "-incoming...").

Can you give the full command line on both ends and what KVM version it is?

Regards,

Anthony Liguori

Re: I/O errors after migration - why?

[Tomasz Chmielewski]

Anthony Liguori schrieb:

(...)

>>>> Similar symptoms with virtio_blk (i.e., when guest is booted off a 
>>>> live CD and tries to access the disk after migration).
>>>>
>>>> The only difference between SCSI and virtio_blk is that SCSI signals 
>>>> errors and aborts, and virtio_blk waits forever and doesn't give a 
>>>> clue.
>>
>> That is interesting (or not?).
>>
>> In monitor, after migration, "info block" says:
>>
>> scsi0-hd0: type=hd ...
>>
>> Before migration, it was:
>>
>> virtio0: type=hd ...
>>
>> ?
>>
>> On both sides the guest was started with the same option (delta 
>> "-incoming...").
> 
> Can you give the full command line on both ends and what KVM version it is?

kvm-84


/usr/bin/kvm -monitor unix:/var/run/qemu-server/117.mon,server,nowait 
-vnc unix:/var/run/qemu-server/117.vnc,password -pidfile 
/var/run/qemu-server/117.pid -daemonize -usbdevice tablet -name opennms1 
-smp 1 -id 117 -cpuunits 1000 -vga cirrus -tdf -k de -drive 
file=/var/lib/vz/template/iso/systemrescuecd-x86-1.1.7-beta3.iso,if=ide,index=2,media=cdrom 
-drive 
file=/dev/disk/by-path/ip-10.2.18.9:3260-iscsi-iqn.2009-03.net.syneticon:san3.opennms1-lun-1,if=scsi,index=0,boot=on 
-m 400 -net 
tap,vlan=6,ifname=vmtab117i6,script=/var/lib/qemu-server/bridge-vlan6 
-net nic,vlan=6,model=e1000,macaddr=F6:13:A3:72:4D:9F -serial 
unix:/var/run/qemu-server/117.serial,server,nowait -incoming tcp:0:4444


file=/dev/disk/by-path... is a symlink to a proper /dev/sdX device (i.e. 
/dev/sdah on host1, /dev/sds on host2).
When checked with lsof, kvm process uses a proper /dev/sdX device on 
both hosts.



-- 
Tomasz Chmielewski
http://wpkg.org

Re: I/O errors after migration - why?

[Nolan]

Tomasz Chmielewski <mangoo <at> wpkg.org> writes:
> I'm trying to perform live migration by following the instructions on 
> http://www.linux-kvm.org/page/Migration.
> Unfortunately, it doesn't work very well - guest is migrated, but looses 
> access to its disk.

The LSI logic scsi device model doesn't implement device state save/restore. 
Any suspend/resume, snapshot or migration will fail.

I sent a patch that partially addresses this (but is buggy in the presence of
in-flight IO):
http://lists.gnu.org/archive/html/qemu-devel/2009-01/msg00744.html

Re: I/O errors after migration - why?

[Tomasz Chmielewski]

Nolan schrieb:
> Tomasz Chmielewski <mangoo <at> wpkg.org> writes:
>> I'm trying to perform live migration by following the instructions on 
>> http://www.linux-kvm.org/page/Migration.
>> Unfortunately, it doesn't work very well - guest is migrated, but looses 
>> access to its disk.
> 
> The LSI logic scsi device model doesn't implement device state save/restore. 
> Any suspend/resume, snapshot or migration will fail.

Oh, that sucks - as not everything supports virtio (which doesn't work 
for me as well for some reason) - like Windows (which should be 
addressed soon with block virtio drivers), but also older installations, 
running older kernels.
Does IDE support migration?


> I sent a patch that partially addresses this (but is buggy in the presence of
> in-flight IO):
> http://lists.gnu.org/archive/html/qemu-devel/2009-01/msg00744.html


-- 
Tomasz Chmielewski
http://wpkg.org

Re: I/O errors after migration - why?

[Nolan]

On Sat, 2009-03-28 at 11:21 +0100, Tomasz Chmielewski wrote:
> Nolan schrieb:
> > Tomasz Chmielewski <mangoo <at> wpkg.org> writes:
> >> I'm trying to perform live migration by following the instructions on 
> >> http://www.linux-kvm.org/page/Migration.
> >> Unfortunately, it doesn't work very well - guest is migrated, but looses 
> >> access to its disk.
> > 
> > The LSI logic scsi device model doesn't implement device state save/restore. 
> > Any suspend/resume, snapshot or migration will fail.
> 
> Oh, that sucks - as not everything supports virtio (which doesn't work 
> for me as well for some reason) - like Windows (which should be 
> addressed soon with block virtio drivers), but also older installations, 
> running older kernels.

It is indeed a shame.  I wish I had the time to investigate and resolve
the problems with my patch that I linked to previously.

LSI in particular is important for interoperability, as that is what
VMware uses.

> Does IDE support migration?

It appears to, but I am not 100% sure that it will always survive
migration under heavy IO load.  I've gotten mixed messages on whether or
not the qemu core waits for all in flight IOs to complete or if the
device models need to checkpoint pending IOs themselves.  Experimental
evidence suggests that it does not.  Also, from ide.c's checkpoint save
code:
    /* XXX: if a transfer is pending, we do not save it yet */

I think the ideal here would be to stop the CPUs, but let the device
models continue to run.  Once all pending IOs have completed (and DMAed
data and/or descriptors into guest memory, or raised interrupts, or
whatever) then checkpoint all device state.  When the guest resumes, it
will see an unusual flurry of IO completions and/or interrupts, but it
should be able to handle that OK.  Shouldn't look much different from
SMM taking over for a while during high IO load.

This would save a lot of (unwritten, complex, hard to test)
checkpointing code in the device models.  Might cause a missed timer
interrupt or two if there is a lot of slow IO, but that can be
compensated for if needed.

> > I sent a patch that partially addresses this (but is buggy in the presence of
> > in-flight IO):
> > http://lists.gnu.org/archive/html/qemu-devel/2009-01/msg00744.html

Re: I/O errors after migration - why?

[Takeshi Sone]

Hello,

I had similar problem regarding block I/O and migration.
And it is worked around by qemu "stop" command and waiting 1 second
before starting migration (and cont after migration).
See the Ubuntu bug report I posted.
https://bugs.launchpad.net/ubuntu/+source/kvm/+bug/341682

I think Nolan description here explains why stop and wait works.


Nolan wrote:
> On Sat, 2009-03-28 at 11:21 +0100, Tomasz Chmielewski wrote:
>> Nolan schrieb:
>>> Tomasz Chmielewski <mangoo <at> wpkg.org> writes:
>>>> I'm trying to perform live migration by following the instructions on 
>>>> http://www.linux-kvm.org/page/Migration.
>>>> Unfortunately, it doesn't work very well - guest is migrated, but looses 
>>>> access to its disk.
>>> The LSI logic scsi device model doesn't implement device state save/restore. 
>>> Any suspend/resume, snapshot or migration will fail.
>> Oh, that sucks - as not everything supports virtio (which doesn't work 
>> for me as well for some reason) - like Windows (which should be 
>> addressed soon with block virtio drivers), but also older installations, 
>> running older kernels.
> 
> It is indeed a shame.  I wish I had the time to investigate and resolve
> the problems with my patch that I linked to previously.
> 
> LSI in particular is important for interoperability, as that is what
> VMware uses.
> 
>> Does IDE support migration?
> 
> It appears to, but I am not 100% sure that it will always survive
> migration under heavy IO load.  I've gotten mixed messages on whether or
> not the qemu core waits for all in flight IOs to complete or if the
> device models need to checkpoint pending IOs themselves.  Experimental
> evidence suggests that it does not.  Also, from ide.c's checkpoint save
> code:
>     /* XXX: if a transfer is pending, we do not save it yet */
> 
> I think the ideal here would be to stop the CPUs, but let the device
> models continue to run.  Once all pending IOs have completed (and DMAed
> data and/or descriptors into guest memory, or raised interrupts, or
> whatever) then checkpoint all device state.  When the guest resumes, it
> will see an unusual flurry of IO completions and/or interrupts, but it
> should be able to handle that OK.  Shouldn't look much different from
> SMM taking over for a while during high IO load.
> 
> This would save a lot of (unwritten, complex, hard to test)
> checkpointing code in the device models.  Might cause a missed timer
> interrupt or two if there is a lot of slow IO, but that can be
> compensated for if needed.
> 
>>> I sent a patch that partially addresses this (but is buggy in the presence of
>>> in-flight IO):
>>> http://lists.gnu.org/archive/html/qemu-devel/2009-01/msg00744.html
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html