Fwd: Corrupted filesystem, possible after livemigration with iSCSI storagebackend.

Fwd: Corrupted filesystem, possible after livemigration with iSCSI storagebackend.

[Espen Berg]

In our KVM system we have two iSCSI backends (master/slave
configuration) with failover and two KVM hosts supporting live migration.

The iSCSI volumes are shared by the host as a block device in KVM, and
the volumes are available on both frontends.  After a reboot one of the
KVMs where not able to start again due to file system corruption.  We
use XFS and have problems to understand what caused the corruption.

We have ruled out the iSCSI backend as both the master and slave data
where consistent at the time.

Anyone else had similar problems?  What is the recommended way to share
an iSCSI drive among the two host machines?

Should XFS be ok as a file system for live migration?  I'm not able to
find any documentation stating that a clustered file system (GFS2 etc.)
is recommended.  Are there any concurrent writes on the two host
machines during a livemigtation?

  <disk type='block' device='disk'>
       <driver name='qemu'/>
       <source dev='/dev/disk/by-path/ip-<ip>:3260-iscsi-test2-lun-0'/>
       <target dev='sda' bus='scsi'/>
       <address type='drive' controller='0' bus='0' unit='0'/>
  </disk>

#virsh version
Compiled against library: libvir 0.7.6
Using library: libvir 0.7.6
Using API: QEMU 0.7.6
Running hypervisor: QEMU 0.11.0

#uname -a
Linux vm01 2.6.32-bpo.2-amd64 #1 SMP Fri Feb 12 16:50:27 UTC 2010 x86_64
GNU/Linux

Regards
Espen

Re: Fwd: Corrupted filesystem, possible after livemigration with iSCSI storagebackend.

[Anthony Liguori]

On 03/15/2010 08:46 AM, Espen Berg wrote:
> In our KVM system we have two iSCSI backends (master/slave
> configuration) with failover and two KVM hosts supporting live migration.
>
> The iSCSI volumes are shared by the host as a block device in KVM, and
> the volumes are available on both frontends.  After a reboot one of the
> KVMs where not able to start again due to file system corruption.  We
> use XFS and have problems to understand what caused the corruption.
>
> We have ruled out the iSCSI backend as both the master and slave data
> where consistent at the time.
>
> Anyone else had similar problems?  What is the recommended way to share
> an iSCSI drive among the two host machines?
>
> Should XFS be ok as a file system for live migration?  I'm not able to
> find any documentation stating that a clustered file system (GFS2 etc.)
> is recommended.  Are there any concurrent writes on the two host
> machines during a livemigtation?
>
> <disk type='block' device='disk'>
> <driver name='qemu'/>
> <source dev='/dev/disk/by-path/ip-<ip>:3260-iscsi-test2-lun-0'/>
> <target dev='sda' bus='scsi'/>
> <address type='drive' controller='0' bus='0' unit='0'/>
> </disk>

You need to use cache=off if you've got one iscsi drive mounted on two 
separate physical machines.

The additional layer of caching will result in inconsistency because 
iSCSI doesn't have a mechanism to provide cache coherence between two nodes.

Regards,

Anthony Liguori

> #virsh version
> Compiled against library: libvir 0.7.6
> Using library: libvir 0.7.6
> Using API: QEMU 0.7.6
> Running hypervisor: QEMU 0.11.0
>
> #uname -a
> Linux vm01 2.6.32-bpo.2-amd64 #1 SMP Fri Feb 12 16:50:27 UTC 2010 x86_64
> GNU/Linux
>
> Regards
> Espen
>
>
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: Corrupted filesystem, possible after livemigration with iSCSI storagebackend.

[Daniel P. Berrange]

On Mon, Mar 15, 2010 at 08:59:10AM -0500, Anthony Liguori wrote:
> On 03/15/2010 08:46 AM, Espen Berg wrote:
> >In our KVM system we have two iSCSI backends (master/slave
> >configuration) with failover and two KVM hosts supporting live migration.
> >
> >The iSCSI volumes are shared by the host as a block device in KVM, and
> >the volumes are available on both frontends.  After a reboot one of the
> >KVMs where not able to start again due to file system corruption.  We
> >use XFS and have problems to understand what caused the corruption.
> >
> >We have ruled out the iSCSI backend as both the master and slave data
> >where consistent at the time.
> >
> >Anyone else had similar problems?  What is the recommended way to share
> >an iSCSI drive among the two host machines?
> >
> >Should XFS be ok as a file system for live migration?  I'm not able to
> >find any documentation stating that a clustered file system (GFS2 etc.)
> >is recommended.  Are there any concurrent writes on the two host
> >machines during a livemigtation?
> >
> ><disk type='block' device='disk'>
> ><driver name='qemu'/>
> ><source dev='/dev/disk/by-path/ip-<ip>:3260-iscsi-test2-lun-0'/>
> ><target dev='sda' bus='scsi'/>
> ><address type='drive' controller='0' bus='0' unit='0'/>
> ></disk>
> 
> You need to use cache=off if you've got one iscsi drive mounted on two 
> separate physical machines.

FYI, this can be done by changing the disk XML driver

  <driver name='qemu'/>

to be

  <driver name='qemu' cache='none'/>

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|