[Qemu-devel] Live Migration with different block devices

[Qemu-devel] Live Migration with different block devices

[Xiongzi Ge]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

Hi,


When I do live migration, in the source and destination host, there are
different block devices, but qemu can not detect this. I used virtio as the
driver in kvm and in the vdi device in the guest is /dev/vda.  So,  the vm
guest can read different data from the same /dev/vda device.  I am studying
this code to let qemu understand that, this is a new device.

Does qemu recognize different block devices after live migration?  Where is
the interface to modify this? Thanks


Gary

[-- Attachment #2: Type: text/html, Size: 671 bytes --]

Re: [Qemu-devel] Live Migration with different block devices

[Brian Jackson]

On 6/24/2014 2:22 PM, Xiongzi Ge wrote:
> Hi,
>
>
> When I do live migration, in the source and destination host, there are
> different block devices, but qemu can not detect this. I used virtio as the
> driver in kvm and in the vdi device in the guest is /dev/vda.  So,  the vm
> guest can read different data from the same /dev/vda device.  I am studying
> this code to let qemu understand that, this is a new device.
>
> Does qemu recognize different block devices after live migration?

You aren't supposed to have different command line options when live 
migrating (with the exception of -incoming if you migrate that way). So 
whatever you are trying to do is unsupported.

> Where is
> the interface to modify this? Thanks
>
>
> Gary
>

Re: [Qemu-devel] Live Migration with different block devices

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1741 bytes --]

On 06/24/2014 03:29 PM, Brian Jackson wrote:
> 
> 
> On 6/24/2014 2:22 PM, Xiongzi Ge wrote:
>> Hi,
>>
>>
>> When I do live migration, in the source and destination host, there are
>> different block devices, but qemu can not detect this. I used virtio
>> as the
>> driver in kvm and in the vdi device in the guest is /dev/vda.  So, 
>> the vm
>> guest can read different data from the same /dev/vda device.  I am
>> studying
>> this code to let qemu understand that, this is a new device.
>>
>> Does qemu recognize different block devices after live migration?
> 
> You aren't supposed to have different command line options when live
> migrating (with the exception of -incoming if you migrate that way). So
> whatever you are trying to do is unsupported.

Caveat - it IS supported to change command line options for things that
do NOT impact guest ABI.  For example, if you have /path/to/file as the
host location containing the guest image on the source, but
/other/path/file as the location on the destination, you CAN rewrite
that parameter (and in fact, libvirt DOES supporting the rewrite such
parameters if you use the 'virsh migrate --xml' option - after first
proving that your changes do not affect guest ABI).  But the onus is on
you to ensure that src:/path/to/file and dst:/other/path/file have the
SAME guest-visible contents at the time of the migration (this generally
requires that the two names be mappings to the same underlying network
resource; you can get trickier, but it starts to be at your peril,
because the guest is hosed if the two files differ in guest contents).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] Live Migration with different block devices

[Xiongzi Ge]

[-- Attachment #1: Type: text/plain, Size: 2567 bytes --]

Hi Eric,

This really works if I set up the same configuration (/dev/vda in the
guest) but the physical block devices are not the same after migration.The
ABI is the same.  When the guest cache stores a page like 'aaa' which is in
the block device of Host A. If the cache in the guest is also migrated to
Host B, but data in the block device actually are "bbb" in Host B now. What
will happen?  Will the cache data in the guest be migrated to the other
host?  The vmstate function in the qemu seems doing something to save the
vm state and the device state.

Should the guest re-open the block device and delete the previous cache
data or just check the consistency of the cache and the block device?
 Thanks.











On Tue, Jun 24, 2014 at 7:16 PM, Eric Blake <eblake@redhat.com> wrote:

> On 06/24/2014 03:29 PM, Brian Jackson wrote:
> >
> >
> > On 6/24/2014 2:22 PM, Xiongzi Ge wrote:
> >> Hi,
> >>
> >>
> >> When I do live migration, in the source and destination host, there are
> >> different block devices, but qemu can not detect this. I used virtio
> >> as the
> >> driver in kvm and in the vdi device in the guest is /dev/vda.  So,
> >> the vm
> >> guest can read different data from the same /dev/vda device.  I am
> >> studying
> >> this code to let qemu understand that, this is a new device.
> >>
> >> Does qemu recognize different block devices after live migration?
> >
> > You aren't supposed to have different command line options when live
> > migrating (with the exception of -incoming if you migrate that way). So
> > whatever you are trying to do is unsupported.
>
> Caveat - it IS supported to change command line options for things that
> do NOT impact guest ABI.  For example, if you have /path/to/file as the
> host location containing the guest image on the source, but
> /other/path/file as the location on the destination, you CAN rewrite
> that parameter (and in fact, libvirt DOES supporting the rewrite such
> parameters if you use the 'virsh migrate --xml' option - after first
> proving that your changes do not affect guest ABI).  But the onus is on
> you to ensure that src:/path/to/file and dst:/other/path/file have the
> SAME guest-visible contents at the time of the migration (this generally
> requires that the two names be mappings to the same underlying network
> resource; you can get trickier, but it starts to be at your peril,
> because the guest is hosed if the two files differ in guest contents).
>
> --
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>
>

[-- Attachment #2: Type: text/html, Size: 3558 bytes --]

Re: [Qemu-devel] Live Migration with different block devices

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

On 06/24/2014 06:08 PM, Xiongzi Ge wrote:
> Hi Eric,

[we tend to avoid top-posting on technical lists]

> 
> This really works if I set up the same configuration (/dev/vda in the
> guest) but the physical block devices are not the same after migration.The
> ABI is the same.  When the guest cache stores a page like 'aaa' which is in
> the block device of Host A. If the cache in the guest is also migrated to
> Host B, but data in the block device actually are "bbb" in Host B now.

The ABI is NOT the same if the two host files do not have identical
contents; it's just that qemu cannot diagnose your bug.

> What
> will happen?  Will the cache data in the guest be migrated to the other
> host?  The vmstate function in the qemu seems doing something to save the
> vm state and the device state.
> 

Who knows what happen? You're in undefined territory, because you
violated the premise that migration does not change disk contents behind
the guest's back.

> Should the guest re-open the block device and delete the previous cache
> data or just check the consistency of the cache and the block device?

Rather than trying to figure out what happens with undefined behavior,
how about you hot-unplug the old host device, then migrate, then
hot-plug the new host device.  That way, the guest will KNOW it it is
connecting to a second disk, and not be confused by anything it cached
about the first.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] Live Migration with different block devices

[Paolo Bonzini]

Il 25/06/2014 02:08, Xiongzi Ge ha scritto:
> Will the cache data in the guest be migrated to the other host?

Again, cache data is _not_ managed by QEMU.  It is managed by the guest 
kernel.  Its content and state is migrated together with the rest of RAM.

Paolo

Re: [Qemu-devel] Live Migration with different block devices

[Xiongzi Ge]

[-- Attachment #1: Type: text/plain, Size: 425 bytes --]

Is there a good way to handle the memory in the guest via qemu?


On Wed, Jun 25, 2014 at 2:18 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 25/06/2014 02:08, Xiongzi Ge ha scritto:
>
>  Will the cache data in the guest be migrated to the other host?
>>
>
> Again, cache data is _not_ managed by QEMU.  It is managed by the guest
> kernel.  Its content and state is migrated together with the rest of RAM.
>
> Paolo
>

[-- Attachment #2: Type: text/html, Size: 935 bytes --]

Re: [Qemu-devel] Live Migration with different block devices

[Paolo Bonzini]

Il 25/06/2014 16:14, Xiongzi Ge ha scritto:
> Is there a good way to handle the memory in the guest via qemu?

I'm not sure what you're asking exactly, but I'm pretty sure the answer 
is no.

You're confusing what QEMU is doing and what the guest is doing.  And 
you still haven't told us what you're trying to achieve (you only told 
us how).

Paolo

Re: [Qemu-devel] Live Migration with different block devices

[Xiongzi Ge]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

Hi Paolo,

I should make it clearer. I want to use a device for caching for the guest
in host A. Then after the guest migrating to Host B, there is also a device
for caching in Host B.   I want to the guest can access the new cache
device in Host B at first automatically. I also want the guest can access
the previous cache device in Host A, if it gets a miss. Here, I may need to
modify some interfaces. Is there a simple way to do this?

Thanks a lot


On Wed, Jun 25, 2014 at 10:18 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 25/06/2014 16:14, Xiongzi Ge ha scritto:
>
>  Is there a good way to handle the memory in the guest via qemu?
>>
>
> I'm not sure what you're asking exactly, but I'm pretty sure the answer is
> no.
>
> You're confusing what QEMU is doing and what the guest is doing.  And you
> still haven't told us what you're trying to achieve (you only told us how).
>
> Paolo
>
>

[-- Attachment #2: Type: text/html, Size: 1521 bytes --]

Re: [Qemu-devel] Live Migration with different block devices

[Paolo Bonzini]

Il 25/06/2014 16:32, Xiongzi Ge ha scritto:
> I should make it clearer. I want to use a device for caching for the
> guest in host A. Then after the guest migrating to Host B, there is also
> a device for caching in Host B.   I want to the guest can access the new
> cache device in Host B at first automatically. I also want the guest can
> access the previous cache device in Host A, if it gets a miss. Here, I
> may need to modify some interfaces. Is there a simple way to do this?

If your caching is done in the guest, the guest must use O_DIRECT to 
open the block device.

Paolo

[Qemu-devel] Live Migration with different block devices

[Xiongzi Ge]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

Hi,


When I do live migration, in the source and destination host, there are
different block devices, but qemu can not detect this. I used virtio as the
driver in kvm and in the vdi device in the guest is /dev/vda.  So,  the vm
guest can read different data from the same /dev/vda device.  I am studying
this code to let qemu understand that, this is a new device.

Does qemu recognize different block devices after live migration?  Where is
the interface to modify this? Thanks


Gary

[-- Attachment #2: Type: text/html, Size: 589 bytes --]

Re: [Qemu-devel] Live Migration with different block devices

[Paolo Bonzini]

Il 24/06/2014 21:18, Xiongzi Ge ha scritto:
>
> When I do live migration, in the source and destination host, there are
> different block devices, but qemu can not detect this. I used virtio as
> the driver in kvm and in the vdi device in the guest is /dev/vda.  So,
> the vm guest can read different data from the same /dev/vda device.

It's probably the guest kernel (not QEMU) that is caching data from the 
source's /dev/vda.  What you are doing is not a valid usage of 
migration.  What are you trying to do?

Paolo

Re: [Qemu-devel] Live Migration with different block devices

[Xiongzi Ge]

[-- Attachment #1: Type: text/plain, Size: 400 bytes --]

Hi Paolo,

I am trying to do migration with two different block devices in the source
and destination hosts.  For example, in the source the data in /dev/vda
are  'aaaaa'. The guest has a cache. These data are cached.  In the other
host, the data in this device are 'bbbbb'.   If we do migration, I want
the  guest vm recognize this change.  How does qemu deal with the cache
data?  Thanks.

Thanks,

[-- Attachment #2: Type: text/html, Size: 486 bytes --]

Re: [Qemu-devel] Live Migration with different block devices

[Paolo Bonzini]

Il 24/06/2014 21:55, Xiongzi Ge ha scritto:
> Hi Paolo,
>
> I am trying to do migration with two different block devices in the
> source and destination hosts.  For example, in the source the data in
> /dev/vda are  'aaaaa'. The guest has a cache. These data are cached.  In
> the other host, the data in this device are 'bbbbb'.

Again, what you are doing is invalid.  The point of migration is to have 
the same thing running on the source and destination, including disk data.

> If we do migration, I want the  guest vm recognize this change.

This is not _what_ you want to do.  This is _how_ you want to do it.

(Example: you eat a big mac.  _What_ you want to do is not being hungry. 
  _How_ you want to do it is going at McDonald's and grabbing a 
sandwich.  Similar to your migration example, there's probably a better 
way to achieve your objective).

So, why do you want data to change upon migration?

> How does qemu deal with the cache data?  Thanks.

It's not QEMU that's caching data, it's the guest.  There's nothing that 
QEMU can do about it.

Paolo

Re: [Qemu-devel] Live Migration with different block devices

[Xiongzi Ge]

[-- Attachment #1: Type: text/plain, Size: 1436 bytes --]

Hi Paolo,

Why the data changed is because the block device is used for a cache device
in each host. So, after migration, such block device will be changed, but I
still want this vm can access the data in the new block device.


Thanks


On Tue, Jun 24, 2014 at 4:25 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:

> Il 24/06/2014 21:55, Xiongzi Ge ha scritto:
>
>  Hi Paolo,
>>
>> I am trying to do migration with two different block devices in the
>> source and destination hosts.  For example, in the source the data in
>> /dev/vda are  'aaaaa'. The guest has a cache. These data are cached.  In
>> the other host, the data in this device are 'bbbbb'.
>>
>
> Again, what you are doing is invalid.  The point of migration is to have
> the same thing running on the source and destination, including disk data.
>
>
>  If we do migration, I want the  guest vm recognize this change.
>>
>
> This is not _what_ you want to do.  This is _how_ you want to do it.
>
> (Example: you eat a big mac.  _What_ you want to do is not being hungry.
>  _How_ you want to do it is going at McDonald's and grabbing a sandwich.
>  Similar to your migration example, there's probably a better way to
> achieve your objective).
>
> So, why do you want data to change upon migration?
>
>
>  How does qemu deal with the cache data?  Thanks.
>>
>
> It's not QEMU that's caching data, it's the guest.  There's nothing that
> QEMU can do about it.
>
> Paolo
>

[-- Attachment #2: Type: text/html, Size: 2363 bytes --]

Re: [Qemu-devel] Live Migration with different block devices

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

On 06/24/2014 01:55 PM, Xiongzi Ge wrote:
> Hi Paolo,
> 
> I am trying to do migration with two different block devices in the source
> and destination hosts.  For example, in the source the data in /dev/vda
> are  'aaaaa'. The guest has a cache. These data are cached.  In the other
> host, the data in this device are 'bbbbb'.   If we do migration, I want
> the  guest vm recognize this change.  How does qemu deal with the cache
> data?  Thanks.

If you want the guest to see different disk contents, then hotunplug the
original and hotplug in the new disk.  Migration is NOT supposed to
change disk contents.  Your problem is independent of migration.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] Live Migration with different block devices

[Paolo Bonzini]

Il 24/06/2014 22:41, Eric Blake ha scritto:
>> >
>> > I am trying to do migration with two different block devices in the source
>> > and destination hosts.  For example, in the source the data in /dev/vda
>> > are  'aaaaa'. The guest has a cache. These data are cached.  In the other
>> > host, the data in this device are 'bbbbb'.   If we do migration, I want
>> > the  guest vm recognize this change.  How does qemu deal with the cache
>> > data?  Thanks.
> If you want the guest to see different disk contents, then hotunplug the
> original and hotplug in the new disk.  Migration is NOT supposed to
> change disk contents.  Your problem is independent of migration.

Or if you really do not want to do that, all access to the disk has to 
be done with O_DIRECT in the guest.

Paolo

Re: [Qemu-devel] Live Migration with different block devices

[Xiongzi Ge]

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

Hi ,

This is just related to migration, but it may be not in the standard
migration.  The cache device in different hosts can only be accessed by the
vm running in the host.  The problem for me is to find the interface of the
cache in the guest or like first doing it in the qemu, after migration,
this cache should recognize that the cache content storing in the disk has
been changed.   DOes anyone has good suggestions? Thanks !


On Tue, Jun 24, 2014 at 4:41 PM, Eric Blake <eblake@redhat.com> wrote:

> On 06/24/2014 01:55 PM, Xiongzi Ge wrote:
> > Hi Paolo,
> >
> > I am trying to do migration with two different block devices in the
> source
> > and destination hosts.  For example, in the source the data in /dev/vda
> > are  'aaaaa'. The guest has a cache. These data are cached.  In the other
> > host, the data in this device are 'bbbbb'.   If we do migration, I want
> > the  guest vm recognize this change.  How does qemu deal with the cache
> > data?  Thanks.
>
> If you want the guest to see different disk contents, then hotunplug the
> original and hotplug in the new disk.  Migration is NOT supposed to
> change disk contents.  Your problem is independent of migration.
>
> --
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>
>

[-- Attachment #2: Type: text/html, Size: 2063 bytes --]

Re: [Qemu-devel] Live Migration with different block devices

[Christopher Covington]

On 06/24/2014 05:26 PM, Xiongzi Ge wrote:
> Hi ,
> 
> This is just related to migration, but it may be not in the standard
> migration.  The cache device in different hosts can only be accessed by the vm
> running in the host.  The problem for me is to find the interface of the cache
> in the guest or like first doing it in the qemu, after migration, this cache
> should recognize that the cache content storing in the disk has been
> changed.   DOes anyone has good suggestions? Thanks ! 

I don't know much about storage and don't really understand what you're
describing (what is a cache device?), but I've been using Checkpoint and
Restore in Userspace (CRIU) [1] to move Linux processes between systems with
different (simulated) hardware. The kernels end up with non-identical state,
but that's fine for my purposes.

http://www.criu.org/Main_Page

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.