Ceph and KVM live migration

Ceph and KVM live migration

[Vladimir Bashkirtsev]

Dear all,

Currently I testing KVMs running on ceph and particularly testing recent 
cache feature. Performance is of course vastly improved but still have 
occasional KVM hold ups - not sure who is at blame ceph of KVM. But I 
will deal with it later. Right now I've got myself a question which I 
could not get answered myself: if I do live migration of KVM while there 
some uncommitted data in ceph cache will this cache be committed prior 
cut-over to another host? Reading through the list I've got an 
impression that it may be left uncommitted and thus it may cause data 
corruption. I just would like a simple confirmation if code which 
commits cache on cut-over to new host does exist and no data corruption 
due to RBD cache+live migration should happen.

Regards,
Vladimir

Re: Ceph and KVM live migration

[Josh Durgin]

On 06/30/2012 05:42 PM, Vladimir Bashkirtsev wrote:
> Dear all,
>
> Currently I testing KVMs running on ceph and particularly testing recent
> cache feature. Performance is of course vastly improved but still have
> occasional KVM hold ups - not sure who is at blame ceph of KVM. But I
> will deal with it later. Right now I've got myself a question which I
> could not get answered myself: if I do live migration of KVM while there
> some uncommitted data in ceph cache will this cache be committed prior
> cut-over to another host? Reading through the list I've got an
> impression that it may be left uncommitted and thus it may cause data
> corruption. I just would like a simple confirmation if code which
> commits cache on cut-over to new host does exist and no data corruption
> due to RBD cache+live migration should happen.
>
> Regards,
> Vladimir

QEMU does a flush on all the disks when it stops the guest on the
original host, so there will be no uncommitted data in the cache.

Josh

Re: Ceph and KVM live migration

[Vladimir Bashkirtsev]

On 01/07/12 10:47, Josh Durgin wrote:
> On 06/30/2012 05:42 PM, Vladimir Bashkirtsev wrote:
>> Dear all,
>>
>> Currently I testing KVMs running on ceph and particularly testing recent
>> cache feature. Performance is of course vastly improved but still have
>> occasional KVM hold ups - not sure who is at blame ceph of KVM. But I
>> will deal with it later. Right now I've got myself a question which I
>> could not get answered myself: if I do live migration of KVM while there
>> some uncommitted data in ceph cache will this cache be committed prior
>> cut-over to another host? Reading through the list I've got an
>> impression that it may be left uncommitted and thus it may cause data
>> corruption. I just would like a simple confirmation if code which
>> commits cache on cut-over to new host does exist and no data corruption
>> due to RBD cache+live migration should happen.
>>
>> Regards,
>> Vladimir
>
> QEMU does a flush on all the disks when it stops the guest on the
> original host, so there will be no uncommitted data in the cache.
>
> Josh
Thank you for quick and precise answer. Now when I actually attempted to 
live migrate ceph based VM I get:

Unable to migrate guest: Invalid relative path 
'rbd/mail.logics.net.au:rbd_cache=true': Invalid argument

I guess KVM does not like having :rbd_cache=true (migration works 
without it). I know that it is most likely KVM problem but still decided 
to ask here in case if you know about it. Any ideas how to fix it?

Regards,
Vladimir

Re: Ceph and KVM live migration

[Josh Durgin]

On 06/30/2012 07:15 PM, Vladimir Bashkirtsev wrote:
> On 01/07/12 10:47, Josh Durgin wrote:
>> On 06/30/2012 05:42 PM, Vladimir Bashkirtsev wrote:
>>> Dear all,
>>>
>>> Currently I testing KVMs running on ceph and particularly testing recent
>>> cache feature. Performance is of course vastly improved but still have
>>> occasional KVM hold ups - not sure who is at blame ceph of KVM. But I
>>> will deal with it later. Right now I've got myself a question which I
>>> could not get answered myself: if I do live migration of KVM while there
>>> some uncommitted data in ceph cache will this cache be committed prior
>>> cut-over to another host? Reading through the list I've got an
>>> impression that it may be left uncommitted and thus it may cause data
>>> corruption. I just would like a simple confirmation if code which
>>> commits cache on cut-over to new host does exist and no data corruption
>>> due to RBD cache+live migration should happen.
>>>
>>> Regards,
>>> Vladimir
>>
>> QEMU does a flush on all the disks when it stops the guest on the
>> original host, so there will be no uncommitted data in the cache.
>>
>> Josh
> Thank you for quick and precise answer. Now when I actually attempted to
> live migrate ceph based VM I get:
>
> Unable to migrate guest: Invalid relative path
> 'rbd/mail.logics.net.au:rbd_cache=true': Invalid argument
>
> I guess KVM does not like having :rbd_cache=true (migration works
> without it). I know that it is most likely KVM problem but still decided
> to ask here in case if you know about it. Any ideas how to fix it?
>
> Regards,
> Vladimir

Is the destination librbd older and not supporting the cache option?

Migrating with rbd_cache=true and other options specified like that
worked in my testing.

Josh

Re: Ceph and KVM live migration

[Vladimir Bashkirtsev]

On 01/07/12 11:59, Josh Durgin wrote:
> On 06/30/2012 07:15 PM, Vladimir Bashkirtsev wrote:
>> On 01/07/12 10:47, Josh Durgin wrote:
>>> On 06/30/2012 05:42 PM, Vladimir Bashkirtsev wrote:
>>>> Dear all,
>>>>
>>>> Currently I testing KVMs running on ceph and particularly testing 
>>>> recent
>>>> cache feature. Performance is of course vastly improved but still have
>>>> occasional KVM hold ups - not sure who is at blame ceph of KVM. But I
>>>> will deal with it later. Right now I've got myself a question which I
>>>> could not get answered myself: if I do live migration of KVM while 
>>>> there
>>>> some uncommitted data in ceph cache will this cache be committed prior
>>>> cut-over to another host? Reading through the list I've got an
>>>> impression that it may be left uncommitted and thus it may cause data
>>>> corruption. I just would like a simple confirmation if code which
>>>> commits cache on cut-over to new host does exist and no data 
>>>> corruption
>>>> due to RBD cache+live migration should happen.
>>>>
>>>> Regards,
>>>> Vladimir
>>>
>>> QEMU does a flush on all the disks when it stops the guest on the
>>> original host, so there will be no uncommitted data in the cache.
>>>
>>> Josh
>> Thank you for quick and precise answer. Now when I actually attempted to
>> live migrate ceph based VM I get:
>>
>> Unable to migrate guest: Invalid relative path
>> 'rbd/mail.logics.net.au:rbd_cache=true': Invalid argument
>>
>> I guess KVM does not like having :rbd_cache=true (migration works
>> without it). I know that it is most likely KVM problem but still decided
>> to ask here in case if you know about it. Any ideas how to fix it?
>>
>> Regards,
>> Vladimir
>
> Is the destination librbd older and not supporting the cache option?
>
> Migrating with rbd_cache=true and other options specified like that
> worked in my testing.
>
> Josh
Both installations are the same:
qemu 1.0.17
ceph 0.47.3
libvirt 0.9.12

I have googled around and found that if I call migration with --unsafe 
option then it should go. And indeed: it works. Apparently this check 
introduced in libvirt 0.9.12 . Did quick downgrade to libvirt 0.9.11 and 
no problems migrating.

Regards,
Vladimir

Re: Ceph and KVM live migration

[Gregory Farnum]

On Sat, Jun 30, 2012 at 8:21 PM, Vladimir Bashkirtsev
<vladimir@bashkirtsev.com> wrote:
> On 01/07/12 11:59, Josh Durgin wrote:
>>
>> On 06/30/2012 07:15 PM, Vladimir Bashkirtsev wrote:
>>>
>>> On 01/07/12 10:47, Josh Durgin wrote:
>>>>
>>>> On 06/30/2012 05:42 PM, Vladimir Bashkirtsev wrote:
>>>>>
>>>>> Dear all,
>>>>>
>>>>> Currently I testing KVMs running on ceph and particularly testing
>>>>> recent
>>>>> cache feature. Performance is of course vastly improved but still have
>>>>> occasional KVM hold ups - not sure who is at blame ceph of KVM. But I
>>>>> will deal with it later. Right now I've got myself a question which I
>>>>> could not get answered myself: if I do live migration of KVM while
>>>>> there
>>>>> some uncommitted data in ceph cache will this cache be committed prior
>>>>> cut-over to another host? Reading through the list I've got an
>>>>> impression that it may be left uncommitted and thus it may cause data
>>>>> corruption. I just would like a simple confirmation if code which
>>>>> commits cache on cut-over to new host does exist and no data corruption
>>>>> due to RBD cache+live migration should happen.
>>>>>
>>>>> Regards,
>>>>> Vladimir
>>>>
>>>>
>>>> QEMU does a flush on all the disks when it stops the guest on the
>>>> original host, so there will be no uncommitted data in the cache.
>>>>
>>>> Josh
>>>
>>> Thank you for quick and precise answer. Now when I actually attempted to
>>> live migrate ceph based VM I get:
>>>
>>> Unable to migrate guest: Invalid relative path
>>> 'rbd/mail.logics.net.au:rbd_cache=true': Invalid argument
>>>
>>> I guess KVM does not like having :rbd_cache=true (migration works
>>> without it). I know that it is most likely KVM problem but still decided
>>> to ask here in case if you know about it. Any ideas how to fix it?
>>>
>>> Regards,
>>> Vladimir
>>
>>
>> Is the destination librbd older and not supporting the cache option?
>>
>> Migrating with rbd_cache=true and other options specified like that
>> worked in my testing.
>>
>> Josh
>
> Both installations are the same:
> qemu 1.0.17
> ceph 0.47.3
> libvirt 0.9.12
>
> I have googled around and found that if I call migration with --unsafe
> option then it should go. And indeed: it works. Apparently this check
> introduced in libvirt 0.9.12 . Did quick downgrade to libvirt 0.9.11 and no
> problems migrating.

Have we checked if the live migrate actually does do the cache flushes
when you use the unsafe flag? That worries me a little!

In either case, I created a bug so we can try and make QEMU play nice:
http://tracker.newdream.net/issues/2685
-Greg

Re: Ceph and KVM live migration

[Josh Durgin]

On 07/02/2012 11:21 AM, Gregory Farnum wrote:
> On Sat, Jun 30, 2012 at 8:21 PM, Vladimir Bashkirtsev
> <vladimir@bashkirtsev.com>  wrote:
>> On 01/07/12 11:59, Josh Durgin wrote:
>>>
>>> On 06/30/2012 07:15 PM, Vladimir Bashkirtsev wrote:
>>>>
>>>> On 01/07/12 10:47, Josh Durgin wrote:
>>>>>
>>>>> On 06/30/2012 05:42 PM, Vladimir Bashkirtsev wrote:
>>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> Currently I testing KVMs running on ceph and particularly testing
>>>>>> recent
>>>>>> cache feature. Performance is of course vastly improved but still have
>>>>>> occasional KVM hold ups - not sure who is at blame ceph of KVM. But I
>>>>>> will deal with it later. Right now I've got myself a question which I
>>>>>> could not get answered myself: if I do live migration of KVM while
>>>>>> there
>>>>>> some uncommitted data in ceph cache will this cache be committed prior
>>>>>> cut-over to another host? Reading through the list I've got an
>>>>>> impression that it may be left uncommitted and thus it may cause data
>>>>>> corruption. I just would like a simple confirmation if code which
>>>>>> commits cache on cut-over to new host does exist and no data corruption
>>>>>> due to RBD cache+live migration should happen.
>>>>>>
>>>>>> Regards,
>>>>>> Vladimir
>>>>>
>>>>>
>>>>> QEMU does a flush on all the disks when it stops the guest on the
>>>>> original host, so there will be no uncommitted data in the cache.
>>>>>
>>>>> Josh
>>>>
>>>> Thank you for quick and precise answer. Now when I actually attempted to
>>>> live migrate ceph based VM I get:
>>>>
>>>> Unable to migrate guest: Invalid relative path
>>>> 'rbd/mail.logics.net.au:rbd_cache=true': Invalid argument
>>>>
>>>> I guess KVM does not like having :rbd_cache=true (migration works
>>>> without it). I know that it is most likely KVM problem but still decided
>>>> to ask here in case if you know about it. Any ideas how to fix it?
>>>>
>>>> Regards,
>>>> Vladimir
>>>
>>>
>>> Is the destination librbd older and not supporting the cache option?
>>>
>>> Migrating with rbd_cache=true and other options specified like that
>>> worked in my testing.
>>>
>>> Josh
>>
>> Both installations are the same:
>> qemu 1.0.17
>> ceph 0.47.3
>> libvirt 0.9.12
>>
>> I have googled around and found that if I call migration with --unsafe
>> option then it should go. And indeed: it works. Apparently this check
>> introduced in libvirt 0.9.12 . Did quick downgrade to libvirt 0.9.11 and no
>> problems migrating.
>
> Have we checked if the live migrate actually does do the cache flushes
> when you use the unsafe flag? That worries me a little!

The unsafe flag is purely a libvirt mechanism for bypassing libvirt's
format whitelist. It does not affect qemu at all.

> In either case, I created a bug so we can try and make QEMU play nice:
> http://tracker.newdream.net/issues/2685

The issue is with libvirt, not qemu. I sent a patch fixing it to the 
libvirt list:

http://www.redhat.com/archives/libvir-list/2012-July/msg00021.html

Josh

Re: Ceph and KVM live migration

[Christian Brunner]

[-- Attachment #1: Type: text/plain, Size: 2989 bytes --]

On Mon, Jul 02, 2012 at 11:21:40AM -0700, Gregory Farnum wrote:
> On Sat, Jun 30, 2012 at 8:21 PM, Vladimir Bashkirtsev
> <vladimir@bashkirtsev.com> wrote:
> > On 01/07/12 11:59, Josh Durgin wrote:
> >>
> >> On 06/30/2012 07:15 PM, Vladimir Bashkirtsev wrote:
> >>>
> >>> On 01/07/12 10:47, Josh Durgin wrote:
> >>>>
> >>>> On 06/30/2012 05:42 PM, Vladimir Bashkirtsev wrote:
> >>>>>
> >>>>> Dear all,
> >>>>>
> >>>>> Currently I testing KVMs running on ceph and particularly testing
> >>>>> recent
> >>>>> cache feature. Performance is of course vastly improved but still have
> >>>>> occasional KVM hold ups - not sure who is at blame ceph of KVM. But I
> >>>>> will deal with it later. Right now I've got myself a question which I
> >>>>> could not get answered myself: if I do live migration of KVM while
> >>>>> there
> >>>>> some uncommitted data in ceph cache will this cache be committed prior
> >>>>> cut-over to another host? Reading through the list I've got an
> >>>>> impression that it may be left uncommitted and thus it may cause data
> >>>>> corruption. I just would like a simple confirmation if code which
> >>>>> commits cache on cut-over to new host does exist and no data corruption
> >>>>> due to RBD cache+live migration should happen.
> >>>>>
> >>>>> Regards,
> >>>>> Vladimir
> >>>>
> >>>>
> >>>> QEMU does a flush on all the disks when it stops the guest on the
> >>>> original host, so there will be no uncommitted data in the cache.
> >>>>
> >>>> Josh
> >>>
> >>> Thank you for quick and precise answer. Now when I actually attempted to
> >>> live migrate ceph based VM I get:
> >>>
> >>> Unable to migrate guest: Invalid relative path
> >>> 'rbd/mail.logics.net.au:rbd_cache=true': Invalid argument
> >>>
> >>> I guess KVM does not like having :rbd_cache=true (migration works
> >>> without it). I know that it is most likely KVM problem but still decided
> >>> to ask here in case if you know about it. Any ideas how to fix it?
> >>>
> >>> Regards,
> >>> Vladimir
> >>
> >>
> >> Is the destination librbd older and not supporting the cache option?
> >>
> >> Migrating with rbd_cache=true and other options specified like that
> >> worked in my testing.
> >>
> >> Josh
> >
> > Both installations are the same:
> > qemu 1.0.17
> > ceph 0.47.3
> > libvirt 0.9.12
> >
> > I have googled around and found that if I call migration with --unsafe
> > option then it should go. And indeed: it works. Apparently this check
> > introduced in libvirt 0.9.12 . Did quick downgrade to libvirt 0.9.11 and no
> > problems migrating.
> 
> Have we checked if the live migrate actually does do the cache flushes
> when you use the unsafe flag? That worries me a little!
> 
> In either case, I created a bug so we can try and make QEMU play nice:
> http://tracker.newdream.net/issues/2685

I took a quick look at the libvirt code and I think this is an issue in 
libvirt only. The unsafe flag is not handed over to qemu.

You could try the attached patch (untested).

Christian

[-- Attachment #2: 0001-libvirt-allow-migration-for-network-protocols.patch --]
[-- Type: text/plain, Size: 1207 bytes --]

From 36314693f8b9be1f3c77621543adf01d7c51cb88 Mon Sep 17 00:00:00 2001
From: Christian Brunner <chb@muc.de>
Date: Tue, 19 Jun 2012 12:23:38 +0200
Subject: [PATCH] libvirt: allow migration for network protocols

Live migration should be possible with most (all ?) network
protocols, as qemu does a flush right before the migration.

Signed-off-by: Christian Brunner <chb@muc.de>

---
 src/qemu/qemu_migration.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c
index aee613e..6392b98 100644
--- a/src/qemu/qemu_migration.c
+++ b/src/qemu/qemu_migration.c
@@ -848,6 +848,12 @@ qemuMigrationIsSafe(virDomainDefPtr def)
                     return false;
             }
 
+            if (disk->protocol == VIR_DOMAIN_DISK_PROTOCOL_RBD || 
+                disk->protocol == VIR_DOMAIN_DISK_PROTOCOL_NBD ||
+                disk->protocol == VIR_DOMAIN_DISK_PROTOCOL_SHEEPDOG ) {
+                continue;
+            }
+
             qemuReportError(VIR_ERR_MIGRATE_UNSAFE, "%s",
                             _("Migration may lead to data corruption if disks"
                               " use cache != none"));
-- 
1.7.1