KVM in HA active/active + fault-tolerant configuration

KVM in HA active/active + fault-tolerant configuration

[g.danti]

Hi all,
I have a question about Linux KVM HA cluster.

I understand that in a HA setup I can live migrate virtual machine 
between host that shares the same storage (via various methods, eg: 
DRDB). This enable us to migrate the VMs based on hosts loads and 
performance.

ìMy current understanding is that, with this setup, an host crash will 
cause the VMs to be restarded on another host.

However, I wonder if there is a method to have a fully fault-tolerant 
HA configuration, where for "fully fault-tolerant" I means that an host 
crash (eg: power failures) will cause the VMs to be migrated to another 
hosts with no state change. In other word: it is possible to have an 
always-synchronized (both disk & memory) VM instance on another host, so 
that the migrated VM does not need to be restarted but only 
restored/unpaused? For disk data synchronization we can use shared 
storages (bypassing the problem) or something similar do DRDB, but what 
about memory?

Thank you,
regards.

Re: KVM in HA active/active + fault-tolerant configuration

[Brian Jackson]

On Wednesday, August 21, 2013 6:02:31 AM CDT, g.danti@assyoma.it wrote:
> Hi all,
> I have a question about Linux KVM HA cluster.
>
> I understand that in a HA setup I can live migrate virtual 
> machine between host that shares the same storage (via various 
> methods, eg: DRDB). This enable us to migrate the VMs based on 
> hosts loads and performance.
>
> ìMy current understanding is that, with this setup, an host 
> crash will cause the VMs to be restarded on another host.
>
> However, I wonder if there is a method to have a fully 
> fault-tolerant HA configuration, where for "fully 
> fault-tolerant" I means that an host crash (eg: power failures) 
> will cause the VMs to be migrated to another hosts with no state 
> change. In other word: it is possible to have an 
> always-synchronized (both disk & memory) VM instance on another 
> host, so that the migrated VM does not need to be restarted but 
> only restored/unpaused? For disk data synchronization we can use 
> shared storages (bypassing the problem) or something similar do 
> DRDB, but what about memory?


You're looking for something that doesn't exist for KVM. There was a project once for it called Kemari, but afaik, it's been abandoned for a while.


>
> Thank you,
> regards.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: KVM in HA active/active + fault-tolerant configuration

[g.danti]

On 2013-08-21 21:40, Brian Jackson wrote:
> On Wednesday, August 21, 2013 6:02:31 AM CDT, g.danti@assyoma.it 
> wrote:
>> Hi all,
>> I have a question about Linux KVM HA cluster.
>> 
>> I understand that in a HA setup I can live migrate virtual machine 
>> between host that shares the same storage (via various methods, eg: 
>> DRDB). This enable us to migrate the VMs based on hosts loads and 
>> performance.
>> 
>> ìMy current understanding is that, with this setup, an host crash 
>> will cause the VMs to be restarded on another host.
>> 
>> However, I wonder if there is a method to have a fully fault-tolerant 
>> HA configuration, where for "fully fault-tolerant" I means that an 
>> host crash (eg: power failures) will cause the VMs to be migrated to 
>> another hosts with no state change. In other word: it is possible to 
>> have an always-synchronized (both disk & memory) VM instance on 
>> another host, so that the migrated VM does not need to be restarted 
>> but only restored/unpaused? For disk data synchronization we can use 
>> shared storages (bypassing the problem) or something similar do DRDB, 
>> but what about memory?
> 
> 
> You're looking for something that doesn't exist for KVM. There was a
> project once for it called Kemari, but afaik, it's been abandoned for
> a while.

Hi Brian,
thank you for your reply.

As I googled extensively without finding anything, I was prepared to a 
similar response.

Anyway, from what I understand, Qemu already use a similar approach 
(tracking dirty memory pages) when live migrating virtual machines to 
another host.

So what is missing is the "glue code" between Qemu and KVM/libvirt 
stack, right?

Thanks again.

> 
> 
>> 
>> Thank you,
>> regards.
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
>> 

Re: KVM in HA active/active + fault-tolerant configuration

[Brian Jackson]

On Wednesday, August 21, 2013 3:49:09 PM CDT, g.danti@assyoma.it wrote:
> On 2013-08-21 21:40, Brian Jackson wrote:
>> On Wednesday, August 21, 2013 6:02:31 AM CDT, 
>> g.danti@assyoma.it wrote: ...
>
> Hi Brian,
> thank you for your reply.
>
> As I googled extensively without finding anything, I was 
> prepared to a similar response.
>
> Anyway, from what I understand, Qemu already use a similar 
> approach (tracking dirty memory pages) when live migrating 
> virtual machines to another host.
>
> So what is missing is the "glue code" between Qemu and 
> KVM/libvirt stack, right?

Live migration isn't what you asked about (at least not from what I understood). Live migration is just moving a VM from one host to another. That is definitely supported by libvirt. Having a constantly running lock-step sync of guest state is what Qemu/KVM does not support. So with Qemu's current live migration abilities, if HostA dies, all it's guests will have downtime while they are restarted on other hosts.


>
> Thanks again.
>
>>  ...
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernelorg
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: KVM in HA active/active + fault-tolerant configuration

[Timon Wang]

On Thu, Aug 22, 2013 at 5:47 AM, Brian Jackson <iggy@theiggy.com> wrote:
> On Wednesday, August 21, 2013 3:49:09 PM CDT, g.danti@assyoma.it wrote:
>>
>> On 2013-08-21 21:40, Brian Jackson wrote:
>>>
>>> On Wednesday, August 21, 2013 6:02:31 AM CDT, g.danti@assyoma.it wrote:
>>> ...
>>
>>
>> Hi Brian,
>> thank you for your reply.
>>
>> As I googled extensively without finding anything, I was prepared to a
>> similar response.
>>
>> Anyway, from what I understand, Qemu already use a similar approach
>> (tracking dirty memory pages) when live migrating virtual machines to
>> another host.
>>
>> So what is missing is the "glue code" between Qemu and KVM/libvirt stack,
>> right?
>
>
> Live migration isn't what you asked about (at least not from what I
> understood). Live migration is just moving a VM from one host to another.
> That is definitely supported by libvirt. Having a constantly running
> lock-step sync of guest state is what Qemu/KVM does not support. So with
> Qemu's current live migration abilities, if HostA dies, all it's guests will
> have downtime while they are restarted on other hosts.
>
>

Live migration is not proper for support VM HA, when host went wrong,
these VMs on the host must be restart on another host based on the
same storage.

I have googled for KVM FT for a while, but only find a project called
Kemari which was no longer updated.
I read some article about KVM FT, which said KVM may be support it in future.


>>
>> Thanks again.
>>
>>>  ...
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernelorg
>>
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: KVM in HA active/active + fault-tolerant configuration

[g.danti]

On 2013-08-22 02:35, Timon Wang wrote:
> On Thu, Aug 22, 2013 at 5:47 AM, Brian Jackson <iggy@theiggy.com> 
> wrote:
>> On Wednesday, August 21, 2013 3:49:09 PM CDT, g.danti@assyoma.it 
>> wrote:
>>> 
>>> On 2013-08-21 21:40, Brian Jackson wrote:
>>>> 
>>>> On Wednesday, August 21, 2013 6:02:31 AM CDT, g.danti@assyoma.it 
>>>> wrote:
>>>> ...
>>> 
>>> 
>>> Hi Brian,
>>> thank you for your reply.
>>> 
>>> As I googled extensively without finding anything, I was prepared to 
>>> a
>>> similar response.
>>> 
>>> Anyway, from what I understand, Qemu already use a similar approach
>>> (tracking dirty memory pages) when live migrating virtual machines 
>>> to
>>> another host.
>>> 
>>> So what is missing is the "glue code" between Qemu and KVM/libvirt 
>>> stack,
>>> right?
>> 
>> 
>> Live migration isn't what you asked about (at least not from what I
>> understood). Live migration is just moving a VM from one host to 
>> another.
>> That is definitely supported by libvirt. Having a constantly running
>> lock-step sync of guest state is what Qemu/KVM does not support. So 
>> with
>> Qemu's current live migration abilities, if HostA dies, all it's 
>> guests will
>> have downtime while they are restarted on other hosts.
>> 
>> 
> 
> Live migration is not proper for support VM HA, when host went wrong,
> these VMs on the host must be restart on another host based on the
> same storage.
> 
> I have googled for KVM FT for a while, but only find a project called
> Kemari which was no longer updated.
> I read some article about KVM FT, which said KVM may be support it in 
> future.
> 
> 

Yes, I understand that live migration is not proper to VM HA/FT, but 
the basic working principles (track and copy dirty pages) is already at 
work in live migration. I was thinking to something along what Zheng 
suggested ("hot backup" with very frequent/continuous memory 
synchronization from a "master" guest).

I totally overlooked Kemari, glad to see that someone point me to it. 
Shame that is dead (at the moment).

Thanks you all guys.

>>> 
>>> Thanks again.
>>> 
>>>>  ...
>>> 
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernelorg
>>> 
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> 
>>> 
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM in HA active/active + fault-tolerant configuration

[Fam Zheng]

On Wed, 08/21 22:49, g.danti@assyoma.it wrote:
> On 2013-08-21 21:40, Brian Jackson wrote:
> >On Wednesday, August 21, 2013 6:02:31 AM CDT, g.danti@assyoma.it
> >wrote:
> >>Hi all,
> >>I have a question about Linux KVM HA cluster.
> >>
> >>I understand that in a HA setup I can live migrate virtual
> >>machine between host that shares the same storage (via various
> >>methods, eg: DRDB). This enable us to migrate the VMs based on
> >>hosts loads and performance.
> >>
> >>ìMy current understanding is that, with this setup, an host
> >>crash will cause the VMs to be restarded on another host.
> >>
> >>However, I wonder if there is a method to have a fully
> >>fault-tolerant HA configuration, where for "fully
> >>fault-tolerant" I means that an host crash (eg: power failures)
> >>will cause the VMs to be migrated to another hosts with no state
> >>change. In other word: it is possible to have an
> >>always-synchronized (both disk & memory) VM instance on another
> >>host, so that the migrated VM does not need to be restarted but
> >>only restored/unpaused? For disk data synchronization we can use
> >>shared storages (bypassing the problem) or something similar do
> >>DRDB, but what about memory?
> >
> >
> >You're looking for something that doesn't exist for KVM. There was a
> >project once for it called Kemari, but afaik, it's been abandoned for
> >a while.
> 
> Hi Brian,
> thank you for your reply.
> 
> As I googled extensively without finding anything, I was prepared to
> a similar response.
> 
> Anyway, from what I understand, Qemu already use a similar approach
> (tracking dirty memory pages) when live migrating virtual machines
> to another host.
> 
Active/active sounds not easy to get, as it seem to me, since you'll
need to make sure the VMs on both nodes are always in the same state all
the time, that sounds impossible for two emulator processes on two
different hosts. I think hot spare is more practical: in background you
repetitively trigger migration of delta memory and copy to hot spare,
but don't start to run it. Once the active one fails, you can resume the
running of hot spare, which is at a latest checkpoint. But I think this
needs to some work on current live migration code.

Fam

Re: KVM in HA active/active + fault-tolerant configuration

[Stefan Hajnoczi]

On Wed, Aug 21, 2013 at 01:02:31PM +0200, g.danti@assyoma.it wrote:
> However, I wonder if there is a method to have a fully
> fault-tolerant HA configuration, where for "fully fault-tolerant" I
> means that an host crash (eg: power failures) will cause the VMs to
> be migrated to another hosts with no state change. In other word: it
> is possible to have an always-synchronized (both disk & memory) VM
> instance on another host, so that the migrated VM does not need to
> be restarted but only restored/unpaused? For disk data
> synchronization we can use shared storages (bypassing the problem)
> or something similar do DRDB, but what about memory?

Remember that you can set up HA inside the guests like you do on
physical machines.  Hypervisor support is not necessary if you use
existing application-level HA features.

Stefan

Re: KVM in HA active/active + fault-tolerant configuration

[Timon Wang]

Application-level HA is not a solution for all environment, I can't
setup an oracle rac cluster or a windows failover cluster based a host
use FC as the storage which already asked in kvm@veger.kernel.org
list.

Hope some body will found same HA solution.

On Thu, Aug 22, 2013 at 3:35 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Wed, Aug 21, 2013 at 01:02:31PM +0200, g.danti@assyoma.it wrote:
>> However, I wonder if there is a method to have a fully
>> fault-tolerant HA configuration, where for "fully fault-tolerant" I
>> means that an host crash (eg: power failures) will cause the VMs to
>> be migrated to another hosts with no state change. In other word: it
>> is possible to have an always-synchronized (both disk & memory) VM
>> instance on another host, so that the migrated VM does not need to
>> be restarted but only restored/unpaused? For disk data
>> synchronization we can use shared storages (bypassing the problem)
>> or something similar do DRDB, but what about memory?
>
> Remember that you can set up HA inside the guests like you do on
> physical machines.  Hypervisor support is not necessary if you use
> existing application-level HA features.
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net