[Qemu-devel] [PATCH 0/3] Live migration without shared storage

[Qemu-devel] [PATCH 0/3] Live migration without shared storage

[lirans]

This series adds support for live migration without shared storage, means 
copy the storage while migrating. It was tested with KVM. Supports 2 ways 
to replicate the storage during migration:
 1. Complete copy of storage to destination 
 2. Assuming the storage is cow based, copy only the allocated 
    data, time of the migration will be linear with the amount of allocated 
    data (user responsibility to verify that the same backend file reside 
    on src and destination).

Live migration will work as follows:
(qemu) migrate -d tcp:0:4444 # for ordinary live migration
(qemu) migrate -d blk tcp:0:4444 # for live migration with complete storage copy
(qemu) migrate -d blk inc tcp:0:4444 # for live migration with incremental storage copy, storage is cow based.

The patches are against git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git 
kvm-87

Re: [Qemu-devel] [PATCH 0/3] Live migration without shared storage

[Pierre Riteau]

On 7 sept. 2009, at 11:24, lirans@il.ibm.com wrote:

> This series adds support for live migration without shared storage,  
> means
> copy the storage while migrating. It was tested with KVM. Supports 2  
> ways
> to replicate the storage during migration:
> 1. Complete copy of storage to destination
> 2. Assuming the storage is cow based, copy only the allocated
>    data, time of the migration will be linear with the amount of  
> allocated
>    data (user responsibility to verify that the same backend file  
> reside
>    on src and destination).
>
> Live migration will work as follows:
> (qemu) migrate -d tcp:0:4444 # for ordinary live migration
> (qemu) migrate -d blk tcp:0:4444 # for live migration with complete  
> storage copy
> (qemu) migrate -d blk inc tcp:0:4444 # for live migration with  
> incremental storage copy, storage is cow based.
>
> The patches are against git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
> kvm-87
>
>
>


I'm trying blk migration (not incremental) between two machines  
connected over Gigabit ethernet.
The transfer is quite slow (about 2 MB/s over the wire).
While the load on the sending end is low (vmstat says ~2000 blocks in/ 
sec, and top says ~1% in io wait), on the receiving end I see almost  
40% CPU in io wait, kjournald takes 20% of the CPU and vmstat reports  
~14000 blocks out/sec.

Hosts are running Debian Lenny (2.6.26 32 bits), kvm-87 + your patches.
The guest is also running Debian Lenny and is idle io wise. I tried  
with both idle and full cpu utilization, it doesn't change anything.

-- 
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/

Re: [Qemu-devel] [PATCH 0/3] Live migration without shared storage

[Lennart Sorensen]

On Mon, Sep 07, 2009 at 06:40:51PM +0200, Pierre Riteau wrote:
> I'm trying blk migration (not incremental) between two machines  
> connected over Gigabit ethernet.
> The transfer is quite slow (about 2 MB/s over the wire).
> While the load on the sending end is low (vmstat says ~2000 blocks in/ 
> sec, and top says ~1% in io wait), on the receiving end I see almost 40% 
> CPU in io wait, kjournald takes 20% of the CPU and vmstat reports ~14000 
> blocks out/sec.
>
> Hosts are running Debian Lenny (2.6.26 32 bits), kvm-87 + your patches.
> The guest is also running Debian Lenny and is idle io wise. I tried with 
> both idle and full cpu utilization, it doesn't change anything.

If you happen to be using ext3, then that might explain it.  ext3 has
long had issues with performance while slowly writing a large file
(this affects mythtv a lot for example).  2.6.30 should improve things a
lot, although ext3 will never be great.  ext4 is better, and hopefully
btrfs will be great (when done).  xfs is also supposed to handle such
tasks very well.

-- 
Len Sorensen

Re: [Qemu-devel] [PATCH 0/3] Live migration without shared storage

[Liran Schour]

lsorense@csclub.uwaterloo.ca (Lennart Sorensen) wrote on 08/09/2009
16:26:31:

> On Mon, Sep 07, 2009 at 06:40:51PM +0200, Pierre Riteau wrote:
> > I'm trying blk migration (not incremental) between two machines
> > connected over Gigabit ethernet.
> > The transfer is quite slow (about 2 MB/s over the wire).
> > While the load on the sending end is low (vmstat says ~2000 blocks in/
> > sec, and top says ~1% in io wait), on the receiving end I see almost
40%
> > CPU in io wait, kjournald takes 20% of the CPU and vmstat reports
~14000
> > blocks out/sec.
> >
> > Hosts are running Debian Lenny (2.6.26 32 bits), kvm-87 + your patches.
> > The guest is also running Debian Lenny and is idle io wise. I tried
with
> > both idle and full cpu utilization, it doesn't change anything.
>
> If you happen to be using ext3, then that might explain it.  ext3 has
> long had issues with performance while slowly writing a large file
> (this affects mythtv a lot for example).  2.6.30 should improve things a
> lot, although ext3 will never be great.  ext4 is better, and hopefully
> btrfs will be great (when done).  xfs is also supposed to handle such
> tasks very well.
>
> --
> Len Sorensen

I found performance issues in the patch itself. I will first submit a new
patch
and maybe then will be able to talk about the fs performance.

Thanks,
- Liran

Re: [Qemu-devel] [PATCH 0/3] Live migration without shared storage

[Pierre Riteau]

On 8 sept. 2009, at 15:26, Lennart Sorensen wrote:

> On Mon, Sep 07, 2009 at 06:40:51PM +0200, Pierre Riteau wrote:
>> I'm trying blk migration (not incremental) between two machines
>> connected over Gigabit ethernet.
>> The transfer is quite slow (about 2 MB/s over the wire).
>> While the load on the sending end is low (vmstat says ~2000 blocks  
>> in/
>> sec, and top says ~1% in io wait), on the receiving end I see  
>> almost 40%
>> CPU in io wait, kjournald takes 20% of the CPU and vmstat reports  
>> ~14000
>> blocks out/sec.
>>
>> Hosts are running Debian Lenny (2.6.26 32 bits), kvm-87 + your  
>> patches.
>> The guest is also running Debian Lenny and is idle io wise. I tried  
>> with
>> both idle and full cpu utilization, it doesn't change anything.
>
> If you happen to be using ext3, then that might explain it.  ext3 has
> long had issues with performance while slowly writing a large file
> (this affects mythtv a lot for example).  2.6.30 should improve  
> things a
> lot, although ext3 will never be great.  ext4 is better, and hopefully
> btrfs will be great (when done).  xfs is also supposed to handle such
> tasks very well.
>
> -- 
> Len Sorensen


Yes, I'm using ext3. When I tested last week, I created the disk image  
on the destination machine using qemu-img create -f raw, which creates  
a sparse file.
I experimented again today, but this time I allocated all the blocks  
in the disk image prior to the migration.
This resulted in increased throughput (between 6 and 8 MB/s on the  
wire), reduced CPU usage on the destination node (now qemu is the top  
CPU user, not kjournald), and the IO interrupt numbers are similar on  
the origin and destination node (about 7000/s).
Still, it could be better, but I see Liran is working on it :)

-- 
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/

Re: [Qemu-devel] [PATCH 0/3] Live migration without shared storage

[Liran Schour]

qemu-devel-bounces+lirans=il.ibm.com@nongnu.org wrote on 07/09/2009
19:40:51:

> On 7 sept. 2009, at 11:24, lirans@il.ibm.com wrote:
>
> > This series adds support for live migration without shared storage,
> > means
> > copy the storage while migrating. It was tested with KVM. Supports 2
> > ways
> > to replicate the storage during migration:
> > 1. Complete copy of storage to destination
> > 2. Assuming the storage is cow based, copy only the allocated
> >    data, time of the migration will be linear with the amount of
> > allocated
> >    data (user responsibility to verify that the same backend file
> > reside
> >    on src and destination).
> >
> > Live migration will work as follows:
> > (qemu) migrate -d tcp:0:4444 # for ordinary live migration
> > (qemu) migrate -d blk tcp:0:4444 # for live migration with complete
> > storage copy
> > (qemu) migrate -d blk inc tcp:0:4444 # for live migration with
> > incremental storage copy, storage is cow based.
> >
> > The patches are against
git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
> > kvm-87
> >
> >
> >
>
>
> I'm trying blk migration (not incremental) between two machines
> connected over Gigabit ethernet.
> The transfer is quite slow (about 2 MB/s over the wire).
> While the load on the sending end is low (vmstat says ~2000 blocks in/
> sec, and top says ~1% in io wait), on the receiving end I see almost
> 40% CPU in io wait, kjournald takes 20% of the CPU and vmstat reports
> ~14000 blocks out/sec.
>
> Hosts are running Debian Lenny (2.6.26 32 bits), kvm-87 + your patches.
> The guest is also running Debian Lenny and is idle io wise. I tried
> with both idle and full cpu utilization, it doesn't change anything.
>
> --
> Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
>

True I see that there is a problem with performance. Need to change the
destination
side to write async to disk And in the source side to submit async read
from disk at
least at the rate of the network bandwidth. I will resend a new patch
series.

Thanks,
- Liran