Re: [Qemu-devel] Memory sync algorithm during migration

Re: [Qemu-devel] Memory sync algorithm during migration

[Takuya Yoshikawa]

Adding qemu-devel ML to CC.

Your question should have been sent to qemu-devel ML because the logic
is implemented in QEMU, not KVM.

(2011/11/11 1:35), Oliver Hookins wrote:
> Hi,
>
> I am performing some benchmarks on KVM migration on two different types of VM.
> One has 4GB RAM and the other 32GB. More or less idle, the 4GB VM takes about 20
> seconds to migrate on our hardware while the 32GB VM takes about a minute.
>
> With a reasonable amount of memory activity going on (in the hundreds of MB per
> second) the 32GB VM takes 3.5 minutes to migrate, but the 4GB VM never
> completes. Intuitively this tells me there is some watermarking of dirty pages
> going on that is not particularly efficient when the dirty pages ratio is high
> compared to total memory, but I may be completely incorrect.

You can change the ratio IIRC.
Hopefully, someone who knows well about QEMU will tell you better ways.

	Takuya

>
> Could anybody fill me in on what might be going on here? We're using libvirt
> 0.8.2 and kvm-83-224.el5.centos.1
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Qemu-devel] Memory sync algorithm during migration

[Juan Quintela]

Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp> wrote:
> Adding qemu-devel ML to CC.
>
> Your question should have been sent to qemu-devel ML because the logic
> is implemented in QEMU, not KVM.
>
> (2011/11/11 1:35), Oliver Hookins wrote:
>> Hi,
>>
>> I am performing some benchmarks on KVM migration on two different types of VM.
>> One has 4GB RAM and the other 32GB. More or less idle, the 4GB VM takes about 20
>> seconds to migrate on our hardware while the 32GB VM takes about a minute.
>>
>> With a reasonable amount of memory activity going on (in the hundreds of MB per
>> second) the 32GB VM takes 3.5 minutes to migrate, but the 4GB VM never
>> completes. Intuitively this tells me there is some watermarking of dirty pages
>> going on that is not particularly efficient when the dirty pages ratio is high
>> compared to total memory, but I may be completely incorrect.

> You can change the ratio IIRC.
> Hopefully, someone who knows well about QEMU will tell you better ways.
>
> 	Takuya
>
>>
>> Could anybody fill me in on what might be going on here? We're using libvirt
>> 0.8.2 and kvm-83-224.el5.centos.1

This is pretty old qemu/kvm code base.
In principle, it makes no sense that with 32GB RAM migration finishes,
and with 4GB RAM it is unable (intuitively it should be, if ever, the
other way around).

Do you have an easy test that makes the problem easily reproducible?
Have you tried ustream qemu.git? (some improvements on that department).

Later, Juan.

Re: [Qemu-devel] Memory sync algorithm during migration

[Oliver Hookins]

On Tue, Nov 15, 2011 at 11:47:58AM +0100, ext Juan Quintela wrote:
> Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp> wrote:
> > Adding qemu-devel ML to CC.
> >
> > Your question should have been sent to qemu-devel ML because the logic
> > is implemented in QEMU, not KVM.
> >
> > (2011/11/11 1:35), Oliver Hookins wrote:
> >> Hi,
> >>
> >> I am performing some benchmarks on KVM migration on two different types of VM.
> >> One has 4GB RAM and the other 32GB. More or less idle, the 4GB VM takes about 20
> >> seconds to migrate on our hardware while the 32GB VM takes about a minute.
> >>
> >> With a reasonable amount of memory activity going on (in the hundreds of MB per
> >> second) the 32GB VM takes 3.5 minutes to migrate, but the 4GB VM never
> >> completes. Intuitively this tells me there is some watermarking of dirty pages
> >> going on that is not particularly efficient when the dirty pages ratio is high
> >> compared to total memory, but I may be completely incorrect.
> 
> > You can change the ratio IIRC.
> > Hopefully, someone who knows well about QEMU will tell you better ways.
> >
> > 	Takuya
> >
> >>
> >> Could anybody fill me in on what might be going on here? We're using libvirt
> >> 0.8.2 and kvm-83-224.el5.centos.1
> 
> This is pretty old qemu/kvm code base.
> In principle, it makes no sense that with 32GB RAM migration finishes,
> and with 4GB RAM it is unable (intuitively it should be, if ever, the
> other way around).

If the syncing of dirty pages is based on some kind of ratio that causes it to
resync everything if more than a certain percentage of total pages are dirty
then this could explain it, as with a smaller amount of memory it could be
continually restarting.

I can certainly state from observation that the network traffic continues at
high speed the whole time so perhaps this makes some sense. I was hoping that
this behaviour would cause someone to recall a problem like this that was solved
- maybe this is not the case!

> 
> Do you have an easy test that makes the problem easily reproducible?
> Have you tried ustream qemu.git? (some improvements on that department).

I haven't just yet, but I'll give this a try. Thanks for the suggestion!

Re: [Qemu-devel] Memory sync algorithm during migration

[Oliver Hookins]

On Tue, Nov 15, 2011 at 11:47:58AM +0100, ext Juan Quintela wrote:
> Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp> wrote:
> > Adding qemu-devel ML to CC.
> >
> > Your question should have been sent to qemu-devel ML because the logic
> > is implemented in QEMU, not KVM.
> >
> > (2011/11/11 1:35), Oliver Hookins wrote:
> >> Hi,
> >>
> >> I am performing some benchmarks on KVM migration on two different types of VM.
> >> One has 4GB RAM and the other 32GB. More or less idle, the 4GB VM takes about 20
> >> seconds to migrate on our hardware while the 32GB VM takes about a minute.
> >>
> >> With a reasonable amount of memory activity going on (in the hundreds of MB per
> >> second) the 32GB VM takes 3.5 minutes to migrate, but the 4GB VM never
> >> completes. Intuitively this tells me there is some watermarking of dirty pages
> >> going on that is not particularly efficient when the dirty pages ratio is high
> >> compared to total memory, but I may be completely incorrect.
> 
> > You can change the ratio IIRC.
> > Hopefully, someone who knows well about QEMU will tell you better ways.
> >
> > 	Takuya
> >
> >>
> >> Could anybody fill me in on what might be going on here? We're using libvirt
> >> 0.8.2 and kvm-83-224.el5.centos.1
> 
> This is pretty old qemu/kvm code base.
> In principle, it makes no sense that with 32GB RAM migration finishes,
> and with 4GB RAM it is unable (intuitively it should be, if ever, the
> other way around).
> 
> Do you have an easy test that makes the problem easily reproducible?
> Have you tried ustream qemu.git? (some improvements on that department).

On the surface this seems to be a bit beyond me. Is it reasonably
straightforward porting the patches from qemu-kvm to qemu? At the moment I'm
going to attempt just building the latest qemu-kvm (0.15 as opposed to 0.9.1)
- should this be enough?

Re: [Qemu-devel] Memory sync algorithm during migration

[Oliver Hookins]

On Tue, Nov 15, 2011 at 11:47:58AM +0100, ext Juan Quintela wrote:
> Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp> wrote:
> > Adding qemu-devel ML to CC.
> >
> > Your question should have been sent to qemu-devel ML because the logic
> > is implemented in QEMU, not KVM.
> >
> > (2011/11/11 1:35), Oliver Hookins wrote:
> >> Hi,
> >>
> >> I am performing some benchmarks on KVM migration on two different types of VM.
> >> One has 4GB RAM and the other 32GB. More or less idle, the 4GB VM takes about 20
> >> seconds to migrate on our hardware while the 32GB VM takes about a minute.
> >>
> >> With a reasonable amount of memory activity going on (in the hundreds of MB per
> >> second) the 32GB VM takes 3.5 minutes to migrate, but the 4GB VM never
> >> completes. Intuitively this tells me there is some watermarking of dirty pages
> >> going on that is not particularly efficient when the dirty pages ratio is high
> >> compared to total memory, but I may be completely incorrect.
> 
> > You can change the ratio IIRC.
> > Hopefully, someone who knows well about QEMU will tell you better ways.
> >
> > 	Takuya
> >
> >>
> >> Could anybody fill me in on what might be going on here? We're using libvirt
> >> 0.8.2 and kvm-83-224.el5.centos.1
> 
> This is pretty old qemu/kvm code base.
> In principle, it makes no sense that with 32GB RAM migration finishes,
> and with 4GB RAM it is unable (intuitively it should be, if ever, the
> other way around).
> 
> Do you have an easy test that makes the problem easily reproducible?
> Have you tried ustream qemu.git? (some improvements on that department).

I've just tried the qemu-kvm 0.14.1 tag which seems to be the latest that builds
on my platform. For some strange reason migrations always seem to fail in one
direction with "Unknown savevm section or instance 'hpet' 0" messages.

This seems to point to different migration protocols on either end but they are
both running the same version of qemu-kvm I built. Does this ring any bells for
anyone?

Re: [Qemu-devel] Memory sync algorithm during migration

[Juan Quintela]

Oliver Hookins <oliver.hookins@nokia.com> wrote:
> On Tue, Nov 15, 2011 at 11:47:58AM +0100, ext Juan Quintela wrote:
>> Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp> wrote:
>> > Adding qemu-devel ML to CC.
>> >
>> > Your question should have been sent to qemu-devel ML because the logic
>> > is implemented in QEMU, not KVM.
>> >
>> > (2011/11/11 1:35), Oliver Hookins wrote:
>> >> Hi,
>> >>
>> >> I am performing some benchmarks on KVM migration on two different types of VM.
>> >> One has 4GB RAM and the other 32GB. More or less idle, the 4GB VM takes about 20
>> >> seconds to migrate on our hardware while the 32GB VM takes about a minute.
>> >>
>> >> With a reasonable amount of memory activity going on (in the hundreds of MB per
>> >> second) the 32GB VM takes 3.5 minutes to migrate, but the 4GB VM never
>> >> completes. Intuitively this tells me there is some watermarking of dirty pages
>> >> going on that is not particularly efficient when the dirty pages ratio is high
>> >> compared to total memory, but I may be completely incorrect.
>> 
>> > You can change the ratio IIRC.
>> > Hopefully, someone who knows well about QEMU will tell you better ways.
>> >
>> > 	Takuya
>> >
>> >>
>> >> Could anybody fill me in on what might be going on here? We're using libvirt
>> >> 0.8.2 and kvm-83-224.el5.centos.1
>> 
>> This is pretty old qemu/kvm code base.
>> In principle, it makes no sense that with 32GB RAM migration finishes,
>> and with 4GB RAM it is unable (intuitively it should be, if ever, the
>> other way around).
>> 
>> Do you have an easy test that makes the problem easily reproducible?
>> Have you tried ustream qemu.git? (some improvements on that department).
>
> I've just tried the qemu-kvm 0.14.1 tag which seems to be the latest that builds
> on my platform. For some strange reason migrations always seem to fail in one
> direction with "Unknown savevm section or instance 'hpet' 0" messages.

What is your platform?  This seems like you are running with hpet in one
side, but without it in the other.  What command line are you using?

> This seems to point to different migration protocols on either end but they are
> both running the same version of qemu-kvm I built. Does this ring any bells for
> anyone?

Command line mismatch.  But, what is your platform?

Later, Juan.

Re: [Qemu-devel] Memory sync algorithm during migration

[Oliver Hookins]

On Tue, Nov 22, 2011 at 10:31:58AM +0100, ext Juan Quintela wrote:
> Oliver Hookins <oliver.hookins@nokia.com> wrote:
> > On Tue, Nov 15, 2011 at 11:47:58AM +0100, ext Juan Quintela wrote:
> >> Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp> wrote:
> >> > Adding qemu-devel ML to CC.
> >> >
> >> > Your question should have been sent to qemu-devel ML because the logic
> >> > is implemented in QEMU, not KVM.
> >> >
> >> > (2011/11/11 1:35), Oliver Hookins wrote:
> >> >> Hi,
> >> >>
> >> >> I am performing some benchmarks on KVM migration on two different types of VM.
> >> >> One has 4GB RAM and the other 32GB. More or less idle, the 4GB VM takes about 20
> >> >> seconds to migrate on our hardware while the 32GB VM takes about a minute.
> >> >>
> >> >> With a reasonable amount of memory activity going on (in the hundreds of MB per
> >> >> second) the 32GB VM takes 3.5 minutes to migrate, but the 4GB VM never
> >> >> completes. Intuitively this tells me there is some watermarking of dirty pages
> >> >> going on that is not particularly efficient when the dirty pages ratio is high
> >> >> compared to total memory, but I may be completely incorrect.
> >> 
> >> > You can change the ratio IIRC.
> >> > Hopefully, someone who knows well about QEMU will tell you better ways.
> >> >
> >> > 	Takuya
> >> >
> >> >>
> >> >> Could anybody fill me in on what might be going on here? We're using libvirt
> >> >> 0.8.2 and kvm-83-224.el5.centos.1
> >> 
> >> This is pretty old qemu/kvm code base.
> >> In principle, it makes no sense that with 32GB RAM migration finishes,
> >> and with 4GB RAM it is unable (intuitively it should be, if ever, the
> >> other way around).
> >> 
> >> Do you have an easy test that makes the problem easily reproducible?
> >> Have you tried ustream qemu.git? (some improvements on that department).
> >
> > I've just tried the qemu-kvm 0.14.1 tag which seems to be the latest that builds
> > on my platform. For some strange reason migrations always seem to fail in one
> > direction with "Unknown savevm section or instance 'hpet' 0" messages.
> 
> What is your platform?  This seems like you are running with hpet in one
> side, but without it in the other.  What command line are you using?

Yes, my mistake. We were also testing later kernels and my test machines managed
to get out of sync. One had support for hpet clocksource but the other one
didn't.

> 
> > This seems to point to different migration protocols on either end but they are
> > both running the same version of qemu-kvm I built. Does this ring any bells for
> > anyone?
> 
> Command line mismatch.  But, what is your platform?

CentOS5.6. Now running the VMs through qemu-kvm 0.14.1, unloaded migrations take
about half the time but with memory I/O load now both VMs never complete the
migration. In practical terms I'm writing about 50MB/s into memory and we have a
10Gbps network (and I've seen real speeds up to 8-9Gbps on the wire) so there
should be enough capacity to sync up the dirty pages.

So now the 32GB and 4GB VMs have matching behaviour (which makes more sense) but
I'm not any closer to figuring out what is going on.

Re: [Qemu-devel] Memory sync algorithm during migration

[Pierre Riteau]

On 22 nov. 2011, at 14:04, Oliver Hookins wrote:

> On Tue, Nov 22, 2011 at 10:31:58AM +0100, ext Juan Quintela wrote:
>> Oliver Hookins <oliver.hookins@nokia.com> wrote:
>>> On Tue, Nov 15, 2011 at 11:47:58AM +0100, ext Juan Quintela wrote:
>>>> Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp> wrote:
>>>>> Adding qemu-devel ML to CC.
>>>>> 
>>>>> Your question should have been sent to qemu-devel ML because the logic
>>>>> is implemented in QEMU, not KVM.
>>>>> 
>>>>> (2011/11/11 1:35), Oliver Hookins wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I am performing some benchmarks on KVM migration on two different types of VM.
>>>>>> One has 4GB RAM and the other 32GB. More or less idle, the 4GB VM takes about 20
>>>>>> seconds to migrate on our hardware while the 32GB VM takes about a minute.
>>>>>> 
>>>>>> With a reasonable amount of memory activity going on (in the hundreds of MB per
>>>>>> second) the 32GB VM takes 3.5 minutes to migrate, but the 4GB VM never
>>>>>> completes. Intuitively this tells me there is some watermarking of dirty pages
>>>>>> going on that is not particularly efficient when the dirty pages ratio is high
>>>>>> compared to total memory, but I may be completely incorrect.
>>>> 
>>>>> You can change the ratio IIRC.
>>>>> Hopefully, someone who knows well about QEMU will tell you better ways.
>>>>> 
>>>>> 	Takuya
>>>>> 
>>>>>> 
>>>>>> Could anybody fill me in on what might be going on here? We're using libvirt
>>>>>> 0.8.2 and kvm-83-224.el5.centos.1
>>>> 
>>>> This is pretty old qemu/kvm code base.
>>>> In principle, it makes no sense that with 32GB RAM migration finishes,
>>>> and with 4GB RAM it is unable (intuitively it should be, if ever, the
>>>> other way around).
>>>> 
>>>> Do you have an easy test that makes the problem easily reproducible?
>>>> Have you tried ustream qemu.git? (some improvements on that department).
>>> 
>>> I've just tried the qemu-kvm 0.14.1 tag which seems to be the latest that builds
>>> on my platform. For some strange reason migrations always seem to fail in one
>>> direction with "Unknown savevm section or instance 'hpet' 0" messages.
>> 
>> What is your platform?  This seems like you are running with hpet in one
>> side, but without it in the other.  What command line are you using?
> 
> Yes, my mistake. We were also testing later kernels and my test machines managed
> to get out of sync. One had support for hpet clocksource but the other one
> didn't.
> 
>> 
>>> This seems to point to different migration protocols on either end but they are
>>> both running the same version of qemu-kvm I built. Does this ring any bells for
>>> anyone?
>> 
>> Command line mismatch.  But, what is your platform?
> 
> CentOS5.6. Now running the VMs through qemu-kvm 0.14.1, unloaded migrations take
> about half the time but with memory I/O load now both VMs never complete the
> migration. In practical terms I'm writing about 50MB/s into memory and we have a
> 10Gbps network (and I've seen real speeds up to 8-9Gbps on the wire) so there
> should be enough capacity to sync up the dirty pages.
> 
> So now the 32GB and 4GB VMs have matching behaviour (which makes more sense) but
> I'm not any closer to figuring out what is going on.

Did you modify the max downtime? The default is 30 ms. At 8 Gbps, this only allows to send 30 MB of data on the wire.

-- 
Pierre Riteau -- PhD student, Myriads team, IRISA, Rennes, France
http://perso.univ-rennes1.fr/pierre.riteau/

Re: [Qemu-devel] Memory sync algorithm during migration

[Pierre Riteau]

On 22 nov. 2011, at 14:04, Oliver Hookins wrote:

> On Tue, Nov 22, 2011 at 10:31:58AM +0100, ext Juan Quintela wrote:
>> Oliver Hookins <oliver.hookins@nokia.com> wrote:
>>> On Tue, Nov 15, 2011 at 11:47:58AM +0100, ext Juan Quintela wrote:
>>>> Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp> wrote:
>>>>> Adding qemu-devel ML to CC.
>>>>> 
>>>>> Your question should have been sent to qemu-devel ML because the logic
>>>>> is implemented in QEMU, not KVM.
>>>>> 
>>>>> (2011/11/11 1:35), Oliver Hookins wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I am performing some benchmarks on KVM migration on two different types of VM.
>>>>>> One has 4GB RAM and the other 32GB. More or less idle, the 4GB VM takes about 20
>>>>>> seconds to migrate on our hardware while the 32GB VM takes about a minute.
>>>>>> 
>>>>>> With a reasonable amount of memory activity going on (in the hundreds of MB per
>>>>>> second) the 32GB VM takes 3.5 minutes to migrate, but the 4GB VM never
>>>>>> completes. Intuitively this tells me there is some watermarking of dirty pages
>>>>>> going on that is not particularly efficient when the dirty pages ratio is high
>>>>>> compared to total memory, but I may be completely incorrect.
>>>> 
>>>>> You can change the ratio IIRC.
>>>>> Hopefully, someone who knows well about QEMU will tell you better ways.
>>>>> 
>>>>> 	Takuya
>>>>> 
>>>>>> 
>>>>>> Could anybody fill me in on what might be going on here? We're using libvirt
>>>>>> 0.8.2 and kvm-83-224.el5.centos.1
>>>> 
>>>> This is pretty old qemu/kvm code base.
>>>> In principle, it makes no sense that with 32GB RAM migration finishes,
>>>> and with 4GB RAM it is unable (intuitively it should be, if ever, the
>>>> other way around).
>>>> 
>>>> Do you have an easy test that makes the problem easily reproducible?
>>>> Have you tried ustream qemu.git? (some improvements on that department).
>>> 
>>> I've just tried the qemu-kvm 0.14.1 tag which seems to be the latest that builds
>>> on my platform. For some strange reason migrations always seem to fail in one
>>> direction with "Unknown savevm section or instance 'hpet' 0" messages.
>> 
>> What is your platform?  This seems like you are running with hpet in one
>> side, but without it in the other.  What command line are you using?
> 
> Yes, my mistake. We were also testing later kernels and my test machines managed
> to get out of sync. One had support for hpet clocksource but the other one
> didn't.
> 
>> 
>>> This seems to point to different migration protocols on either end but they are
>>> both running the same version of qemu-kvm I built. Does this ring any bells for
>>> anyone?
>> 
>> Command line mismatch.  But, what is your platform?
> 
> CentOS5.6. Now running the VMs through qemu-kvm 0.14.1, unloaded migrations take
> about half the time but with memory I/O load now both VMs never complete the
> migration. In practical terms I'm writing about 50MB/s into memory and we have a
> 10Gbps network (and I've seen real speeds up to 8-9Gbps on the wire) so there
> should be enough capacity to sync up the dirty pages.
> 
> So now the 32GB and 4GB VMs have matching behaviour (which makes more sense) but
> I'm not any closer to figuring out what is going on.

You say you write 50 MB/s in memory, but this does not provide enough information to analyze the problem.
How distributed in memory are these writes? If your writes are not restricted to a small memory region, they could dirty many pages. In this case, live migration would have to transfer much more than 50 MB/s of pages to the destination.

-- 
Pierre Riteau -- PhD student, Myriads team, IRISA, Rennes, France
http://perso.univ-rennes1.fr/pierre.riteau/

Re: [Qemu-devel] Memory sync algorithm during migration

[oliver.hookins]

[-- Attachment #1: Type: text/plain, Size: 3905 bytes --]

Apologies for the top post as my mobile device doesn't allow anything else. I have no set the maximum permissible migration time but that default certainly points to a possibility of a solution. As for the writing semantics it was a straight dd from disk to /dev/shm so I can't speak for the kernel but naively I would think it may be contiguous space.


On 22.11.11 18:09 ext Pierre Riteau wrote:

On 22 nov. 2011, at 14:04, Oliver Hookins wrote:

> On Tue, Nov 22, 2011 at 10:31:58AM +0100, ext Juan Quintela wrote:
>> Oliver Hookins <oliver.hookins@nokia.com> wrote:
>>> On Tue, Nov 15, 2011 at 11:47:58AM +0100, ext Juan Quintela wrote:
>>>> Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp> wrote:
>>>>> Adding qemu-devel ML to CC.
>>>>>
>>>>> Your question should have been sent to qemu-devel ML because the logic
>>>>> is implemented in QEMU, not KVM.
>>>>>
>>>>> (2011/11/11 1:35), Oliver Hookins wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I am performing some benchmarks on KVM migration on two different types of VM.
>>>>>> One has 4GB RAM and the other 32GB. More or less idle, the 4GB VM takes about 20
>>>>>> seconds to migrate on our hardware while the 32GB VM takes about a minute.
>>>>>>
>>>>>> With a reasonable amount of memory activity going on (in the hundreds of MB per
>>>>>> second) the 32GB VM takes 3.5 minutes to migrate, but the 4GB VM never
>>>>>> completes. Intuitively this tells me there is some watermarking of dirty pages
>>>>>> going on that is not particularly efficient when the dirty pages ratio is high
>>>>>> compared to total memory, but I may be completely incorrect.
>>>>
>>>>> You can change the ratio IIRC.
>>>>> Hopefully, someone who knows well about QEMU will tell you better ways.
>>>>>
>>>>> Takuya
>>>>>
>>>>>>
>>>>>> Could anybody fill me in on what might be going on here? We're using libvirt
>>>>>> 0.8.2 and kvm-83-224.el5.centos.1
>>>>
>>>> This is pretty old qemu/kvm code base.
>>>> In principle, it makes no sense that with 32GB RAM migration finishes,
>>>> and with 4GB RAM it is unable (intuitively it should be, if ever, the
>>>> other way around).
>>>>
>>>> Do you have an easy test that makes the problem easily reproducible?
>>>> Have you tried ustream qemu.git? (some improvements on that department).
>>>
>>> I've just tried the qemu-kvm 0.14.1 tag which seems to be the latest that builds
>>> on my platform. For some strange reason migrations always seem to fail in one
>>> direction with "Unknown savevm section or instance 'hpet' 0" messages.
>>
>> What is your platform? This seems like you are running with hpet in one
>> side, but without it in the other. What command line are you using?
>
> Yes, my mistake. We were also testing later kernels and my test machines managed
> to get out of sync. One had support for hpet clocksource but the other one
> didn't.
>
>>
>>> This seems to point to different migration protocols on either end but they are
>>> both running the same version of qemu-kvm I built. Does this ring any bells for
>>> anyone?
>>
>> Command line mismatch. But, what is your platform?
>
> CentOS5.6. Now running the VMs through qemu-kvm 0.14.1, unloaded migrations take
> about half the time but with memory I/O load now both VMs never complete the
> migration. In practical terms I'm writing about 50MB/s into memory and we have a
> 10Gbps network (and I've seen real speeds up to 8-9Gbps on the wire) so there
> should be enough capacity to sync up the dirty pages.
>
> So now the 32GB and 4GB VMs have matching behaviour (which makes more sense) but
> I'm not any closer to figuring out what is going on.

Did you modify the max downtime? The default is 30 ms. At 8 Gbps, this only allows to send 30 MB of data on the wire.

--
Pierre Riteau -- PhD student, Myriads team, IRISA, Rennes, France
http://perso.univ-rennes1.fr/pierre.riteau/




[-- Attachment #2: Type: text/html, Size: 5409 bytes --]

Re: [Qemu-devel] Memory sync algorithm during migration

[Oliver Hookins]

On Tue, Nov 22, 2011 at 02:04:17PM +0100, Oliver Hookins wrote:
> On Tue, Nov 22, 2011 at 10:31:58AM +0100, ext Juan Quintela wrote:
> > Oliver Hookins <oliver.hookins@nokia.com> wrote:
> > > On Tue, Nov 15, 2011 at 11:47:58AM +0100, ext Juan Quintela wrote:
> > >> Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp> wrote:
> > >> > Adding qemu-devel ML to CC.
> > >> >
> > >> > Your question should have been sent to qemu-devel ML because the logic
> > >> > is implemented in QEMU, not KVM.
> > >> >
> > >> > (2011/11/11 1:35), Oliver Hookins wrote:
> > >> >> Hi,
> > >> >>
> > >> >> I am performing some benchmarks on KVM migration on two different types of VM.
> > >> >> One has 4GB RAM and the other 32GB. More or less idle, the 4GB VM takes about 20
> > >> >> seconds to migrate on our hardware while the 32GB VM takes about a minute.
> > >> >>
> > >> >> With a reasonable amount of memory activity going on (in the hundreds of MB per
> > >> >> second) the 32GB VM takes 3.5 minutes to migrate, but the 4GB VM never
> > >> >> completes. Intuitively this tells me there is some watermarking of dirty pages
> > >> >> going on that is not particularly efficient when the dirty pages ratio is high
> > >> >> compared to total memory, but I may be completely incorrect.
> > >> 
> > >> > You can change the ratio IIRC.
> > >> > Hopefully, someone who knows well about QEMU will tell you better ways.
> > >> >
> > >> > 	Takuya
> > >> >
> > >> >>
> > >> >> Could anybody fill me in on what might be going on here? We're using libvirt
> > >> >> 0.8.2 and kvm-83-224.el5.centos.1
> > >> 
> > >> This is pretty old qemu/kvm code base.
> > >> In principle, it makes no sense that with 32GB RAM migration finishes,
> > >> and with 4GB RAM it is unable (intuitively it should be, if ever, the
> > >> other way around).
> > >> 
> > >> Do you have an easy test that makes the problem easily reproducible?
> > >> Have you tried ustream qemu.git? (some improvements on that department).
> > >
> > > I've just tried the qemu-kvm 0.14.1 tag which seems to be the latest that builds
> > > on my platform. For some strange reason migrations always seem to fail in one
> > > direction with "Unknown savevm section or instance 'hpet' 0" messages.
> > 
> > What is your platform?  This seems like you are running with hpet in one
> > side, but without it in the other.  What command line are you using?
> 
> Yes, my mistake. We were also testing later kernels and my test machines managed
> to get out of sync. One had support for hpet clocksource but the other one
> didn't.
> 
> > 
> > > This seems to point to different migration protocols on either end but they are
> > > both running the same version of qemu-kvm I built. Does this ring any bells for
> > > anyone?
> > 
> > Command line mismatch.  But, what is your platform?
> 
> CentOS5.6. Now running the VMs through qemu-kvm 0.14.1, unloaded migrations take
> about half the time but with memory I/O load now both VMs never complete the
> migration. In practical terms I'm writing about 50MB/s into memory and we have a
> 10Gbps network (and I've seen real speeds up to 8-9Gbps on the wire) so there
> should be enough capacity to sync up the dirty pages.
> 
> So now the 32GB and 4GB VMs have matching behaviour (which makes more sense) but
> I'm not any closer to figuring out what is going on.

Resending as it got rejected the first time:

I have not set the maximum permissible migration time but that default
certainly points to a possibility of a solution. As for the writing semantics
it was a straight dd from disk to /dev/shm so I can't speak for the kernel but
naively I would think it may be contiguous space.