colo: qemu 4.2.0 vs. qemu 5.0.0-rc2 performance regression

colo: qemu 4.2.0 vs. qemu 5.0.0-rc2 performance regression

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 1255 bytes --]

Hello Everyone,
I did some Benchmarking with iperf3 and memtester (to dirty some guest memory)
of colo performance in qemu 4.2.0 and in qemu 5.0.0-rc2
with my bugfixes on top.( https://lists.nongnu.org/archive/html/qemu-devel/2020-04/msg01432.html )

I have taken the average over 4 runs.
Client-to-server tcp bandwidth rose slightly from ~83.98 Mbit/s to ~89.40 Mbits.
Server-to-client tcp bandwidth fell from ~9.73 Mbit/s to ~1.79 Mbit/s.
Client-to-server udp bandwidth stayed the same at 1.05 Mbit/s
and jitter rose from ~5.12 ms to ~10.77 ms.
Server-to-client udp bandwidth fell from ~380.5 Kbit/s to ~33.6 Kbit/s
and jitter rose from ~41.74 ms to ~83976.15 ms (!).

I haven't looked closely into it, but i think
0393031a16735835a441b6d6e0495a1bd14adb90 "COLO: Optimize memory back-up process"
is the culprint as it reduces vm downtime for the checkpoints but increases
the overall checkpoint time and we can only release miscompared primary packets
after the checkpoint is completely finished.

Another thing that I noticed: With 4.2.0, the secondary qemu uses thrice
the amount of gest memory. With 5.0.0-rc2 it's just double the amount of
guest memory. So maybe the ram cache isn't working properly?

Regards,
Lukas Straub

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

答复: colo: qemu 4.2.0 vs. qemu 5.0.0-rc2 performance regression

[Zhanghailiang]

Hi,

This patch " COLO: Optimize memory back-up process " should only affects VM's migration process before COLO compare starting to work.
Have you tried to revert this patch to see if it affects your tests ?

For memory size we used for secondary qemu, we only need a backup of VM's ram, so it should be double amount.


Thanks,
Hailiang


-----邮件原件-----
发件人: Lukas Straub [mailto:lukasstraub2@web.de] 
发送时间: 2020年4月12日 1:17
收件人: qemu-devel@nongnu.org
抄送: dgilbert@redhat.com; quintela@redhat.com; Zhanghailiang <zhang.zhanghailiang@huawei.com>; Zhang Chen <chen.zhang@intel.com>
主题: colo: qemu 4.2.0 vs. qemu 5.0.0-rc2 performance regression

Hello Everyone,
I did some Benchmarking with iperf3 and memtester (to dirty some guest memory) of colo performance in qemu 4.2.0 and in qemu 5.0.0-rc2 with my bugfixes on top.( https://lists.nongnu.org/archive/html/qemu-devel/2020-04/msg01432.html )

I have taken the average over 4 runs.
Client-to-server tcp bandwidth rose slightly from ~83.98 Mbit/s to ~89.40 Mbits.
Server-to-client tcp bandwidth fell from ~9.73 Mbit/s to ~1.79 Mbit/s.
Client-to-server udp bandwidth stayed the same at 1.05 Mbit/s and jitter rose from ~5.12 ms to ~10.77 ms.
Server-to-client udp bandwidth fell from ~380.5 Kbit/s to ~33.6 Kbit/s and jitter rose from ~41.74 ms to ~83976.15 ms (!).

I haven't looked closely into it, but i think
0393031a16735835a441b6d6e0495a1bd14adb90 "COLO: Optimize memory back-up process"
is the culprint as it reduces vm downtime for the checkpoints but increases the overall checkpoint time and we can only release miscompared primary packets after the checkpoint is completely finished.

Another thing that I noticed: With 4.2.0, the secondary qemu uses thrice the amount of gest memory. With 5.0.0-rc2 it's just double the amount of guest memory. So maybe the ram cache isn't working properly?

Regards,
Lukas Straub

Re: colo: qemu 4.2.0 vs. qemu 5.0.0-rc2 performance regression

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 1738 bytes --]

On Sat, 11 Apr 2020 19:16:54 +0200
Lukas Straub <lukasstraub2@web.de> wrote:

> Hello Everyone,
> I did some Benchmarking with iperf3 and memtester (to dirty some guest memory)
> of colo performance in qemu 4.2.0 and in qemu 5.0.0-rc2
> with my bugfixes on top.( https://lists.nongnu.org/archive/html/qemu-devel/2020-04/msg01432.html )
> 
> I have taken the average over 4 runs.
> Client-to-server tcp bandwidth rose slightly from ~83.98 Mbit/s to ~89.40 Mbits.
> Server-to-client tcp bandwidth fell from ~9.73 Mbit/s to ~1.79 Mbit/s.
> Client-to-server udp bandwidth stayed the same at 1.05 Mbit/s
> and jitter rose from ~5.12 ms to ~10.77 ms.
> Server-to-client udp bandwidth fell from ~380.5 Kbit/s to ~33.6 Kbit/s
> and jitter rose from ~41.74 ms to ~83976.15 ms (!).
> 
> I haven't looked closely into it, but i think
> 0393031a16735835a441b6d6e0495a1bd14adb90 "COLO: Optimize memory back-up process"
> is the culprint as it reduces vm downtime for the checkpoints but increases
> the overall checkpoint time and we can only release miscompared primary packets
> after the checkpoint is completely finished.
> 
> Another thing that I noticed: With 4.2.0, the secondary qemu uses thrice
> the amount of gest memory. With 5.0.0-rc2 it's just double the amount of
> guest memory. So maybe the ram cache isn't working properly?
> 
> Regards,
> Lukas Straub

Hmm,
I looked at my test again and saw that the results where very noisy, so qemu 5.0.0-rc2
being slower was just a coincidence. I did increase the test time and the results are 
more meaningful now. Now qemu 5.0.0-rc2 is around the same speed and still faster
in the client-to-server tcp case.

Sorry for the noise.

Regards,
Lukas Straub

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: colo: qemu 4.2.0 vs. qemu 5.0.0-rc2 performance regression

[Dr. David Alan Gilbert]

* Lukas Straub (lukasstraub2@web.de) wrote:
> On Sat, 11 Apr 2020 19:16:54 +0200
> Lukas Straub <lukasstraub2@web.de> wrote:
> 
> > Hello Everyone,
> > I did some Benchmarking with iperf3 and memtester (to dirty some guest memory)
> > of colo performance in qemu 4.2.0 and in qemu 5.0.0-rc2
> > with my bugfixes on top.( https://lists.nongnu.org/archive/html/qemu-devel/2020-04/msg01432.html )
> > 
> > I have taken the average over 4 runs.
> > Client-to-server tcp bandwidth rose slightly from ~83.98 Mbit/s to ~89.40 Mbits.
> > Server-to-client tcp bandwidth fell from ~9.73 Mbit/s to ~1.79 Mbit/s.
> > Client-to-server udp bandwidth stayed the same at 1.05 Mbit/s
> > and jitter rose from ~5.12 ms to ~10.77 ms.
> > Server-to-client udp bandwidth fell from ~380.5 Kbit/s to ~33.6 Kbit/s
> > and jitter rose from ~41.74 ms to ~83976.15 ms (!).
> > 
> > I haven't looked closely into it, but i think
> > 0393031a16735835a441b6d6e0495a1bd14adb90 "COLO: Optimize memory back-up process"
> > is the culprint as it reduces vm downtime for the checkpoints but increases
> > the overall checkpoint time and we can only release miscompared primary packets
> > after the checkpoint is completely finished.
> > 
> > Another thing that I noticed: With 4.2.0, the secondary qemu uses thrice
> > the amount of gest memory. With 5.0.0-rc2 it's just double the amount of
> > guest memory. So maybe the ram cache isn't working properly?
> > 
> > Regards,
> > Lukas Straub
> 
> Hmm,
> I looked at my test again and saw that the results where very noisy, so qemu 5.0.0-rc2
> being slower was just a coincidence. I did increase the test time and the results are 
> more meaningful now. Now qemu 5.0.0-rc2 is around the same speed and still faster
> in the client-to-server tcp case.
> 
> Sorry for the noise.

Is it back to using 3x RAM in the secondary?

Dave

> 
> Regards,
> Lukas Straub


--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: colo: qemu 4.2.0 vs. qemu 5.0.0-rc2 performance regression

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 2200 bytes --]

On Mon, 27 Apr 2020 11:34:32 +0100
"Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:

> * Lukas Straub (lukasstraub2@web.de) wrote:
> > On Sat, 11 Apr 2020 19:16:54 +0200
> > Lukas Straub <lukasstraub2@web.de> wrote:
> >   
> > > Hello Everyone,
> > > I did some Benchmarking with iperf3 and memtester (to dirty some guest memory)
> > > of colo performance in qemu 4.2.0 and in qemu 5.0.0-rc2
> > > with my bugfixes on top.( https://lists.nongnu.org/archive/html/qemu-devel/2020-04/msg01432.html )
> > > 
> > > I have taken the average over 4 runs.
> > > Client-to-server tcp bandwidth rose slightly from ~83.98 Mbit/s to ~89.40 Mbits.
> > > Server-to-client tcp bandwidth fell from ~9.73 Mbit/s to ~1.79 Mbit/s.
> > > Client-to-server udp bandwidth stayed the same at 1.05 Mbit/s
> > > and jitter rose from ~5.12 ms to ~10.77 ms.
> > > Server-to-client udp bandwidth fell from ~380.5 Kbit/s to ~33.6 Kbit/s
> > > and jitter rose from ~41.74 ms to ~83976.15 ms (!).
> > > 
> > > I haven't looked closely into it, but i think
> > > 0393031a16735835a441b6d6e0495a1bd14adb90 "COLO: Optimize memory back-up process"
> > > is the culprint as it reduces vm downtime for the checkpoints but increases
> > > the overall checkpoint time and we can only release miscompared primary packets
> > > after the checkpoint is completely finished.
> > > 
> > > Another thing that I noticed: With 4.2.0, the secondary qemu uses thrice
> > > the amount of gest memory. With 5.0.0-rc2 it's just double the amount of
> > > guest memory. So maybe the ram cache isn't working properly?
> > > 
> > > Regards,
> > > Lukas Straub  
> > 
> > Hmm,
> > I looked at my test again and saw that the results where very noisy, so qemu 5.0.0-rc2
> > being slower was just a coincidence. I did increase the test time and the results are 
> > more meaningful now. Now qemu 5.0.0-rc2 is around the same speed and still faster
> > in the client-to-server tcp case.
> > 
> > Sorry for the noise.  
> 
> Is it back to using 3x RAM in the secondary?

No.

> Dave
> 
> > 
> > Regards,
> > Lukas Straub  
> 
> 
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]