[Qemu-devel] ridiculously slow VM memory performance on Ryzen CPU

[Qemu-devel] ridiculously slow VM memory performance on Ryzen CPU

[Qu Wenruo]

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]

Hi,

When testing IO heavy work on my VM backed by Ryzen 1700 CPU, I turned
to brd modules, but surprisingly, the speed is even slower than some HDD:

---
$ sudo modprobe brd rd_nr=1 rd_size=1048576
$ dd if=/dev/zero of=/dev/ram0 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 9.9928 s, 107 MB/s
---
107MB is pretty lame...
Even some HDD could be faster than this.

On host, it's much better:
---
$ if=/dev/zero of=/dev/ram0 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.754641 s, 1.4 GB/s
---

For host hardware:
CPU: Ryzen 1700 All cores @ 3.8G
Mem: DDR4 2400 dual channel (8G x 2)

For host software:
Kernel:       4.16.3-1-ARCH
Qemu:         2.11.1-2
Distribution: Archlinux

VM setup is mostly default setup done by libvirt.

I'm not sure if this is related to this bug:
https://www.redhat.com/archives/vfio-users/2017-April/msg00019.html

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] ridiculously slow VM memory performance on Ryzen CPU

[Daniel P. Berrangé]

On Wed, Apr 25, 2018 at 05:48:42PM +0800, Qu Wenruo wrote:
> Hi,
> 
> When testing IO heavy work on my VM backed by Ryzen 1700 CPU, I turned
> to brd modules, but surprisingly, the speed is even slower than some HDD:
> 
> ---
> $ sudo modprobe brd rd_nr=1 rd_size=1048576
> $ dd if=/dev/zero of=/dev/ram0 bs=1M count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 9.9928 s, 107 MB/s
> ---
> 107MB is pretty lame...
> Even some HDD could be faster than this.

You are sure that QEMU is actually using KVM, and not TCG emulation
mode, as that could account for such terrible numbers. The QEMU
argv would confirm this or can query it live with

   virsh qemu-monitor-command --hmp $GUESTNAME "info kvm"

> On host, it's much better:
> ---
> $ if=/dev/zero of=/dev/ram0 bs=1M count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.754641 s, 1.4 GB/s
> ---
> 
> For host hardware:
> CPU: Ryzen 1700 All cores @ 3.8G
> Mem: DDR4 2400 dual channel (8G x 2)
> 
> For host software:
> Kernel:       4.16.3-1-ARCH
> Qemu:         2.11.1-2
> Distribution: Archlinux
> 
> VM setup is mostly default setup done by libvirt.

Ideally please share the guest XML description from 'virsh dumpxml $GUESTNAME'
and the corresponding QEMU argv from /var/log/libvirt/qemu/$GUESTNAME.log

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] ridiculously slow VM memory performance on Ryzen CPU

[Qu Wenruo]

[-- Attachment #1: Type: text/plain, Size: 1842 bytes --]



On 2018年04月25日 21:18, Daniel P. Berrangé wrote:
> On Wed, Apr 25, 2018 at 05:48:42PM +0800, Qu Wenruo wrote:
>> Hi,
>>
>> When testing IO heavy work on my VM backed by Ryzen 1700 CPU, I turned
>> to brd modules, but surprisingly, the speed is even slower than some HDD:
>>
>> ---
>> $ sudo modprobe brd rd_nr=1 rd_size=1048576
>> $ dd if=/dev/zero of=/dev/ram0 bs=1M count=1024
>> 1024+0 records in
>> 1024+0 records out
>> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 9.9928 s, 107 MB/s
>> ---
>> 107MB is pretty lame...
>> Even some HDD could be faster than this.
> 
> You are sure that QEMU is actually using KVM, and not TCG emulation
> mode, as that could account for such terrible numbers. The QEMU
> argv would confirm this or can query it live with
> 
>    virsh qemu-monitor-command --hmp $GUESTNAME "info kvm"
> 
>> On host, it's much better:
>> ---
>> $ if=/dev/zero of=/dev/ram0 bs=1M count=1024
>> 1024+0 records in
>> 1024+0 records out
>> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.754641 s, 1.4 GB/s
>> ---
>>
>> For host hardware:
>> CPU: Ryzen 1700 All cores @ 3.8G
>> Mem: DDR4 2400 dual channel (8G x 2)
>>
>> For host software:
>> Kernel:       4.16.3-1-ARCH
>> Qemu:         2.11.1-2
>> Distribution: Archlinux
>>
>> VM setup is mostly default setup done by libvirt.
> 
> Ideally please share the guest XML description from 'virsh dumpxml $GUESTNAME'
> and the corresponding QEMU argv from /var/log/libvirt/qemu/$GUESTNAME.log

It turns out that my debugging config (KASAN + lockdep, maybe something
else) is causing the problem.

With lockdep only, the guest degrades to about 50% of host memory
performance.
With lockde + KASAN, only 10% of host.

So, in short, I'm just an idiot.
Sorry for the trouble.

Thanks,
Qu

> 
> Regards,
> Daniel
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]