Re: [Qemu-devel] Following up questions related to QEMU and I/O Thread

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 2913 bytes --]

On Mon, Apr 22, 2019 at 09:21:53PM -0700, Wei Li wrote:
>         2. kvm_stat or perf record -a -e kvm:\* counters for vmexits and
>            interrupt injections.  If these counters vary greatly between queue
>            sizes, then that is usually a clue.  It's possible to get higher
>            performance by spending more CPU cycles although your system doesn't
>            have many CPUs available, so I'm not sure if this is the case.
> 
> [wei]: vmexits looks like a reason. I am using FIO tool to read/write block storage via following sample command, interesting thing is that kvm:kvm_exit count decreased from 846K to 395K after I increased num_queues from 2 to 4 while the vCPU count is 2.
>            1). Does this mean using more queues than vCPU count may increase IOPS via spending more CPU cycle? 
>            2). Could you please help me better understand how more queues is able to spend more CPU cycle? Thanks!
>            FIO command: fio --filename=/dev/sdb --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=64 --numjobs=4 --time_based --group_reporting --name=iops --runtime=60 --eta-newline=1
> 
>         3. Power management and polling (kvm.ko halt_poll_ns, tuned profiles,
>            and QEMU iothread poll-max-ns).  It's expensive to wake a CPU when it
>            goes into a low power mode due to idle.  There are several features
>            that can keep the CPU awake or even poll so that request latency is
>            reduced.  The reason why the number of queues may matter is that
>            kicking multiple queues may keep the CPU awake more than batching
>            multiple requests onto a small number of queues.
> [wei]: CPU awake could be another reason, I noticed that kvm:kvm_vcpu_wakeup count decreased from 151K to 47K after I increased num_queues from 2 to 4 while the vCPU count is 2.

This suggests that wakeups are involved in the performance difference.

>            1). Does this mean more queues may keep CPU more busy and awake which reduced the vcpu wakeup time?

Yes, although it depends on how I/O requests are distributed across the
queues.  You can check /proc/interrupts inside the guest to see
interrupt counts for the virtqueues.

>            2). If using more num_queues than vCPU count is able to get higher IOPS for this case, is it safe to use 4 queues while it only have 2 vCPU, or there is any concern or impact by using more queues than vCPU count which I should keep in mind?

2 vs 4 queues should be functionally identical.  The only difference is
performance.

> In addition, does Virtio-scsi support Batch I/O Submission feature which may be able to increase the IOPS via reducing the number of system calls?

I don't see obvious batching support in drivers/scsi/virtio_scsi.c.  The
Linux block layer supports batching but I'm not sure if the SCSI layer
does.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]