From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48978) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XSQ8D-0005NR-9K for qemu-devel@nongnu.org; Fri, 12 Sep 2014 08:39:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1XSQ84-0006P8-7O for qemu-devel@nongnu.org; Fri, 12 Sep 2014 08:39:17 -0400 Received: from mail-wi0-x22d.google.com ([2a00:1450:400c:c05::22d]:53618) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XSQ83-0006P2-Ti for qemu-devel@nongnu.org; Fri, 12 Sep 2014 08:39:08 -0400 Received: by mail-wi0-f173.google.com with SMTP id em10so538878wid.6 for ; Fri, 12 Sep 2014 05:39:01 -0700 (PDT) Date: Fri, 12 Sep 2014 13:38:57 +0100 From: Stefan Hajnoczi Message-ID: <20140912123857.GA6207@stefanha-thinkpad.redhat.com> References: <20140829143849.GA8909@grmbl.mre> <201409012038178763909@sangfor.com> <201409012052442706397@sangfor.com> <5404701F.5070302@de.ibm.com> <540470CD.9060800@redhat.com> <5404731A.8020405@de.ibm.com> <201409041556259426013@sangfor.com> <540C2972.5070707@gmail.com> <20140911061133.GG19202@grmbl.mre> <201409121121358332189@sangfor.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Kj7319i9nmIyA2yE" Content-Disposition: inline In-Reply-To: <201409121121358332189@sangfor.com> Subject: Re: [Qemu-devel] [question] virtio-blk performance degradation happened with virito-serial List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Zhang Haoyu Cc: kvm , qemu-devel , Zhang Haoyu , Max Reitz , Christian Borntraeger , Amit Shah , Paolo Bonzini --Kj7319i9nmIyA2yE Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Sep 12, 2014 at 11:21:37AM +0800, Zhang Haoyu wrote: > >>> > > If virtio-blk and virtio-serial share an IRQ, the guest operating= system has to check each virtqueue for activity. Maybe there is some ineff= iciency doing that. > >>> > > AFAIK virtio-serial registers 64 virtqueues (on 31 ports + consol= e) even if everything is unused. > >>> >=20 > >>> > That could be the case if MSI is disabled. > >>>=20 > >>> Do the windows virtio drivers enable MSIs, in their inf file? > >> > >>It depends on the version of the drivers, but it is a reasonable guess > >>at what differs between Linux and Windows. Haoyu, can you give us the > >>output of lspci from a Linux guest? > >> > >I made a test with fio on rhel-6.5 guest, the same degradation happened = too, this degradation can be reproduced on rhel6.5 guest 100%. > >virtio_console module installed: > >64K-write-sequence: 285 MBPS, 4380 IOPS > >virtio_console module uninstalled: > >64K-write-sequence: 370 MBPS, 5670 IOPS > > > I use top -d 1 -H -p to monitor the cpu usage, and found that, > virtio_console module installed: > qemu main thread cpu usage: 98% > virtio_console module uninstalled: > qemu main thread cpu usage: 60% >=20 > perf top -p result, > virtio_console module installed: > PerfTop: 9868 irqs/sec kernel:76.4% exact: 0.0% [4000Hz cycles],= (target_pid: 88381) > -------------------------------------------------------------------------= ---------------------------------------------------------------------------= ------------------ >=20 > 11.80% [kernel] [k] _raw_spin_lock_irqsave > 8.42% [kernel] [k] _raw_spin_unlock_irqrestore > 7.33% [kernel] [k] fget_light > 6.28% [kernel] [k] fput > 3.61% [kernel] [k] do_sys_poll > 3.30% qemu-system-x86_64 [.] qcow2_check_metadata_overlap > 3.10% [kernel] [k] __pollwait > 2.15% qemu-system-x86_64 [.] qemu_iohandler_poll > 1.44% libglib-2.0.so.0.3200.4 [.] g_array_append_vals > 1.36% libc-2.13.so [.] 0x000000000011fc2a > 1.31% libpthread-2.13.so [.] pthread_mutex_lock > 1.24% libglib-2.0.so.0.3200.4 [.] 0x000000000001f961 > 1.20% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt > 0.99% [kernel] [k] eventfd_poll > 0.98% [vdso] [.] 0x0000000000000771 > 0.97% [kernel] [k] remove_wait_queue > 0.96% qemu-system-x86_64 [.] qemu_iohandler_fill > 0.95% [kernel] [k] add_wait_queue > 0.69% [kernel] [k] __srcu_read_lock > 0.58% [kernel] [k] poll_freewait > 0.57% [kernel] [k] _raw_spin_lock_irq > 0.54% [kernel] [k] __srcu_read_unlock > 0.47% [kernel] [k] copy_user_enhanced_fast_string > 0.46% [kvm_intel] [k] vmx_vcpu_run > 0.46% [kvm] [k] vcpu_enter_guest > 0.42% [kernel] [k] tcp_poll > 0.41% [kernel] [k] system_call_after_swapgs > 0.40% libglib-2.0.so.0.3200.4 [.] g_slice_alloc > 0.40% [kernel] [k] system_call > 0.38% libpthread-2.13.so [.] 0x000000000000e18d > 0.38% libglib-2.0.so.0.3200.4 [.] g_slice_free1 > 0.38% qemu-system-x86_64 [.] address_space_translate_internal > 0.38% [kernel] [k] _raw_spin_lock > 0.37% qemu-system-x86_64 [.] phys_page_find > 0.36% [kernel] [k] get_page_from_freelist > 0.35% [kernel] [k] sock_poll > 0.34% [kernel] [k] fsnotify > 0.31% libglib-2.0.so.0.3200.4 [.] g_main_context_check > 0.30% [kernel] [k] do_direct_IO > 0.29% libpthread-2.13.so [.] pthread_getspecific >=20 > virtio_console module uninstalled: > PerfTop: 9138 irqs/sec kernel:71.7% exact: 0.0% [4000Hz cycles],= (target_pid: 88381) > -------------------------------------------------------------------------= ----------------------------------------------------- >=20 > 5.72% qemu-system-x86_64 [.] qcow2_check_metadata_overlap > 4.51% [kernel] [k] fget_light > 3.98% [kernel] [k] _raw_spin_lock_irqsave > 2.55% [kernel] [k] fput > 2.48% libpthread-2.13.so [.] pthread_mutex_lock > 2.46% [kernel] [k] _raw_spin_unlock_irqrestore > 2.21% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt > 1.71% [vdso] [.] 0x000000000000060c > 1.68% libc-2.13.so [.] 0x00000000000e751f > 1.64% libglib-2.0.so.0.3200.4 [.] 0x000000000004fca0 > 1.20% [kernel] [k] __srcu_read_lock > 1.14% [kernel] [k] do_sys_poll > 0.96% [kernel] [k] _raw_spin_lock_irq > 0.95% [kernel] [k] __pollwait > 0.91% [kernel] [k] __srcu_read_unlock > 0.78% [kernel] [k] tcp_poll > 0.74% [kvm] [k] vcpu_enter_guest > 0.73% [kvm_intel] [k] vmx_vcpu_run > 0.72% [kernel] [k] _raw_spin_lock > 0.72% [kernel] [k] system_call_after_swapgs > 0.70% [kernel] [k] copy_user_enhanced_fast_string > 0.67% libglib-2.0.so.0.3200.4 [.] g_slice_free1 > 0.66% libpthread-2.13.so [.] 0x000000000000e12d > 0.65% [kernel] [k] system_call > 0.61% [kernel] [k] do_direct_IO > 0.57% qemu-system-x86_64 [.] qemu_iohandler_poll > 0.57% [kernel] [k] fsnotify > 0.54% libglib-2.0.so.0.3200.4 [.] g_slice_alloc > 0.50% [kernel] [k] vfs_write > 0.49% libpthread-2.13.so [.] pthread_getspecific > 0.48% qemu-system-x86_64 [.] qemu_event_reset > 0.47% libglib-2.0.so.0.3200.4 [.] g_main_context_check > 0.46% qemu-system-x86_64 [.] address_space_translate_internal > 0.46% [kernel] [k] sock_poll > 0.46% libpthread-2.13.so [.] __pthread_disable_asynccancel > 0.44% [kernel] [k] resched_task > 0.43% libpthread-2.13.so [.] __pthread_enable_asynccancel > 0.42% qemu-system-x86_64 [.] phys_page_find > 0.39% qemu-system-x86_64 [.] object_dynamic_cast_assert Max: Unrelated to this performance issue but I notice that the qcow2 metadata overlap check is high in the host CPU profile. Have you had any thoughts about optimizing the check? Stefan --Kj7319i9nmIyA2yE Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUEulhAAoJEJykq7OBq3PI+XAIAL+JgkVT36NdxFTSSJCYDQfs X4L4qHpkWHUy8Q8QU06Pp89BurMjk0cvpDS8FnnizIaSnTx5MvztcnlIT2Aerx+s ELf80oeowXHRgG1V/CJkSvKa1Web/mP20CXBQL5lLJfFhjtji/z8XSmhZ0yhbwLM 2dDYRbMq58H38o7yRypY9eHMcScRyqZzbl8RJcbuxxcK1vt9RQHLdBF/9GK1Y70H tfSWYlke/Je6G3M8gh73NNd0sDo78YnzRZAH49MX8KGTnFS6dZ7MJaHgiTjnGxkJ fz4I7u8NfaqnZZu3hhg8IEDQEpTq5ZAbPkxghgfb397Tw8rEh5gk0yL5kZ03PG4= =V8L0 -----END PGP SIGNATURE----- --Kj7319i9nmIyA2yE--