From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39831) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ab4xc-0006l1-Sp for qemu-devel@nongnu.org; Wed, 02 Mar 2016 06:28:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ab4xY-0001Ih-KQ for qemu-devel@nongnu.org; Wed, 02 Mar 2016 06:28:56 -0500 Received: from mail-wm0-x236.google.com ([2a00:1450:400c:c09::236]:37359) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ab4xY-0001Ib-DA for qemu-devel@nongnu.org; Wed, 02 Mar 2016 06:28:52 -0500 Received: by mail-wm0-x236.google.com with SMTP id p65so73326069wmp.0 for ; Wed, 02 Mar 2016 03:28:52 -0800 (PST) Date: Wed, 2 Mar 2016 11:28:49 +0000 From: Stefan Hajnoczi Message-ID: <20160302112849.GD11268@stefanha-x1.localdomain> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="FFoLq8A0u+X9iRU8" Content-Disposition: inline In-Reply-To: Subject: Re: [Qemu-devel] Performance Profiling 2 VMs List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: kalyan tata Cc: qemu-devel@nongnu.org --FFoLq8A0u+X9iRU8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Mar 01, 2016 at 04:06:16PM -0800, kalyan tata wrote: > Hi All, >=20 > I am new to qemu development. > Sorry If this is not the correct forum for this question, it would be gre= at > if you could direct me to correct forum. >=20 > I am seeing very low virtio network throughput on an older (2.6.18) linux > guest vs another newer guest (3.10) both running on the same host. (same > config 2 vcpus, no multi Q etc.) I see very high CPU usage on the 2.6.18 > guest at very low network throughput and want to profile to find > bottleneck. >=20 > I tried to use "perf kvm" but the analysis shows overhead as max .25 % > where as top in VM shows 100% cpu. (I used following as a guide > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/= html-single/Virtualization_Tuning_and_Optimization_Guide/index.html#sect-Vi= rtualization_Tuning_Optimization_Guide-Monitoring_Tools-perf_kvm > ) >=20 > 0.25% :5235 [uhci_hcd] [g] 0xffffffff80182236 > 0.24% :5235 [uhci_hcd] [g] 0xffffffff8018226a > 0.23% :5235 [virtio_ring] [g] vring_new_virtqueue > 0.20% :5236 [uhci_hcd] [g] 0xffffffff80182236 > 0.18% :5236 [uhci_hcd] [g] 0xffffffff8018226a > 0.18% :5235 [uhci_hcd] [g] 0xffffffff8016f385 > 0.14% :5236 [uhci_hcd] [g] 0xffffffff802fbe0f > 0.14% :5235 [uhci_hcd] [g] 0xffffffff8001161a > 0.14% :5235 [virtio_ring] [g] virtqueue_is_broken >=20 >=20 > My basic question is - Is there a way I can profile the older version of > linux guest so i can see the bottleneck (where the guest is spending CPU > cycles) My aim is to see if i can patch the older version in the critical > path with improvements made in newer version What is the output of "mpstat 5" in the guest and on the host? mpstat is part of the "sysstat" package. mpstat is similar to vmstat but also shows "guest time" and "steal time". Both are relevant to virtualization and will help show which component is using so much CPU time. Stefan --FFoLq8A0u+X9iRU8 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJW1s5xAAoJEJykq7OBq3PIek8H/i9XDbb6oBXofR5kV9c8UUhA OvG2EEOmWE+wVez6Uy1yb74R83g8h9LsvoS2O/hAqdIbEM74EtvCViW37msaLqgE sLHyDgxCLi6ejZ+/MvZIilNmcHknCU1aCaT0t6TGW7i8zJFAe2Fd1sgUy9D1tvS1 GNGLDppRkxuz4kyuEnfZjWSkflRslNXpd+zUH8abzlHaFVJDoDQN9lo65QzBo2iL yMMZZR87HoPMDU2GaPl/wFXp4Q2jv/kOKPwT6B1HXNtkLmNMjEGmayZ9i9214j0b nPoEx3PGNva04Jfj+Zi1nj3LQkzB8PsnSZysG1hX2SuYC7Y82qkq2s9lRoPrKso= =JVLx -----END PGP SIGNATURE----- --FFoLq8A0u+X9iRU8--