From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54804) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1X6brL-00039z-I0 for qemu-devel@nongnu.org; Mon, 14 Jul 2014 04:44:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1X6brG-0001vA-SG for qemu-devel@nongnu.org; Mon, 14 Jul 2014 04:43:43 -0400 Received: from mx1.redhat.com ([209.132.183.28]:8727) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1X6brG-0001uQ-KX for qemu-devel@nongnu.org; Mon, 14 Jul 2014 04:43:38 -0400 Date: Mon, 14 Jul 2014 10:43:27 +0200 From: Stefan Hajnoczi Message-ID: <20140714084327.GH2185@stefanha-thinkpad.redhat.com> References: <87B246BB5ED53A4C98E4F9A35839EDE128F6F49D@nkgeml506-mbs.china.huawei.com> <20140711130413.GA4255@stefanha-thinkpad.redhat.com> <87B246BB5ED53A4C98E4F9A35839EDE128F6FC93@nkgeml506-mbs.china.huawei.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jaoouwwPWoQSJZYp" Content-Disposition: inline In-Reply-To: <87B246BB5ED53A4C98E4F9A35839EDE128F6FC93@nkgeml506-mbs.china.huawei.com> Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Wangkai (Kevin,C)" Cc: Lee yang , "qemu-devel@nongnu.org" , "aliguori@amazon.com" --jaoouwwPWoQSJZYp Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jul 14, 2014 at 01:55:05AM +0000, Wangkai (Kevin,C) wrote: >=20 >=20 > > -----Original Message----- > > From: Stefan Hajnoczi [mailto:stefanha@redhat.com] > > Sent: Friday, July 11, 2014 9:04 PM > > To: Wangkai (Kevin,C) > > Cc: qemu-devel@nongnu.org; aliguori@amazon.com; Lee yang > > Subject: Re: [PATCH] Tap: fix vcpu long time io blocking on tap > >=20 > > On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote: > > > When used a tap as net driver for vm, if too many packets was > > > delivered to the guest os via tap interface, the guest os will be > > > blocked on io events for a long time, while tap driver was busying > > process packets. > > > > > > kvm vcpu thread block on io lock call trace: > > > __lll_lock_wait > > > _L_lock_1004 > > > __pthread_mutex_lock > > > qemu_mutex_lock > > > kvm_cpu_exec > > > qemu_kvm_cpu_thread_fn > > > start_thread > > > > > > qemu io thread call trace: > > > ... > > > qemu_net_queue_send > > > tap_send > > > qemu_iohandler_poll > > > main_loop_wait > > > main_loop > > > > > > > > > I think the qemu io lock time should be as small as possible, and the > > > io work slice should be limited at a particular ration or time. > > > > > > --- > > > Signed-off-by: Wangkai > >=20 > > How many packets are you seeing in a single tap_send() call? > >=20 > > Have you profiled the tap_send() code path? Maybe it is performing > > some operation that is very slow. > >=20 > > By the way, if you want good performance you should use vhost_net > > instead of userspace vhost_net. Userspace virtio-net is not very > > optimized. > >=20 > > Stefan >=20 >=20 > Hi Stefan, >=20 > I am not use profile, just debug with gdb and code review. It's worth understanding the root cause for this behavior because something is probably wrong. > When packets delivered, I found the VM was hung, and I check qemu run >=20 > State by gdb, I see the call trace for IO thread and vcpu thread, and >=20 > I add debug info to check how many packets within tap_send, the info belo= w: >=20 > total recv 393520 time 1539821 us=20 > total recv 1270 time 4931 us > total recv 257872 time 995828 us=20 > total recv 10745 time 41438 us > total recv 505387 time 2000925 us 505387 packets or 505387 bytes? If that's packets, then even with small 64-byte packets that would mean 32 MB of pending data! Are you running a networking benchmark where you'd expect lots of packets? Have you checked how lot the time between tap_send() calls is? Perhaps something else in QEMU is blocking the event loop so packets accumulate in the host kernel. Stefan --jaoouwwPWoQSJZYp Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJTw5gvAAoJEJykq7OBq3PIAt0IAJxPWQoVK4NekHhERNDr4x4b NTkYB/aUQrrArILFAf4X1xvV3tCen6YCU5NI16T/NUIAX7pXCAU82ejFTihTfb44 FqtY02XfbUlC2kaabXvCamZ3jSk9eHdsShAHQeM51jWup5T/WaUitexhTdWRw82T Le+Ix0VSmazgoncI2B3Z18DMDrDqIubOmOGD1OXLJwJv7u4TFCQ8CFD2uDu6stmh hK3fU+psdVQWEpkblaUzRH04WOY6LzR8gl1T/ETTOxwMkgKBPdBJ0ww1Lm+cCsjB Rw/ZFvwtpUDWvGhLM25ymtqdTm4cDXHjw6mfbgbLETIFz53tIkDn4hsllH4FpNw= =8hgG -----END PGP SIGNATURE----- --jaoouwwPWoQSJZYp--