From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34068) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bPpjQ-0005oV-Kj for qemu-devel@nongnu.org; Wed, 20 Jul 2016 07:32:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bPpjO-0001Xu-WD for qemu-devel@nongnu.org; Wed, 20 Jul 2016 07:32:04 -0400 Received: from mail-wm0-x243.google.com ([2a00:1450:400c:c09::243]:35932) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bPpjO-0001Xe-Ki for qemu-devel@nongnu.org; Wed, 20 Jul 2016 07:32:02 -0400 Received: by mail-wm0-x243.google.com with SMTP id x83so6550533wma.3 for ; Wed, 20 Jul 2016 04:32:02 -0700 (PDT) Date: Wed, 20 Jul 2016 10:08:11 +0100 From: Stefan Hajnoczi Message-ID: <20160720090811.GC13233@stefanha-x1.localdomain> References: <1468931263-32667-1-git-send-email-roman.penyaev@profitbricks.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="4jXrM3lyYWu4nBt5" Content-Disposition: inline In-Reply-To: <1468931263-32667-1-git-send-email-roman.penyaev@profitbricks.com> Subject: Re: [Qemu-devel] [PATCH v2 0/3] linux-aio: reduce completion latency List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Roman Pen Cc: Paolo Bonzini , Stefan Hajnoczi , qemu-devel@nongnu.org --4jXrM3lyYWu4nBt5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 19, 2016 at 02:27:40PM +0200, Roman Pen wrote: > v2: > o For the third patch do not introduce extra member for LinuxAioState > structure, reuse ret =3D=3D -EINPROGRESS. >=20 > o Add explicit comment which explains why we do not hang if requests > are still pended. >=20 >=20 > This series are intended to reduce completion latencies by two changes: >=20 > 1. QEMU does not use any timeout value for harvesting completed AIO > requests from the ring buffer, thus io_getevents() can be implemented > in userspace (first patch). >=20 > 2. In order to reduce completion latency it makes sense to harvest comple= ted > requests ASAP. Very fast backend device can complete requests just af= ter > submission, so it is worth trying to check ring buffer and peek comple= ted > requests directly after io_submit() has been called (third patch). >=20 > Indeed, the series reduces the completions latencies and increases the > overall throughput, e.g. the following is the percentiles of number of > completed requests at once: >=20 > 1th 10th 20th 30th 40th 50th 60th 70th 80th 90th 99.99th > Before 2 4 42 112 128 128 128 128 128 128 128 > After 1 1 4 14 33 45 47 48 50 51 108 >=20 > That means, that before the third patch is applied the ring buffer is > observed as full (128 requests were consumed at once) in 60% of calls. >=20 > After the third patch is applied the distribution of number of completed > requests is "smoother" and the queue (requests in-flight) is almost never > full. >=20 > The fio read results are the following (write results are almost the > same and are not showed here): >=20 > Before > ------ > job: (groupid=3D0, jobs=3D8): err=3D 0: pid=3D2227: Tue Jul 19 11:29:50 2= 016 > Description : [Emulation of Storage Server Access Pattern] > read : io=3D54681MB, bw=3D1822.7MB/s, iops=3D179779, runt=3D 30001msec > slat (usec): min=3D172, max=3D16883, avg=3D338.35, stdev=3D109.66 > clat (usec): min=3D1, max=3D21977, avg=3D1051.45, stdev=3D299.29 > lat (usec): min=3D317, max=3D22521, avg=3D1389.83, stdev=3D300.73 > clat percentiles (usec): > | 1.00th=3D[ 346], 5.00th=3D[ 596], 10.00th=3D[ 708], 20.00th= =3D[ 852], > | 30.00th=3D[ 932], 40.00th=3D[ 996], 50.00th=3D[ 1048], 60.00th= =3D[ 1112], > | 70.00th=3D[ 1176], 80.00th=3D[ 1256], 90.00th=3D[ 1384], 95.00th= =3D[ 1496], > | 99.00th=3D[ 1800], 99.50th=3D[ 1928], 99.90th=3D[ 2320], 99.95th= =3D[ 2672], > | 99.99th=3D[ 4704] > bw (KB /s): min=3D205229, max=3D553181, per=3D12.50%, avg=3D233278.2= 6, stdev=3D18383.51 >=20 > After > ------ > job: (groupid=3D0, jobs=3D8): err=3D 0: pid=3D2220: Tue Jul 19 11:31:51 2= 016 > Description : [Emulation of Storage Server Access Pattern] > read : io=3D57637MB, bw=3D1921.2MB/s, iops=3D189529, runt=3D 30002msec > slat (usec): min=3D169, max=3D20636, avg=3D329.61, stdev=3D124.18 > clat (usec): min=3D2, max=3D19592, avg=3D988.78, stdev=3D251.04 > lat (usec): min=3D381, max=3D21067, avg=3D1318.42, stdev=3D243.58 > clat percentiles (usec): > | 1.00th=3D[ 310], 5.00th=3D[ 580], 10.00th=3D[ 748], 20.00th= =3D[ 876], > | 30.00th=3D[ 908], 40.00th=3D[ 948], 50.00th=3D[ 1012], 60.00th= =3D[ 1064], > | 70.00th=3D[ 1080], 80.00th=3D[ 1128], 90.00th=3D[ 1224], 95.00th= =3D[ 1288], > | 99.00th=3D[ 1496], 99.50th=3D[ 1608], 99.90th=3D[ 1960], 99.95th= =3D[ 2256], > | 99.99th=3D[ 5408] > bw (KB /s): min=3D212149, max=3D390160, per=3D12.49%, avg=3D245746.0= 4, stdev=3D11606.75 >=20 > Throughput increased from 1822MB/s to 1921MB/s, average completion latenc= ies > decreased from 1051us to 988us. >=20 > Roman Pen (3): > linux-aio: consume events in userspace instead of calling io_getevents > linux-aio: split processing events function > linux-aio: process completions from ioq_submit() >=20 > block/linux-aio.c | 178 ++++++++++++++++++++++++++++++++++++++++++------= ------ > 1 file changed, 141 insertions(+), 37 deletions(-) >=20 > Signed-off-by: Roman Pen > Cc: Stefan Hajnoczi > Cc: Paolo Bonzini > Cc: qemu-devel@nongnu.org Thanks, applied to my block-next tree for QEMU 2.8: https://github.com/stefanha/qemu/commits/block-next Stefan --4jXrM3lyYWu4nBt5 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXjz97AAoJEJykq7OBq3PI2p8IAMKpQ1FAinYwxrYhUD9NGJrv jMQ8tQuy8YCui4mbWM8wPNkhZQq7EO/LeHDUtAQGloxQ4cq7BGfMYiarxp5s7rGf L0hX0byVaInV4+9XGjOERaCjIgj5FBMTbQLTjUgcef54QY0GIHIlDJOsHPQdIzqT o++sTwL/eBqHlNZoOYu24Ae1oOKq7As9BoS0Uf+g3f1WICuSL71BTzpGBjZWhWQl c/IW7qVzUIZooQyG6jpUNdH7qi4ge4+3Hjfg6oUl2s7eDfAC/nNlo0oIdw6cQzKT kPbxSIJWNmIbvxo0fOyESL56leGxMKWthPiTtq0Ytrb698OlZ00yh5EbdTUlAUU= =PTea -----END PGP SIGNATURE----- --4jXrM3lyYWu4nBt5--