From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1KjHn1-0003Cc-8u for qemu-devel@nongnu.org; Fri, 26 Sep 2008 14:11:39 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1KjHn0-0003C6-Bg for qemu-devel@nongnu.org; Fri, 26 Sep 2008 14:11:38 -0400 Received: from [199.232.76.173] (port=53136 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1KjHn0-0003Bz-5I for qemu-devel@nongnu.org; Fri, 26 Sep 2008 14:11:38 -0400 Received: from e1.ny.us.ibm.com ([32.97.182.141]:37326) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1KjHmz-0006E7-VK for qemu-devel@nongnu.org; Fri, 26 Sep 2008 14:11:38 -0400 Received: from d01relay02.pok.ibm.com (d01relay02.pok.ibm.com [9.56.227.234]) by e1.ny.us.ibm.com (8.13.8/8.13.8) with ESMTP id m8QHxT2g001560 for ; Fri, 26 Sep 2008 13:59:29 -0400 Received: from d01av01.pok.ibm.com (d01av01.pok.ibm.com [9.56.224.215]) by d01relay02.pok.ibm.com (8.13.8/8.13.8/NCO v9.1) with ESMTP id m8QHxTCH218436 for ; Fri, 26 Sep 2008 13:59:29 -0400 Received: from d01av01.pok.ibm.com (loopback [127.0.0.1]) by d01av01.pok.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id m8QHxSm2031519 for ; Fri, 26 Sep 2008 13:59:28 -0400 Date: Fri, 26 Sep 2008 12:59:27 -0500 From: Ryan Harper Subject: Re: [Qemu-devel] [5323] Implement an fd pool to get real AIO with posix-aio Message-ID: <20080926175927.GZ31395@us.ibm.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Reply-To: qemu-devel@nongnu.org List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: Anthony Liguori , kvm@vger.kernel.org * Anthony Liguori [2008-09-26 11:03]: > Revision: 5323 > http://svn.sv.gnu.org/viewvc/?view=rev&root=qemu&revision=5323 > Author: aliguori > Date: 2008-09-26 15:59:29 +0000 (Fri, 26 Sep 2008) > > Log Message: > ----------- > Implement an fd pool to get real AIO with posix-aio > > This patch implements a simple fd pool to allow many AIO requests with > posix-aio. The result is significantly improved performance (identical to that > reported for linux-aio) for both cache=on and cache=off. > > The fundamental problem with posix-aio is that it limits itself to one thread > per-file descriptor. I don't know why this is, but this patch provides a simple > mechanism to work around this (duplicating the file descriptor). > > This isn't a great solution, but it seems like a reasonable intermediate step > between posix-aio and a custom thread-pool to replace it. > > Ryan Harper will be posting some performance analysis he did comparing posix-aio > with fd pooling against linux-aio. The size of the posix-aio thread pool and > the fd pool were largely determined by him based on this analysis. I'll have some more data to post in a bit, but for now, bumping the fd pool up to 64 and ensuring we init aio to support a thread per fd, we mostly match linux aio performance with a simpler implementation. For randomwrites, fd_pool lags a bit, but I've got other data that shows in most scenarios, fd_pool matches linux aio performance and does so with less CPU consumption. Results: 16k randwrite 1 thread, 74 iodepth | MB/s | avg sub lat (us) | avg comp lat (ms) -----------------------------------+------+------------------+------------------ baremetal (O_DIRECT, aka cache=off)| 61.2 | 13.07 | 19.59 kvm: cache=off posix-aio w/o patch | 4.7 | 3467.44 | 254.08 kvm: cache=off linux-aio | 61.1 | 75.35 | 19.57 kvm: cache=on posix-aio w/o patch |127.0 | 115.78 | 9.19 kvm: cache=on posix-aio w/ patch |126.0 | 67.35 | 9.30 ------------ new results ----------+------+------------------+------------------ kvm:cache=off posix-aio fd_pool[16]| 33.5 | 14.28 | 49.19 kvm:cache=off posix-aio fd_pool[64]| 51.1 | 14.86 | 23.66 16k write 1 thread, 74 iodepth | MB/s | avg sub lat (us) | avg comp lat (ms) -----------------------------------+------+------------------+------------------ baremetal (O_DIRECT, aka cache=off)|128.1 | 10.90 | 9.45 kvm: cache=off posix-aio w/o patch | 5.1 | 3152.00 | 231.06 kvm: cache=off linux-aio |130.0 | 83.83 | 8.99 kvm: cache=on posix-aio w/o patch |184.0 | 80.46 | 6.35 kvm: cache=on posix-aio w/ patch |165.0 | 70.90 | 7.09 ------------ new results ----------+------+------------------+------------------ kvm:cache=off posix-aio fd_pool[16]| 78.2 | 58.24 | 15.43 kvm:cache=off posix-aio fd_pool[64]|129.0 | 71.62 | 9.11 -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com