From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1KjIP8-000231-Op for qemu-devel@nongnu.org; Fri, 26 Sep 2008 14:51:02 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1KjIP6-00022d-TW for qemu-devel@nongnu.org; Fri, 26 Sep 2008 14:51:02 -0400 Received: from [199.232.76.173] (port=33736 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1KjIP6-00022a-O0 for qemu-devel@nongnu.org; Fri, 26 Sep 2008 14:51:00 -0400 Received: from e1.ny.us.ibm.com ([32.97.182.141]:32804) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1KjIP6-00054L-O4 for qemu-devel@nongnu.org; Fri, 26 Sep 2008 14:51:00 -0400 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by e1.ny.us.ibm.com (8.13.8/8.13.8) with ESMTP id m8QIowpH021329 for ; Fri, 26 Sep 2008 14:50:58 -0400 Received: from d01av01.pok.ibm.com (d01av01.pok.ibm.com [9.56.224.215]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v9.1) with ESMTP id m8QIoweJ235600 for ; Fri, 26 Sep 2008 14:50:58 -0400 Received: from d01av01.pok.ibm.com (loopback [127.0.0.1]) by d01av01.pok.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id m8QIowXN000640 for ; Fri, 26 Sep 2008 14:50:58 -0400 Date: Fri, 26 Sep 2008 13:50:57 -0500 From: Ryan Harper Subject: Re: [Qemu-devel] [5323] Implement an fd pool to get real AIO with posix-aio Message-ID: <20080926185057.GA31395@us.ibm.com> References: <20080926175927.GZ31395@us.ibm.com> <48DD2B67.407@us.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <48DD2B67.407@us.ibm.com> Reply-To: qemu-devel@nongnu.org List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: Ryan Harper , qemu-devel@nongnu.org, kvm@vger.kernel.org * Anthony Liguori [2008-09-26 13:37]: > Ryan Harper wrote: > >* Anthony Liguori [2008-09-26 11:03]: > >kvm: cache=on posix-aio w/o patch |127.0 | 115.78 | 9.19 > >kvm: cache=on posix-aio w/ patch |126.0 | 67.35 | 9.30 > > > > It looks like 127mb/s is pretty close to the optimal cached write time. > When using caching, writes can complete almost immediately so it's not > surprising that submission latency is so low (even though it's blocking > during submission). > > I am surprised that w/patch has a latency that's so high. I think that > suggests that requests are queuing up. I bet increasing the aio_num > field would reduce this number. Yeah, there is plenty of room to twiddle with the threads and number of outstanding ios, but that'll take quite a bit of time to generate the data and compare. > >------------ new results > >----------+------+------------------+------------------ > >kvm:cache=off posix-aio fd_pool[16]| 33.5 | 14.28 | 49.19 > >kvm:cache=off posix-aio fd_pool[64]| 51.1 | 14.86 | 23.66 > > > > I assume you tried to bump from 64 to something higher and couldn't make > up the lost bandwidth? Very slightly, switching to 128 threads/fds gave another 1MB/s. > >16k write 1 thread, 74 iodepth | MB/s | avg sub lat (us) | avg comp > >lat (ms) > >-----------------------------------+------+------------------+------------------ > >baremetal (O_DIRECT, aka cache=off)|128.1 | 10.90 | 9.45 > >kvm: cache=off posix-aio w/o patch | 5.1 | 3152.00 | 231.06 > >kvm: cache=off linux-aio |130.0 | 83.83 | 8.99 > >kvm: cache=on posix-aio w/o patch |184.0 | 80.46 | 6.35 > >kvm: cache=on posix-aio w/ patch |165.0 | 70.90 | 7.09 > >------------ new results > >----------+------+------------------+------------------ > >kvm:cache=off posix-aio fd_pool[16]| 78.2 | 58.24 | 15.43 > >kvm:cache=off posix-aio fd_pool[64]|129.0 | 71.62 | 9.11 > > > > That's a nice result. We could probably improve the latency by tweaking > the queue sizes. Yeah, I was quite pleased to see a simpler solution perform so well. > > Very nice work! Thanks for doing the thorough analysis. Thanks, very happy to see a signficant improvement in IO here. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com