From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=52598 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OWQef-00075U-Df for qemu-devel@nongnu.org; Wed, 07 Jul 2010 05:10:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OWQed-00088c-IM for qemu-devel@nongnu.org; Wed, 07 Jul 2010 05:10:57 -0400 Received: from adelie.canonical.com ([91.189.90.139]:57676) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OWQed-00088R-Ad for qemu-devel@nongnu.org; Wed, 07 Jul 2010 05:10:55 -0400 Received: from loganberry.canonical.com ([91.189.90.37]) by adelie.canonical.com with esmtp (Exim 4.69 #1 (Debian)) id 1OWQea-000092-Mj for ; Wed, 07 Jul 2010 10:10:52 +0100 Received: from loganberry.canonical.com (localhost [127.0.0.1]) by loganberry.canonical.com (Postfix) with ESMTP id 7C8EB2E8391 for ; Wed, 7 Jul 2010 10:10:50 +0100 (BST) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Date: Wed, 07 Jul 2010 09:01:52 -0000 From: Stephane Chazelas Sender: bounces@canonical.com References: <20100616143321.24259.15137.malonedeb@palladium.canonical.com> <20100623161348.15581.21305.launchpad@gandwana.canonical.com> <20100624001603.GI7058@shareable.org> Message-Id: Subject: Re: [Qemu-devel] [Bug 595117] Re: qemu-nbd slow and missing "writeback" cache option Errors-To: bounces@canonical.com Reply-To: Bug 595117 <595117@bugs.launchpad.net> List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org 2010-06-24 00:16:03 -0000, Jamie Lokier: > Serge Hallyn wrote: > > The default of qemu-img (of using O_SYNC) is not very sensible > > because anyway, the client (the kernel) uses caches (write-back), > > (and "qemu-nbd -d" doesn't flush those by the way). So if for > > instance qemu-nbd is killed, regardless of whether qemu-nbd uses > > O_SYNC, O_DIRECT or not, the data in the image will not be > > consistent anyway, unless "syncs" are done by the client (like fsync > > on the nbd device or sync mount option), and with qemu-nbd's O_SYNC > > mode, those "sync"s will be extremely slow. > = > Do the "client syncs" cause the nbd server to fsync or fdatasync the > file? The clients syncs cause the data to be sent to the server. The server then writes it to disk and each write blocks until the data is written physically on disk with O_SYNC. > > It appears it is because by default the disk image it serves is open > > with O_SYNC. The --nocache option, unintuitively, makes matters a > > bit better because it causes the image to be open with O_DIRECT > > instead of O_SYNC. > [...] > > --cache=3Doff is the same as --nocache (that is use O_DIRECT), > > writethrough is using O_SYNC and is still the default so this patch > > doesn't change the functionality. writeback is none of those flags, > > so is the addition of this patch. The patch also does an fsync upon > > "qemu-nbd -d" to make sure data is flushed to the image before > > removing the nbd. > = > I really wish qemu's options didn't give the false impression > "nocache" does less caching than "writethrough". O_DIRECT does > caching in the disk controller/hardware, while O_SYNC hopefully does > not, nowadays. [...] Note that I use the same "none", "writethrough", "writeback" as another utility shipped with qemu for consistency (see vl.c in the source), I don't mind about the words as long as the "writeback" functionality is available. Cheers, Stephane -- = qemu-nbd slow and missing "writeback" cache option https://bugs.launchpad.net/bugs/595117 You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. Status in QEMU: Invalid Status in =E2=80=9Cqemu-kvm=E2=80=9D package in Ubuntu: Incomplete Bug description: Binary package hint: qemu-kvm dpkg -l | grep qemu ii kvm 1:84+dfsg-0ubuntu16+0.12.3+noroms+= 0ubuntu9 dummy transitional pacakge from kvm to qemu- ii qemu 0.12.3+noroms-0ubuntu9 = dummy transitional pacakge from qemu to qemu ii qemu-common 0.12.3+noroms-0ubuntu9 = qemu common functionality (bios, documentati ii qemu-kvm 0.12.3+noroms-0ubuntu9 = Full virtualization on i386 and amd64 hardwa ii qemu-kvm-extras 0.12.3+noroms-0ubuntu9 = fast processor emulator binaries for non-x86 ii qemu-launcher 1.7.4-1ubuntu2 = GTK+ front-end to QEMU computer emulator ii qemuctl 0.2-2 = controlling GUI for qemu lucid amd64. qemu-nbd is a lot slower when writing to disk than say nbd-server. It appears it is because by default the disk image it serves is open with O= _SYNC. The --nocache option, unintuitively, makes matters a bit better beca= use it causes the image to be open with O_DIRECT instead of O_SYNC. The qemu code allows an image to be open without any of those flags, but un= fortunately qemu-nbd doesn't have the option to do that (qemu doesn't allow= the image to be open with both O_SYNC and O_DIRECT though). The default of qemu-img (of using O_SYNC) is not very sensible because anyw= ay, the client (the kernel) uses caches (write-back), (and "qemu-nbd -d" do= esn't flush those by the way). So if for instance qemu-nbd is killed, regar= dless of whether qemu-nbd uses O_SYNC, O_DIRECT or not, the data in the ima= ge will not be consistent anyway, unless "syncs" are done by the client (li= ke fsync on the nbd device or sync mount option), and with qemu-nbd's O_SYN= C mode, those "sync"s will be extremely slow. Attached is a patch that adds a --cache=3D{off,none,writethrough,writeback}= option to qemu-nbd. --cache=3Doff is the same as --nocache (that is use O_DIRECT), writethrough= is using O_SYNC and is still the default so this patch doesn't change the = functionality. writeback is none of those flags, so is the addition of this= patch. The patch also does an fsync upon "qemu-nbd -d" to make sure data i= s flushed to the image before removing the nbd. Consider this test scenario: dd bs=3D1M count=3D100 of=3Da < /dev/null qemu-nbd --cache=3D -c /dev/nbd0 a cp /dev/zero /dev/nbd0 time perl -MIO::Handle -e 'STDOUT->sync or die$!' 1<> /dev/nbd0 With cache=3Dwritethrough (the default), it takes over 10 minutes to write = those 100MB worth of zeroes. Running a strace, we see the recvfrom and sent= os delayed by each 1kb write(2)s to disk (10 to 30 ms per write). With cache=3Doff, it takes about 30 seconds. With cache=3Dwriteback, it takes about 3 seconds, which is similar to the p= erformance you get with nbd-server Note that the cp command runs instantly as the data is buffered by the clie= nt (the kernel), and not sent to qemu-nbd until the fsync(2) is called.