From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40440) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YrqsC-0006zG-2i for qemu-devel@nongnu.org; Mon, 11 May 2015 12:48:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YrqsA-0006SE-ON for qemu-devel@nongnu.org; Mon, 11 May 2015 12:48:08 -0400 Message-ID: <5550DD2D.8000407@odin.com> Date: Mon, 11 May 2015 19:47:41 +0300 From: "Denis V. Lunev" MIME-Version: 1.0 References: <1430746944-27347-1-git-send-email-den@openvz.org> <20150511150817.GK16270@stefanha-thinkpad.redhat.com> <5550D3B5.2050703@openvz.org> In-Reply-To: <5550D3B5.2050703@openvz.org> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [Qemu-block] [PATCH v5 0/2] block: enforce minimal 4096 alignment in qemu_blockalign List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: Paolo Bonzini , Stefan Hajnoczi , qemu-devel@nongnu.org, qemu-block@nongnu.org On 11/05/15 19:07, Denis V. Lunev wrote: > On 11/05/15 18:08, Stefan Hajnoczi wrote: >> On Mon, May 04, 2015 at 04:42:22PM +0300, Denis V. Lunev wrote: >>> The difference is quite reliable and the same 5%. >>> qemu-io -n -c 'write -P 0xaa 0 1G' 1.img >>> for image in qcow2 format is 1% faster. >> I looked a little at the qemu-io invocation but am not clear why there >> would be a measurable performance difference. Can you explain? >> >> What about real qemu-img or QEMU use cases? >> >> I'm okay with the patches themselves, but I don't really understand why >> this code change is justified. >> >> Stefan > There is a problem in the Linux kernel when the buffer > is not aligned to the page size. Actually the strict requirement > is the alignment to the 512 (one physical sector). > > This comes into the account in qemu-img and qemu-io > when buffers are allocated inside the application. QEMU > is free of this problem as the guest sends buffers > aligned to page already. > > You can see below results of qemu-img, they are exactly > the same as for qemu-io. > > qemu-img create -f qcow2 1.img 64G > qemu-io -n -c 'write -P 0xaa 0 1G' 1.img > time for i in `seq 1 30` ; do /home/den/src/qemu/qemu-img convert > 1.img -t none -O raw 2.img ; rm -rf 2.img ; done > > ==== without patches ====: > real 2m6.287s > user 0m1.322s > sys 0m8.819s > > real 2m7.483s > user 0m1.614s > sys 0m9.096s > > ==== with patches ====: > real 1m59.715s > user 0m1.453s > sys 0m9.365s > > real 1m58.739s > user 0m1.419s > sys 0m8.530s > > I could not exactly say where the difference comes, but > the problem comes from the fact that real IO operation > over the block device should be > a) page aligned for the buffer > b) page aligned for the offset > This is how buffer cache is working in the kernel. And > with non-aligned buffer in userspace the kernel should collect > kernel page for IO from 2 userspaces pages instead of one. > Something is not optimal here I presume. I can assume > that the user page could be sent immediately to the > controller is buffer is aligned and no additional memory > allocation is needed. Though I don't know exactly. > > Regards, > Den Here are results of blktrace on my host. Logs are collected using sudo blktrace -d /dev/md0 -o - | blkparse -i - Test command: /home/den/src/qemu/qemu-img convert 1.img -t none -O raw 2.img In general, not patched qemu-img IO pattern looks like this: 9,0 11 1 0.000000000 11151 Q WS 312737792 + 1023 [qemu-img] 9,0 11 2 0.000007938 11151 Q WS 312738815 + 8 [qemu-img] 9,0 11 3 0.000030735 11151 Q WS 312738823 + 1016 [qemu-img] 9,0 11 4 0.000032482 11151 Q WS 312739839 + 8 [qemu-img] 9,0 11 5 0.000041379 11151 Q WS 312739847 + 1016 [qemu-img] 9,0 11 6 0.000042818 11151 Q WS 312740863 + 8 [qemu-img] 9,0 11 7 0.000051236 11151 Q WS 312740871 + 1017 [qemu-img] 9,0 5 1 0.169071519 11151 Q WS 312741888 + 1023 [qemu-img] 9,0 5 2 0.169075331 11151 Q WS 312742911 + 8 [qemu-img] 9,0 5 3 0.169085244 11151 Q WS 312742919 + 1016 [qemu-img] 9,0 5 4 0.169086786 11151 Q WS 312743935 + 8 [qemu-img] 9,0 5 5 0.169095740 11151 Q WS 312743943 + 1016 [qemu-img] and patched one: 9,0 6 1 0.000000000 12422 Q WS 314834944 + 1024 [qemu-img] 9,0 6 2 0.000038527 12422 Q WS 314835968 + 1024 [qemu-img] 9,0 6 3 0.000072849 12422 Q WS 314836992 + 1024 [qemu-img] 9,0 6 4 0.000106276 12422 Q WS 314838016 + 1024 [qemu-img] 9,0 2 1 0.171038202 12422 Q WS 314839040 + 1024 [qemu-img] 9,0 2 2 0.171073156 12422 Q WS 314840064 + 1024 [qemu-img] Thus the load to the disk is MUCH higher without the patch! Total amount of lines (IO requests sent to disks) are the following: hades ~ $ wc -l *.blk 3622 non-patched.blk 2086 patched.blk 5708 total hades ~ $ and this from my point of view explains everything! With aligned buffers the amount of IO requests is almost 2 times less. Regards, Den P.S. sorry for previous letter with big attachments. I was excited that I have found good explanation.