From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:41663) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Su0b3-0000FK-GK for qemu-devel@nongnu.org; Wed, 25 Jul 2012 08:21:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Su0az-00075s-79 for qemu-devel@nongnu.org; Wed, 25 Jul 2012 08:21:45 -0400 Received: from e06smtp17.uk.ibm.com ([195.75.94.113]:49529) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Su0ay-00075A-US for qemu-devel@nongnu.org; Wed, 25 Jul 2012 08:21:41 -0400 Received: from /spool/local by e06smtp17.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 25 Jul 2012 13:21:37 +0100 Received: from d06av04.portsmouth.uk.ibm.com (d06av04.portsmouth.uk.ibm.com [9.149.37.216]) by d06nrmr1407.portsmouth.uk.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q6PCLVC11687662 for ; Wed, 25 Jul 2012 13:21:31 +0100 Received: from d06av04.portsmouth.uk.ibm.com (loopback [127.0.0.1]) by d06av04.portsmouth.uk.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q6PCLVBl003160 for ; Wed, 25 Jul 2012 06:21:31 -0600 From: Stefan Hajnoczi Date: Wed, 25 Jul 2012 13:21:17 +0100 Message-Id: <1343218884-14980-1-git-send-email-stefanha@linux.vnet.ibm.com> Subject: [Qemu-devel] [PATCH v2 0/7] qcow2: implement lazy refcounts optimization List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: Kevin Wolf , Khoa Huynh , Anthony Liguori , Stefan Hajnoczi This series aims to improve qcow2 performance with cache=writethrough and cache=directsync. In particular it reduces the impact of metadata updates for allocating writes. This optimization should help make qcow2 a good choice even for cache=writethrough and cache=directsync where QED has traditionally had an advantage due to less metadata - this allows us to converge image format development in QEMU around the qcow2v3 format. Allocating writes are expensive because they involve updating L2 tables and refcount blocks. In addition they can also cause L2 table allocation and refcount block allocation but these remain unaffected by this optimization. The key insight is that refcounts are not required to access data in the image. This means that we can postpone refcount updates without violating the data integrity guarantee that cache=writethrough and cache=directsync give. The trade-off for postponing refcount updates is that the image may not be completely consistent in case of power failure or crash. If the image is dirty then it must be repaired before performing further modifications, in other words we need an fsck-like scan on startup. fio benchmark results: WRITE lazy_refcounts=off io=552650KB WRITE lazy_refcounts=on io=708264KB (+28%) Disk: Intel X25-M 120GB 2nd Gen Host file system: ext4 Host kernel: Linux 3.2.0-3-amd64 Debian Note that these random 8k writes are to an empty qcow2 file. This is the worst case scenario for cache=writethrough and cache=directsync because each I/O is likely to involve metadata I/O, including refcount updates. $ qemu-system-x86_64 -enable-kvm -m 1024 \ -drive if=virtio,cache=none,file=../vm/rhel-6.img \ -drive if=virtio,cache=writethrough,file=a.qcow2 Job file: [global] ioengine=aio buffered=0 rw=write bs=8k runtime=3m time_based=1 iodepth=1 filename=/dev/vdb [job1] v2: * Fixed feature bit qcow2 spec formatting [Kevin] * Record feature bit names in qcow2 header for better error messages [Kevin] * Use separate enums for different feature bit types [Kevin] * Check that dirty bit is still set after read-only image access [Kevin] * Check that the dirty flag is never set with lazy_refcounts=off [Kevin] Stefan Hajnoczi (7): docs: add dirty bit to qcow2 specification qcow2: introduce dirty bit docs: add lazy refcounts bit to qcow2 specification qemu-iotests: ignore qemu-img create lazy_refcounts output qcow2: implement lazy refcounts qemu-io: add "abort" command to simulate program crash qemu-iotests: add 039 qcow2 lazy refcounts test block/qcow2-cluster.c | 5 +- block/qcow2.c | 121 +++++++++++++++++++++++++++++++++++++++--- block/qcow2.h | 21 ++++++++ block_int.h | 26 ++++----- docs/specs/qcow2.txt | 14 ++++- qemu-io.c | 12 +++++ tests/qemu-iotests/031.out | 20 +++---- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/039 | 117 ++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/039.out | 42 +++++++++++++++ tests/qemu-iotests/common.rc | 3 +- tests/qemu-iotests/group | 1 + 12 files changed, 351 insertions(+), 35 deletions(-) create mode 100755 tests/qemu-iotests/039 create mode 100644 tests/qemu-iotests/039.out -- 1.7.10.4