[Qemu-devel] [PATCH 1.3 v3 0/2] Documentation: Block related updates for 1.3

[Qemu-devel] [PATCH 1.3 v3 0/2] Documentation: Block related updates for 1.3

[Kevin Wolf]

Kevin Wolf (2):
  Documentation: Update block cache mode information
  Documentation: Update image format information

 qemu-doc.texi   |  167 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 qemu-img.texi   |   84 +++++++++-------------------
 qemu-options.hx |   38 ++++++------
 3 files changed, 213 insertions(+), 76 deletions(-)

-- 
1.7.6.5

[Qemu-devel] [PATCH 1.3 v3 1/2] Documentation: Update block cache mode information

[Kevin Wolf]

Somehow we forgot to update this when cache=writeback became the
default. While changing the information on the default, also make the
description of all caches modes a bit more accurate.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-options.hx |   38 +++++++++++++++++++-------------------
 1 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/qemu-options.hx b/qemu-options.hx
index 9bb29d3..b045ff8 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -206,33 +206,33 @@ Open drive @option{file} as read-only. Guest write attempts will fail.
 file sectors into the image file.
 @end table
 
-By default, writethrough caching is used for all block device.  This means that
-the host page cache will be used to read and write data but write notification
-will be sent to the guest only when the data has been reported as written by
-the storage subsystem.
-
-Writeback caching will report data writes as completed as soon as the data is
-present in the host page cache.  This is safe as long as you trust your host.
-If your host crashes or loses power, then the guest may experience data
-corruption.
+By default, the @option{cache=writeback} mode is used. It will report data
+writes as completed as soon as the data is present in the host page cache.
+This is safe as long as your guest OS makes sure to correctly flush disk caches
+where needed. If your guest OS does not handle volatile disk write caches
+correctly and your host crashes or loses power, then the guest may experience
+data corruption.
+
+For such guests, you should consider using @option{cache=writethrough}. This
+means that the host page cache will be used to read and write data, but write
+notification will be sent to the guest only after QEMU has made sure to flush
+each write to the disk. Be aware that this has a major impact on performance.
 
 The host page cache can be avoided entirely with @option{cache=none}.  This will
-attempt to do disk IO directly to the guests memory.  QEMU may still perform
-an internal copy of the data.
+attempt to do disk IO directly to the guest's memory.  QEMU may still perform
+an internal copy of the data. Note that this is considered a writeback mode and
+the guest OS must handle the disk write cache correctly in order to avoid data
+corruption on host crashes.
 
 The host page cache can be avoided while only sending write notifications to
-the guest when the data has been reported as written by the storage subsystem
-using @option{cache=directsync}.
-
-Some block drivers perform badly with @option{cache=writethrough}, most notably,
-qcow2.  If performance is more important than correctness,
-@option{cache=writeback} should be used with qcow2.
+the guest when the data has been flushed to the disk using
+@option{cache=directsync}.
 
 In case you don't care about data integrity over host failures, use
-cache=unsafe. This option tells QEMU that it never needs to write any data
-to the disk but can instead keeps things in cache. If anything goes wrong,
+@option{cache=unsafe}. This option tells QEMU that it never needs to write any
+data to the disk but can instead keep things in cache. If anything goes wrong,
 like your host losing power, the disk storage getting disconnected accidentally,
-etc. you're image will most probably be rendered unusable.   When using
+etc. your image will most probably be rendered unusable.   When using
 the @option{-snapshot} option, unsafe caching is always used.
 
 Copy-on-read avoids accessing the same backing file sectors repeatedly and is
-- 
1.7.6.5

[Qemu-devel] [PATCH 1.3 v3 2/2] Documentation: Update image format information

[Kevin Wolf]

Document new and yet undocumented options and image formats. The
qemu-img man page contains information only for raw and qcow2 now and
references the HTML documentation for a more detailed description of
other formats.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-doc.texi |  167 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 qemu-img.texi |   84 +++++++++-------------------
 2 files changed, 194 insertions(+), 57 deletions(-)

diff --git a/qemu-doc.texi b/qemu-doc.texi
index 6ff309d..cf62843 100644
--- a/qemu-doc.texi
+++ b/qemu-doc.texi
@@ -416,6 +416,7 @@ snapshots.
 * vm_snapshots::              VM snapshots
 * qemu_img_invocation::       qemu-img Invocation
 * qemu_nbd_invocation::       qemu-nbd Invocation
+* disk_images_formats::       Disk image file formats
 * host_drives::               Using host drives
 * disk_images_fat_images::    Virtual FAT disk images
 * disk_images_nbd::           NBD access
@@ -507,6 +508,172 @@ state is not saved or restored properly (in particular USB).
 
 @include qemu-nbd.texi
 
+@node disk_images_formats
+@subsection Disk image file formats
+
+QEMU supports many image file formats that can be used with VMs as well as with
+any of the tools (like @code{qemu-img}). This includes the preferred formats
+raw and qcow2 as well as formats that are supported for compatibility with
+older QEMU versions or other hypervisors.
+
+Depending on the image format, different options can be passed to
+@code{qemu-img create} and @code{qemu-img convert} using the @code{-o} option.
+This section describes each format and the options that are supported for it.
+
+@table @option
+@item raw
+
+Raw disk image format). This format has the advantage of
+being simple and easily exportable to all other emulators. If your
+file system supports @emph{holes} (for example in ext2 or ext3 on
+Linux or NTFS on Windows), then only the written sectors will reserve
+space. Use @code{qemu-img info} to know the real size used by the
+image or @code{ls -ls} on Unix/Linux.
+
+@item qcow2
+QEMU image format, the most versatile format. Use it to have smaller
+images (useful if your filesystem does not supports holes, for example
+on Windows), optional AES encryption, zlib based compression and
+support of multiple VM snapshots.
+
+Supported options:
+@table @code
+@item compat
+Determines the qcow2 version to use. @code{compat=0.10} uses the traditional
+image format that can be read by any QEMU since 0.10 (this is the default).
+@code{compat=1.1} enables image format extensions that only QEMU 1.1 and
+newer understand. Amongst others, this includes zero clusters, which allow
+efficient copy-on-read for sparse images.
+
+@item backing_file
+File name of a base image (see @option{create} subcommand)
+@item backing_fmt
+Image format of the base image
+@item encryption
+If this option is set to @code{on}, the image is encrypted.
+
+Encryption uses the AES format which is very secure (128 bit keys). Use
+a long password (16 characters) to get maximum protection.
+
+@item cluster_size
+Changes the qcow2 cluster size (must be between 512 and 2M). Smaller cluster
+sizes can improve the image file size whereas larger cluster sizes generally
+provide better performance.
+
+@item preallocation
+Preallocation mode (allowed values: off, metadata). An image with preallocated
+metadata is initially larger but can improve performance when the image needs
+to grow.
+
+@item lazy_refcounts
+If this option is set to @code{on}, reference count updates are postponed with
+the goal of avoiding metadata I/O and improving performance. This is
+particularly interesting with @option{cache=writethrough} which doesn't batch
+metadata updates. The tradeoff is that after a host crash, the reference count
+tables must be rebuilt, i.e. on the next open an (automatic) @code{qemu-img
+check -r all} is required, which may take some time.
+
+This option can only be enabled if @code{compat=1.1} is specified.
+
+@end table
+
+@item qed
+Old QEMU image format with support for backing files and compact image files
+(when your filesystem or transport medium does not support holes).
+
+When converting QED images to qcow2, you might want to consider using the
+@code{lazy_refcounts=on} option to get a more QED-like behaviour.
+
+Supported options:
+@table @code
+@item backing_file
+File name of a base image (see @option{create} subcommand).
+@item backing_fmt
+Image file format of backing file (optional).  Useful if the format cannot be
+autodetected because it has no header, like some vhd/vpc files.
+@item cluster_size
+Changes the cluster size (must be power-of-2 between 4K and 64K). Smaller
+cluster sizes can improve the image file size whereas larger cluster sizes
+generally provide better performance.
+@item table_size
+Changes the number of clusters per L1/L2 table (must be power-of-2 between 1
+and 16).  There is normally no need to change this value but this option can be
+used for performance benchmarking.
+@end table
+
+@item qcow
+Old QEMU image format with support for backing files, compact image files,
+encryption and compression.
+
+Supported options:
+@table @code
+@item backing_file
+File name of a base image (see @option{create} subcommand)
+@item encryption
+If this option is set to @code{on}, the image is encrypted.
+@end table
+
+@item cow
+User Mode Linux Copy On Write image format. It is supported only for
+compatibility with previous versions.
+Supported options:
+@table @code
+@item backing_file
+File name of a base image (see @option{create} subcommand)
+@end table
+
+@item vdi
+VirtualBox 1.1 compatible image format.
+Supported options:
+@table @code
+@item static
+If this option is set to @code{on}, the image is created with metadata
+preallocation.
+@end table
+
+@item vmdk
+VMware 3 and 4 compatible image format.
+
+Supported options:
+@table @code
+@item backing_file
+File name of a base image (see @option{create} subcommand).
+@item compat6
+Create a VMDK version 6 image (instead of version 4)
+@item subformat
+Specifies which VMDK subformat to use. Valid options are
+@code{monolithicSparse} (default),
+@code{monolithicFlat},
+@code{twoGbMaxExtentSparse},
+@code{twoGbMaxExtentFlat} and
+@code{streamOptimized}.
+@end table
+
+@item vpc
+VirtualPC compatible image format (VHD).
+Supported options:
+@table @code
+@item subformat
+Specifies which VHD subformat to use. Valid options are
+@code{dynamic} (default) and @code{fixed}.
+@end table
+@end table
+
+@subsubsection Read-only formats
+More disk image file formats are supported in a read-only mode.
+@table @option
+@item bochs
+Bochs images of @code{growing} type.
+@item cloop
+Linux Compressed Loop image, useful only to reuse directly compressed
+CD-ROM images present for example in the Knoppix CD-ROMs.
+@item dmg
+Apple disk image.
+@item parallels
+Parallels disk image format.
+@end table
+
+
 @node host_drives
 @subsection Using host drives
 
diff --git a/qemu-img.texi b/qemu-img.texi
index 60b83fc..00fca8d 100644
--- a/qemu-img.texi
+++ b/qemu-img.texi
@@ -226,7 +226,10 @@ After using this command to grow a disk image, you must use file system and
 partitioning tools inside the VM to actually begin using the new space on the
 device.
 @end table
+@c man end
 
+@ignore
+@c man begin NOTES
 Supported image file formats:
 
 @table @option
@@ -247,6 +250,13 @@ support of multiple VM snapshots.
 
 Supported options:
 @table @code
+@item compat
+Determines the qcow2 version to use. @code{compat=0.10} uses the traditional
+image format that can be read by any QEMU since 0.10 (this is the default).
+@code{compat=1.1} enables image format extensions that only QEMU 1.1 and
+newer understand. Amongst others, this includes zero clusters, which allow
+efficient copy-on-read for sparse images.
+
 @item backing_file
 File name of a base image (see @option{create} subcommand)
 @item backing_fmt
@@ -267,73 +277,33 @@ Preallocation mode (allowed values: off, metadata). An image with preallocated
 metadata is initially larger but can improve performance when the image needs
 to grow.
 
-@end table
+@item lazy_refcounts
+If this option is set to @code{on}, reference count updates are postponed with
+the goal of avoiding metadata I/O and improving performance. This is
+particularly interesting with @option{cache=writethrough} which doesn't batch
+metadata updates. The tradeoff is that after a host crash, the reference count
+tables must be rebuilt, i.e. on the next open an (automatic) @code{qemu-img
+check -r all} is required, which may take some time.
 
-@item qed
-Image format with support for backing files and compact image files (when your
-filesystem or transport medium does not support holes).  Good performance due
-to less metadata than the more featureful qcow2 format, especially with
-cache=writethrough or cache=directsync.  Consider using qcow2 which will soon
-have a similar optimization and is most actively developed.
+This option can only be enabled if @code{compat=1.1} is specified.
 
-Supported options:
-@table @code
-@item backing_file
-File name of a base image (see @option{create} subcommand).
-@item backing_fmt
-Image file format of backing file (optional).  Useful if the format cannot be
-autodetected because it has no header, like some vhd/vpc files.
-@item cluster_size
-Changes the cluster size (must be power-of-2 between 4K and 64K). Smaller
-cluster sizes can improve the image file size whereas larger cluster sizes
-generally provide better performance.
-@item table_size
-Changes the number of clusters per L1/L2 table (must be power-of-2 between 1
-and 16).  There is normally no need to change this value but this option can be
-used for performance benchmarking.
 @end table
 
-@item qcow
-Old QEMU image format. Left for compatibility.
+@item Other
+QEMU also supports various other image file formats for compatibility with
+older QEMU versions or other hypervisors, including VMDK, VDI, VHD (vpc), qcow1
+and QED. For a full list of supported formats see @code{qemu-img --help}.
+For a more detailed description of these formats, see the QEMU Emulation User
+Documentation.
 
-Supported options:
-@table @code
-@item backing_file
-File name of a base image (see @option{create} subcommand)
-@item encryption
-If this option is set to @code{on}, the image is encrypted.
-@end table
-
-@item cow
-User Mode Linux Copy On Write image format. Used to be the only growable
-image format in QEMU. It is supported only for compatibility with
-previous versions. It does not work on win32.
-@item vdi
-VirtualBox 1.1 compatible image format.
-@item vmdk
-VMware 3 and 4 compatible image format.
-
-Supported options:
-@table @code
-@item backing_fmt
-Image format of the base image
-@item compat6
-Create a VMDK version 6 image (instead of version 4)
-@end table
-
-@item vpc
-VirtualPC compatible image format (VHD).
-
-@item cloop
-Linux Compressed Loop image, useful only to reuse directly compressed
-CD-ROM images present for example in the Knoppix CD-ROMs.
+The main purpose of the block drivers for these formats is image conversion.
+For running VMs, it is recommended to convert the disk images to either raw or
+qcow2 in order to achieve good performance.
 @end table
 
 
 @c man end
 
-@ignore
-
 @setfilename qemu-img
 @settitle QEMU disk image utility
 
-- 
1.7.6.5

Re: [Qemu-devel] [PATCH 1.3 v3 2/2] Documentation: Update image format information

[Stefan Hajnoczi]

On Mon, Nov 26, 2012 at 2:21 PM, Kevin Wolf <kwolf@redhat.com> wrote:
> +@table @option
> +@item raw
> +
> +Raw disk image format). This format has the advantage of

s/format)/format/

Otherwise, looks good.

Re: [Qemu-devel] [PATCH 1.3 v3 0/2] Documentation: Block related updates for 1.3

[Stefan Hajnoczi]

Noticed a typo in Patch 2.  Otherwise looks good.

Acked-by: Stefan Hajnoczi <stefanha@redhat.com>

On Mon, Nov 26, 2012 at 2:21 PM, Kevin Wolf <kwolf@redhat.com> wrote:
> Kevin Wolf (2):
>   Documentation: Update block cache mode information
>   Documentation: Update image format information
>
>  qemu-doc.texi   |  167 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  qemu-img.texi   |   84 +++++++++-------------------
>  qemu-options.hx |   38 ++++++------
>  3 files changed, 213 insertions(+), 76 deletions(-)
>
> --
> 1.7.6.5
>
>

Re: [Qemu-devel] [PATCH 1.3 v3 2/2] Documentation: Update image format information

[Kevin Wolf]

Am 26.11.2012 14:53, schrieb Stefan Hajnoczi:
> On Mon, Nov 26, 2012 at 2:21 PM, Kevin Wolf <kwolf@redhat.com> wrote:
>> +@table @option
>> +@item raw
>> +
>> +Raw disk image format). This format has the advantage of
> 
> s/format)/format/
> 
> Otherwise, looks good.

Thanks for catching this typo. Fixed it locally and applied both patches
to the block branch for 1.3.

Kevin