[Qemu-devel] [PATCH 1.3 0/2] Manpage updates for 1.3

[Qemu-devel] [PATCH 1.3 0/2] Manpage updates for 1.3

[Kevin Wolf]

Kevin Wolf (2):
  Documentation: Update block cache mode information
  Documentation: Update image format information

 qemu-img.texi   |   37 ++++++++++++++++++++++++++++++-------
 qemu-options.hx |   34 +++++++++++++++++-----------------
 2 files changed, 47 insertions(+), 24 deletions(-)

-- 
1.7.6.5

[Qemu-devel] [PATCH 1/2] Documentation: Update block cache mode information

[Kevin Wolf]

Somehow we forgot to update this when cache=writeback became the
default. While changing the information on the default, also make the
description of all caches modes a bit more accurate.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-options.hx |   34 +++++++++++++++++-----------------
 1 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/qemu-options.hx b/qemu-options.hx
index 9bb29d3..cb59e3e 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -206,31 +206,31 @@ Open drive @option{file} as read-only. Guest write attempts will fail.
 file sectors into the image file.
 @end table
 
-By default, writethrough caching is used for all block device.  This means that
-the host page cache will be used to read and write data but write notification
-will be sent to the guest only when the data has been reported as written by
-the storage subsystem.
-
-Writeback caching will report data writes as completed as soon as the data is
-present in the host page cache.  This is safe as long as you trust your host.
-If your host crashes or loses power, then the guest may experience data
-corruption.
+By default, the @option{cache=writeback} mode is used. It will report data
+writes as completed as soon as the data is present in the host page cache.
+This is safe as long as your guest OS makes sure to correctly flush disk caches
+where needed. If your guest OS does not handle volatile disk write caches
+correctly and your host crashes or loses power, then the guest may experience
+data corruption.
+
+For such guests, you should consider using @option{cache=writethrough}. This
+means that the host page cache will be used to read and write data, but write
+notification will be sent to the guest only after qemu has made sure to flush
+each write to the disk. Be aware that this has a major impact on performance.
 
 The host page cache can be avoided entirely with @option{cache=none}.  This will
 attempt to do disk IO directly to the guests memory.  QEMU may still perform
-an internal copy of the data.
+an internal copy of the data. Note that this is considered a writeback mode and
+the guest OS must handle the disk write cache correctly in order to avoid data
+corruption on host crashes.
 
 The host page cache can be avoided while only sending write notifications to
-the guest when the data has been reported as written by the storage subsystem
-using @option{cache=directsync}.
-
-Some block drivers perform badly with @option{cache=writethrough}, most notably,
-qcow2.  If performance is more important than correctness,
-@option{cache=writeback} should be used with qcow2.
+the guest when the data has been flushed to the disk using
+@option{cache=directsync}.
 
 In case you don't care about data integrity over host failures, use
-cache=unsafe. This option tells QEMU that it never needs to write any data
-to the disk but can instead keeps things in cache. If anything goes wrong,
+@option{cache=unsafe}. This option tells QEMU that it never needs to write any
+data to the disk but can instead keeps things in cache. If anything goes wrong,
 like your host losing power, the disk storage getting disconnected accidentally,
 etc. you're image will most probably be rendered unusable.   When using
 the @option{-snapshot} option, unsafe caching is always used.
-- 
1.7.6.5

Re: [Qemu-devel] [PATCH 1/2] Documentation: Update block cache mode information

[Peter Maydell]

On 21 November 2012 13:23, Kevin Wolf <kwolf@redhat.com> wrote:
> Somehow we forgot to update this when cache=writeback became the
> default. While changing the information on the default, also make the
> description of all caches modes a bit more accurate.

Some minor nits below, mostly in text around the areas you've
changed rather than the changes themselves...

> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -206,31 +206,31 @@ Open drive @option{file} as read-only. Guest write attempts will fail.
>  file sectors into the image file.
>  @end table
>
> -By default, writethrough caching is used for all block device.  This means that
> -the host page cache will be used to read and write data but write notification
> -will be sent to the guest only when the data has been reported as written by
> -the storage subsystem.
> -
> -Writeback caching will report data writes as completed as soon as the data is
> -present in the host page cache.  This is safe as long as you trust your host.
> -If your host crashes or loses power, then the guest may experience data
> -corruption.
> +By default, the @option{cache=writeback} mode is used. It will report data
> +writes as completed as soon as the data is present in the host page cache.
> +This is safe as long as your guest OS makes sure to correctly flush disk caches
> +where needed. If your guest OS does not handle volatile disk write caches
> +correctly and your host crashes or loses power, then the guest may experience
> +data corruption.
> +
> +For such guests, you should consider using @option{cache=writethrough}. This
> +means that the host page cache will be used to read and write data, but write
> +notification will be sent to the guest only after qemu has made sure to flush

"QEMU"

> +each write to the disk. Be aware that this has a major impact on performance.
>
>  The host page cache can be avoided entirely with @option{cache=none}.  This will
>  attempt to do disk IO directly to the guests memory.  QEMU may still perform

"guest's"

> -an internal copy of the data.
> +an internal copy of the data. Note that this is considered a writeback mode and
> +the guest OS must handle the disk write cache correctly in order to avoid data
> +corruption on host crashes.
>
>  The host page cache can be avoided while only sending write notifications to
> -the guest when the data has been reported as written by the storage subsystem
> -using @option{cache=directsync}.
> -
> -Some block drivers perform badly with @option{cache=writethrough}, most notably,
> -qcow2.  If performance is more important than correctness,
> -@option{cache=writeback} should be used with qcow2.
> +the guest when the data has been flushed to the disk using
> +@option{cache=directsync}.
>
>  In case you don't care about data integrity over host failures, use

I think "If you don't" is more straightforward English here...

> -cache=unsafe. This option tells QEMU that it never needs to write any data
> -to the disk but can instead keeps things in cache. If anything goes wrong,
> +@option{cache=unsafe}. This option tells QEMU that it never needs to write any
> +data to the disk but can instead keeps things in cache. If anything goes wrong,

"can instead keep"

>  like your host losing power, the disk storage getting disconnected accidentally,
>  etc. you're image will most probably be rendered unusable.   When using

"your image"

>  the @option{-snapshot} option, unsafe caching is always used.

thanks
-- PMM

[Qemu-devel] [PATCH v2 1/2] Documentation: Update block cache mode information

[Kevin Wolf]

Somehow we forgot to update this when cache=writeback became the
default. While changing the information on the default, also make the
description of all caches modes a bit more accurate.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-options.hx |   38 +++++++++++++++++++-------------------
 1 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/qemu-options.hx b/qemu-options.hx
index 9bb29d3..b045ff8 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -206,33 +206,33 @@ Open drive @option{file} as read-only. Guest write attempts will fail.
 file sectors into the image file.
 @end table
 
-By default, writethrough caching is used for all block device.  This means that
-the host page cache will be used to read and write data but write notification
-will be sent to the guest only when the data has been reported as written by
-the storage subsystem.
-
-Writeback caching will report data writes as completed as soon as the data is
-present in the host page cache.  This is safe as long as you trust your host.
-If your host crashes or loses power, then the guest may experience data
-corruption.
+By default, the @option{cache=writeback} mode is used. It will report data
+writes as completed as soon as the data is present in the host page cache.
+This is safe as long as your guest OS makes sure to correctly flush disk caches
+where needed. If your guest OS does not handle volatile disk write caches
+correctly and your host crashes or loses power, then the guest may experience
+data corruption.
+
+For such guests, you should consider using @option{cache=writethrough}. This
+means that the host page cache will be used to read and write data, but write
+notification will be sent to the guest only after QEMU has made sure to flush
+each write to the disk. Be aware that this has a major impact on performance.
 
 The host page cache can be avoided entirely with @option{cache=none}.  This will
-attempt to do disk IO directly to the guests memory.  QEMU may still perform
-an internal copy of the data.
+attempt to do disk IO directly to the guest's memory.  QEMU may still perform
+an internal copy of the data. Note that this is considered a writeback mode and
+the guest OS must handle the disk write cache correctly in order to avoid data
+corruption on host crashes.
 
 The host page cache can be avoided while only sending write notifications to
-the guest when the data has been reported as written by the storage subsystem
-using @option{cache=directsync}.
-
-Some block drivers perform badly with @option{cache=writethrough}, most notably,
-qcow2.  If performance is more important than correctness,
-@option{cache=writeback} should be used with qcow2.
+the guest when the data has been flushed to the disk using
+@option{cache=directsync}.
 
 In case you don't care about data integrity over host failures, use
-cache=unsafe. This option tells QEMU that it never needs to write any data
-to the disk but can instead keeps things in cache. If anything goes wrong,
+@option{cache=unsafe}. This option tells QEMU that it never needs to write any
+data to the disk but can instead keep things in cache. If anything goes wrong,
 like your host losing power, the disk storage getting disconnected accidentally,
-etc. you're image will most probably be rendered unusable.   When using
+etc. your image will most probably be rendered unusable.   When using
 the @option{-snapshot} option, unsafe caching is always used.
 
 Copy-on-read avoids accessing the same backing file sectors repeatedly and is
-- 
1.7.6.5

[Qemu-devel] [PATCH 2/2] Documentation: Update image format information

[Kevin Wolf]

Document new options, mark QED as deprecated.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-img.texi |   37 ++++++++++++++++++++++++++++++-------
 1 files changed, 30 insertions(+), 7 deletions(-)

diff --git a/qemu-img.texi b/qemu-img.texi
index 60b83fc..2fdb3ef 100644
--- a/qemu-img.texi
+++ b/qemu-img.texi
@@ -247,6 +247,13 @@ support of multiple VM snapshots.
 
 Supported options:
 @table @code
+@item compat
+Determines the qcow2 version to use. @code{compat=0.10} uses the traditional
+image format that can be read by any QEMU since 0.10 (this is the default).
+@code{compat=1.1} enables image format extensions that only QEMU 1.1 and
+newer understand. Amongst others, this includes zero clusters, which allow
+efficient copy-on-read for sparse images.
+
 @item backing_file
 File name of a base image (see @option{create} subcommand)
 @item backing_fmt
@@ -267,14 +274,23 @@ Preallocation mode (allowed values: off, metadata). An image with preallocated
 metadata is initially larger but can improve performance when the image needs
 to grow.
 
+@item lazy_refcounts
+If this option is set to @code{on}, reference count updates are postponed with
+the goal of avoiding metadata I/O and improving performance. This is
+particularly interesting with @option{cache=writethrough} which doesn't batch
+metadata updates. The tradeoff is that after a host crash, the reference count
+tables must be rebuilt, i.e. on the next open an (automatic) @code{qemu-img
+check -r all} is required, which may take some time.
+
+This option can only be enabled if @code{compat=1.1} is specified.
+
 @end table
 
 @item qed
-Image format with support for backing files and compact image files (when your
-filesystem or transport medium does not support holes).  Good performance due
-to less metadata than the more featureful qcow2 format, especially with
-cache=writethrough or cache=directsync.  Consider using qcow2 which will soon
-have a similar optimization and is most actively developed.
+Old QEMU image format. Left for compatibility.
+
+For new images, use qcow2 instead. You might want to consider using the
+@code{lazy_refcounts=on} option to get a more QED-like behaviour.
 
 Supported options:
 @table @code
@@ -315,10 +331,17 @@ VMware 3 and 4 compatible image format.
 
 Supported options:
 @table @code
-@item backing_fmt
-Image format of the base image
+@item backing_file
+File name of a base image (see @option{create} subcommand).
 @item compat6
 Create a VMDK version 6 image (instead of version 4)
+@item subformat
+Specifies which VMDK subformat to use. Valid options are
+@code{monolithicSparse} (default),
+@code{monolithicFlat},
+@code{twoGbMaxExtentSparse},
+@code{twoGbMaxExtentFlat} and
+@code{streamOptimized}.
 @end table
 
 @item vpc
-- 
1.7.6.5

Re: [Qemu-devel] [PATCH 2/2] Documentation: Update image format information

[Stefan Hajnoczi]

On Wed, Nov 21, 2012 at 02:23:57PM +0100, Kevin Wolf wrote:
>  @item qed
> -Image format with support for backing files and compact image files (when your
> -filesystem or transport medium does not support holes).  Good performance due
> -to less metadata than the more featureful qcow2 format, especially with
> -cache=writethrough or cache=directsync.  Consider using qcow2 which will soon
> -have a similar optimization and is most actively developed.
> +Old QEMU image format. Left for compatibility.
> +
> +For new images, use qcow2 instead. You might want to consider using the
> +@code{lazy_refcounts=on} option to get a more QED-like behaviour.

The first sentence should be kept, it describes the general feature set
and scope of this image format.  I agree that the rest of the paragraph
can be dropped.

You could insert a statement saying that qcow2 is now preferred because
it is actively developed and offers advanced features and performance as
the very first sentence.

Stefan

Re: [Qemu-devel] [PATCH 2/2] Documentation: Update image format information

[Kevin Wolf]

Am 21.11.2012 16:14, schrieb Stefan Hajnoczi:
> On Wed, Nov 21, 2012 at 02:23:57PM +0100, Kevin Wolf wrote:
>>  @item qed
>> -Image format with support for backing files and compact image files (when your
>> -filesystem or transport medium does not support holes).  Good performance due
>> -to less metadata than the more featureful qcow2 format, especially with
>> -cache=writethrough or cache=directsync.  Consider using qcow2 which will soon
>> -have a similar optimization and is most actively developed.
>> +Old QEMU image format. Left for compatibility.
>> +
>> +For new images, use qcow2 instead. You might want to consider using the
>> +@code{lazy_refcounts=on} option to get a more QED-like behaviour.
> 
> The first sentence should be kept, it describes the general feature set
> and scope of this image format.  I agree that the rest of the paragraph
> can be dropped.
> 
> You could insert a statement saying that qcow2 is now preferred because
> it is actively developed and offers advanced features and performance as
> the very first sentence.

It's the same terse description as for qcow1. If we decided to describe
the format in more detail, we should do so consistently, and probably
also for non-native formats. (But why? The only use case is
compatibility with other/older hypervisors, which is mentioned.)

Kevin

Re: [Qemu-devel] [PATCH 2/2] Documentation: Update image format information

[Stefan Hajnoczi]

On Wed, Nov 21, 2012 at 04:22:07PM +0100, Kevin Wolf wrote:
> Am 21.11.2012 16:14, schrieb Stefan Hajnoczi:
> > On Wed, Nov 21, 2012 at 02:23:57PM +0100, Kevin Wolf wrote:
> >>  @item qed
> >> -Image format with support for backing files and compact image files (when your
> >> -filesystem or transport medium does not support holes).  Good performance due
> >> -to less metadata than the more featureful qcow2 format, especially with
> >> -cache=writethrough or cache=directsync.  Consider using qcow2 which will soon
> >> -have a similar optimization and is most actively developed.
> >> +Old QEMU image format. Left for compatibility.
> >> +
> >> +For new images, use qcow2 instead. You might want to consider using the
> >> +@code{lazy_refcounts=on} option to get a more QED-like behaviour.
> > 
> > The first sentence should be kept, it describes the general feature set
> > and scope of this image format.  I agree that the rest of the paragraph
> > can be dropped.
> > 
> > You could insert a statement saying that qcow2 is now preferred because
> > it is actively developed and offers advanced features and performance as
> > the very first sentence.
> 
> It's the same terse description as for qcow1. If we decided to describe
> the format in more detail, we should do so consistently, and probably
> also for non-native formats. (But why? The only use case is
> compatibility with other/older hypervisors, which is mentioned.)

Yes, we should have descriptions of other formats too.

Users dealing with existing guests using these formats should still have
access to general information about the formats.  For example, if I'm a
new user and it's my job to work with an existing qcow1 disk image then
it's not very helpful to just see a terse "Use qcow2 instead" message.

It's not good to drop documentation on a feature because it is
deprecated.

Stefan

Re: [Qemu-devel] [PATCH 2/2] Documentation: Update image format information

[Kevin Wolf]

Am 22.11.2012 08:37, schrieb Stefan Hajnoczi:
> On Wed, Nov 21, 2012 at 04:22:07PM +0100, Kevin Wolf wrote:
>> Am 21.11.2012 16:14, schrieb Stefan Hajnoczi:
>>> On Wed, Nov 21, 2012 at 02:23:57PM +0100, Kevin Wolf wrote:
>>>>  @item qed
>>>> -Image format with support for backing files and compact image files (when your
>>>> -filesystem or transport medium does not support holes).  Good performance due
>>>> -to less metadata than the more featureful qcow2 format, especially with
>>>> -cache=writethrough or cache=directsync.  Consider using qcow2 which will soon
>>>> -have a similar optimization and is most actively developed.
>>>> +Old QEMU image format. Left for compatibility.
>>>> +
>>>> +For new images, use qcow2 instead. You might want to consider using the
>>>> +@code{lazy_refcounts=on} option to get a more QED-like behaviour.
>>>
>>> The first sentence should be kept, it describes the general feature set
>>> and scope of this image format.  I agree that the rest of the paragraph
>>> can be dropped.
>>>
>>> You could insert a statement saying that qcow2 is now preferred because
>>> it is actively developed and offers advanced features and performance as
>>> the very first sentence.
>>
>> It's the same terse description as for qcow1. If we decided to describe
>> the format in more detail, we should do so consistently, and probably
>> also for non-native formats. (But why? The only use case is
>> compatibility with other/older hypervisors, which is mentioned.)
> 
> Yes, we should have descriptions of other formats too.
> 
> Users dealing with existing guests using these formats should still have
> access to general information about the formats.  For example, if I'm a
> new user and it's my job to work with an existing qcow1 disk image then
> it's not very helpful to just see a terse "Use qcow2 instead" message.
>
> It's not good to drop documentation on a feature because it is
> deprecated.

I can see your point, but the whole section starts to become a bit
lengthy then for a man page. Also, I don't think that your qcow1 user
really needs a detailed description - he already has a format and
doesn't have to choose one, so the characteristics aren't that
important. If he considers converting the image, we're back to raw and
qcow2.

Maybe we should keep the current descriptions for raw and qcow2 (as they
are what users really choose between for new images), and extend and
move the documentation of other formats into qemu-doc.texi and mention
them in the man page of qemu-img only as "Also supported for
compatibility with other hypervisors or older QEMU versions: vmdk, vdi,
qed, ... See the QEMU Emulator User Documentation for more information
on these."

Kevin

Re: [Qemu-devel] [PATCH 2/2] Documentation: Update image format information

[Stefan Hajnoczi]

On Thu, Nov 22, 2012 at 10:08:06AM +0100, Kevin Wolf wrote:
> Am 22.11.2012 08:37, schrieb Stefan Hajnoczi:
> > On Wed, Nov 21, 2012 at 04:22:07PM +0100, Kevin Wolf wrote:
> >> Am 21.11.2012 16:14, schrieb Stefan Hajnoczi:
> >>> On Wed, Nov 21, 2012 at 02:23:57PM +0100, Kevin Wolf wrote:
> >>>>  @item qed
> >>>> -Image format with support for backing files and compact image files (when your
> >>>> -filesystem or transport medium does not support holes).  Good performance due
> >>>> -to less metadata than the more featureful qcow2 format, especially with
> >>>> -cache=writethrough or cache=directsync.  Consider using qcow2 which will soon
> >>>> -have a similar optimization and is most actively developed.
> >>>> +Old QEMU image format. Left for compatibility.
> >>>> +
> >>>> +For new images, use qcow2 instead. You might want to consider using the
> >>>> +@code{lazy_refcounts=on} option to get a more QED-like behaviour.
> >>>
> >>> The first sentence should be kept, it describes the general feature set
> >>> and scope of this image format.  I agree that the rest of the paragraph
> >>> can be dropped.
> >>>
> >>> You could insert a statement saying that qcow2 is now preferred because
> >>> it is actively developed and offers advanced features and performance as
> >>> the very first sentence.
> >>
> >> It's the same terse description as for qcow1. If we decided to describe
> >> the format in more detail, we should do so consistently, and probably
> >> also for non-native formats. (But why? The only use case is
> >> compatibility with other/older hypervisors, which is mentioned.)
> > 
> > Yes, we should have descriptions of other formats too.
> > 
> > Users dealing with existing guests using these formats should still have
> > access to general information about the formats.  For example, if I'm a
> > new user and it's my job to work with an existing qcow1 disk image then
> > it's not very helpful to just see a terse "Use qcow2 instead" message.
> >
> > It's not good to drop documentation on a feature because it is
> > deprecated.
> 
> I can see your point, but the whole section starts to become a bit
> lengthy then for a man page. Also, I don't think that your qcow1 user
> really needs a detailed description - he already has a format and
> doesn't have to choose one, so the characteristics aren't that
> important. If he considers converting the image, we're back to raw and
> qcow2.
> 
> Maybe we should keep the current descriptions for raw and qcow2 (as they
> are what users really choose between for new images), and extend and
> move the documentation of other formats into qemu-doc.texi and mention
> them in the man page of qemu-img only as "Also supported for
> compatibility with other hypervisors or older QEMU versions: vmdk, vdi,
> qed, ... See the QEMU Emulator User Documentation for more information
> on these."

Sounds good.

Stefan