[Qemu-devel] Default cache mode

[Qemu-devel] Default cache mode

[Kevin Wolf]

Hi,

I think we have touched this topic before during some IRC discussions or
somewhere deep in a mailing list thread, but I think it hasn't been
discussed on the list.

Our default cache mode of cache=writethrough is extremely conservative
and provides absolute safety at the cost of performance, and most people
don't use it if they know that it can be changed because it just
performs too bad. There are use cases where you need it (broken guest
OS), but none and writeback are just as correct with respect to the
specs and they are safe to use with current OSes. And even with broken
OSes, in many use cases it doesn't really matter if you lose a VM  and
have to reinstall it (which is probably true even more for users
invoking qemu directly instead of using libvirt).

I think the motivation to switch from writeback to writethrough as
default was that writeback was entirely unsafe back then. This isn't
true any more, so is there still enough reason to have the slow
writethrough mode as default?

I'm not entirely sure if I should suggest writeback or none as the new
default, but I think it could make sense to change it.

Kevin

Re: [Qemu-devel] Default cache mode

[Anthony Liguori]

On 06/29/2011 06:59 AM, Kevin Wolf wrote:
> Hi,
>
> I think we have touched this topic before during some IRC discussions or
> somewhere deep in a mailing list thread, but I think it hasn't been
> discussed on the list.
>
> Our default cache mode of cache=writethrough is extremely conservative
> and provides absolute safety at the cost of performance,

But for the most part, we track bare metal fairly well in terms of block 
performance, no?

Or are you really referring to qcow2 as a specific example?  In the 
past, we used a different default caching mode for qcow2.  I think that 
could be done again if there was a compelling reason.

Regards,

Anthony Liguori

Re: [Qemu-devel] Default cache mode

[Kevin Wolf]

Am 29.06.2011 14:06, schrieb Anthony Liguori:
> On 06/29/2011 06:59 AM, Kevin Wolf wrote:
>> Hi,
>>
>> I think we have touched this topic before during some IRC discussions or
>> somewhere deep in a mailing list thread, but I think it hasn't been
>> discussed on the list.
>>
>> Our default cache mode of cache=writethrough is extremely conservative
>> and provides absolute safety at the cost of performance,
> 
> But for the most part, we track bare metal fairly well in terms of block 
> performance, no?
> 
> Or are you really referring to qcow2 as a specific example?  In the 
> past, we used a different default caching mode for qcow2.  I think that 
> could be done again if there was a compelling reason.

No, people are also complaining about bad performance with raw. Which
isn't really surprising when you do a flush after each single write
request. O_SYNC is really much more than is needed in the average case.

Kevin

Re: [Qemu-devel] Default cache mode

[Anthony Liguori]

On 06/29/2011 07:16 AM, Kevin Wolf wrote:
> Am 29.06.2011 14:06, schrieb Anthony Liguori:
>> On 06/29/2011 06:59 AM, Kevin Wolf wrote:
>>> Hi,
>>>
>>> I think we have touched this topic before during some IRC discussions or
>>> somewhere deep in a mailing list thread, but I think it hasn't been
>>> discussed on the list.
>>>
>>> Our default cache mode of cache=writethrough is extremely conservative
>>> and provides absolute safety at the cost of performance,
>>
>> But for the most part, we track bare metal fairly well in terms of block
>> performance, no?
>>
>> Or are you really referring to qcow2 as a specific example?  In the
>> past, we used a different default caching mode for qcow2.  I think that
>> could be done again if there was a compelling reason.
>
> No, people are also complaining about bad performance with raw. Which
> isn't really surprising when you do a flush after each single write
> request. O_SYNC is really much more than is needed in the average case.

Which file system on the host?

At any rate, I'm a big fan of making wce tunable in the guest and then I 
think setting wce=1 is quite reasonable to do by default.

Regards,

Anthony Liguori

>
> Kevin

Re: [Qemu-devel] Default cache mode

[Kevin Wolf]

Am 29.06.2011 14:23, schrieb Anthony Liguori:
> On 06/29/2011 07:16 AM, Kevin Wolf wrote:
>> Am 29.06.2011 14:06, schrieb Anthony Liguori:
>>> On 06/29/2011 06:59 AM, Kevin Wolf wrote:
>>>> Hi,
>>>>
>>>> I think we have touched this topic before during some IRC discussions or
>>>> somewhere deep in a mailing list thread, but I think it hasn't been
>>>> discussed on the list.
>>>>
>>>> Our default cache mode of cache=writethrough is extremely conservative
>>>> and provides absolute safety at the cost of performance,
>>>
>>> But for the most part, we track bare metal fairly well in terms of block
>>> performance, no?
>>>
>>> Or are you really referring to qcow2 as a specific example?  In the
>>> past, we used a different default caching mode for qcow2.  I think that
>>> could be done again if there was a compelling reason.
>>
>> No, people are also complaining about bad performance with raw. Which
>> isn't really surprising when you do a flush after each single write
>> request. O_SYNC is really much more than is needed in the average case.
> 
> Which file system on the host?

Honestly, I don't know. Usually the problem is resolved with setting a
different cache option, so nobody bothers to ask for details. I'd guess
that it's ext4 in most cases.

> At any rate, I'm a big fan of making wce tunable in the guest and then I 
> think setting wce=1 is quite reasonable to do by default.

Okay, I think that's a fair requirement. So let's plan to implement this
in 0.16.

Christoph, it looks like we're back to your WCE patches then. Do you
still work on them?

Kevin

Re: [Qemu-devel] Default cache mode

[Christoph Hellwig]

On Wed, Jun 29, 2011 at 02:32:34PM +0200, Kevin Wolf wrote:
> Christoph, it looks like we're back to your WCE patches then. Do you
> still work on them?

The IDE patch is ready once the bdrv_reopen from the hostcache patches
goes in.  Virtio still needs to be redesigned to use a command instead
of config space, and SCSI is not doable without a major SCSI layer
redesign.

Re: [Qemu-devel] Default cache mode

[Michael Tokarev]

29.06.2011 16:32, Kevin Wolf wrote:
> Am 29.06.2011 14:23, schrieb Anthony Liguori:
>> On 06/29/2011 07:16 AM, Kevin Wolf wrote:

>>> No, people are also complaining about bad performance with raw. Which
>>> isn't really surprising when you do a flush after each single write
>>> request. O_SYNC is really much more than is needed in the average case.
>>
>> Which file system on the host?
> 
> Honestly, I don't know. Usually the problem is resolved with setting a
> different cache option, so nobody bothers to ask for details. I'd guess
> that it's ext4 in most cases.

Extremly poor performance also happens on raw devices --
be it lvm volumes or plain partitions, so that's w/o
any filesystem.  Sure, a filesystem adds quite noticeable
overhead (because of mtime updates and that sort of things),
but _just_ syncronous writes are enough to kill performance.

/mjt

Re: [Qemu-devel] Default cache mode

[Christoph Hellwig]

On Wed, Jun 29, 2011 at 06:17:02PM +0400, Michael Tokarev wrote:
> > Honestly, I don't know. Usually the problem is resolved with setting a
> > different cache option, so nobody bothers to ask for details. I'd guess
> > that it's ext4 in most cases.
> 
> Extremly poor performance also happens on raw devices --
> be it lvm volumes or plain partitions, so that's w/o
> any filesystem.

Depends on your setup.  If you have ATA devices that have WCE=1
it will suck due to the non-queueable FLUSH command.  If you have
an older kernel with the draining barriers semantics it will to.

If you run on a SAS disk or FC array with WCE=0 performance will
be quite good and close to native performance on these.

That's all assuming you use O_DIRECT.  using cache=writhrough
as-is will suck everywher.

Re: [Qemu-devel] Default cache mode

[Michael Tokarev]

29.06.2011 18:50, Christoph Hellwig wrote:
> On Wed, Jun 29, 2011 at 06:17:02PM +0400, Michael Tokarev wrote:
>>> Honestly, I don't know. Usually the problem is resolved with setting a
>>> different cache option, so nobody bothers to ask for details. I'd guess
>>> that it's ext4 in most cases.
>>
>> Extremly poor performance also happens on raw devices --
>> be it lvm volumes or plain partitions, so that's w/o
>> any filesystem.
> 
> Depends on your setup.  If you have ATA devices that have WCE=1
> it will suck due to the non-queueable FLUSH command.  If you have
> an older kernel with the draining barriers semantics it will to.
> 
> If you run on a SAS disk or FC array with WCE=0 performance will
> be quite good and close to native performance on these.
> 
> That's all assuming you use O_DIRECT.  using cache=writhrough
> as-is will suck everywher.

The whole question (sorry I cut a bit more details than needed)
was about cache=writhrough :)

O_DIRECT on consumer drives with WCE=1 works fairy well too.

/mjt

Re: [Qemu-devel] Default cache mode

[Christoph Hellwig]

On Wed, Jun 29, 2011 at 07:23:31AM -0500, Anthony Liguori wrote:
> Which file system on the host?

Any filesystem.  Although extN and btrfs are particularly bad.

> At any rate, I'm a big fan of making wce tunable in the guest and then I 
> think setting wce=1 is quite reasonable to do by default.

Note that this means a manual override during each boot.  Windows and
modern Linux just use what's available.  Only Solaris with ZFS actually
turns WCE on if it's not set AFAIK.  And to me it's a rather questionable
behaviour at least for SCSI disks where the admin might have chosen
those intentionally.  (unlike ATA disks SCSI allows to store the setting
permanently)

Re: [Qemu-devel] Default cache mode

[Anthony Liguori]

On 06/29/2011 08:50 AM, Christoph Hellwig wrote:
> On Wed, Jun 29, 2011 at 07:23:31AM -0500, Anthony Liguori wrote:
>> Which file system on the host?
>
> Any filesystem.  Although extN and btrfs are particularly bad.
>
>> At any rate, I'm a big fan of making wce tunable in the guest and then I
>> think setting wce=1 is quite reasonable to do by default.
>
> Note that this means a manual override during each boot.  Windows and
> modern Linux just use what's available.  Only Solaris with ZFS actually
> turns WCE on if it's not set AFAIK.  And to me it's a rather questionable
> behaviour at least for SCSI disks where the admin might have chosen
> those intentionally.  (unlike ATA disks SCSI allows to store the setting
> permanently)

Is WCE ever persisted by a disk?  Would it make sense to have a 
mechanism to persist the WCE setting for a guest?

Regards,

Anthony Liguori

>

Re: [Qemu-devel] Default cache mode

[Christoph Hellwig]

On Wed, Jun 29, 2011 at 09:13:35AM -0500, Anthony Liguori wrote:
> Is WCE ever persisted by a disk?  Would it make sense to have a mechanism 
> to persist the WCE setting for a guest?

Only for SCSI, which has persistant mode pages.  For ATA I've not found
a way to make it persistent yet.

Re: [Qemu-devel] Default cache mode

[Stefan Hajnoczi]

On Wed, Jun 29, 2011 at 3:13 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> On 06/29/2011 08:50 AM, Christoph Hellwig wrote:
>>
>> On Wed, Jun 29, 2011 at 07:23:31AM -0500, Anthony Liguori wrote:
>>>
>>> Which file system on the host?
>>
>> Any filesystem.  Although extN and btrfs are particularly bad.
>>
>>> At any rate, I'm a big fan of making wce tunable in the guest and then I
>>> think setting wce=1 is quite reasonable to do by default.
>>
>> Note that this means a manual override during each boot.  Windows and
>> modern Linux just use what's available.  Only Solaris with ZFS actually
>> turns WCE on if it's not set AFAIK.  And to me it's a rather questionable
>> behaviour at least for SCSI disks where the admin might have chosen
>> those intentionally.  (unlike ATA disks SCSI allows to store the setting
>> permanently)
>
> Is WCE ever persisted by a disk?  Would it make sense to have a mechanism to
> persist the WCE setting for a guest?

Disks may store it in non-volatile memory, but it depends on the disk
and is reported in the cache mode page for SCSI.

Stefan

Re: [Qemu-devel] Default cache mode

[Frediano Ziglio]

2011/6/29 Anthony Liguori <anthony@codemonkey.ws>:
> On 06/29/2011 07:16 AM, Kevin Wolf wrote:
>>
>> Am 29.06.2011 14:06, schrieb Anthony Liguori:
>>>
>>> On 06/29/2011 06:59 AM, Kevin Wolf wrote:
>>>>
>>>> Hi,
>>>>
>>>> I think we have touched this topic before during some IRC discussions or
>>>> somewhere deep in a mailing list thread, but I think it hasn't been
>>>> discussed on the list.
>>>>
>>>> Our default cache mode of cache=writethrough is extremely conservative
>>>> and provides absolute safety at the cost of performance,
>>>
>>> But for the most part, we track bare metal fairly well in terms of block
>>> performance, no?
>>>
>>> Or are you really referring to qcow2 as a specific example?  In the
>>> past, we used a different default caching mode for qcow2.  I think that
>>> could be done again if there was a compelling reason.
>>
>> No, people are also complaining about bad performance with raw. Which
>> isn't really surprising when you do a flush after each single write
>> request. O_SYNC is really much more than is needed in the average case.
>
> Which file system on the host?
>
> At any rate, I'm a big fan of making wce tunable in the guest and then I
> think setting wce=1 is quite reasonable to do by default.
>
> Regards,
>
> Anthony Liguori
>
>>
>> Kevin
>
>
>

Personally I think default lead to poor performance but currently
people know that even critical application are handled correctly.
Assuming a client connect to a database server on a qemu guest when
server reply "transaction ok" you know that even a host crash lead to
a successful transaction. At least the fact that this assumption won't
be true has to be stated.

Lately I'm doing some extensive testing on image performance and I
noted that are not only image dependent but also filesystem (where
image is stored) dependent but when data are allocated speed is good.
raw disks are not so fast as expected? Try to use fallocate command to
allocate the file than do a dd to fill the entire file with zeros and
writethrough will perform very well. Allocating raw with qemu-img lead
to a normal sparse file which written with O_DSYNC flag is quite slow
due to extents allocation and fragmentation.

Regards
 Frediano Ziglio

Re: [Qemu-devel] Default cache mode

[Kevin Wolf]

Am 29.06.2011 15:53, schrieb Frediano Ziglio:
> 2011/6/29 Anthony Liguori <anthony@codemonkey.ws>:
>> On 06/29/2011 07:16 AM, Kevin Wolf wrote:
>>>
>>> Am 29.06.2011 14:06, schrieb Anthony Liguori:
>>>>
>>>> On 06/29/2011 06:59 AM, Kevin Wolf wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I think we have touched this topic before during some IRC discussions or
>>>>> somewhere deep in a mailing list thread, but I think it hasn't been
>>>>> discussed on the list.
>>>>>
>>>>> Our default cache mode of cache=writethrough is extremely conservative
>>>>> and provides absolute safety at the cost of performance,
>>>>
>>>> But for the most part, we track bare metal fairly well in terms of block
>>>> performance, no?
>>>>
>>>> Or are you really referring to qcow2 as a specific example?  In the
>>>> past, we used a different default caching mode for qcow2.  I think that
>>>> could be done again if there was a compelling reason.
>>>
>>> No, people are also complaining about bad performance with raw. Which
>>> isn't really surprising when you do a flush after each single write
>>> request. O_SYNC is really much more than is needed in the average case.
>>
>> Which file system on the host?
>>
>> At any rate, I'm a big fan of making wce tunable in the guest and then I
>> think setting wce=1 is quite reasonable to do by default.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>>
>>> Kevin
>>
>>
>>
> 
> Personally I think default lead to poor performance but currently
> people know that even critical application are handled correctly.
> Assuming a client connect to a database server on a qemu guest when
> server reply "transaction ok" you know that even a host crash lead to
> a successful transaction. At least the fact that this assumption won't
> be true has to be stated.

If the guest is acting correctly, then this assumption is definitely
true with cache=none/writeback. What needs to happen is that the guest
issues a disk flush before sending your "transaction ok" message. If it
doesn't do that it can fail even on real hardware. cache=writethrough
makes things safe that aren't even safe on real hardware, that's why I
consider it extremely conservative.

Kevin

Re: [Qemu-devel] Default cache mode

[Christoph Hellwig]

On Wed, Jun 29, 2011 at 04:11:53PM +0200, Kevin Wolf wrote:
> If the guest is acting correctly, then this assumption is definitely
> true with cache=none/writeback. What needs to happen is that the guest
> issues a disk flush before sending your "transaction ok" message. If it
> doesn't do that it can fail even on real hardware. cache=writethrough
> makes things safe that aren't even safe on real hardware, that's why I
> consider it extremely conservative.

Depends on your defintion of real hardware.  Traditionally disks always
had WCE=0 semantics.  When ATA disks added volatile writecaches and people
started seeing problems due to it the advertisement of cache modes and
cache flush commands were added.  For non-ATA disks like SAS or FC
WCE=1 still is the typical delivery setting.

Re: [Qemu-devel] Default cache mode

[Kevin Wolf]

Am 29.06.2011 16:12, schrieb Christoph Hellwig:
> On Wed, Jun 29, 2011 at 04:11:53PM +0200, Kevin Wolf wrote:
>> If the guest is acting correctly, then this assumption is definitely
>> true with cache=none/writeback. What needs to happen is that the guest
>> issues a disk flush before sending your "transaction ok" message. If it
>> doesn't do that it can fail even on real hardware. cache=writethrough
>> makes things safe that aren't even safe on real hardware, that's why I
>> consider it extremely conservative.
> 
> Depends on your defintion of real hardware.  Traditionally disks always
> had WCE=0 semantics.  When ATA disks added volatile writecaches and people
> started seeing problems due to it the advertisement of cache modes and
> cache flush commands were added.  For non-ATA disks like SAS or FC
> WCE=1 still is the typical delivery setting.

Sure, I'm not saying that it definitely breaks on any hardware, just
that it can break.

Kevin

Re: [Qemu-devel] Default cache mode

[Christoph Hellwig]

On Wed, Jun 29, 2011 at 07:06:03AM -0500, Anthony Liguori wrote:
> But for the most part, we track bare metal fairly well in terms of block 
> performance, no?

only if using cache=none and fmt=raw on either a blockdevice or a fully
preallocated (not fallocated) file on a modern filesystem.

Re: [Qemu-devel] Default cache mode

[Anthony Liguori]

On 06/29/2011 06:59 AM, Kevin Wolf wrote:
> Hi,
>
> I think we have touched this topic before during some IRC discussions or
> somewhere deep in a mailing list thread, but I think it hasn't been
> discussed on the list.
>
> Our default cache mode of cache=writethrough is extremely conservative
> and provides absolute safety at the cost of performance,


Thinking twice about this, shouldn't we just move to a simplified model:

-drive file=foo.img,cache=[on|off],hd0  -device 
virtio-blk-pci,drive=hd0,wce=1

As long as we advertise wce and wce can be toggled from the guest, I 
don't think the default is all that important.  I think cache=on is the 
right default for most common use cases.

Regards,

Anthony Liguori

Re: [Qemu-devel] Default cache mode

[Christoph Hellwig]

On Wed, Jun 29, 2011 at 07:08:18AM -0500, Anthony Liguori wrote:
> As long as we advertise wce and wce can be toggled from the guest, I don't 
> think the default is all that important.  I think cache=on is the right 
> default for most common use cases.

What do you mean with cache=on?  We have
cache=[none,writeback,writethrough,unsafe].  As discussed about a dozen
times before these are an incomplete choice of the [odirect=on,off;
osync=on,off, ignore cache flushes=on,off] matrix.

Not using O_DIRECT when it is available is almost always the wrong choice.
It means an additional data copy, VM pressure and exercise of often suck
fsync/O_SYNC codepatheses, with only a benefit if you have multiple VMs
actually using the same dataset with the same COW backing image.

Using WCE = 0 or 1 is a tradeoff of safety vs performance.  WCE=0 performance
generally sucks badly, but offers data integrity for older or non-PCish host
operating systems.

Re: [Qemu-devel] Default cache mode

[Anthony Liguori]

On 06/29/2011 08:55 AM, Christoph Hellwig wrote:
> On Wed, Jun 29, 2011 at 07:08:18AM -0500, Anthony Liguori wrote:
>> As long as we advertise wce and wce can be toggled from the guest, I don't
>> think the default is all that important.  I think cache=on is the right
>> default for most common use cases.
>
> What do you mean with cache=on?

Sorry, I meant having a new cache option (maybe diskcache).  Semantics 
would be:

diskcache=on,wce=on    0
diskcache=on,wce=off   O_SYNC
diskcache=off,wce=on   O_DIRECT
diskcache=off,wce=off  O_DIRECT | O_SYNC

ignore_flush could be another option too.

> We have
> cache=[none,writeback,writethrough,unsafe].  As discussed about a dozen
> times before these are an incomplete choice of the [odirect=on,off;
> osync=on,off, ignore cache flushes=on,off] matrix.

Yup, I agree that we need to deprecate 
cache=[none,writeback,writethrough,unsafe] and add more meaningful tunables.

> Not using O_DIRECT when it is available is almost always the wrong choice.
> It means an additional data copy, VM pressure and exercise of often suck
> fsync/O_SYNC codepatheses, with only a benefit if you have multiple VMs
> actually using the same dataset with the same COW backing image.

Can we set globals for -drive via -readconfig?

If so, we could defer the default value to a config file setting which I 
think would be the best possible scenario.

I think developers usually want diskcache=on but I also agree that 
usually in production, you want diskcache=off.

Regards,

Anthony Liguori

Re: [Qemu-devel] Default cache mode

[Christoph Hellwig]

On Wed, Jun 29, 2011 at 09:20:03AM -0500, Anthony Liguori wrote:
> Sorry, I meant having a new cache option (maybe diskcache).  Semantics 
> would be:
>
> diskcache=on,wce=on    0
> diskcache=on,wce=off   O_SYNC
> diskcache=off,wce=on   O_DIRECT
> diskcache=off,wce=off  O_DIRECT | O_SYNC
>
> ignore_flush could be another option too.

Please make it two options instead of a two-dimensional one.

E.g. e hostcache=[on,off] option for the drive command line,
where fake is the equivalent of the current code together with

> I think developers usually want diskcache=on but I also agree that usually 
> in production, you want diskcache=off.

Why would you want to not use O_DIRECT as a developer?  Especially for
desktop setups where the VM images might be on a common use filesystem
not using O_DIRECT a deathly option for performance.

Re: [Qemu-devel] Default cache mode

[Anthony Liguori]

On 06/29/2011 09:52 AM, Christoph Hellwig wrote:
> On Wed, Jun 29, 2011 at 09:20:03AM -0500, Anthony Liguori wrote:
>> Sorry, I meant having a new cache option (maybe diskcache).  Semantics
>> would be:
>>
>> diskcache=on,wce=on    0
>> diskcache=on,wce=off   O_SYNC
>> diskcache=off,wce=on   O_DIRECT
>> diskcache=off,wce=off  O_DIRECT | O_SYNC
>>
>> ignore_flush could be another option too.
>
> Please make it two options instead of a two-dimensional one.

Yeah, that was just for the table.  diskcache/hostcache belongs in 
-drive, wce belongs in -device.

> E.g. e hostcache=[on,off] option for the drive command line,
> where fake is the equivalent of the current code together with
>
>> I think developers usually want diskcache=on but I also agree that usually
>> in production, you want diskcache=off.
>
> Why would you want to not use O_DIRECT as a developer?

Relaunching the same guest:

cache=writethrough:

real	0m9.500s
user	0m4.720s
sys	0m3.780s

cache=none:

real	0m15.263s
user	0m4.870s
sys	0m6.390s

If you're doing kernel development where you're frequently launching the 
same guest with a different kernel, it using the page cache makes a big 
difference in guest boot time.

Regards,

Anthony Liguori

Re: [Qemu-devel] Default cache mode

[Alexander Graf]

On 29.06.2011, at 13:59, Kevin Wolf wrote:

> Hi,
> 
> I think we have touched this topic before during some IRC discussions or
> somewhere deep in a mailing list thread, but I think it hasn't been
> discussed on the list.
> 
> Our default cache mode of cache=writethrough is extremely conservative
> and provides absolute safety at the cost of performance, and most people
> don't use it if they know that it can be changed because it just
> performs too bad. There are use cases where you need it (broken guest
> OS), but none and writeback are just as correct with respect to the
> specs and they are safe to use with current OSes. And even with broken
> OSes, in many use cases it doesn't really matter if you lose a VM  and
> have to reinstall it (which is probably true even more for users
> invoking qemu directly instead of using libvirt).
> 
> I think the motivation to switch from writeback to writethrough as
> default was that writeback was entirely unsafe back then. This isn't
> true any more, so is there still enough reason to have the slow
> writethrough mode as default?
> 
> I'm not entirely sure if I should suggest writeback or none as the new
> default, but I think it could make sense to change it.

None as default would be a bad choice, as not all underlying file systems support it. Try cache=none on tmpfs or nfs. It will just fail on you.
What I'd personally like to see is some sort of detection that can determine which caching mode would perform best. So it would do cache=none when it makes sense (raw block device), otherwise cache=writeback.


Alex

Re: [Qemu-devel] Default cache mode

[Stefan Hajnoczi]

On Wed, Jun 29, 2011 at 1:21 PM, Alexander Graf <agraf@suse.de> wrote:
> On 29.06.2011, at 13:59, Kevin Wolf wrote:
>> I'm not entirely sure if I should suggest writeback or none as the new
>> default, but I think it could make sense to change it.
>
> None as default would be a bad choice, as not all underlying file systems support it. Try cache=none on tmpfs or nfs. It will just fail on you.
> What I'd personally like to see is some sort of detection that can determine which caching mode would perform best. So it would do cache=none when it makes sense (raw block device), otherwise cache=writeback.

O_DIRECT works on the Linux NFS client.  The performance currently
isn't great with vectored I/O (which QEMU does) but there is work
ongoing to improve it.

Your point makes sense though.  The default (no explicit cache=)
should mean "the most appropriate caching mode".

Stefan

Re: [Qemu-devel] Default cache mode

[Alexander Graf]

On 29.06.2011, at 15:57, Stefan Hajnoczi wrote:

> On Wed, Jun 29, 2011 at 1:21 PM, Alexander Graf <agraf@suse.de> wrote:
>> On 29.06.2011, at 13:59, Kevin Wolf wrote:
>>> I'm not entirely sure if I should suggest writeback or none as the new
>>> default, but I think it could make sense to change it.
>> 
>> None as default would be a bad choice, as not all underlying file systems support it. Try cache=none on tmpfs or nfs. It will just fail on you.
>> What I'd personally like to see is some sort of detection that can determine which caching mode would perform best. So it would do cache=none when it makes sense (raw block device), otherwise cache=writeback.
> 
> O_DIRECT works on the Linux NFS client.  The performance currently
> isn't great with vectored I/O (which QEMU does) but there is work
> ongoing to improve it.

It does? One of the things I stumbled over when doing autotest was that cache=none was the default - and that simply failed on my nfs-root system.

> Your point makes sense though.  The default (no explicit cache=)
> should mean "the most appropriate caching mode".

Yeah, that's what defaults should be like in most cases. The same reason people are for going with -cpu host as default :).


Alex

Re: [Qemu-devel] Default cache mode

[Stefan Hajnoczi]

On Wed, Jun 29, 2011 at 12:59 PM, Kevin Wolf <kwolf@redhat.com> wrote:
> I'm not entirely sure if I should suggest writeback or none as the new
> default, but I think it could make sense to change it.

I agree that cache=none is safe and fast with correct guests and local
disks.  It is beaten by cache=writeback in cases where the host has a
lot of buffer cache to play with and the guest has very little - for
example when running qemu without a -m argument.  If I was running
qemu-system-arm I might like cache=writeback.  If I was virtualizing
x86 servers I might like cache=none.

cache=none as a default - which some management tools inherit - would
be reasonable.  Folks using command-line qemu can always tweak to use
cache=writeback or even cache=unsafe if performance matters but data
integrity does not.

Whether it is safe to transition to cache=none or not depends on what
broken guests are still widely deployed.  Does CentOS 5 flush the disk
cache?

Stefan

Re: [Qemu-devel] Default cache mode

[Christoph Hellwig]

On Wed, Jun 29, 2011 at 01:52:19PM +0100, Stefan Hajnoczi wrote:
> Whether it is safe to transition to cache=none or not depends on what
> broken guests are still widely deployed.  Does CentOS 5 flush the disk
> cache?

Unless you use ext3 it does.  The same unfortunate story still applies to
current Linux mainline.

Re: [Qemu-devel] Default cache mode

[Avi Kivity]

On 06/29/2011 02:59 PM, Kevin Wolf wrote:
> Hi,
>
> I think we have touched this topic before during some IRC discussions or
> somewhere deep in a mailing list thread, but I think it hasn't been
> discussed on the list.
>
> Our default cache mode of cache=writethrough is extremely conservative
> and provides absolute safety at the cost of performance, and most people
> don't use it if they know that it can be changed because it just
> performs too bad. There are use cases where you need it (broken guest
> OS), but none and writeback are just as correct with respect to the
> specs and they are safe to use with current OSes. And even with broken
> OSes, in many use cases it doesn't really matter if you lose a VM  and
> have to reinstall it (which is probably true even more for users
> invoking qemu directly instead of using libvirt).
>
> I think the motivation to switch from writeback to writethrough as
> default was that writeback was entirely unsafe back then. This isn't
> true any more, so is there still enough reason to have the slow
> writethrough mode as default?
>
> I'm not entirely sure if I should suggest writeback or none as the new
> default, but I think it could make sense to change it.

So long as -M old retains the old behaviour, I'm in favour.

I think writeback is probably a better default than none.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Default cache mode

[Anthony Liguori]

On 06/29/2011 08:00 AM, Avi Kivity wrote:
> On 06/29/2011 02:59 PM, Kevin Wolf wrote:
>> Hi,
>>
>> I think we have touched this topic before during some IRC discussions or
>> somewhere deep in a mailing list thread, but I think it hasn't been
>> discussed on the list.
>>
>> Our default cache mode of cache=writethrough is extremely conservative
>> and provides absolute safety at the cost of performance, and most people
>> don't use it if they know that it can be changed because it just
>> performs too bad. There are use cases where you need it (broken guest
>> OS), but none and writeback are just as correct with respect to the
>> specs and they are safe to use with current OSes. And even with broken
>> OSes, in many use cases it doesn't really matter if you lose a VM and
>> have to reinstall it (which is probably true even more for users
>> invoking qemu directly instead of using libvirt).
>>
>> I think the motivation to switch from writeback to writethrough as
>> default was that writeback was entirely unsafe back then. This isn't
>> true any more, so is there still enough reason to have the slow
>> writethrough mode as default?
>>
>> I'm not entirely sure if I should suggest writeback or none as the new
>> default, but I think it could make sense to change it.
>
> So long as -M old retains the old behaviour, I'm in favour.

wce needs to be preserved but cache wouldn't need to be AFAICT.

Regards,

Anthony Liguori

>
> I think writeback is probably a better default than none.
>

Re: [Qemu-devel] Default cache mode

[Avi Kivity]

On 06/29/2011 04:05 PM, Anthony Liguori wrote:
>> So long as -M old retains the old behaviour, I'm in favour.
>
>
> wce needs to be preserved but cache wouldn't need to be AFAICT.

Right.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Default cache mode

[Kevin Wolf]

Am 29.06.2011 15:00, schrieb Avi Kivity:
> On 06/29/2011 02:59 PM, Kevin Wolf wrote:
>> Hi,
>>
>> I think we have touched this topic before during some IRC discussions or
>> somewhere deep in a mailing list thread, but I think it hasn't been
>> discussed on the list.
>>
>> Our default cache mode of cache=writethrough is extremely conservative
>> and provides absolute safety at the cost of performance, and most people
>> don't use it if they know that it can be changed because it just
>> performs too bad. There are use cases where you need it (broken guest
>> OS), but none and writeback are just as correct with respect to the
>> specs and they are safe to use with current OSes. And even with broken
>> OSes, in many use cases it doesn't really matter if you lose a VM  and
>> have to reinstall it (which is probably true even more for users
>> invoking qemu directly instead of using libvirt).
>>
>> I think the motivation to switch from writeback to writethrough as
>> default was that writeback was entirely unsafe back then. This isn't
>> true any more, so is there still enough reason to have the slow
>> writethrough mode as default?
>>
>> I'm not entirely sure if I should suggest writeback or none as the new
>> default, but I think it could make sense to change it.
> 
> So long as -M old retains the old behaviour, I'm in favour.

Means that we need to move the WCE flag into guest state first, but
that's what Anthony suggested, too. So I think we can do that.

> I think writeback is probably a better default than none.

Alex had a good point that O_DIRECT doesn't work everywhere, so I agree.

Kevin