[Qemu-devel] Is cache=writeback safe yet?

[Qemu-devel] Is cache=writeback safe yet?

[Virtbie]

Dear qemuers,
thanks for your exellent software.

I would like to use cache=writeback, but I still can't understand if 
this is safe or not in case of power loss.
In particular with virtio-blk on bare LVM device.

Qemu manpage still says cache=writeback isn't safe, but I seem to find 
some discordant information reading around.

Is the WCE+volatile flag exposed to the guest already? I am logging into 
a 2.6.38 guest and trying to find an indicator of disk cache in 
/sys/block/vda/device but I am not able to. There is an obscure very 
long "features" bitmask which I don't know what it shows. I think this 
mode is not safe until WBC is exposed, AFAIU, right?


Also please have a look at this:
https://events.linuxfoundation.org/slides/2011/linuxcon-japan/lcj2011_hajnoczi.pdf
page "Caching modes in Qemu"
In the table it is written that writeback has "Guest disk write cache = on".
Does that mean that the guest can commit to real platters by issuing a 
flush on the virtual device?
So the problem lies in the fact that no guest will ever spontaneously 
issue the flush because they don't see a wce=1?


There is another thing I don't understand:
I think I read somewhere that cache=none is safer than cache=writeback. 
Is that true? I think that both have a writeback cache, one is in the 
physical disks attached to the host, the other is in the host's page 
cache, so they should have about the same level of safety. What do you 
think?


Lastly, regarding the newly introduced cache=directsync. Do I understand 
correctly that it is supposed to be as safe as cache=writeback but also 
as slow as cache=writeback, for writes, and in addition it cannot use 
the host pagecache for reads, is that correct?


BTW I also wanted to say I very much like the simplification proposed by 
Anthony Liguori on 06/29/2011:

> Thinking twice about this, shouldn't we just move to a simplified model:
>
> -drive file=foo.img,cache=[on|off],hd0  -device 
> virtio-blk-pci,drive=hd0,wce=1

splitting the cache features in two (host side / guest side) would very 
much clarify what is happening in Qemu, for us ignorant users.


Thank you
Vb.

Re: [Qemu-devel] Is cache=writeback safe yet?

[Anthony Liguori]

On 02/20/2012 08:18 AM, Virtbie wrote:
> Dear qemuers,
> thanks for your exellent software.
>
> I would like to use cache=writeback, but I still can't understand if this is
> safe or not in case of power loss.

"Safe" is too simplistic of a view.  The documentation in qemu-options.hx 
probably needs to be revisited.

cache=writeback emulates a large disk cache much the same as every modern hard 
drive has a builtin cache.

The only real difference is that the host cache is very, very large.  Some 
modern file systems did not take the necessary steps to ensure consistency when 
a volatile write cache is present (ext3 up until very recently when barrier=1 
became default).

In practice, this didn't create a huge issue because disk write caches are 
flushed often.  Ted T'so has written a lot about the practical bits here.

But since the host cache is very large, and may not be flushed for many minutes 
after the initial write, this can exacerbate the problem.

In short, if you're using a recent kernel with ext3 or ext4, cache=writeback is 
absolutely safe.  If you're using an older version of ext3, cache=writeback is 
still safe but ext3 itself isn't.  cache=writeback can make the situation worse.

cache=writethrough presents a non-volatile cache which makes even older ext3 
filesystems safe.

Regards,

Anthony Liguori

> In particular with virtio-blk on bare LVM device.
>
> Qemu manpage still says cache=writeback isn't safe, but I seem to find some
> discordant information reading around.
>
> Is the WCE+volatile flag exposed to the guest already? I am logging into a
> 2.6.38 guest and trying to find an indicator of disk cache in
> /sys/block/vda/device but I am not able to. There is an obscure very long
> "features" bitmask which I don't know what it shows. I think this mode is not
> safe until WBC is exposed, AFAIU, right?
>
>
> Also please have a look at this:
> https://events.linuxfoundation.org/slides/2011/linuxcon-japan/lcj2011_hajnoczi.pdf
> page "Caching modes in Qemu"
> In the table it is written that writeback has "Guest disk write cache = on".
> Does that mean that the guest can commit to real platters by issuing a flush on
> the virtual device?
> So the problem lies in the fact that no guest will ever spontaneously issue the
> flush because they don't see a wce=1?
>
>
> There is another thing I don't understand:
> I think I read somewhere that cache=none is safer than cache=writeback. Is that
> true? I think that both have a writeback cache, one is in the physical disks
> attached to the host, the other is in the host's page cache, so they should have
> about the same level of safety. What do you think?
>
>
> Lastly, regarding the newly introduced cache=directsync. Do I understand
> correctly that it is supposed to be as safe as cache=writeback but also as slow
> as cache=writeback, for writes, and in addition it cannot use the host pagecache
> for reads, is that correct?
>
>
> BTW I also wanted to say I very much like the simplification proposed by Anthony
> Liguori on 06/29/2011:
>
>> Thinking twice about this, shouldn't we just move to a simplified model:
>>
>> -drive file=foo.img,cache=[on|off],hd0 -device virtio-blk-pci,drive=hd0,wce=1
>
> splitting the cache features in two (host side / guest side) would very much
> clarify what is happening in Qemu, for us ignorant users.
>
>
> Thank you
> Vb.
>

Re: [Qemu-devel] Is cache=writeback safe yet?

[Kevin Wolf]

Am 20.02.2012 15:18, schrieb Virtbie:
> Dear qemuers,
> thanks for your exellent software.
> 
> I would like to use cache=writeback, but I still can't understand if 
> this is safe or not in case of power loss.
> In particular with virtio-blk on bare LVM device.
> 
> Qemu manpage still says cache=writeback isn't safe, but I seem to find 
> some discordant information reading around.
> 
> Is the WCE+volatile flag exposed to the guest already? I am logging into 
> a 2.6.38 guest and trying to find an indicator of disk cache in 
> /sys/block/vda/device but I am not able to. There is an obscure very 
> long "features" bitmask which I don't know what it shows. I think this 
> mode is not safe until WBC is exposed, AFAIU, right?

qemu exposes the WCE flag and if your guest handles it correctly (i.e.
it sends flush requests where needed), cache=writeback is safe. This is
true for recent Linux versions.

> Also please have a look at this:
> https://events.linuxfoundation.org/slides/2011/linuxcon-japan/lcj2011_hajnoczi.pdf
> page "Caching modes in Qemu"
> In the table it is written that writeback has "Guest disk write cache = on".
> Does that mean that the guest can commit to real platters by issuing a 
> flush on the virtual device?

Yes.

> So the problem lies in the fact that no guest will ever spontaneously 
> issue the flush because they don't see a wce=1?

This might have been true in very old versions of qemu, I'm not sure.
The real problem is that some guests could see the wce=1 if they
checked, but they simply don't care about it and don't send flushes anyway.

> There is another thing I don't understand:
> I think I read somewhere that cache=none is safer than cache=writeback. 
> Is that true? I think that both have a writeback cache, one is in the 
> physical disks attached to the host, the other is in the host's page 
> cache, so they should have about the same level of safety. What do you 
> think?

Yes, they are both considered writeback cache modes that require the
guest to issue flushes in order to be correct. The only difference is
that the cache is much bigger with cache=writeback (the whole host RAM
as opposed to a small disk cache).

> Lastly, regarding the newly introduced cache=directsync. Do I understand 
> correctly that it is supposed to be as safe as cache=writeback but also 
> as slow as cache=writeback, for writes, and in addition it cannot use 
> the host pagecache for reads, is that correct?

s/writeback/writethrough/ (I think this also is what you really meant)

Yes, it is a writethrough mode that doesn't use the host cache. Or
translated into open() flags, it's O_DIRECT | O_DSYNC. You usually don't
want to cache the data in the host because you already have a cache in
the guest. Caching twice is just a waste of memory. Also, Linux AIO only
works with O_DIRECT.

> BTW I also wanted to say I very much like the simplification proposed by 
> Anthony Liguori on 06/29/2011:
> 
>> Thinking twice about this, shouldn't we just move to a simplified model:
>>
>> -drive file=foo.img,cache=[on|off],hd0  -device 
>> virtio-blk-pci,drive=hd0,wce=1
> 
> splitting the cache features in two (host side / guest side) would very 
> much clarify what is happening in Qemu, for us ignorant users.

Yes, we're planning to move to something like this eventually.

Kevin

Re: [Qemu-devel] Is cache=writeback safe yet?

[Peter Maydell]

On 20 February 2012 15:20, Kevin Wolf <kwolf@redhat.com> wrote:
> Am 20.02.2012 15:18, schrieb Virtbie:
>> Qemu manpage still says cache=writeback isn't safe, but I seem to find
>> some discordant information reading around.
>>
>> Is the WCE+volatile flag exposed to the guest already? I am logging into
>> a 2.6.38 guest and trying to find an indicator of disk cache in
>> /sys/block/vda/device but I am not able to. There is an obscure very
>> long "features" bitmask which I don't know what it shows. I think this
>> mode is not safe until WBC is exposed, AFAIU, right?
>
> qemu exposes the WCE flag and if your guest handles it correctly (i.e.
> it sends flush requests where needed), cache=writeback is safe. This is
> true for recent Linux versions.

The thing that confuses me when this subject comes up is that "cache=writeback"
is a property of the block layer, but the WCE flag is a SCSI parameter,
right? How does this work on non-SCSI disks? Is there something that
eg hw/sd.c should be doing to tell the block layer "writeback cache is
safe/unsafe" ?

-- PMM

Re: [Qemu-devel] Is cache=writeback safe yet?

[Virtbie]

On 02/20/12 16:06, Anthony Liguori wrote:
> On 02/20/2012 08:18 AM, Virtbie wrote:
>> Dear qemuers,
>> thanks for your exellent software.
>>
>> I would like to use cache=writeback, but I still can't understand if 
>> this is
>> safe or not in case of power loss.
>
> "Safe" is too simplistic of a view.  The documentation in 
> qemu-options.hx probably needs to be revisited.
>
> cache=writeback emulates a large disk cache much the same as every 
> modern hard drive has a builtin cache.
>
> The only real difference is that the host cache is very, very large.  
> Some modern file systems did not take the necessary steps to ensure 
> consistency when a volatile write cache is present (ext3 up until very 
> recently when barrier=1 became default).
>
> In practice, this didn't create a huge issue because disk write caches 
> are flushed often.  Ted T'so has written a lot about the practical 
> bits here.
>
> But since the host cache is very large, and may not be flushed for 
> many minutes after the initial write, this can exacerbate the problem.
>
> In short, if you're using a recent kernel with ext3 or ext4, 
> cache=writeback is absolutely safe.  If you're using an older version 
> of ext3, cache=writeback is still safe but ext3 itself isn't.  
> cache=writeback can make the situation worse.
>
> cache=writethrough presents a non-volatile cache which makes even 
> older ext3 filesystems safe.
>
> Regards,
>
> Anthony Liguori


Great explanation Anthony,

may I still ask:

1)
Is WCE + volatile flag exposed to the guest, by all three virtual devices:
- virtio
- scsi
- ide
?
(if not, I still don't understand how this works)


2) Is there a minimum guest kernel and a minimum viostor Windows driver 
version, to see such WCE+volatile flag in a virtio disk, so that the 
guest OS can actually see the cache?


Because I can't seem to find such flag for a virtio disk on a linux 
2.6.38 guest and that seems serious to me.
Is it visible somewhere in /sys hierarchy? Or is 2.6.38 too old?


Thank you
Vb.

Re: [Qemu-devel] Is cache=writeback safe yet?

[Kevin Wolf]

Am 20.02.2012 16:29, schrieb Peter Maydell:
> On 20 February 2012 15:20, Kevin Wolf <kwolf@redhat.com> wrote:
>> Am 20.02.2012 15:18, schrieb Virtbie:
>>> Qemu manpage still says cache=writeback isn't safe, but I seem to find
>>> some discordant information reading around.
>>>
>>> Is the WCE+volatile flag exposed to the guest already? I am logging into
>>> a 2.6.38 guest and trying to find an indicator of disk cache in
>>> /sys/block/vda/device but I am not able to. There is an obscure very
>>> long "features" bitmask which I don't know what it shows. I think this
>>> mode is not safe until WBC is exposed, AFAIU, right?
>>
>> qemu exposes the WCE flag and if your guest handles it correctly (i.e.
>> it sends flush requests where needed), cache=writeback is safe. This is
>> true for recent Linux versions.
> 
> The thing that confuses me when this subject comes up is that "cache=writeback"
> is a property of the block layer, but the WCE flag is a SCSI parameter,
> right? How does this work on non-SCSI disks? Is there something that
> eg hw/sd.c should be doing to tell the block layer "writeback cache is
> safe/unsafe" ?

IDE and virtio-blk have some kind of WCE bit as well.

If your hardware doesn't have it, I think you need to check whether your
hardware never has any write cache or if it always has one (if it
sometimes has one but doesn't expose the information, it's already the
hardware that is broken).

You should probably just fail device creation with an inappropriate
cache option.

Kevin

Re: [Qemu-devel] Is cache=writeback safe yet?

[Peter Maydell]

On 20 February 2012 15:56, Kevin Wolf <kwolf@redhat.com> wrote:
> Am 20.02.2012 16:29, schrieb Peter Maydell:
>> The thing that confuses me when this subject comes up is that "cache=writeback"
>> is a property of the block layer, but the WCE flag is a SCSI parameter,
>> right? How does this work on non-SCSI disks? Is there something that
>> eg hw/sd.c should be doing to tell the block layer "writeback cache is
>> safe/unsafe" ?
>
> IDE and virtio-blk have some kind of WCE bit as well.
>
> If your hardware doesn't have it, I think you need to check whether your
> hardware never has any write cache or if it always has one (if it
> sometimes has one but doesn't expose the information, it's already the
> hardware that is broken).

The nature of the SD card protocol is that you feed a 512 byte block
to the thing and when it's written it's written. (The best you can do
is that some cards support feeding one block to the card while it's
still digesting the previous block; we don't emulate this in QEMU
though.)

> You should probably just fail device creation with an inappropriate
> cache option.

The trouble with that idea is that it's slower, so you want to give the
user the option of saying "go fast and I accept data loss if my host
kernel crashes"...

Also it seems to me that it would be a cleaner API for the sd/ide/scsi
layers to tell the block layer what their capabilities are and have the
block layer fail the device creation if that doesn't match with the
user's requests.

-- PMM

Re: [Qemu-devel] Is cache=writeback safe yet?

[Paolo Bonzini]

On 02/20/2012 05:08 PM, Peter Maydell wrote:
> On 20 February 2012 15:56, Kevin Wolf <kwolf@redhat.com> wrote:
>> Am 20.02.2012 16:29, schrieb Peter Maydell:
>>> The thing that confuses me when this subject comes up is that "cache=writeback"
>>> is a property of the block layer, but the WCE flag is a SCSI parameter,
>>> right? How does this work on non-SCSI disks? Is there something that
>>> eg hw/sd.c should be doing to tell the block layer "writeback cache is
>>> safe/unsafe" ?
>>
>> IDE and virtio-blk have some kind of WCE bit as well.
>>
>> If your hardware doesn't have it, I think you need to check whether your
>> hardware never has any write cache or if it always has one (if it
>> sometimes has one but doesn't expose the information, it's already the
>> hardware that is broken).
> 
> The nature of the SD card protocol is that you feed a 512 byte block
> to the thing and when it's written it's written. (The best you can do
> is that some cards support feeding one block to the card while it's
> still digesting the previous block; we don't emulate this in QEMU
> though.)

You need to send a bdrv_flush after each write then.

>> You should probably just fail device creation with an inappropriate
>> cache option.
> 
> The trouble with that idea is that it's slower, so you want to give the
> user the option of saying "go fast and I accept data loss if my host
> kernel crashes"...

That's cache=unsafe.

> Also it seems to me that it would be a cleaner API for the sd/ide/scsi
> layers to tell the block layer what their capabilities are and have the
> block layer fail the device creation if that doesn't match with the
> user's requests.

I think it's not necessary.  Just fix SD to always be safe, and the user
will still have the option to make it fast with cache=unsafe.  Unless
they manually set cache=none users won't notice the difference anyway,
because the default is cache=writethrough and it already does a flush
after each write.

Paolo

Re: [Qemu-devel] Is cache=writeback safe yet?

[Peter Maydell]

On 20 February 2012 17:03, Paolo Bonzini <pbonzini@redhat.com> wrote:
> the default is cache=writethrough and it already does a flush
> after each write.

This runs so slowly that SD is basically unusable for anything beyond
trivial stuff.

-- PMM

Re: [Qemu-devel] Is cache=writeback safe yet?

[Virtbie]

On 02/20/12 16:43, Virtbie wrote:
>
> Great explanation Anthony,
>
> may I still ask:
>
> 1)
> Is WCE + volatile flag exposed to the guest, by all three virtual 
> devices:
> - virtio
> - scsi
> - ide
> ?
> (if not, I still don't understand how this works)
>
>
> 2) Is there a minimum guest kernel and a minimum viostor Windows 
> driver version, to see such WCE+volatile flag in a virtio disk, so 
> that the guest OS can actually see the cache?
>
>
> Because I can't seem to find such flag for a virtio disk on a linux 
> 2.6.38 guest and that seems serious to me.
> Is it visible somewhere in /sys hierarchy? Or is 2.6.38 too old?
>

Is it this commit?

commit f1b0ef062602713c2c7cfa12362d5d90ed01c5f6
Author: Christoph Hellwig <hch@lst.de>
Date:   Thu Sep 17 19:57:42 2009 +0200

     virtio_blk: add support for cache flush

     Recent qemu has added a VIRTIO_BLK_F_FLUSH flag to advertise that the
     virtual disk has a volatile write cache that needs to be flushed.  
In case
     we see this feature implement tell the Linux block layer about the fact
     and use the new VIRTIO_BLK_T_FLUSH to flush the cache when 
required.  This
     allows for an correct and simple implementation of write barriers.

     Signed-off-by: Christoph Hellwig <hch@lst.de>
     Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>



That was introduced in 2.6.32 .
So can I suppose that a guest running linux 2.6.32+ detects the write 
cache in a virtio disk and actually flushes? (suppose a wise filesystem)


The flag seems to go into :

     /* Feature bits */
     #define VIRTIO_BLK_F_BARRIER    0       /* Does host support 
barriers? */
     #define VIRTIO_BLK_F_SIZE_MAX   1       /* Indicates maximum 
segment size */
     #define VIRTIO_BLK_F_SEG_MAX    2       /* Indicates maximum # of 
segments */
     #define VIRTIO_BLK_F_GEOMETRY   4       /* Legacy geometry 
available  */
     #define VIRTIO_BLK_F_RO         5       /* Disk is read-only */
     #define VIRTIO_BLK_F_BLK_SIZE   6       /* Block size of disk is 
available*/
     #define VIRTIO_BLK_F_SCSI       7       /* Supports scsi command 
passthru */
     #define VIRTIO_BLK_F_FLUSH      9       /* Cache flush command 
support */
     #define VIRTIO_BLK_F_TOPOLOGY   10      /* Topology information is 
available */

     #define VIRTIO_BLK_ID_BYTES     20      /* ID string length */

     ........

     static unsigned int features[] = {
             VIRTIO_BLK_F_SEG_MAX, VIRTIO_BLK_F_SIZE_MAX, 
VIRTIO_BLK_F_GEOMETRY,
             VIRTIO_BLK_F_RO, VIRTIO_BLK_F_BLK_SIZE, VIRTIO_BLK_F_SCSI,
             VIRTIO_BLK_F_FLUSH, VIRTIO_BLK_F_TOPOLOGY
     };

so the presence of volatile cache is probably exposed by:

     $ cat /sys/block/vda/device/features
     0010101101100000000000000000100000000000000000000000000000000000

I don't understand if I should look at the 10th bit (9 in 0-based as per 
#define's ) which would be 0 == "no cache" in this case, or at the 7th 
bit (by the position in features[] array), which would be 1 == "has 
cache" in this case .


Thank you
Vb.