[Qemu-devel] qemu-img create: set "nocow" flag to solve performance issue on btrfs

[Qemu-devel] qemu-img create: set "nocow" flag to solve performance issue on btrfs

[Chunyan Liu]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

Hi, List,

Btrfs has terrible performance when hosting VM images, even more when the
guest in those VM are also using btrfs as file system.
One way to mitigate this bad performance would be to turn off COW
attributes on VM files (since having copy on write for this kind of data is
not useful). We could improve qemu-img to ensure they flag newly created
images as "nocow". For those who want to use Copy-on-write (for
snapshotting, to share snapshots across VM, etc..) could be able to change
this behaviour by 'chattr', either globally or per VM.

I wonder could we add a patch to improve qemu-img create, to set 'nocow'
flag by default on newly created images?

Thanks,
Chunyan

[-- Attachment #2: Type: text/html, Size: 785 bytes --]

Re: [Qemu-devel] qemu-img create: set "nocow" flag to solve performance issue on btrfs

[Stefan Hajnoczi]

On Wed, Sep 25, 2013 at 02:38:36PM +0800, Chunyan Liu wrote:
> Btrfs has terrible performance when hosting VM images, even more when the
> guest in those VM are also using btrfs as file system.
> One way to mitigate this bad performance would be to turn off COW
> attributes on VM files (since having copy on write for this kind of data is
> not useful). We could improve qemu-img to ensure they flag newly created
> images as "nocow". For those who want to use Copy-on-write (for
> snapshotting, to share snapshots across VM, etc..) could be able to change
> this behaviour by 'chattr', either globally or per VM.

The full implications of the NOCOW attribute aren't clear to me.  Does
it really mean the file cannot be snapshotted?  Or is it purely a data
integrity issue where overwriting data in-place puts that data at risk
in case of hardware/power failure?

> I wonder could we add a patch to improve qemu-img create, to set 'nocow'
> flag by default on newly created images?

I think that would be fine.  It's a ioctl(FS_IOC_SETFLAGS, FS_NOCOW_FL)
call so not even too btrfs-specific.

Stefan

Re: [Qemu-devel] qemu-img create: set "nocow" flag to solve performance issue on btrfs

[Paolo Bonzini]

Il 26/09/2013 09:58, Stefan Hajnoczi ha scritto:
> On Wed, Sep 25, 2013 at 02:38:36PM +0800, Chunyan Liu wrote:
>> Btrfs has terrible performance when hosting VM images, even more when the
>> guest in those VM are also using btrfs as file system.
>> One way to mitigate this bad performance would be to turn off COW
>> attributes on VM files (since having copy on write for this kind of data is
>> not useful). We could improve qemu-img to ensure they flag newly created
>> images as "nocow". For those who want to use Copy-on-write (for
>> snapshotting, to share snapshots across VM, etc..) could be able to change
>> this behaviour by 'chattr', either globally or per VM.
> 
> The full implications of the NOCOW attribute aren't clear to me.  Does
> it really mean the file cannot be snapshotted?  Or is it purely a data
> integrity issue where overwriting data in-place puts that data at risk
> in case of hardware/power failure?
> 
>> I wonder could we add a patch to improve qemu-img create, to set 'nocow'
>> flag by default on newly created images?
> 
> I think that would be fine.  It's a ioctl(FS_IOC_SETFLAGS, FS_NOCOW_FL)
> call so not even too btrfs-specific.

I'm not sure...  I have some questions:

1) Does btrfs cow mean that one could run with cache=unsafe, for
example?  If we create the image with nocow, this would not be true.

2) Does ZFS have the same problem?  In other words, could this just be
considered a btrfs bug?

Paolo

Re: [Qemu-devel] qemu-img create: set "nocow" flag to solve performance issue on btrfs

[Chunyan Liu]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

2013/9/26 Stefan Hajnoczi <stefanha@gmail.com>

> On Wed, Sep 25, 2013 at 02:38:36PM +0800, Chunyan Liu wrote:
> > Btrfs has terrible performance when hosting VM images, even more when the
> > guest in those VM are also using btrfs as file system.
> > One way to mitigate this bad performance would be to turn off COW
> > attributes on VM files (since having copy on write for this kind of data
> is
> > not useful). We could improve qemu-img to ensure they flag newly created
> > images as "nocow". For those who want to use Copy-on-write (for
> > snapshotting, to share snapshots across VM, etc..) could be able to
> change
> > this behaviour by 'chattr', either globally or per VM.
>
> The full implications of the NOCOW attribute aren't clear to me.  Does
> it really mean the file cannot be snapshotted?


Yes, I think so. The benefits brought by COW: data integrity and convenient
snapshot, would be disappears.

Or is it purely a data
> integrity issue where overwriting data in-place puts that data at risk
> in case of hardware/power failure?


> > I wonder could we add a patch to improve qemu-img create, to set 'nocow'
> > flag by default on newly created images?
>
> I think that would be fine.  It's a ioctl(FS_IOC_SETFLAGS, FS_NOCOW_FL)
> call so not even too btrfs-specific.
>
> OK. I'll prepare the patch. Thanks.

Regards,
Chunyan


> Stefan
>
>

[-- Attachment #2: Type: text/html, Size: 2446 bytes --]

Re: [Qemu-devel] qemu-img create: set "nocow" flag to solve performance issue on btrfs

[Chunyan Liu]

[-- Attachment #1: Type: text/plain, Size: 2249 bytes --]

2013/9/26 Paolo Bonzini <pbonzini@redhat.com>

> Il 26/09/2013 09:58, Stefan Hajnoczi ha scritto:
> > On Wed, Sep 25, 2013 at 02:38:36PM +0800, Chunyan Liu wrote:
> >> Btrfs has terrible performance when hosting VM images, even more when
> the
> >> guest in those VM are also using btrfs as file system.
> >> One way to mitigate this bad performance would be to turn off COW
> >> attributes on VM files (since having copy on write for this kind of
> data is
> >> not useful). We could improve qemu-img to ensure they flag newly created
> >> images as "nocow". For those who want to use Copy-on-write (for
> >> snapshotting, to share snapshots across VM, etc..) could be able to
> change
> >> this behaviour by 'chattr', either globally or per VM.
> >
> > The full implications of the NOCOW attribute aren't clear to me.  Does
> > it really mean the file cannot be snapshotted?  Or is it purely a data
> > integrity issue where overwriting data in-place puts that data at risk
> > in case of hardware/power failure?
> >
> >> I wonder could we add a patch to improve qemu-img create, to set 'nocow'
> >> flag by default on newly created images?
> >
> > I think that would be fine.  It's a ioctl(FS_IOC_SETFLAGS, FS_NOCOW_FL)
> > call so not even too btrfs-specific.
>
> I'm not sure...  I have some questions:
>
> 1) Does btrfs cow mean that one could run with cache=unsafe, for
> example?  If we create the image with nocow, this would not be true.
>
> I don't know if I understand correctly. I think you mentioned cache=unsafe
here, due to the snapshot function? cache=unsafe could enhance snapshot
performance. But btrfs snapshot (btrfs subvolume snapshot xx xx) and qemu
snapshot function are two different levels. With cow attribute, btrfs
snapshot could be achieved very easily. With nocow attribute, the btrfs
snapshot function should be not working on the file.


> 2) Does ZFS have the same problem?  In other words, could this just be
> considered a btrfs bug?
>
> I think the performance issue is due to the COW ifself. With COW, there
are more read/write IO(s) when first writing a place, so random small write
on a large file would get bad performance. But I don't know how ZFS is
affected. Perhaps it degrades not so much?


> Paolo
>
>

[-- Attachment #2: Type: text/html, Size: 3263 bytes --]

Re: [Qemu-devel] qemu-img create: set "nocow" flag to solve performance issue on btrfs

[Paolo Bonzini]

Il 26/09/2013 12:30, Chunyan Liu ha scritto:
> 
> 
> 
> 2013/9/26 Paolo Bonzini <pbonzini@redhat.com <mailto:pbonzini@redhat.com>>
> 
>     Il 26/09/2013 09:58, Stefan Hajnoczi ha scritto:
>     > On Wed, Sep 25, 2013 at 02:38:36PM +0800, Chunyan Liu wrote:
>     >> Btrfs has terrible performance when hosting VM images, even more
>     when the
>     >> guest in those VM are also using btrfs as file system.
>     >> One way to mitigate this bad performance would be to turn off COW
>     >> attributes on VM files (since having copy on write for this kind
>     of data is
>     >> not useful). We could improve qemu-img to ensure they flag newly
>     created
>     >> images as "nocow". For those who want to use Copy-on-write (for
>     >> snapshotting, to share snapshots across VM, etc..) could be able
>     to change
>     >> this behaviour by 'chattr', either globally or per VM.
>     >
>     > The full implications of the NOCOW attribute aren't clear to me.  Does
>     > it really mean the file cannot be snapshotted?  Or is it purely a data
>     > integrity issue where overwriting data in-place puts that data at risk
>     > in case of hardware/power failure?
>     >
>     >> I wonder could we add a patch to improve qemu-img create, to set
>     'nocow'
>     >> flag by default on newly created images?
>     >
>     > I think that would be fine.  It's a ioctl(FS_IOC_SETFLAGS,
>     FS_NOCOW_FL)
>     > call so not even too btrfs-specific.
> 
>     I'm not sure...  I have some questions:
> 
>     1) Does btrfs cow mean that one could run with cache=unsafe, for
>     example?  If we create the image with nocow, this would not be true.
> 
> I don't know if I understand correctly. I think you mentioned
> cache=unsafe here, due to the snapshot function? cache=unsafe could
> enhance snapshot performance. But btrfs snapshot (btrfs subvolume
> snapshot xx xx) and qemu snapshot function are two different levels.
> With cow attribute, btrfs snapshot could be achieved very easily. With
> nocow attribute, the btrfs snapshot function should be not working on
> the file.

Does COW preserve the order of writes even after a power loss (i.e. you
might lose a write, but then you will always lose all the ones that come
after it)?  If so, you could run QEMU with "cache=unsafe" and have
basically the same data safety guarantees as "cache=writeback" on every
other file system.

Similarly, you could use "cache.no-flush=true,cache.direct=true" instead
of "cache=none".

Paolo

Re: [Qemu-devel] qemu-img create: set "nocow" flag to solve performance issue on btrfs

[Chun Yan Liu]

>>> On 9/27/2013 at 12:56 AM, in message <5244673F.4000203@redhat.com>, Paolo
Bonzini <pbonzini@redhat.com> wrote: 
> Il 26/09/2013 12:30, Chunyan Liu ha scritto: 
> >  
> >  
> >  
> > 2013/9/26 Paolo Bonzini <pbonzini@redhat.com <mailto:pbonzini@redhat.com>> 
> >  
> >     Il 26/09/2013 09:58, Stefan Hajnoczi ha scritto: 
> >     > On Wed, Sep 25, 2013 at 02:38:36PM +0800, Chunyan Liu wrote: 
> >     >> Btrfs has terrible performance when hosting VM images, even more 
> >     when the 
> >     >> guest in those VM are also using btrfs as file system. 
> >     >> One way to mitigate this bad performance would be to turn off COW 
> >     >> attributes on VM files (since having copy on write for this kind 
> >     of data is 
> >     >> not useful). We could improve qemu-img to ensure they flag newly 
> >     created 
> >     >> images as "nocow". For those who want to use Copy-on-write (for 
> >     >> snapshotting, to share snapshots across VM, etc..) could be able 
> >     to change 
> >     >> this behaviour by 'chattr', either globally or per VM. 
> >     > 
> >     > The full implications of the NOCOW attribute aren't clear to me.  Does 
> >     > it really mean the file cannot be snapshotted?  Or is it purely a data 
> >     > integrity issue where overwriting data in-place puts that data at risk 
> >     > in case of hardware/power failure? 
> >     > 
> >     >> I wonder could we add a patch to improve qemu-img create, to set 
> >     'nocow' 
> >     >> flag by default on newly created images? 
> >     > 
> >     > I think that would be fine.  It's a ioctl(FS_IOC_SETFLAGS, 
> >     FS_NOCOW_FL) 
> >     > call so not even too btrfs-specific. 
> >  
> >     I'm not sure...  I have some questions: 
> >  
> >     1) Does btrfs cow mean that one could run with cache=unsafe, for 
> >     example?  If we create the image with nocow, this would not be true. 
> >  
> > I don't know if I understand correctly. I think you mentioned 
> > cache=unsafe here, due to the snapshot function? cache=unsafe could 
> > enhance snapshot performance. But btrfs snapshot (btrfs subvolume 
> > snapshot xx xx) and qemu snapshot function are two different levels. 
> > With cow attribute, btrfs snapshot could be achieved very easily. With 
> > nocow attribute, the btrfs snapshot function should be not working on 
> > the file. 
>  
> Does COW preserve the order of writes even after a power loss (i.e. you 
> might lose a write, but then you will always lose all the ones that come 
> after it)? 

Yes, I think so. COW could make sure at any time its FS tree points to a valid root (has all data in a consistent status), if due to the power loss, write task is not completed, it still points to the older root; if write task is completed, it will points to a new root. No matter in which case, it is a consistent status.

> If so, you could run QEMU with "cache=unsafe" and have
> basically the same data safety guarantees as "cache=writeback" on every 
> other file system. 

"cache=unsafe" means it never calls fsync() ? If so, it seems to be not proper to use this option to disk image of a running VM. In case someone has run the VM for a very long time and made many changes already on the disk image, but all those are in cache (?) Then due to a power off, all these changes are lost. Although the disk image is still in a consistent data status, but the data is maybe too old.

- Chunyan
>  
> Similarly, you could use "cache.no-flush=true,cache.direct=true" instead 
> of "cache=none". 
>  
> Paolo 
>  
>  

Re: [Qemu-devel] qemu-img create: set "nocow" flag to solve performance issue on btrfs

[Paolo Bonzini]

Il 27/09/2013 10:58, Chun Yan Liu ha scritto:
>> If so, you could run QEMU with "cache=unsafe" and have
>> basically the same data safety guarantees as "cache=writeback" on
>> every other file system.
> "cache=unsafe" means it never calls fsync() ?

Yes.  However, metadata writes are still done and ordered correctly with
respect to data writes.

> If so, it seems to be
> not proper to use this option to disk image of a running VM. In case
> someone has run the VM for a very long time and made many changes
> already on the disk image, but all those are in cache (?) Then due to
> a power off, all these changes are lost. Although the disk image is
> still in a consistent data status, but the data is maybe too old.

Perhaps for btrfs-cow a different strategy is better, with fsyncs
coalesced and issued by QEMU every N seconds.

Paolo