[Qemu-devel] Should 'qemu-img commit' sparsify the backing file?

[Qemu-devel] Should 'qemu-img commit' sparsify the backing file?

[Richard W.M. Jones]

This doesn't work (I tested both qemu 2.3 and qemu from git).  Should it?

(1) Create a non-sparse guest image:

$ virt-builder fedora-22
$ mv fedora-22.img fedora-22.img.sparse
$ cp --sparse=never fedora-22.img.sparse fedora-22.img
$ du -sh fedora-22.img
6.1G fedora-22.img

(2) Add a snapshot on top:

$ qemu-img create -f qcow2 -o compat=1.1 -b fedora-22.img overlay.qcow2
$ du -sh fedora-22.img overlay.qcow2 
6.1G fedora-22.img
196K overlay.qcow2

(3) Sparsify the overlay:

$ virt-sparsify --in-place overlay.qcow2 
$ du -sh fedora-22.img overlay.qcow2 
6.1G fedora-22.img
3.2M overlay.qcow2

(4) Commit to the backing file:

$ qemu-img commit overlay.qcow2 
Image committed.
$ du -sh fedora-22.img overlay.qcow2 
6.1G fedora-22.img                         <------
260K overlay.qcow2

Notice that the backing file (fedora-22.img) doesn't get any smaller.

I'm expecting the backing file to shrink to around 800 MB, which is
does if you run virt-sparsify directly on the backing file.

For background, `virt-sparsify --in-place' mounts each filesystem
(using a small Linux/qemu appliance), and runs fstrim on each one.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Re: [Qemu-devel] Should 'qemu-img commit' sparsify the backing file?

[Fam Zheng]

On Wed, 11/04 16:37, Richard W.M. Jones wrote:
> 
> This doesn't work (I tested both qemu 2.3 and qemu from git).  Should it?
> 
> (1) Create a non-sparse guest image:
> 
> $ virt-builder fedora-22
> $ mv fedora-22.img fedora-22.img.sparse
> $ cp --sparse=never fedora-22.img.sparse fedora-22.img
> $ du -sh fedora-22.img
> 6.1G fedora-22.img
> 
> (2) Add a snapshot on top:
> 
> $ qemu-img create -f qcow2 -o compat=1.1 -b fedora-22.img overlay.qcow2
> $ du -sh fedora-22.img overlay.qcow2 
> 6.1G fedora-22.img
> 196K overlay.qcow2
> 
> (3) Sparsify the overlay:
> 
> $ virt-sparsify --in-place overlay.qcow2 
> $ du -sh fedora-22.img overlay.qcow2 
> 6.1G fedora-22.img
> 3.2M overlay.qcow2
> 
> (4) Commit to the backing file:
> 
> $ qemu-img commit overlay.qcow2 
> Image committed.
> $ du -sh fedora-22.img overlay.qcow2 
> 6.1G fedora-22.img                         <------
> 260K overlay.qcow2
> 
> Notice that the backing file (fedora-22.img) doesn't get any smaller.
> 
> I'm expecting the backing file to shrink to around 800 MB, which is
> does if you run virt-sparsify directly on the backing file.
> 

I don't think this the purpose of "qemu-img commit". Committing "new" data in
overlay.qcow2 has little to do with discarding backing image's fragments where
there are only explict zeroes. However you can try

    qemu-img convert fedora-22.img fedora-22.img.1

as the last step, which I think will yield a sparse image.

I don't think we have an in-place sparsifier now.

Fam

Re: [Qemu-devel] Should 'qemu-img commit' sparsify the backing file?

[Richard W.M. Jones]

On Thu, Nov 05, 2015 at 03:14:49PM +0800, Fam Zheng wrote:
> I don't think this the purpose of "qemu-img commit". Committing
> "new" data in overlay.qcow2 has little to do with discarding backing
> image's fragments where there are only explict zeroes.

In that case, why didn't this step sparsify the backing file?

> > (3) Sparsify the overlay:
> > 
> > $ virt-sparsify --in-place overlay.qcow2 
> > $ du -sh fedora-22.img overlay.qcow2 
> > 6.1G fedora-22.img
> > 3.2M overlay.qcow2
>     qemu-img convert fedora-22.img fedora-22.img.1
> 
> I don't think we have an in-place sparsifier now.

virt-sparsify ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Re: [Qemu-devel] Should 'qemu-img commit' sparsify the backing file?

[Fam Zheng]

On Thu, 11/05 08:56, Richard W.M. Jones wrote:
> On Thu, Nov 05, 2015 at 03:14:49PM +0800, Fam Zheng wrote:
> > I don't think this the purpose of "qemu-img commit". Committing
> > "new" data in overlay.qcow2 has little to do with discarding backing
> > image's fragments where there are only explict zeroes.
> 
> In that case, why didn't this step sparsify the backing file?

Commit should be (by default, at least) as fast as possible, so committing
overlay.qcow2 shouldn't involve scanning the whole backing file.

But I think we can implement a dedicate "sparsify" command to do that, though.

> 
> > > (3) Sparsify the overlay:
> > > 
> > > $ virt-sparsify --in-place overlay.qcow2 
> > > $ du -sh fedora-22.img overlay.qcow2 
> > > 6.1G fedora-22.img
> > > 3.2M overlay.qcow2
> >     qemu-img convert fedora-22.img fedora-22.img.1
> > 
> > I don't think we have an in-place sparsifier now.
> 
> virt-sparsify ...
> 

Does virt-sparsify work if there is no file system present (e.g. the disk is
fully filled with zeroes) on the device?

Fam

Re: [Qemu-devel] Should 'qemu-img commit' sparsify the backing file?

[Richard W.M. Jones]

On Thu, Nov 05, 2015 at 05:46:01PM +0800, Fam Zheng wrote:
> Does virt-sparsify work if there is no file system present (e.g. the disk is
> fully filled with zeroes) on the device?

It doesn't do anything bad, but it doesn't currently do that.  You
could do that using:

  guestfish <<EOF
    add zeroes.img discard:enable format:raw
    run
    blkdiscard /dev/sda
  EOF

To add this to virt-sparsify, we'd need a different primitive command,
one which only trims blocks that contain zero (blkdiscard trims
everything), which we don't have right now.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

Re: [Qemu-devel] Should 'qemu-img commit' sparsify the backing file?

[Kevin Wolf]

Am 05.11.2015 um 10:46 hat Fam Zheng geschrieben:
> On Thu, 11/05 08:56, Richard W.M. Jones wrote:
> > On Thu, Nov 05, 2015 at 03:14:49PM +0800, Fam Zheng wrote:
> > > I don't think this the purpose of "qemu-img commit". Committing
> > > "new" data in overlay.qcow2 has little to do with discarding backing
> > > image's fragments where there are only explict zeroes.
> > 
> > In that case, why didn't this step sparsify the backing file?
> 
> Commit should be (by default, at least) as fast as possible, so committing
> overlay.qcow2 shouldn't involve scanning the whole backing file.

But if we do need to commit some data to the backing file, we should try
to be efficient (i.e. use zero clusters in the backing file). I think
your mirror fixes achieve this.

This should actually be enough for virt-sparsify because IIUC it fills
the overlay with zero clusters in the right places before committing.

Kevin