[Qemu-devel] vm-tests images disks filling up?

[Qemu-devel] vm-tests images disks filling up?

[Peter Maydell]

I just ran into a build failure using the tests/vm/ BSD build tests,
because the NetBSD build image's disk filled up.

Looking more closely there seemed to be 9 stale build trees in
the VM's /var/tmp/qemu-test.* , which is why the disk was full
(they'd used up about 18GB between them).

The other VMs (freebsd, openbsd) also had the same problem of
/var/tmp gradually filling with stale trees, they just hadn't
quite run out of space yet...

What's the process for managing the disk space on these images?
How are stale or completed build trees deleted ?

thanks
-- PMM

Re: [Qemu-devel] vm-tests images disks filling up?

[Daniel P. Berrangé]

On Fri, Aug 17, 2018 at 11:26:39AM +0100, Peter Maydell wrote:
> I just ran into a build failure using the tests/vm/ BSD build tests,
> because the NetBSD build image's disk filled up.
> 
> Looking more closely there seemed to be 9 stale build trees in
> the VM's /var/tmp/qemu-test.* , which is why the disk was full
> (they'd used up about 18GB between them).
> 
> The other VMs (freebsd, openbsd) also had the same problem of
> /var/tmp gradually filling with stale trees, they just hadn't
> quite run out of space yet...
> 
> What's the process for managing the disk space on these images?
> How are stale or completed build trees deleted ?

I'd prefer to see the test process honouring the build directory instead
of putting stuff in /var/tmp, so that a developer's normal approach to
cleaning up build artifacts works.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] vm-tests images disks filling up?

[Peter Maydell]

On 17 August 2018 at 11:47, Daniel P. Berrangé <berrange@redhat.com> wrote:
> I'd prefer to see the test process honouring the build directory instead
> of putting stuff in /var/tmp, so that a developer's normal approach to
> cleaning up build artifacts works.

That's tricky because the build directory is outside the VM and
the build is done inside it (using a copy of the sources): the
VM doesn't have access to the build directory.

thanks
-- PMM

Re: [Qemu-devel] vm-tests images disks filling up?

[Daniel P. Berrangé]

On Fri, Aug 17, 2018 at 11:47:06AM +0100, Daniel P. Berrangé wrote:
> On Fri, Aug 17, 2018 at 11:26:39AM +0100, Peter Maydell wrote:
> > I just ran into a build failure using the tests/vm/ BSD build tests,
> > because the NetBSD build image's disk filled up.
> > 
> > Looking more closely there seemed to be 9 stale build trees in
> > the VM's /var/tmp/qemu-test.* , which is why the disk was full
> > (they'd used up about 18GB between them).
> > 
> > The other VMs (freebsd, openbsd) also had the same problem of
> > /var/tmp gradually filling with stale trees, they just hadn't
> > quite run out of space yet...
> > 
> > What's the process for managing the disk space on these images?
> > How are stale or completed build trees deleted ?
> 
> I'd prefer to see the test process honouring the build directory instead
> of putting stuff in /var/tmp, so that a developer's normal approach to
> cleaning up build artifacts works.

Oh, wait I'm stupid,  you're talking about the /var/tmp inside the guest
disk images, not the host !

IIUC, the problem is that we create the tests/vm/freebsd.img file
by deep copying the template cached under $HOME and never purge it,
so it gradually fills up.

It would be desirable to guarantee a clean image for every build test
to avoid risk of previous problems affecting subsequent builds.

So how about we create tests/vm/freebsd.img as a qcow2 overlay that
points back to the cached image in $HOME. That makes it cheap to create,
so on every test we can just throw away the previous overlay to get
pristine state.

NB, don't delete the overlay at end of build - only start of the next
build. That gives devs a way to launch the VM to debug a failed build

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] vm-tests images disks filling up?

[Fam Zheng]

On Fri, 08/17 11:26, Peter Maydell wrote:
> I just ran into a build failure using the tests/vm/ BSD build tests,
> because the NetBSD build image's disk filled up.
> 
> Looking more closely there seemed to be 9 stale build trees in
> the VM's /var/tmp/qemu-test.* , which is why the disk was full
> (they'd used up about 18GB between them).
> 
> The other VMs (freebsd, openbsd) also had the same problem of
> /var/tmp gradually filling with stale trees, they just hadn't
> quite run out of space yet...
> 
> What's the process for managing the disk space on these images?
> How are stale or completed build trees deleted ?

Ouch. We used to always use snapshot when running tests until 983c2a777be.

One possibility is to use a fixed temp dir (image locking makes sure no two
tests boot the same image concurrently). I.e. something like:

  $ sed -i -e 's:cd.*mktemp.*:mkdir -p /var/tmp/qemu-vm-test; cd /var/tmp/qemu-vm-test:' tests/vm/*

Fam