'make check-functional' uses lots of disk space in build tree

'make check-functional' uses lots of disk space in build tree

[Peter Maydell]

I notice that with the recent migration to check-functional we
seem to be using a lot of disk space in the build tree: for
one of my build trees
 du -h build/arm-clang/tests/functional/
returns a total of 4.5GB used, for instance, most of which seems
to be guest binary files.

Shouldn't something be cleaning these up after a test run?

thanks
-- PMM

Re: 'make check-functional' uses lots of disk space in build tree

[Daniel P. Berrangé]

On Fri, Oct 25, 2024 at 02:50:50PM +0100, Peter Maydell wrote:
> I notice that with the recent migration to check-functional we
> seem to be using a lot of disk space in the build tree: for
> one of my build trees
>  du -h build/arm-clang/tests/functional/
> returns a total of 4.5GB used, for instance, most of which seems
> to be guest binary files.
> 
> Shouldn't something be cleaning these up after a test run?

The big problem seems to be tests/functional/test_aarch64_sbsaref.py
which is creating a pair of 256 MB firmware files on every test
and not deleting them.

The tests/functional/test_arm_raspi2.py is also uncompressing some
archives and not deleting them.

Perhaps we need to define a standard "scratch dir" in the framework
that files can be put into, and then have the framework force delete
them at the end.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: 'make check-functional' uses lots of disk space in build tree

[Philippe Mathieu-Daudé]

On 25/10/24 10:57, Daniel P. Berrangé wrote:
> On Fri, Oct 25, 2024 at 02:50:50PM +0100, Peter Maydell wrote:
>> I notice that with the recent migration to check-functional we
>> seem to be using a lot of disk space in the build tree: for
>> one of my build trees
>>   du -h build/arm-clang/tests/functional/
>> returns a total of 4.5GB used, for instance, most of which seems
>> to be guest binary files.
>>
>> Shouldn't something be cleaning these up after a test run?
> 
> The big problem seems to be tests/functional/test_aarch64_sbsaref.py
> which is creating a pair of 256 MB firmware files on every test
> and not deleting them.
> 
> The tests/functional/test_arm_raspi2.py is also uncompressing some
> archives and not deleting them.

Some files could be added back to the cache (with the proper hash)
so we don't need to re-extract or re-create them everytime.

> Perhaps we need to define a standard "scratch dir" in the framework
> that files can be put into, and then have the framework force delete
> them at the end.
> 
> With regards,
> Daniel

Re: 'make check-functional' uses lots of disk space in build tree

[Daniel P. Berrangé]

On Fri, Oct 25, 2024 at 12:29:13PM -0300, Philippe Mathieu-Daudé wrote:
> On 25/10/24 10:57, Daniel P. Berrangé wrote:
> > On Fri, Oct 25, 2024 at 02:50:50PM +0100, Peter Maydell wrote:
> > > I notice that with the recent migration to check-functional we
> > > seem to be using a lot of disk space in the build tree: for
> > > one of my build trees
> > >   du -h build/arm-clang/tests/functional/
> > > returns a total of 4.5GB used, for instance, most of which seems
> > > to be guest binary files.
> > > 
> > > Shouldn't something be cleaning these up after a test run?
> > 
> > The big problem seems to be tests/functional/test_aarch64_sbsaref.py
> > which is creating a pair of 256 MB firmware files on every test
> > and not deleting them.
> > 
> > The tests/functional/test_arm_raspi2.py is also uncompressing some
> > archives and not deleting them.
> 
> Some files could be added back to the cache (with the proper hash)
> so we don't need to re-extract or re-create them everytime.

Unless we've seen a significant time penalty from the extraction,
I think we shouldn't complicate our cache handling. Simplicity
is of great value when it comes to debugging problems with tests.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|