[PATCH v2 0/2] Clean up the functional download cache after some months

[PATCH v2 0/2] Clean up the functional download cache after some months

[Thomas Huth]

The download cache of the functional tests is currently only growing.
But sometimes tests get removed or changed to use different assets,
thus we should clean up the stale old assets after a while when they
are not in use anymore.

This little patch series now introduces time stamps for the assets in
the cache (first patch) that we update during each test run, so we can
check these time stamps regularly and evict stale assets after half
of a year not being used anymore (that should be enough time to still
keep the assets around for a while in case you want to diagnose a
problem that is still some months old).

v2:
- Add some wordings to the 1st patch why we cannot use mtime/atime here
- Use functions from Path object instead from "os" (thanks to Daniel!)

Thomas Huth (2):
  tests/functional: Set current time stamp of assets when using them
  tests: Evict stale files in the functional download cache after a
    while

 MAINTAINERS                         |  1 +
 scripts/clean_functional_cache.py   | 45 +++++++++++++++++++++++++++++
 tests/Makefile.include              |  1 +
 tests/functional/qemu_test/asset.py | 15 ++++++++++
 4 files changed, 62 insertions(+)
 create mode 100755 scripts/clean_functional_cache.py

-- 
2.51.0

[PATCH v2 1/2] tests/functional: Set current time stamp of assets when using them

[Thomas Huth]

From: Thomas Huth <thuth@redhat.com>

We are going to remove obsolete assets from the cache, so keep
the time stamps of the assets that we use up-to-date to have a way
to detect stale assets later.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/functional/qemu_test/asset.py | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py
index f666125bfaf..d5c4ad04bb5 100644
--- a/tests/functional/qemu_test/asset.py
+++ b/tests/functional/qemu_test/asset.py
@@ -10,6 +10,7 @@
 import os
 import stat
 import sys
+import time
 import unittest
 import urllib.request
 from time import sleep
@@ -113,6 +114,18 @@ def _wait_for_other_download(self, tmp_cache_file):
         self.log.debug("Time out while waiting for %s!", tmp_cache_file)
         raise
 
+    def _save_time_stamp(self):
+        '''
+        Update the time stamp of the asset in the cache. Unfortunately, we
+        cannot use the modification or access time of the asset file itself,
+        since e.g. the functional jobs in the gitlab CI reload the files
+        from the gitlab cache and thus always have recent file time stamps,
+        so we have to save our asset time stamp to a separate file instead.
+        '''
+        with open(self.cache_file.with_suffix(".stamp"), 'w',
+                  encoding='utf-8') as fh:
+            fh.write(f"{int(time.time())}")
+
     def fetch(self):
         if not self.cache_dir.exists():
             self.cache_dir.mkdir(parents=True, exist_ok=True)
@@ -120,6 +133,7 @@ def fetch(self):
         if self.valid():
             self.log.debug("Using cached asset %s for %s",
                            self.cache_file, self.url)
+            self._save_time_stamp()
             return str(self.cache_file)
 
         if not self.fetchable():
@@ -208,6 +222,7 @@ def fetch(self):
             tmp_cache_file.unlink()
             raise AssetError(self, "Hash does not match %s" % self.hash)
         tmp_cache_file.replace(self.cache_file)
+        self._save_time_stamp()
         # Remove write perms to stop tests accidentally modifying them
         os.chmod(self.cache_file, stat.S_IRUSR | stat.S_IRGRP)
 
-- 
2.51.0

[PATCH v2 2/2] tests: Evict stale files in the functional download cache after a while

[Thomas Huth]

From: Thomas Huth <thuth@redhat.com>

The download cache of the functional tests is currently only growing.
But sometimes tests get removed or changed to use different assets,
thus we should clean up the stale old assets after a while when they
are not in use anymore. So add a script that looks at the time stamps
of the assets and removes them if they haven't been touched for more
than half of a year. Since there might also be some assets around that
have been added to the cache before we added the time stamp files,
assume a default time stamp that is close to the creation date of this
patch, so that we don't delete these files too early (so we still have
all assets around in case we have to bisect an issue in the recent past
of QEMU).

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 MAINTAINERS                       |  1 +
 scripts/clean_functional_cache.py | 45 +++++++++++++++++++++++++++++++
 tests/Makefile.include            |  1 +
 3 files changed, 47 insertions(+)
 create mode 100755 scripts/clean_functional_cache.py

diff --git a/MAINTAINERS b/MAINTAINERS
index 84cfd85e1fa..4c468d45337 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4398,6 +4398,7 @@ M: Thomas Huth <thuth@redhat.com>
 R: Philippe Mathieu-Daudé <philmd@linaro.org>
 R: Daniel P. Berrange <berrange@redhat.com>
 F: docs/devel/testing/functional.rst
+F: scripts/clean_functional_cache.py
 F: tests/functional/qemu_test/
 
 Windows Hosted Continuous Integration
diff --git a/scripts/clean_functional_cache.py b/scripts/clean_functional_cache.py
new file mode 100755
index 00000000000..c3370ffbb87
--- /dev/null
+++ b/scripts/clean_functional_cache.py
@@ -0,0 +1,45 @@
+#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+"""Delete stale assets from the download cache of the functional tests"""
+
+import os
+import stat
+import sys
+import time
+from pathlib import Path
+
+
+cache_dir_env = os.getenv('QEMU_TEST_CACHE_DIR')
+if cache_dir_env:
+    cache_dir = Path(cache_dir_env, "download")
+else:
+    cache_dir = Path(Path("~").expanduser(), ".cache", "qemu", "download")
+
+if not cache_dir.exists():
+    print(f"Cache dir {cache_dir} does not exist!", file=sys.stderr)
+    sys.exit(1)
+
+os.chdir(cache_dir)
+
+for file in cache_dir.iterdir():
+    # Only consider the files that use a sha256 as filename:
+    if len(file.name) != 64:
+        continue
+
+    try:
+        timestamp = int(file.with_suffix(".stamp").read_text())
+    except FileNotFoundError:
+        # Assume it's an old file that was already in the cache before we
+        # added the code for evicting stale assets. Use the release date
+        # of QEMU v10.1 as a default timestamp.
+        timestamp = time.mktime((2025, 8, 26, 0, 0, 0, 0, 0, 0))
+
+    age = time.time() - timestamp
+
+    # Delete files older than half of a year (183 days * 24h * 60m * 60s)
+    if age > 15811200:
+        print(f"Removing {cache_dir}/{file.name}.")
+        file.chmod(stat.S_IWRITE)
+        file.unlink()
diff --git a/tests/Makefile.include b/tests/Makefile.include
index e47ef4d45c9..d4dfbf3716d 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -111,6 +111,7 @@ $(FUNCTIONAL_TARGETS): check-venv
 .PHONY: check-functional
 check-functional: check-venv
 	@$(NINJA) precache-functional
+	@$(PYTHON) $(SRC_PATH)/scripts/clean_functional_cache.py
 	@QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func check-func-quick
 
 .PHONY: check-func check-func-quick
-- 
2.51.0

Re: [PATCH v2 1/2] tests/functional: Set current time stamp of assets when using them

[Daniel P. Berrangé]

On Mon, Oct 13, 2025 at 02:17:18PM +0200, Thomas Huth wrote:
> From: Thomas Huth <thuth@redhat.com>
> 
> We are going to remove obsolete assets from the cache, so keep
> the time stamps of the assets that we use up-to-date to have a way
> to detect stale assets later.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  tests/functional/qemu_test/asset.py | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py
> index f666125bfaf..d5c4ad04bb5 100644
> --- a/tests/functional/qemu_test/asset.py
> +++ b/tests/functional/qemu_test/asset.py
> @@ -10,6 +10,7 @@
>  import os
>  import stat
>  import sys
> +import time
>  import unittest
>  import urllib.request
>  from time import sleep
> @@ -113,6 +114,18 @@ def _wait_for_other_download(self, tmp_cache_file):
>          self.log.debug("Time out while waiting for %s!", tmp_cache_file)
>          raise
>  
> +    def _save_time_stamp(self):
> +        '''
> +        Update the time stamp of the asset in the cache. Unfortunately, we
> +        cannot use the modification or access time of the asset file itself,
> +        since e.g. the functional jobs in the gitlab CI reload the files
> +        from the gitlab cache and thus always have recent file time stamps,
> +        so we have to save our asset time stamp to a separate file instead.
> +        '''
> +        with open(self.cache_file.with_suffix(".stamp"), 'w',
> +                  encoding='utf-8') as fh:
> +            fh.write(f"{int(time.time())}")

Realized we can simplify that too

   self.cache_file.with_suffix(".stamp").write_text(f"{int(time.time())}")

The 'encoding' arg is redundant because we're only writing an integer
time value which is identical in every encoding.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 2/2] tests: Evict stale files in the functional download cache after a while

[Daniel P. Berrangé]

On Mon, Oct 13, 2025 at 02:17:19PM +0200, Thomas Huth wrote:
> From: Thomas Huth <thuth@redhat.com>
> 
> The download cache of the functional tests is currently only growing.
> But sometimes tests get removed or changed to use different assets,
> thus we should clean up the stale old assets after a while when they
> are not in use anymore. So add a script that looks at the time stamps
> of the assets and removes them if they haven't been touched for more
> than half of a year. Since there might also be some assets around that
> have been added to the cache before we added the time stamp files,
> assume a default time stamp that is close to the creation date of this
> patch, so that we don't delete these files too early (so we still have
> all assets around in case we have to bisect an issue in the recent past
> of QEMU).
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  MAINTAINERS                       |  1 +
>  scripts/clean_functional_cache.py | 45 +++++++++++++++++++++++++++++++
>  tests/Makefile.include            |  1 +
>  3 files changed, 47 insertions(+)
>  create mode 100755 scripts/clean_functional_cache.py

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|