* [PATCH 0/2] Clean up the functional download cache after some months
@ 2025-10-10 9:32 Thomas Huth
2025-10-10 9:32 ` [PATCH 1/2] tests/functional: Set current time stamp of assets when using them Thomas Huth
2025-10-10 9:32 ` [PATCH 2/2] tests: Evict stale files in the functional download cache after a while Thomas Huth
0 siblings, 2 replies; 9+ messages in thread
From: Thomas Huth @ 2025-10-10 9:32 UTC (permalink / raw)
To: qemu-devel, Daniel P. Berrangé
Cc: John Snow, Alex Bennée, Philippe Mathieu-Daudé
The download cache of the functional tests is currently only growing.
But sometimes tests get removed or changed to use different assets,
thus we should clean up the stale old assets after a while when they
are not in use anymore.
This little patch series now introduces time stamps for the assets in
the cache (first patch) that we update during each test run, so we can
check these time stamps regularly and evict stale assets after half
of a year not being used anymore (that should be enough time to still
keep the assets around for a while in case you want to diagnose a
problem that is still some months old).
Thomas Huth (2):
tests/functional: Set current time stamp of assets when using them
tests: Evict stale files in the functional download cache after a
while
MAINTAINERS | 1 +
scripts/clean_functional_cache.py | 47 +++++++++++++++++++++++++++++
tests/Makefile.include | 1 +
tests/functional/qemu_test/asset.py | 8 +++++
4 files changed, 57 insertions(+)
create mode 100755 scripts/clean_functional_cache.py
--
2.51.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 1/2] tests/functional: Set current time stamp of assets when using them 2025-10-10 9:32 [PATCH 0/2] Clean up the functional download cache after some months Thomas Huth @ 2025-10-10 9:32 ` Thomas Huth 2025-10-10 9:39 ` Daniel P. Berrangé 2025-10-10 9:32 ` [PATCH 2/2] tests: Evict stale files in the functional download cache after a while Thomas Huth 1 sibling, 1 reply; 9+ messages in thread From: Thomas Huth @ 2025-10-10 9:32 UTC (permalink / raw) To: qemu-devel, Daniel P. Berrangé Cc: John Snow, Alex Bennée, Philippe Mathieu-Daudé From: Thomas Huth <thuth@redhat.com> We are going to remove obsolete assets from the cache, so keep the time stamps of the assets that we use up-to-date to have a way to detect stale assets later. Signed-off-by: Thomas Huth <thuth@redhat.com> --- tests/functional/qemu_test/asset.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py index 2971a989d1e..251953ed99f 100644 --- a/tests/functional/qemu_test/asset.py +++ b/tests/functional/qemu_test/asset.py @@ -10,6 +10,7 @@ import os import stat import sys +import time import unittest import urllib.request from time import sleep @@ -113,6 +114,11 @@ def _wait_for_other_download(self, tmp_cache_file): self.log.debug("Time out while waiting for %s!", tmp_cache_file) raise + def _save_time_stamp(self): + with open(self.cache_file.with_suffix(".stamp"), 'w', + encoding='utf-8') as fh: + fh.write(f"{int(time.time())}") + def fetch(self): if not self.cache_dir.exists(): self.cache_dir.mkdir(parents=True, exist_ok=True) @@ -120,6 +126,7 @@ def fetch(self): if self.valid(): self.log.debug("Using cached asset %s for %s", self.cache_file, self.url) + self._save_time_stamp() return str(self.cache_file) if not self.fetchable(): @@ -208,6 +215,7 @@ def fetch(self): tmp_cache_file.unlink() raise AssetError(self, "Hash does not match %s" % self.hash) tmp_cache_file.replace(self.cache_file) + self._save_time_stamp() # Remove write perms to stop tests accidentally modifying them os.chmod(self.cache_file, stat.S_IRUSR | stat.S_IRGRP) -- 2.51.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] tests/functional: Set current time stamp of assets when using them 2025-10-10 9:32 ` [PATCH 1/2] tests/functional: Set current time stamp of assets when using them Thomas Huth @ 2025-10-10 9:39 ` Daniel P. Berrangé 2025-10-10 9:46 ` Thomas Huth 0 siblings, 1 reply; 9+ messages in thread From: Daniel P. Berrangé @ 2025-10-10 9:39 UTC (permalink / raw) To: Thomas Huth Cc: qemu-devel, John Snow, Alex Bennée, Philippe Mathieu-Daudé On Fri, Oct 10, 2025 at 11:32:42AM +0200, Thomas Huth wrote: > From: Thomas Huth <thuth@redhat.com> > > We are going to remove obsolete assets from the cache, so keep > the time stamps of the assets that we use up-to-date to have a way > to detect stale assets later. > > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > tests/functional/qemu_test/asset.py | 8 ++++++++ > 1 file changed, 8 insertions(+) > > diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py > index 2971a989d1e..251953ed99f 100644 > --- a/tests/functional/qemu_test/asset.py > +++ b/tests/functional/qemu_test/asset.py > @@ -10,6 +10,7 @@ > import os > import stat > import sys > +import time > import unittest > import urllib.request > from time import sleep > @@ -113,6 +114,11 @@ def _wait_for_other_download(self, tmp_cache_file): > self.log.debug("Time out while waiting for %s!", tmp_cache_file) > raise > > + def _save_time_stamp(self): > + with open(self.cache_file.with_suffix(".stamp"), 'w', > + encoding='utf-8') as fh: > + fh.write(f"{int(time.time())}") Rather than creating a parallel timestamp file, it feels like we could just call 'os.utime(self.cache_file)' which will set atime + mtime to the current timestamp, which we can check later with os.stat(). > + > def fetch(self): > if not self.cache_dir.exists(): > self.cache_dir.mkdir(parents=True, exist_ok=True) > @@ -120,6 +126,7 @@ def fetch(self): > if self.valid(): > self.log.debug("Using cached asset %s for %s", > self.cache_file, self.url) > + self._save_time_stamp() > return str(self.cache_file) > > if not self.fetchable(): > @@ -208,6 +215,7 @@ def fetch(self): > tmp_cache_file.unlink() > raise AssetError(self, "Hash does not match %s" % self.hash) > tmp_cache_file.replace(self.cache_file) > + self._save_time_stamp() > # Remove write perms to stop tests accidentally modifying them > os.chmod(self.cache_file, stat.S_IRUSR | stat.S_IRGRP) > > -- > 2.51.0 > With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] tests/functional: Set current time stamp of assets when using them 2025-10-10 9:39 ` Daniel P. Berrangé @ 2025-10-10 9:46 ` Thomas Huth 2025-10-10 9:53 ` Daniel P. Berrangé 0 siblings, 1 reply; 9+ messages in thread From: Thomas Huth @ 2025-10-10 9:46 UTC (permalink / raw) To: Daniel P. Berrangé Cc: qemu-devel, John Snow, Alex Bennée, Philippe Mathieu-Daudé On 10/10/2025 11.39, Daniel P. Berrangé wrote: > On Fri, Oct 10, 2025 at 11:32:42AM +0200, Thomas Huth wrote: >> From: Thomas Huth <thuth@redhat.com> >> >> We are going to remove obsolete assets from the cache, so keep >> the time stamps of the assets that we use up-to-date to have a way >> to detect stale assets later. >> >> Signed-off-by: Thomas Huth <thuth@redhat.com> >> --- >> tests/functional/qemu_test/asset.py | 8 ++++++++ >> 1 file changed, 8 insertions(+) >> >> diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py >> index 2971a989d1e..251953ed99f 100644 >> --- a/tests/functional/qemu_test/asset.py >> +++ b/tests/functional/qemu_test/asset.py >> @@ -10,6 +10,7 @@ >> import os >> import stat >> import sys >> +import time >> import unittest >> import urllib.request >> from time import sleep >> @@ -113,6 +114,11 @@ def _wait_for_other_download(self, tmp_cache_file): >> self.log.debug("Time out while waiting for %s!", tmp_cache_file) >> raise >> >> + def _save_time_stamp(self): >> + with open(self.cache_file.with_suffix(".stamp"), 'w', >> + encoding='utf-8') as fh: >> + fh.write(f"{int(time.time())}") > > Rather than creating a parallel timestamp file, it feels like we could > just call 'os.utime(self.cache_file)' which will set atime + mtime > to the current timestamp, which we can check later with os.stat(). That was my first idea, too (sorry, I should maybe have mentioned it in the patch description), but it does not work: In the gitlab CI runners, the files get re-initialized from the runners cache each time the functional test job is started, so the atime and mtime is always close to the the current date there --> we would never expire files in the gitlab CI that way. Thomas ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] tests/functional: Set current time stamp of assets when using them 2025-10-10 9:46 ` Thomas Huth @ 2025-10-10 9:53 ` Daniel P. Berrangé 0 siblings, 0 replies; 9+ messages in thread From: Daniel P. Berrangé @ 2025-10-10 9:53 UTC (permalink / raw) To: Thomas Huth Cc: qemu-devel, John Snow, Alex Bennée, Philippe Mathieu-Daudé On Fri, Oct 10, 2025 at 11:46:44AM +0200, Thomas Huth wrote: > On 10/10/2025 11.39, Daniel P. Berrangé wrote: > > On Fri, Oct 10, 2025 at 11:32:42AM +0200, Thomas Huth wrote: > > > From: Thomas Huth <thuth@redhat.com> > > > > > > We are going to remove obsolete assets from the cache, so keep > > > the time stamps of the assets that we use up-to-date to have a way > > > to detect stale assets later. > > > > > > Signed-off-by: Thomas Huth <thuth@redhat.com> > > > --- > > > tests/functional/qemu_test/asset.py | 8 ++++++++ > > > 1 file changed, 8 insertions(+) > > > > > > diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py > > > index 2971a989d1e..251953ed99f 100644 > > > --- a/tests/functional/qemu_test/asset.py > > > +++ b/tests/functional/qemu_test/asset.py > > > @@ -10,6 +10,7 @@ > > > import os > > > import stat > > > import sys > > > +import time > > > import unittest > > > import urllib.request > > > from time import sleep > > > @@ -113,6 +114,11 @@ def _wait_for_other_download(self, tmp_cache_file): > > > self.log.debug("Time out while waiting for %s!", tmp_cache_file) > > > raise > > > + def _save_time_stamp(self): > > > + with open(self.cache_file.with_suffix(".stamp"), 'w', > > > + encoding='utf-8') as fh: > > > + fh.write(f"{int(time.time())}") > > > > Rather than creating a parallel timestamp file, it feels like we could > > just call 'os.utime(self.cache_file)' which will set atime + mtime > > to the current timestamp, which we can check later with os.stat(). > > That was my first idea, too (sorry, I should maybe have mentioned it in the > patch description), but it does not work: In the gitlab CI runners, the > files get re-initialized from the runners cache each time the functional > test job is started, so the atime and mtime is always close to the the > current date there --> we would never expire files in the gitlab CI that > way. Oh well, I had assumed gitlab would preserve timestamps when taring/untaring files for the cache :-( With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/2] tests: Evict stale files in the functional download cache after a while 2025-10-10 9:32 [PATCH 0/2] Clean up the functional download cache after some months Thomas Huth 2025-10-10 9:32 ` [PATCH 1/2] tests/functional: Set current time stamp of assets when using them Thomas Huth @ 2025-10-10 9:32 ` Thomas Huth 2025-10-10 9:50 ` Daniel P. Berrangé 1 sibling, 1 reply; 9+ messages in thread From: Thomas Huth @ 2025-10-10 9:32 UTC (permalink / raw) To: qemu-devel, Daniel P. Berrangé Cc: John Snow, Alex Bennée, Philippe Mathieu-Daudé From: Thomas Huth <thuth@redhat.com> The download cache of the functional tests is currently only growing. But sometimes tests get removed or changed to use different assets, thus we should clean up the stale old assets after a while when they are not in use anymore. So add a script that looks at the time stamps of the assets and removes them if they haven't been touched for more than half of a year. Since there might also be some assets around that have been added to the cache before we added the time stamp files, assume a default time stamp that is close to the creation date of this patch, so that we don't delete these files too early. Signed-off-by: Thomas Huth <thuth@redhat.com> --- MAINTAINERS | 1 + scripts/clean_functional_cache.py | 47 +++++++++++++++++++++++++++++++ tests/Makefile.include | 1 + 3 files changed, 49 insertions(+) create mode 100755 scripts/clean_functional_cache.py diff --git a/MAINTAINERS b/MAINTAINERS index 84cfd85e1fa..4c468d45337 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4398,6 +4398,7 @@ M: Thomas Huth <thuth@redhat.com> R: Philippe Mathieu-Daudé <philmd@linaro.org> R: Daniel P. Berrange <berrange@redhat.com> F: docs/devel/testing/functional.rst +F: scripts/clean_functional_cache.py F: tests/functional/qemu_test/ Windows Hosted Continuous Integration diff --git a/scripts/clean_functional_cache.py b/scripts/clean_functional_cache.py new file mode 100755 index 00000000000..e5c4d1acaf3 --- /dev/null +++ b/scripts/clean_functional_cache.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +"""Delete stale assets from the download cache of the functional tests""" + +import os +import stat +import sys +import time +from pathlib import Path + + +cache_dir_env = os.getenv('QEMU_TEST_CACHE_DIR') +if cache_dir_env: + cache_dir = Path(cache_dir_env, "download") +else: + cache_dir = Path(Path("~").expanduser(), ".cache", "qemu", "download") + +if not os.path.exists(cache_dir): + print(f"Cache dir {cache_dir} does not exist!", file=sys.stderr) + sys.exit(1) + +os.chdir(cache_dir) + +for file in os.listdir(cache_dir): + filename = os.fsdecode(file) + # Only consider the files that use a sha256 as filename: + if len(filename) != 64: + continue + + try: + with open(filename + ".stamp", "r", encoding='utf-8') as fh: + timestamp = int(fh.read()) + except FileNotFoundError: + # Assume it's an old file that was already in the cache before we + # added the code for evicting stale assets. Use the release date + # of QEMU v10.1 as a default timestamp. + timestamp = time.mktime((2025, 8, 26, 0, 0, 0, 0, 0, 0)) + + age = time.time() - timestamp + + # Delete files older than half of a year (183 days * 24h * 60m * 60s) + if age > 15811200: + print(f"Removing {cache_dir}/{filename}.") + os.chmod(filename, stat.S_IWRITE) + os.remove(filename) diff --git a/tests/Makefile.include b/tests/Makefile.include index e47ef4d45c9..d4dfbf3716d 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -111,6 +111,7 @@ $(FUNCTIONAL_TARGETS): check-venv .PHONY: check-functional check-functional: check-venv @$(NINJA) precache-functional + @$(PYTHON) $(SRC_PATH)/scripts/clean_functional_cache.py @QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func check-func-quick .PHONY: check-func check-func-quick -- 2.51.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] tests: Evict stale files in the functional download cache after a while 2025-10-10 9:32 ` [PATCH 2/2] tests: Evict stale files in the functional download cache after a while Thomas Huth @ 2025-10-10 9:50 ` Daniel P. Berrangé 2025-10-13 11:47 ` Thomas Huth 0 siblings, 1 reply; 9+ messages in thread From: Daniel P. Berrangé @ 2025-10-10 9:50 UTC (permalink / raw) To: Thomas Huth Cc: qemu-devel, John Snow, Alex Bennée, Philippe Mathieu-Daudé On Fri, Oct 10, 2025 at 11:32:43AM +0200, Thomas Huth wrote: > From: Thomas Huth <thuth@redhat.com> > > The download cache of the functional tests is currently only growing. > But sometimes tests get removed or changed to use different assets, > thus we should clean up the stale old assets after a while when they > are not in use anymore. So add a script that looks at the time stamps > of the assets and removes them if they haven't been touched for more > than half of a year. Since there might also be some assets around that > have been added to the cache before we added the time stamp files, > assume a default time stamp that is close to the creation date of this > patch, so that we don't delete these files too early. > > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > MAINTAINERS | 1 + > scripts/clean_functional_cache.py | 47 +++++++++++++++++++++++++++++++ > tests/Makefile.include | 1 + > 3 files changed, 49 insertions(+) > create mode 100755 scripts/clean_functional_cache.py > > diff --git a/MAINTAINERS b/MAINTAINERS > index 84cfd85e1fa..4c468d45337 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -4398,6 +4398,7 @@ M: Thomas Huth <thuth@redhat.com> > R: Philippe Mathieu-Daudé <philmd@linaro.org> > R: Daniel P. Berrange <berrange@redhat.com> > F: docs/devel/testing/functional.rst > +F: scripts/clean_functional_cache.py > F: tests/functional/qemu_test/ > > Windows Hosted Continuous Integration > diff --git a/scripts/clean_functional_cache.py b/scripts/clean_functional_cache.py > new file mode 100755 > index 00000000000..e5c4d1acaf3 > --- /dev/null > +++ b/scripts/clean_functional_cache.py > @@ -0,0 +1,47 @@ > +#!/usr/bin/env python3 > +# > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +"""Delete stale assets from the download cache of the functional tests""" > + > +import os > +import stat > +import sys > +import time > +from pathlib import Path > + > + > +cache_dir_env = os.getenv('QEMU_TEST_CACHE_DIR') > +if cache_dir_env: > + cache_dir = Path(cache_dir_env, "download") > +else: > + cache_dir = Path(Path("~").expanduser(), ".cache", "qemu", "download") This creates a Path object but then doesn't take advantage of any of its functionality, calling os. functions still.... > + > +if not os.path.exists(cache_dir): cache_dir.exists() > + print(f"Cache dir {cache_dir} does not exist!", file=sys.stderr) > + sys.exit(1) > + > +os.chdir(cache_dir) > + > +for file in os.listdir(cache_dir): for file in cache_dir.iterdir(): > + filename = os.fsdecode(file) Wouldn't be required since 'file' would be a Path object > + # Only consider the files that use a sha256 as filename: > + if len(filename) != 64: if len(file.name) != 64 > + continue > + > + try: > + with open(filename + ".stamp", "r", encoding='utf-8') as fh: > + timestamp = int(fh.read()) timestamp = file.read_text() > + except FileNotFoundError: > + # Assume it's an old file that was already in the cache before we > + # added the code for evicting stale assets. Use the release date > + # of QEMU v10.1 as a default timestamp. > + timestamp = time.mktime((2025, 8, 26, 0, 0, 0, 0, 0, 0)) The prev patch will make the precache task create the .stamp for all files that are currently in use by the current branch. So the only thing this does is to prevent us deleting cached files that might still be needed by a different branch. There will be few of them, so if we prematurely delete a handful that's not a big deal. If we switch to checking mtime, this except won't even exist. > + > + age = time.time() - timestamp > + > + # Delete files older than half of a year (183 days * 24h * 60m * 60s) > + if age > 15811200: > + print(f"Removing {cache_dir}/{filename}.") > + os.chmod(filename, stat.S_IWRITE) file.chmod(stat.S_IWRITE) > + os.remove(filename) file.unlink() > diff --git a/tests/Makefile.include b/tests/Makefile.include > index e47ef4d45c9..d4dfbf3716d 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -111,6 +111,7 @@ $(FUNCTIONAL_TARGETS): check-venv > .PHONY: check-functional > check-functional: check-venv > @$(NINJA) precache-functional > + @$(PYTHON) $(SRC_PATH)/scripts/clean_functional_cache.py > @QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func check-func-quick > > .PHONY: check-func check-func-quick > -- > 2.51.0 > With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] tests: Evict stale files in the functional download cache after a while 2025-10-10 9:50 ` Daniel P. Berrangé @ 2025-10-13 11:47 ` Thomas Huth 2025-10-13 11:50 ` Daniel P. Berrangé 0 siblings, 1 reply; 9+ messages in thread From: Thomas Huth @ 2025-10-13 11:47 UTC (permalink / raw) To: Daniel P. Berrangé Cc: qemu-devel, John Snow, Alex Bennée, Philippe Mathieu-Daudé On 10/10/2025 11.50, Daniel P. Berrangé wrote: > On Fri, Oct 10, 2025 at 11:32:43AM +0200, Thomas Huth wrote: >> From: Thomas Huth <thuth@redhat.com> >> >> The download cache of the functional tests is currently only growing. >> But sometimes tests get removed or changed to use different assets, >> thus we should clean up the stale old assets after a while when they >> are not in use anymore. So add a script that looks at the time stamps >> of the assets and removes them if they haven't been touched for more >> than half of a year. Since there might also be some assets around that >> have been added to the cache before we added the time stamp files, >> assume a default time stamp that is close to the creation date of this >> patch, so that we don't delete these files too early. >> >> Signed-off-by: Thomas Huth <thuth@redhat.com> >> --- >> MAINTAINERS | 1 + >> scripts/clean_functional_cache.py | 47 +++++++++++++++++++++++++++++++ >> tests/Makefile.include | 1 + >> 3 files changed, 49 insertions(+) >> create mode 100755 scripts/clean_functional_cache.py >> >> diff --git a/MAINTAINERS b/MAINTAINERS >> index 84cfd85e1fa..4c468d45337 100644 >> --- a/MAINTAINERS >> +++ b/MAINTAINERS >> @@ -4398,6 +4398,7 @@ M: Thomas Huth <thuth@redhat.com> >> R: Philippe Mathieu-Daudé <philmd@linaro.org> >> R: Daniel P. Berrange <berrange@redhat.com> >> F: docs/devel/testing/functional.rst >> +F: scripts/clean_functional_cache.py >> F: tests/functional/qemu_test/ >> >> Windows Hosted Continuous Integration >> diff --git a/scripts/clean_functional_cache.py b/scripts/clean_functional_cache.py >> new file mode 100755 >> index 00000000000..e5c4d1acaf3 >> --- /dev/null >> +++ b/scripts/clean_functional_cache.py >> @@ -0,0 +1,47 @@ >> +#!/usr/bin/env python3 >> +# >> +# SPDX-License-Identifier: GPL-2.0-or-later >> +# >> +"""Delete stale assets from the download cache of the functional tests""" >> + >> +import os >> +import stat >> +import sys >> +import time >> +from pathlib import Path >> + >> + >> +cache_dir_env = os.getenv('QEMU_TEST_CACHE_DIR') >> +if cache_dir_env: >> + cache_dir = Path(cache_dir_env, "download") >> +else: >> + cache_dir = Path(Path("~").expanduser(), ".cache", "qemu", "download") > > This creates a Path object but then doesn't take advantage of > any of its functionality, calling os. functions still.... Ok, you got me, looks like I'm still a python ignorant after one year of hacking the functional testing framework ;-) Thanks for the hints how to do it better! >> + try: >> + with open(filename + ".stamp", "r", encoding='utf-8') as fh: >> + timestamp = int(fh.read()) > > timestamp = file.read_text() Hmm, but "file" points to the asset, not to the .stamp file, doesn't it? >> + except FileNotFoundError: >> + # Assume it's an old file that was already in the cache before we >> + # added the code for evicting stale assets. Use the release date >> + # of QEMU v10.1 as a default timestamp. >> + timestamp = time.mktime((2025, 8, 26, 0, 0, 0, 0, 0, 0)) > > The prev patch will make the precache task create the .stamp for all > files that are currently in use by the current branch. So the only > thing this does is to prevent us deleting cached files that might > still be needed by a different branch. There will be few of them, > so if we prematurely delete a handful that's not a big deal. If we > switch to checking mtime, this except won't even exist. When hunting regressions that have been introduced recently, I often have to do bisecting on revisions from the previous 1 or 2 QEMU releases, so I'd prefer keeping the assets of the last few months, even if they have been removed from the master branch in a very recent commit. Thomas ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] tests: Evict stale files in the functional download cache after a while 2025-10-13 11:47 ` Thomas Huth @ 2025-10-13 11:50 ` Daniel P. Berrangé 0 siblings, 0 replies; 9+ messages in thread From: Daniel P. Berrangé @ 2025-10-13 11:50 UTC (permalink / raw) To: Thomas Huth Cc: qemu-devel, John Snow, Alex Bennée, Philippe Mathieu-Daudé On Mon, Oct 13, 2025 at 01:47:57PM +0200, Thomas Huth wrote: > On 10/10/2025 11.50, Daniel P. Berrangé wrote: > > On Fri, Oct 10, 2025 at 11:32:43AM +0200, Thomas Huth wrote: > > > From: Thomas Huth <thuth@redhat.com> > > > > > > The download cache of the functional tests is currently only growing. > > > But sometimes tests get removed or changed to use different assets, > > > thus we should clean up the stale old assets after a while when they > > > are not in use anymore. So add a script that looks at the time stamps > > > of the assets and removes them if they haven't been touched for more > > > than half of a year. Since there might also be some assets around that > > > have been added to the cache before we added the time stamp files, > > > assume a default time stamp that is close to the creation date of this > > > patch, so that we don't delete these files too early. > > > > > > Signed-off-by: Thomas Huth <thuth@redhat.com> > > > --- > > > MAINTAINERS | 1 + > > > scripts/clean_functional_cache.py | 47 +++++++++++++++++++++++++++++++ > > > tests/Makefile.include | 1 + > > > 3 files changed, 49 insertions(+) > > > create mode 100755 scripts/clean_functional_cache.py > > > > > > diff --git a/MAINTAINERS b/MAINTAINERS > > > index 84cfd85e1fa..4c468d45337 100644 > > > --- a/MAINTAINERS > > > +++ b/MAINTAINERS > > > @@ -4398,6 +4398,7 @@ M: Thomas Huth <thuth@redhat.com> > > > R: Philippe Mathieu-Daudé <philmd@linaro.org> > > > R: Daniel P. Berrange <berrange@redhat.com> > > > F: docs/devel/testing/functional.rst > > > +F: scripts/clean_functional_cache.py > > > F: tests/functional/qemu_test/ > > > Windows Hosted Continuous Integration > > > diff --git a/scripts/clean_functional_cache.py b/scripts/clean_functional_cache.py > > > new file mode 100755 > > > index 00000000000..e5c4d1acaf3 > > > --- /dev/null > > > +++ b/scripts/clean_functional_cache.py > > > @@ -0,0 +1,47 @@ > > > +#!/usr/bin/env python3 > > > +# > > > +# SPDX-License-Identifier: GPL-2.0-or-later > > > +# > > > +"""Delete stale assets from the download cache of the functional tests""" > > > + > > > +import os > > > +import stat > > > +import sys > > > +import time > > > +from pathlib import Path > > > + > > > + > > > +cache_dir_env = os.getenv('QEMU_TEST_CACHE_DIR') > > > +if cache_dir_env: > > > + cache_dir = Path(cache_dir_env, "download") > > > +else: > > > + cache_dir = Path(Path("~").expanduser(), ".cache", "qemu", "download") > > > > This creates a Path object but then doesn't take advantage of > > any of its functionality, calling os. functions still.... > > Ok, you got me, looks like I'm still a python ignorant after one year of > hacking the functional testing framework ;-) Thanks for the hints how to do > it better! > > > > + try: > > > + with open(filename + ".stamp", "r", encoding='utf-8') as fh: > > > + timestamp = int(fh.read()) > > > > timestamp = file.read_text() > > Hmm, but "file" points to the asset, not to the .stamp file, doesn't it? Opps, yes, you'll need file.with_stem(".stamp").read_text() > > > + except FileNotFoundError: > > > + # Assume it's an old file that was already in the cache before we > > > + # added the code for evicting stale assets. Use the release date > > > + # of QEMU v10.1 as a default timestamp. > > > + timestamp = time.mktime((2025, 8, 26, 0, 0, 0, 0, 0, 0)) > > > > The prev patch will make the precache task create the .stamp for all > > files that are currently in use by the current branch. So the only > > thing this does is to prevent us deleting cached files that might > > still be needed by a different branch. There will be few of them, > > so if we prematurely delete a handful that's not a big deal. If we > > switch to checking mtime, this except won't even exist. > > When hunting regressions that have been introduced recently, I often have to > do bisecting on revisions from the previous 1 or 2 QEMU releases, so I'd > prefer keeping the assets of the last few months, even if they have been > removed from the master branch in a very recent commit. Ok. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2025-10-13 11:51 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-10-10 9:32 [PATCH 0/2] Clean up the functional download cache after some months Thomas Huth 2025-10-10 9:32 ` [PATCH 1/2] tests/functional: Set current time stamp of assets when using them Thomas Huth 2025-10-10 9:39 ` Daniel P. Berrangé 2025-10-10 9:46 ` Thomas Huth 2025-10-10 9:53 ` Daniel P. Berrangé 2025-10-10 9:32 ` [PATCH 2/2] tests: Evict stale files in the functional download cache after a while Thomas Huth 2025-10-10 9:50 ` Daniel P. Berrangé 2025-10-13 11:47 ` Thomas Huth 2025-10-13 11:50 ` Daniel P. Berrangé
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).