[REPORT] Double deletion of output directory in functional test

[REPORT] Double deletion of output directory in functional test

[Aditya Gupta]

Hello,

I notice a test failure with ppc64 functional tests, which only occurs when
tests are run in parallel ('make check-functional-ppc64 -j8').

I believe the issue might be with the generic migration testcase. The following
backtrace happens on error:

	Traceback (most recent call last):
	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
	    super().tearDown()
	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
	    shutil.rmtree(self.workdir)
	  File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
	    onerror(os.lstat, path, sys.exc_info())
	  File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
	    orig_st = os.lstat(path, dir_fd=dir_fd)
	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/scratch'

The core issue is that during setUp, the `self.workdir` is correctly created,
but during tearDown, the `self.workdir` is somehow already deleted, thus causing
shutil.rmtree to fail with directory doesn't exist.

Since this only occurs with parallel tests, I believe multiple subtests
are trying to remove the same dir ?

I haven't been able to figure exactly where the issue happens. Any ideas ?

Two possible workarounds:
1. `export QEMU_TEST_KEEP_SCRATCH=1`, this causes qemu to skip the shutil.rmtree
2. Have if condition (or try-except) around the shutl.rmtree to skip double
deletion issues.

Thanks,
- Aditya G

QEMU Commit Id: 2339d0a1cfac6ecc667e6e062a593865c1541c35

To reproduce issue:
	./configure
	make check-functional-ppc64 -j8

Log:
	$ make check-functional-ppc64 -j8
	
	>>> LD_LIBRARY_PATH=<SRC_DIR>/build/tests/tcg/plugins:<SRC_DIR>/build/contrib/plugins QEMU_TEST_GDB=/usr/bin/gdb PYTHONPATH=<SRC_DIR>/python:<SRC_DIR>/tests/functional MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MESON_TEST_ITERATION=1 G_TEST_SLOW=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 RUST_BACKTRACE=1 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 QEMU_BUILD_ROOT=<SRC_DIR>/build QEMU_TEST_QEMU_BINARY=<SRC_DIR>/build/qemu-system-ppc64 MALLOC_PERTURB_=145 QEMU_TEST_QEMU_IMG=<SRC_DIR>/build/qemu-img <SRC_DIR>/build/pyvenv/bin/python3 <SRC_DIR>/tests/functional/ppc64/test_pseries.py
	―――――――――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――――――――――
	stderr:
	Traceback (most recent call last):
	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
	    super().tearDown()
	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
	    shutil.rmtree(self.workdir)
	  File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
	    onerror(os.lstat, path, sys.exc_info())
	  File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
	    orig_st = os.lstat(path, dir_fd=dir_fd)
	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/scratch'
	
	Traceback (most recent call last):
	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
	    super().tearDown()
	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
	    shutil.rmtree(self.workdir)
	  File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
	    onerror(os.lstat, path, sys.exc_info())
	  File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
	    orig_st = os.lstat(path, dir_fd=dir_fd)
	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_tcp_localhost/scratch'
	
	Traceback (most recent call last):
	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
	    super().tearDown()
	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
	    shutil.rmtree(self.workdir)
	  File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
	    onerror(os.lstat, path, sys.exc_info())
	  File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
	    orig_st = os.lstat(path, dir_fd=dir_fd)
	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_unix/scratch'
	
	More information on test_migration.PpcMigrationTest.test_migration_with_exec could be found here:
	 <SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/base.log
	 <SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/console.log
	More information on test_migration.PpcMigrationTest.test_migration_with_tcp_localhost could be found here:
	 <SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_tcp_localhost/base.log
	 <SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_tcp_localhost/console.log
	More information on test_migration.PpcMigrationTest.test_migration_with_unix could be found here:
	 <SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_unix/base.log
	 <SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_unix/console.log
	
	(test program exited with status code 1)
	―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

Re: [REPORT] Double deletion of output directory in functional test

[Aditya Gupta]

+harsh

On 23/01/26 20:07, Aditya Gupta wrote:

> Hello,
>
> I notice a test failure with ppc64 functional tests, which only occurs when
> tests are run in parallel ('make check-functional-ppc64 -j8').
>
> I believe the issue might be with the generic migration testcase. The following
> backtrace happens on error:
>
> 	Traceback (most recent call last):
> 	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
> 	    super().tearDown()
> 	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
> 	    shutil.rmtree(self.workdir)
> 	  File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
> 	    onerror(os.lstat, path, sys.exc_info())
> 	  File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
> 	    orig_st = os.lstat(path, dir_fd=dir_fd)
> 	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 	FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/scratch'

Git bisect points to this:


     f4e34d0fd56c25dd0c90722ed80a83a175160a6c is the first bad commit
     commit f4e34d0fd56c25dd0c90722ed80a83a175160a6c
     Date:   Fri Jan 9 09:35:19 2026 -0300

         tests/functional: Add a OS level migration test for pseries


- Aditya G

Re: [REPORT] Double deletion of output directory in functional test

[Fabiano Rosas]

Aditya Gupta <adityag@linux.ibm.com> writes:

> +harsh
>
> On 23/01/26 20:07, Aditya Gupta wrote:
>
>> Hello,
>>
>> I notice a test failure with ppc64 functional tests, which only occurs when
>> tests are run in parallel ('make check-functional-ppc64 -j8').
>>
>> I believe the issue might be with the generic migration testcase. The following
>> backtrace happens on error:
>>
>> 	Traceback (most recent call last):
>> 	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
>> 	    super().tearDown()
>> 	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
>> 	    shutil.rmtree(self.workdir)
>> 	  File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
>> 	    onerror(os.lstat, path, sys.exc_info())
>> 	  File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
>> 	    orig_st = os.lstat(path, dir_fd=dir_fd)
>> 	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> 	FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/scratch'
>
> Git bisect points to this:
>
>
>      f4e34d0fd56c25dd0c90722ed80a83a175160a6c is the first bad commit
>      commit f4e34d0fd56c25dd0c90722ed80a83a175160a6c
>      Date:   Fri Jan 9 09:35:19 2026 -0300
>
>          tests/functional: Add a OS level migration test for pseries
>

I see, the test I added imports PpcMigrationTest which causes it to be
instantiated by unittest and the setup method races with another test
that uses the same class. I'll send a patch.

>
> - Aditya G

Re: [REPORT] Double deletion of output directory in functional test

[Daniel P. Berrangé]

On Mon, Jan 26, 2026 at 12:48:00PM -0300, Fabiano Rosas wrote:
> Aditya Gupta <adityag@linux.ibm.com> writes:
> 
> > +harsh
> >
> > On 23/01/26 20:07, Aditya Gupta wrote:
> >
> >> Hello,
> >>
> >> I notice a test failure with ppc64 functional tests, which only occurs when
> >> tests are run in parallel ('make check-functional-ppc64 -j8').
> >>
> >> I believe the issue might be with the generic migration testcase. The following
> >> backtrace happens on error:
> >>
> >> 	Traceback (most recent call last):
> >> 	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
> >> 	    super().tearDown()
> >> 	  File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
> >> 	    shutil.rmtree(self.workdir)
> >> 	  File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
> >> 	    onerror(os.lstat, path, sys.exc_info())
> >> 	  File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
> >> 	    orig_st = os.lstat(path, dir_fd=dir_fd)
> >> 	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >> 	FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/scratch'
> >
> > Git bisect points to this:
> >
> >
> >      f4e34d0fd56c25dd0c90722ed80a83a175160a6c is the first bad commit
> >      commit f4e34d0fd56c25dd0c90722ed80a83a175160a6c
> >      Date:   Fri Jan 9 09:35:19 2026 -0300
> >
> >          tests/functional: Add a OS level migration test for pseries
> >
> 
> I see, the test I added imports PpcMigrationTest which causes it to be
> instantiated by unittest and the setup method races with another test
> that uses the same class. I'll send a patch.

While looking at the issues in mail, I noticed another minor bug in
tests/functional/migration.py.  It is directly using the TemporaryDirectory
class to create a dir for UNIX sockets, when it should use  self.socket_dir()
so we get consistent dir prefixes.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|