* [REPORT] Double deletion of output directory in functional test
@ 2026-01-23 14:37 Aditya Gupta
2026-01-23 19:45 ` Aditya Gupta
0 siblings, 1 reply; 4+ messages in thread
From: Aditya Gupta @ 2026-01-23 14:37 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Philippe Mathieu-Daudé, Daniel P. Berrangé,
Peter Xu, Fabiano Rosas
Hello,
I notice a test failure with ppc64 functional tests, which only occurs when
tests are run in parallel ('make check-functional-ppc64 -j8').
I believe the issue might be with the generic migration testcase. The following
backtrace happens on error:
Traceback (most recent call last):
File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
super().tearDown()
File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
shutil.rmtree(self.workdir)
File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
orig_st = os.lstat(path, dir_fd=dir_fd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/scratch'
The core issue is that during setUp, the `self.workdir` is correctly created,
but during tearDown, the `self.workdir` is somehow already deleted, thus causing
shutil.rmtree to fail with directory doesn't exist.
Since this only occurs with parallel tests, I believe multiple subtests
are trying to remove the same dir ?
I haven't been able to figure exactly where the issue happens. Any ideas ?
Two possible workarounds:
1. `export QEMU_TEST_KEEP_SCRATCH=1`, this causes qemu to skip the shutil.rmtree
2. Have if condition (or try-except) around the shutl.rmtree to skip double
deletion issues.
Thanks,
- Aditya G
QEMU Commit Id: 2339d0a1cfac6ecc667e6e062a593865c1541c35
To reproduce issue:
./configure
make check-functional-ppc64 -j8
Log:
$ make check-functional-ppc64 -j8
>>> LD_LIBRARY_PATH=<SRC_DIR>/build/tests/tcg/plugins:<SRC_DIR>/build/contrib/plugins QEMU_TEST_GDB=/usr/bin/gdb PYTHONPATH=<SRC_DIR>/python:<SRC_DIR>/tests/functional MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MESON_TEST_ITERATION=1 G_TEST_SLOW=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 RUST_BACKTRACE=1 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 QEMU_BUILD_ROOT=<SRC_DIR>/build QEMU_TEST_QEMU_BINARY=<SRC_DIR>/build/qemu-system-ppc64 MALLOC_PERTURB_=145 QEMU_TEST_QEMU_IMG=<SRC_DIR>/build/qemu-img <SRC_DIR>/build/pyvenv/bin/python3 <SRC_DIR>/tests/functional/ppc64/test_pseries.py
―――――――――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――――――――――
stderr:
Traceback (most recent call last):
File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
super().tearDown()
File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
shutil.rmtree(self.workdir)
File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
orig_st = os.lstat(path, dir_fd=dir_fd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/scratch'
Traceback (most recent call last):
File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
super().tearDown()
File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
shutil.rmtree(self.workdir)
File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
orig_st = os.lstat(path, dir_fd=dir_fd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_tcp_localhost/scratch'
Traceback (most recent call last):
File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
super().tearDown()
File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
shutil.rmtree(self.workdir)
File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
orig_st = os.lstat(path, dir_fd=dir_fd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_unix/scratch'
More information on test_migration.PpcMigrationTest.test_migration_with_exec could be found here:
<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/base.log
<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/console.log
More information on test_migration.PpcMigrationTest.test_migration_with_tcp_localhost could be found here:
<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_tcp_localhost/base.log
<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_tcp_localhost/console.log
More information on test_migration.PpcMigrationTest.test_migration_with_unix could be found here:
<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_unix/base.log
<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_unix/console.log
(test program exited with status code 1)
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [REPORT] Double deletion of output directory in functional test
2026-01-23 14:37 [REPORT] Double deletion of output directory in functional test Aditya Gupta
@ 2026-01-23 19:45 ` Aditya Gupta
2026-01-26 15:48 ` Fabiano Rosas
0 siblings, 1 reply; 4+ messages in thread
From: Aditya Gupta @ 2026-01-23 19:45 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Philippe Mathieu-Daudé, Daniel P. Berrangé,
Peter Xu, Fabiano Rosas, Harsh Prateek Bora
+harsh
On 23/01/26 20:07, Aditya Gupta wrote:
> Hello,
>
> I notice a test failure with ppc64 functional tests, which only occurs when
> tests are run in parallel ('make check-functional-ppc64 -j8').
>
> I believe the issue might be with the generic migration testcase. The following
> backtrace happens on error:
>
> Traceback (most recent call last):
> File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
> super().tearDown()
> File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
> shutil.rmtree(self.workdir)
> File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
> onerror(os.lstat, path, sys.exc_info())
> File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
> orig_st = os.lstat(path, dir_fd=dir_fd)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/scratch'
Git bisect points to this:
f4e34d0fd56c25dd0c90722ed80a83a175160a6c is the first bad commit
commit f4e34d0fd56c25dd0c90722ed80a83a175160a6c
Date: Fri Jan 9 09:35:19 2026 -0300
tests/functional: Add a OS level migration test for pseries
- Aditya G
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [REPORT] Double deletion of output directory in functional test
2026-01-23 19:45 ` Aditya Gupta
@ 2026-01-26 15:48 ` Fabiano Rosas
2026-01-26 15:53 ` Daniel P. Berrangé
0 siblings, 1 reply; 4+ messages in thread
From: Fabiano Rosas @ 2026-01-26 15:48 UTC (permalink / raw)
To: Aditya Gupta, qemu-devel
Cc: Thomas Huth, Philippe Mathieu-Daudé, Daniel P. Berrangé,
Peter Xu, Harsh Prateek Bora
Aditya Gupta <adityag@linux.ibm.com> writes:
> +harsh
>
> On 23/01/26 20:07, Aditya Gupta wrote:
>
>> Hello,
>>
>> I notice a test failure with ppc64 functional tests, which only occurs when
>> tests are run in parallel ('make check-functional-ppc64 -j8').
>>
>> I believe the issue might be with the generic migration testcase. The following
>> backtrace happens on error:
>>
>> Traceback (most recent call last):
>> File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
>> super().tearDown()
>> File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
>> shutil.rmtree(self.workdir)
>> File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
>> onerror(os.lstat, path, sys.exc_info())
>> File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
>> orig_st = os.lstat(path, dir_fd=dir_fd)
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/scratch'
>
> Git bisect points to this:
>
>
> f4e34d0fd56c25dd0c90722ed80a83a175160a6c is the first bad commit
> commit f4e34d0fd56c25dd0c90722ed80a83a175160a6c
> Date: Fri Jan 9 09:35:19 2026 -0300
>
> tests/functional: Add a OS level migration test for pseries
>
I see, the test I added imports PpcMigrationTest which causes it to be
instantiated by unittest and the setup method races with another test
that uses the same class. I'll send a patch.
>
> - Aditya G
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [REPORT] Double deletion of output directory in functional test
2026-01-26 15:48 ` Fabiano Rosas
@ 2026-01-26 15:53 ` Daniel P. Berrangé
0 siblings, 0 replies; 4+ messages in thread
From: Daniel P. Berrangé @ 2026-01-26 15:53 UTC (permalink / raw)
To: Fabiano Rosas
Cc: Aditya Gupta, qemu-devel, Thomas Huth,
Philippe Mathieu-Daudé, Peter Xu, Harsh Prateek Bora
On Mon, Jan 26, 2026 at 12:48:00PM -0300, Fabiano Rosas wrote:
> Aditya Gupta <adityag@linux.ibm.com> writes:
>
> > +harsh
> >
> > On 23/01/26 20:07, Aditya Gupta wrote:
> >
> >> Hello,
> >>
> >> I notice a test failure with ppc64 functional tests, which only occurs when
> >> tests are run in parallel ('make check-functional-ppc64 -j8').
> >>
> >> I believe the issue might be with the generic migration testcase. The following
> >> backtrace happens on error:
> >>
> >> Traceback (most recent call last):
> >> File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 421, in tearDown
> >> super().tearDown()
> >> File "<SRC_DIR>/tests/functional/qemu_test/testcase.py", line 238, in tearDown
> >> shutil.rmtree(self.workdir)
> >> File "/usr/lib64/python3.11/shutil.py", line 742, in rmtree
> >> onerror(os.lstat, path, sys.exc_info())
> >> File "/usr/lib64/python3.11/shutil.py", line 740, in rmtree
> >> orig_st = os.lstat(path, dir_fd=dir_fd)
> >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >> FileNotFoundError: [Errno 2] No such file or directory: '<SRC_DIR>/build/tests/functional/ppc64/test_migration.PpcMigrationTest.test_migration_with_exec/scratch'
> >
> > Git bisect points to this:
> >
> >
> > f4e34d0fd56c25dd0c90722ed80a83a175160a6c is the first bad commit
> > commit f4e34d0fd56c25dd0c90722ed80a83a175160a6c
> > Date: Fri Jan 9 09:35:19 2026 -0300
> >
> > tests/functional: Add a OS level migration test for pseries
> >
>
> I see, the test I added imports PpcMigrationTest which causes it to be
> instantiated by unittest and the setup method races with another test
> that uses the same class. I'll send a patch.
While looking at the issues in mail, I noticed another minor bug in
tests/functional/migration.py. It is directly using the TemporaryDirectory
class to create a dir for UNIX sockets, when it should use self.socket_dir()
so we get consistent dir prefixes.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-01-26 15:56 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-01-23 14:37 [REPORT] Double deletion of output directory in functional test Aditya Gupta
2026-01-23 19:45 ` Aditya Gupta
2026-01-26 15:48 ` Fabiano Rosas
2026-01-26 15:53 ` Daniel P. Berrangé
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.