test harness hang running functional test

test harness hang running functional test

[Peter Maydell]

I was trying to track down why one of the functional tests was
hanging, so I tried running it directly:

UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1
QEMU_TEST_QEMU_BINARY=build/clang/qemu-system-loongarch64
PYTHONPATH=./build/clang/pyvenv:./python time
./build/clang/pyvenv/bin/python3
./tests/functional/test_loongarch64_virt.py

(This is with a QEMU built with the undefined-behaviour
sanitizer.)

Mostly this seems to succeed, but this time it's hung. Looking
at the process tree:

petmay01 3616444  0.0  0.0   4768  1124 pts/2    S+   15:53   0:00
 \_ time ./build/clang/pyvenv/bin/python3
./tests/functional/test_loongarch64_virt.py
petmay01 3616445 99.8  0.0  38480 23620 pts/2    R+   15:53  13:20
     \_ ./build/clang/pyvenv/bin/python3
./tests/functional/test_loongarch64_virt.py
petmay01 3616448  0.0  0.0      0     0 pts/2    Z+   15:53   0:00
         \_ [qemu-system-loo] <defunct>

The QEMU process itself has exited, but the test harness has
not yet noticed or reaped the zombie process. Instead it's sitting
there eating CPU. Presumably this is a bug in the test harness?

thanks
-- PMM

Re: test harness hang running functional test

[Peter Maydell]

On Thu, 7 Nov 2024 at 16:09, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> I was trying to track down why one of the functional tests was
> hanging, so I tried running it directly:
>
> UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1
> QEMU_TEST_QEMU_BINARY=build/clang/qemu-system-loongarch64
> PYTHONPATH=./build/clang/pyvenv:./python time
> ./build/clang/pyvenv/bin/python3
> ./tests/functional/test_loongarch64_virt.py
>
> (This is with a QEMU built with the undefined-behaviour
> sanitizer.)
>
> Mostly this seems to succeed, but this time it's hung. Looking
> at the process tree:
>
> petmay01 3616444  0.0  0.0   4768  1124 pts/2    S+   15:53   0:00
>  \_ time ./build/clang/pyvenv/bin/python3
> ./tests/functional/test_loongarch64_virt.py
> petmay01 3616445 99.8  0.0  38480 23620 pts/2    R+   15:53  13:20
>      \_ ./build/clang/pyvenv/bin/python3
> ./tests/functional/test_loongarch64_virt.py
> petmay01 3616448  0.0  0.0      0     0 pts/2    Z+   15:53   0:00
>          \_ [qemu-system-loo] <defunct>
>
> The QEMU process itself has exited, but the test harness has
> not yet noticed or reaped the zombie process. Instead it's sitting
> there eating CPU. Presumably this is a bug in the test harness?

Eventually I hit ^C and got this backtrace:

  File "/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/./tests/functional/test_loongarch64_virt.py",
line 62, in <module>
    QemuSystemTest.main()
  File "/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/tests/functional/qemu_test/testcase.py",
line 71, in main
    unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
  File "/usr/lib/python3.10/unittest/main.py", line 101, in __init__
    self.runTests()
  File "/usr/lib/python3.10/unittest/main.py", line 271, in runTests
    self.result = testRunner.run(self.test)
  File "/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/clang/pyvenv/lib/python3.10/site-packages/pycotap/__init__.py",
line 156, in run
    test(result)
  File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.10/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.10/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.10/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.10/unittest/case.py", line 650, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.10/unittest/case.py", line 591, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
    method()
  File "/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/tests/functional/test_loongarch64_virt.py",
line 57, in test_loongarch64_devices
    self.wait_for_console_pattern('Run /sbin/init as init process')
  File "/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/tests/functional/test_loongarch64_virt.py",
line 33, in wait_for_console_pattern
    wait_for_console_pattern(self, success_message,
  File "/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/tests/functional/qemu_test/cmd.py",
line 152, in wait_for_console_pattern
    _console_interaction(test, success_message, failure_message, None, vm=vm)
  File "/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/tests/functional/qemu_test/cmd.py",
line 101, in _console_interaction
    msg = console.readline().decode().strip()
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)


Something has gone wrong if we can be blocked waiting for
guest input when the QEMU process has already exited...

-- PMM

Re: test harness hang running functional test

[Thomas Huth]

On 07/11/2024 17.09, Peter Maydell wrote:
> I was trying to track down why one of the functional tests was
> hanging, so I tried running it directly:
> 
> UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1
> QEMU_TEST_QEMU_BINARY=build/clang/qemu-system-loongarch64
> PYTHONPATH=./build/clang/pyvenv:./python time
> ./build/clang/pyvenv/bin/python3
> ./tests/functional/test_loongarch64_virt.py
> 
> (This is with a QEMU built with the undefined-behaviour
> sanitizer.)
> 
> Mostly this seems to succeed, but this time it's hung. Looking
> at the process tree:
> 
> petmay01 3616444  0.0  0.0   4768  1124 pts/2    S+   15:53   0:00
>   \_ time ./build/clang/pyvenv/bin/python3
> ./tests/functional/test_loongarch64_virt.py
> petmay01 3616445 99.8  0.0  38480 23620 pts/2    R+   15:53  13:20
>       \_ ./build/clang/pyvenv/bin/python3
> ./tests/functional/test_loongarch64_virt.py
> petmay01 3616448  0.0  0.0      0     0 pts/2    Z+   15:53   0:00
>           \_ [qemu-system-loo] <defunct>
> 
> The QEMU process itself has exited, but the test harness has
> not yet noticed or reaped the zombie process. Instead it's sitting
> there eating CPU. Presumably this is a bug in the test harness?

Sounds like either a bug in the test harness or in the "python/qemu" module 
... I'll try to look at it, but maybe not in the next two weeks (I'm 
currently quite busy with other stuff), so if someone else wants to have a 
look already, that would be great!

  Thomas