* [PATCH] tests/qtest/npcm7xx_rng-test: Disable randomness tests
@ 2020-11-02 15:24 Peter Maydell
2020-11-02 15:43 ` Philippe Mathieu-Daudé
2020-11-02 16:34 ` Havard Skinnemoen
0 siblings, 2 replies; 3+ messages in thread
From: Peter Maydell @ 2020-11-02 15:24 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Tyrone Ting, Daniel P. Berrange, Havard Skinnemoen
The randomness tests in the NPCM7xx RNG test fail intermittently
but fairly frequently. On my machine running the test in a loop:
while QTEST_QEMU_BINARY=./qemu-system-aarch64 ./tests/qtest/npcm7xx_rng-test; do true; done
will fail in less than a minute with an error like:
ERROR:../../tests/qtest/npcm7xx_rng-test.c:256:test_first_byte_runs:
assertion failed (calc_runs_p(buf.l, sizeof(buf) * BITS_PER_BYTE) > 0.01): (0.00286205989 > 0.01)
(Failures have been observed on all 4 of the randomness tests,
not just first_byte_runs.)
It's not clear why these tests are failing like this, but intermittent
failures make CI and merge testing awkward, so disable running them
unless a developer specifically sets QEMU_TEST_FLAKY_RNG_TESTS when
running the test suite, until we work out the cause.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
tests/qtest/npcm7xx_rng-test.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/tests/qtest/npcm7xx_rng-test.c b/tests/qtest/npcm7xx_rng-test.c
index da6e639bf6f..e7cde85fbbc 100644
--- a/tests/qtest/npcm7xx_rng-test.c
+++ b/tests/qtest/npcm7xx_rng-test.c
@@ -265,10 +265,16 @@ int main(int argc, char **argv)
qtest_add_func("npcm7xx_rng/enable_disable", test_enable_disable);
qtest_add_func("npcm7xx_rng/rosel", test_rosel);
- qtest_add_func("npcm7xx_rng/continuous/monobit", test_continuous_monobit);
- qtest_add_func("npcm7xx_rng/continuous/runs", test_continuous_runs);
- qtest_add_func("npcm7xx_rng/first_byte/monobit", test_first_byte_monobit);
- qtest_add_func("npcm7xx_rng/first_byte/runs", test_first_byte_runs);
+ /*
+ * These tests fail intermittently; only run them on explicit
+ * request until we figure out why.
+ */
+ if (getenv("QEMU_TEST_FLAKY_RNG_TESTS")) {
+ qtest_add_func("npcm7xx_rng/continuous/monobit", test_continuous_monobit);
+ qtest_add_func("npcm7xx_rng/continuous/runs", test_continuous_runs);
+ qtest_add_func("npcm7xx_rng/first_byte/monobit", test_first_byte_monobit);
+ qtest_add_func("npcm7xx_rng/first_byte/runs", test_first_byte_runs);
+ }
qtest_start("-machine npcm750-evb");
ret = g_test_run();
--
2.20.1
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [PATCH] tests/qtest/npcm7xx_rng-test: Disable randomness tests
2020-11-02 15:24 [PATCH] tests/qtest/npcm7xx_rng-test: Disable randomness tests Peter Maydell
@ 2020-11-02 15:43 ` Philippe Mathieu-Daudé
2020-11-02 16:34 ` Havard Skinnemoen
1 sibling, 0 replies; 3+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-11-02 15:43 UTC (permalink / raw)
To: Peter Maydell, qemu-arm, qemu-devel
Cc: Tyrone Ting, Daniel P. Berrange, Havard Skinnemoen
On 11/2/20 4:24 PM, Peter Maydell wrote:
> The randomness tests in the NPCM7xx RNG test fail intermittently
> but fairly frequently. On my machine running the test in a loop:
> while QTEST_QEMU_BINARY=./qemu-system-aarch64 ./tests/qtest/npcm7xx_rng-test; do true; done
>
> will fail in less than a minute with an error like:
> ERROR:../../tests/qtest/npcm7xx_rng-test.c:256:test_first_byte_runs:
> assertion failed (calc_runs_p(buf.l, sizeof(buf) * BITS_PER_BYTE) > 0.01): (0.00286205989 > 0.01)
>
> (Failures have been observed on all 4 of the randomness tests,
> not just first_byte_runs.)
>
> It's not clear why these tests are failing like this, but intermittent
> failures make CI and merge testing awkward, so disable running them
> unless a developer specifically sets QEMU_TEST_FLAKY_RNG_TESTS when
> running the test suite, until we work out the cause.
>
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---
> tests/qtest/npcm7xx_rng-test.c | 14 ++++++++++----
> 1 file changed, 10 insertions(+), 4 deletions(-)
>
> diff --git a/tests/qtest/npcm7xx_rng-test.c b/tests/qtest/npcm7xx_rng-test.c
> index da6e639bf6f..e7cde85fbbc 100644
> --- a/tests/qtest/npcm7xx_rng-test.c
> +++ b/tests/qtest/npcm7xx_rng-test.c
> @@ -265,10 +265,16 @@ int main(int argc, char **argv)
>
> qtest_add_func("npcm7xx_rng/enable_disable", test_enable_disable);
> qtest_add_func("npcm7xx_rng/rosel", test_rosel);
> - qtest_add_func("npcm7xx_rng/continuous/monobit", test_continuous_monobit);
> - qtest_add_func("npcm7xx_rng/continuous/runs", test_continuous_runs);
> - qtest_add_func("npcm7xx_rng/first_byte/monobit", test_first_byte_monobit);
> - qtest_add_func("npcm7xx_rng/first_byte/runs", test_first_byte_runs);
> + /*
> + * These tests fail intermittently; only run them on explicit
> + * request until we figure out why.
> + */
> + if (getenv("QEMU_TEST_FLAKY_RNG_TESTS")) {
> + qtest_add_func("npcm7xx_rng/continuous/monobit", test_continuous_monobit);
> + qtest_add_func("npcm7xx_rng/continuous/runs", test_continuous_runs);
> + qtest_add_func("npcm7xx_rng/first_byte/monobit", test_first_byte_monobit);
> + qtest_add_func("npcm7xx_rng/first_byte/runs", test_first_byte_runs);
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> + }
>
> qtest_start("-machine npcm750-evb");
> ret = g_test_run();
>
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [PATCH] tests/qtest/npcm7xx_rng-test: Disable randomness tests
2020-11-02 15:24 [PATCH] tests/qtest/npcm7xx_rng-test: Disable randomness tests Peter Maydell
2020-11-02 15:43 ` Philippe Mathieu-Daudé
@ 2020-11-02 16:34 ` Havard Skinnemoen
1 sibling, 0 replies; 3+ messages in thread
From: Havard Skinnemoen @ 2020-11-02 16:34 UTC (permalink / raw)
To: Peter Maydell; +Cc: qemu-arm, QEMU Developers, Tyrone Ting, Daniel P. Berrange
On Mon, Nov 2, 2020 at 7:24 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> The randomness tests in the NPCM7xx RNG test fail intermittently
> but fairly frequently. On my machine running the test in a loop:
> while QTEST_QEMU_BINARY=./qemu-system-aarch64 ./tests/qtest/npcm7xx_rng-test; do true; done
>
> will fail in less than a minute with an error like:
> ERROR:../../tests/qtest/npcm7xx_rng-test.c:256:test_first_byte_runs:
> assertion failed (calc_runs_p(buf.l, sizeof(buf) * BITS_PER_BYTE) > 0.01): (0.00286205989 > 0.01)
>
> (Failures have been observed on all 4 of the randomness tests,
> not just first_byte_runs.)
>
> It's not clear why these tests are failing like this, but intermittent
> failures make CI and merge testing awkward, so disable running them
> unless a developer specifically sets QEMU_TEST_FLAKY_RNG_TESTS when
> running the test suite, until we work out the cause.
>
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---
> tests/qtest/npcm7xx_rng-test.c | 14 ++++++++++----
> 1 file changed, 10 insertions(+), 4 deletions(-)
>
> diff --git a/tests/qtest/npcm7xx_rng-test.c b/tests/qtest/npcm7xx_rng-test.c
> index da6e639bf6f..e7cde85fbbc 100644
> --- a/tests/qtest/npcm7xx_rng-test.c
> +++ b/tests/qtest/npcm7xx_rng-test.c
> @@ -265,10 +265,16 @@ int main(int argc, char **argv)
>
> qtest_add_func("npcm7xx_rng/enable_disable", test_enable_disable);
> qtest_add_func("npcm7xx_rng/rosel", test_rosel);
> - qtest_add_func("npcm7xx_rng/continuous/monobit", test_continuous_monobit);
> - qtest_add_func("npcm7xx_rng/continuous/runs", test_continuous_runs);
> - qtest_add_func("npcm7xx_rng/first_byte/monobit", test_first_byte_monobit);
> - qtest_add_func("npcm7xx_rng/first_byte/runs", test_first_byte_runs);
> + /*
> + * These tests fail intermittently; only run them on explicit
> + * request until we figure out why.
> + */
> + if (getenv("QEMU_TEST_FLAKY_RNG_TESTS")) {
> + qtest_add_func("npcm7xx_rng/continuous/monobit", test_continuous_monobit);
> + qtest_add_func("npcm7xx_rng/continuous/runs", test_continuous_runs);
> + qtest_add_func("npcm7xx_rng/first_byte/monobit", test_first_byte_monobit);
> + qtest_add_func("npcm7xx_rng/first_byte/runs", test_first_byte_runs);
> + }
Reviewed-by: Havard Skinnemoen <hskinnemoen@google.com>
Thanks!
> qtest_start("-machine npcm750-evb");
> ret = g_test_run();
> --
> 2.20.1
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-11-02 16:39 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-11-02 15:24 [PATCH] tests/qtest/npcm7xx_rng-test: Disable randomness tests Peter Maydell
2020-11-02 15:43 ` Philippe Mathieu-Daudé
2020-11-02 16:34 ` Havard Skinnemoen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).