Re: [PATCH] tests/qtest/npcm7xx_rng-test: Disable randomness tests

["Philippe Mathieu-Daudé" <philmd@redhat.com>]

On 11/2/20 4:24 PM, Peter Maydell wrote:
> The randomness tests in the NPCM7xx RNG test fail intermittently
> but fairly frequently. On my machine running the test in a loop:
>  while QTEST_QEMU_BINARY=./qemu-system-aarch64 ./tests/qtest/npcm7xx_rng-test; do true; done
> 
> will fail in less than a minute with an error like:
> ERROR:../../tests/qtest/npcm7xx_rng-test.c:256:test_first_byte_runs:
> assertion failed (calc_runs_p(buf.l, sizeof(buf) * BITS_PER_BYTE) > 0.01): (0.00286205989 > 0.01)
> 
> (Failures have been observed on all 4 of the randomness tests,
> not just first_byte_runs.)
> 
> It's not clear why these tests are failing like this, but intermittent
> failures make CI and merge testing awkward, so disable running them
> unless a developer specifically sets QEMU_TEST_FLAKY_RNG_TESTS when
> running the test suite, until we work out the cause.
> 
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---
>  tests/qtest/npcm7xx_rng-test.c | 14 ++++++++++----
>  1 file changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/qtest/npcm7xx_rng-test.c b/tests/qtest/npcm7xx_rng-test.c
> index da6e639bf6f..e7cde85fbbc 100644
> --- a/tests/qtest/npcm7xx_rng-test.c
> +++ b/tests/qtest/npcm7xx_rng-test.c
> @@ -265,10 +265,16 @@ int main(int argc, char **argv)
>  
>      qtest_add_func("npcm7xx_rng/enable_disable", test_enable_disable);
>      qtest_add_func("npcm7xx_rng/rosel", test_rosel);
> -    qtest_add_func("npcm7xx_rng/continuous/monobit", test_continuous_monobit);
> -    qtest_add_func("npcm7xx_rng/continuous/runs", test_continuous_runs);
> -    qtest_add_func("npcm7xx_rng/first_byte/monobit", test_first_byte_monobit);
> -    qtest_add_func("npcm7xx_rng/first_byte/runs", test_first_byte_runs);
> +    /*
> +     * These tests fail intermittently; only run them on explicit
> +     * request until we figure out why.
> +     */
> +    if (getenv("QEMU_TEST_FLAKY_RNG_TESTS")) {
> +        qtest_add_func("npcm7xx_rng/continuous/monobit", test_continuous_monobit);
> +        qtest_add_func("npcm7xx_rng/continuous/runs", test_continuous_runs);
> +        qtest_add_func("npcm7xx_rng/first_byte/monobit", test_first_byte_monobit);
> +        qtest_add_func("npcm7xx_rng/first_byte/runs", test_first_byte_runs);

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> +    }
>  
>      qtest_start("-machine npcm750-evb");
>      ret = g_test_run();
>