Re: kvm-unit-tests: inconsistent test result between run_tests.sh and standalone test

[Guoqing Jiang <guoqing.jiang@linux.dev>]

On 11/15/22 6:29 PM, Andrew Jones wrote:
> On Tue, Nov 15, 2022 at 04:53:55PM +0800, Guoqing Jiang wrote:
>> Hi,
>>
>> Thanks for the quick reply!
>>
>> On 11/15/22 4:42 PM, Andrew Jones wrote:
>>> On Tue, Nov 15, 2022 at 04:11:48PM +0800, Guoqing Jiang wrote:
>>>> Hi,
>>>>
>>>> I find the two test results (pmu and intel_cet) are quite different, but
>>>> other
>>>> test results are consistent.
>>>>
>>>> gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh
>>>> ...
>>>> *PASS pmu (142 tests)*
>>>> ...
>>>> *FAIL intel_cet*
>>>> ...
>>>>
>>>> 1. pmu standalone test
>>>> gjiang@x1:~/source/kvm-unit-tests/tests> ./pmu
>>>> BUILD_HEAD=73d9d850
>>>> timeout -k 1s --foreground 90s /usr/bin/qemu-system-x86_64 --no-reboot
>>>> -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4
>>>> -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel
>>>> /tmp/tmp.Bai8UEIh2F -smp 1 -cpu max # -initrd /tmp/tmp.DFE9VFPOdp
>>>> enabling apic
>>>> smp: waiting for 0 APs
>>>> paging enabled
>>>> cr0 = 80010011
>>>> cr3 = 1007000
>>>> cr4 = 20
>>>> PMU version:         2
>>>> GP counters:         4
>>>> GP counter width:    48
>>>> Mask length:         7
>>>> Fixed counters:      3
>>>> Fixed counter width: 48
>>>> PASS: core cycles-0
>>>> ...
>>>> FAIL: llc misses-0
>>>> FAIL: llc misses-1
>>>> FAIL: llc misses-2
>>>> FAIL: llc misses-3
>>>> ...
>>>> SUMMARY: 142 tests, 4 unexpected failures
>>>> *FAIL pmu (142 tests, 4 unexpected failures)
>>>>
>>>> *And
>>>>
>>>> gjiang@x1:~/source/kvm-unit-tests> ./x86-run ./x86/pmu.flat
>>>> /usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev
>>>> -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio
>>>> -device pci-testdev -machine accel=kvm -kernel ./x86/pmu.flat # -initrd
>>>> /tmp/tmp.jiEHps3KLW
>>>> enabling apic
>>>> smp: waiting for 0 APs
>>>> paging enabled
>>>> cr0 = 80010011
>>>> cr3 = 1007000
>>>> cr4 = 20
>>>> *SKIP: No pmu is detected!**
>>>> **SUMMARY: 1 tests, 1 skipped*
>>>>
>>> ./x86-run doesn't look at x86/unittests.cfg, which is where the pmu test
>>> states that it needs '-cpu max'. You either need to add it yourself, e.g.
>>> './x86-run ./x86/pmu.flat -cpu max' or use run_tests.sh, e.g.
>>> './run_tests.sh pmu'. standalone tests get their parameters from
>>> x86/unittests.cfg, which is why it's already using '-cpu max'.
>> Thanks for the tips. And I still see two different results, one is PASS
>> while another has failures.
>>
>> gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu
>> PASS pmu (142 tests)
>>
>> gjiang@x1:~/source/kvm-unit-tests> ./x86-run ./x86/pmu.flat -cpu max
>> /usr/bin/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev
>> -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio
>> -device pci-testdev -machine accel=kvm -kernel ./x86/pmu.flat -cpu max #
>> -initrd /tmp/tmp.wBVHPW1XUr
>> enabling apic
>> smp: waiting for 0 APs
>> paging enabled
>> cr0 = 80010011
>> cr3 = 1007000
>> cr4 = 20
>> PMU version:         2
>> GP counters:         4
>> GP counter width:    48
>> Mask length:         7
>> Fixed counters:      3
>> Fixed counter width: 48
>> ...
>> FAIL: llc misses-0
>> FAIL: llc misses-1
>> FAIL: llc misses-2
>> FAIL: llc misses-3
>> ...
>> SUMMARY: 142 tests, 4 unexpected failures
> There shouldn't be any difference. Do the exact same failures occur every
> time with standalone and never with run_tests.sh? I'll bet the failures
> sometimes occur and sometimes not occur with either test invocation. In
> which case, the test harness is fine, and the test cases have found
> something to debug within the test and/or kvm.

The pmu standalone test always has failures though the failure numbers are
not same after every run. And './run_tests.sh pmu' which rarely reports 
failure.

gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu
PASS pmu (142 tests)
gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu
PASS pmu (142 tests)
gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu
PASS pmu (142 tests)
gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu
PASS pmu (142 tests)
gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu
FAIL pmu (142 tests, 4 unexpected failures)
gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu
PASS pmu (142 tests)
gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu
PASS pmu (142 tests)
gjiang@x1:~/source/kvm-unit-tests> ./run_tests.sh pmu
PASS pmu (142 tests)

>>> **
>>> 2. intel_cet
>>> gjiang@x1:~/source/kvm-unit-tests/tests> ./intel_cet
>>> BUILD_HEAD=73d9d850
>>> *skip intel_cet (test kernel not present)*
>>> This error looks like x86/cet.c wasn't built. Maybe do a 'make clean' and
>>> 'make standalone' again and watch that cet.c doesn't fail to compile.
>> Indeed, seems cet.c is not compiled since only cet.c exists even after run
>> 'make clean' and 'make standalone', but tests/intel_cet is generated.
>>
>> gjiang@x1:~/source/kvm-unit-tests> ls x86/cet.*
>> x86/cet.c
> Take a look at the makefile. It shows that your compiler needs to support
> -fcf-protection=full.
>
> We could consider a patch for mkstandalone like below, but I think I
> prefer the way it is now, rather than silently dropping tests.

Thanks for the patch, is it possible to print more intuitive information
instead of 'test kernel not present'? I think access-reduced-maxphyaddr
is better.

gjiang@x1:~/source/kvm-unit-tests> ./tests/access-reduced-maxphyaddr
BUILD_HEAD=73d9d850
SKIP access-reduced-maxphyaddr 
(/sys/module/kvm_intel/parameters/allow_smaller_maxphyaddr not equal to Y)
> Thanks,
> drew
>
> diff --git a/scripts/mkstandalone.sh b/scripts/mkstandalone.sh
> index 86c7e5498246..3d61f24d3d6e 100755
> --- a/scripts/mkstandalone.sh
> +++ b/scripts/mkstandalone.sh
> @@ -47,9 +47,7 @@ generate_test ()
>          echo "echo BUILD_HEAD=$(cat build-head)"
>
>          if [ ! -f $kernel ]; then
> -               echo 'echo "skip '"$testname"' (test kernel not present)"'
> -               echo 'exit 2'
> -               return
> +               return 2
>          fi
>
>          echo "trap 'rm -f \$cleanup' EXIT"
> @@ -89,7 +87,10 @@ function mkstandalone()
>
>          standalone=tests/$testname
>
> -       generate_test "$@" > $standalone
> +       if ! generate_test "$@" > $standalone; then
> +               rm -f $standalone
> +               return
> +       fi
>
>          chmod +x $standalone
>          echo Written $standalone.

Thanks,
Guoqing