kvm-unit-tests failures

kvm-unit-tests failures

[Chris J Arges]

Hi,

We are planning on running kvm-unit-tests as part of our test suite; but
I've noticed that many tests fail (even running the latest kvm tip).
After searching I found many BZ entires that seem to point at this
master bug for tracking these issues:
https://bugzilla.redhat.com/show_bug.cgi?id=1079979
However, this bug is private; and cannot be viewed by the public.

I'd like to know how to help report issues that we observe with testing
in order to help fix these tests, or understand any progress being made
to fix them already. Is there a public bug that everybody can view to
track these issues? Should I be reporting new bugs with failures in the
unit tests? Where is the appropriate place to file bugs against
kvm-unit-tests and discuss issues?

Thanks,
--chris j arges

Re: kvm-unit-tests failures

[Paolo Bonzini]

Il 01/08/2014 23:09, Chris J Arges ha scritto:
> Hi,
> 
> We are planning on running kvm-unit-tests as part of our test suite; but
> I've noticed that many tests fail (even running the latest kvm tip).
> After searching I found many BZ entires that seem to point at this
> master bug for tracking these issues:
> https://bugzilla.redhat.com/show_bug.cgi?id=1079979

This bug is for failures in the RHEL7 kernel, which is older than
kvm.git.  Which tests fail for you?.

kvm.git bugs are tracked in bugzilla.kernel.org, but for now I think
it's better if you just reply here.

Paolo

> However, this bug is private; and cannot be viewed by the public.
> 
> I'd like to know how to help report issues that we observe with testing
> in order to help fix these tests, or understand any progress being made
> to fix them already. Is there a public bug that everybody can view to
> track these issues? Should I be reporting new bugs with failures in the
> unit tests? Where is the appropriate place to file bugs against
> kvm-unit-tests and discuss issues?
> 
> Thanks,
> --chris j arges
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: kvm-unit-tests failures

[Chris J Arges]

On 08/02/2014 01:35 AM, Paolo Bonzini wrote:
> Il 01/08/2014 23:09, Chris J Arges ha scritto:
>> Hi,
>>
>> We are planning on running kvm-unit-tests as part of our test suite; but
>> I've noticed that many tests fail (even running the latest kvm tip).
>> After searching I found many BZ entires that seem to point at this
>> master bug for tracking these issues:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1079979
> 
> This bug is for failures in the RHEL7 kernel, which is older than
> kvm.git.  Which tests fail for you?.
> 
> kvm.git bugs are tracked in bugzilla.kernel.org, but for now I think
> it's better if you just reply here.
> 
> Paolo
> 
<snip>

Paulo,
Hopefully this is enough to get started; I can file this as a bug if its
easier to track. Let me know if there is additional information that is
needed. I'm thinking some may be userspace qemu-system-x86_64 issues
since three of them can be resolved by adding "-cpu host"; however it
may be something to add in the test-suite. Let me know and I am happy to
write a patch depending on what you think will work.

All tests below are done with the kvm tip (fresh as of today) merged
with 3.16-rc4.

Thanks,
--chris j arges

--

* Failures:

- pmu
$ sudo ./x86-run x86/pmu.flat -smp 1
qemu-system-x86_64 -enable-kvm -device pc-testdev -device
isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio
-device pci-testdev -kernel x86/pmu.flat -smp 1
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
PMU version:         33
GP counters:         1
GP counter width:    0
Mask length:         0
Fixed counters:      1
Fixed counter width: 0
GP event 'core cycles' is disabled
GP event 'instructions' is disabled
GP event 'ref cycles' is disabled
GP event 'llc refference' is disabled
GP event 'llc misses' is disabled
GP event 'branches' is disabled
unhandled excecption 13
Return value from qemu: 15

As suggested in comment 2 on this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1079796
Adding the -cpu host allows this test to PASS.

- tsc_adjust
$ sudo ./x86-run x86/tsc_adjust.flat -smp 1
qemu-system-x86_64 -enable-kvm -device pc-testdev -device
isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio
-device pci-testdev -kernel x86/tsc_adjust.flat -smp 1
enabling apic
failure: IA32_TSC_ADJUST msr read / write incorrect
failure: TSC did not adjust for IA32_TSC_ADJUST value
failure: IA32_TSC_ADJUST msr incorrectly adjusted on tsc write
Return value from qemu: 3

Here again adding "-cpu host" allows this to pass.

- pcid
$ sudo ./x86-run x86/tsc_adjust.flat -smp 1
qemu-system-x86_64 -enable-kvm -device pc-testdev -device
isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio
-device pci-testdev -kernel x86/tsc_adjust.flat -smp 1
enabling apic
failure: IA32_TSC_ADJUST msr read / write incorrect
failure: TSC did not adjust for IA32_TSC_ADJUST value
failure: IA32_TSC_ADJUST msr incorrectly adjusted on tsc write
Return value from qemu: 3
ubuntu@zoot:~/kvm-unit-tests$ sudo ./x86-run x86/pcid.flat -smp 1 -cpu
qemu64,+pcid
qemu-system-x86_64 -enable-kvm -device pc-testdev -device
isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio
-device pci-testdev -kernel x86/pcid.flat -smp 1 -cpu qemu64,+pcid
enabling apic
PASS: CPUID consistency
FAIL: Test on PCID when enabled
PASS: Test on INVPCID when disabled

SUMMARY: 3 tests, 1 unexpected failures
Return value from qemu: 3

Here again adding "-cpu host" allows this to pass.

- msr (HANG)
./x86-run x86/msr.flat -smp 1

$ sudo ./x86-run x86/msr.flat -smp 1qemu-system-x86_64 -enable-kvm
-device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4
-display none -serial stdio -device pci-testdev -kernel x86/msr.flat -smp 1
enabling apic
enabling apic
enabling apic

"enabling apic" repeats and adding -cpu host seems to have no effect.
Interestingly enough on an Ubuntu 3.13 series kernel this PASSes.

- vmx (HANG)
$ sudo ./x86-run x86/vmx.flat -smp 1 -cpu host,+vmx
qemu-system-x86_64 -enable-kvm -device pc-testdev -device
isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio
-device pci-testdev -kernel x86/vmx.flat -smp 1 -cpu host,+vmx
enabling apic
paging enabled
cr0 = 80010011
cr3 = 7fff000
cr4 = 20
PASS: test vmxon with FEATURE_CONTROL cleared
PASS: test vmxon without FEATURE_CONTROL lock
PASS: test enable VMX in FEATURE_CONTROL
PASS: test FEATURE_CONTROL lock bit
PASS: test vmxon with unaligned vmxon region
PASS: test vmxon with bits set beyond physical address width
PASS: test vmxon with invalid revision identifier
PASS: test vmxon with valid vmxon region
PASS: test vmptrld with unaligned vmcs
PASS: test vmptrld with vmcs address bits set beyond physical address width
PASS: test vmptrld with vmxon region
PASS: test vmptrld with valid vmcs region
PASS: test vmclear with unaligned vmcs
PASS: test vmclear with vmcs address bits set beyond physical address width
PASS: test vmclear with vmxon region
PASS: test vmclear with valid vmcs region
PASS: test vmptrst
PASS: test vmxoff

Test suite: VMX capability reporting
PASS: MSR_IA32_VMX_BASIC
PASS: MSR_IA32_VMX_MISC
PASS: MSR_IA32_VMX_PINBASED_CTLS
PASS: MSR_IA32_VMX_PROCBASED_CTLS
PASS: MSR_IA32_VMX_PROCBASED_CTLS2
PASS: MSR_IA32_VMX_EXIT_CTLS
PASS: MSR_IA32_VMX_ENTRY_CTLS
PASS: MSR_IA32_VMX_IA32_VMX_CR0_FIXED0/1
PASS: MSR_IA32_VMX_IA32_VMX_CR4_FIXED0/1
PASS: MSR_IA32_VMX_VMCS_ENUM
PASS: MSR_IA32_VMX_EPT_VPID_CAP

Test suite: vmenter
PASS: test vmlaunch
PASS: test vmresume

Test suite: preemption timer
PASS: Keep preemption value
PASS: Save preemption value
PASS: busy-wait for preemption timer
PASS: preemption timer during hlt
PASS: preemption timer with 0 value

Test suite: control field PAT
PASS: Exit save PAT
PASS: Exit load PAT
PASS: Entry load PAT

Test suite: control field EFER
PASS: Exit save EFER
PASS: Exit load EFER
PASS: Entry load EFER

Test suite: CR shadowing
PASS: Read through CR0
PASS: Read through CR4
PASS: Write through CR0
PASS: Write through CR4
PASS: Read shadowing CR0
PASS: Read shadowing CR4
PASS: Write shadowing CR0 (same value)
PASS: Write shadowing CR4 (same value)
PASS: Write shadowing different X86_CR0_TS
PASS: Write shadowing different X86_CR0_MP
PASS: Write shadowing different X86_CR4_TSD
PASS: Write shadowing different X86_CR4_DE

Test suite: I/O bitmap
PASS: I/O bitmap - I/O pass
PASS: I/O bitmap - I/O width, byte
PASS: I/O bitmap - I/O direction, in
PASS: I/O bitmap - trap in
PASS: I/O bitmap - I/O width, word
PASS: I/O bitmap - I/O direction, out
PASS: I/O bitmap - trap out
PASS: I/O bitmap - I/O width, long
PASS: I/O bitmap - I/O port, low part
PASS: I/O bitmap - I/O port, high part
PASS: I/O bitmap - partial pass
PASS: I/O bitmap - overrun
PASS: I/O bitmap - ignore unconditional exiting
PASS: I/O bitmap - unconditional exiting

Test suite: instruction intercept
PASS: execute HLT
PASS: intercept HLT
PASS: execute INVLPG
PASS: intercept INVLPG
PASS: execute MWAIT
PASS: intercept MWAIT
PASS: execute RDPMC
PASS: intercept RDPMC
PASS: execute RDTSC
PASS: intercept RDTSC
PASS: execute CR3 load
PASS: intercept CR3 load
PASS: execute CR3 store
PASS: intercept CR3 store
PASS: execute CR8 load
PASS: intercept CR8 load
PASS: execute CR8 store
PASS: intercept CR8 store
PASS: execute MONITOR
PASS: intercept MONITOR
PASS: execute PAUSE
PASS: intercept PAUSE
PASS: execute WBINVD
PASS: intercept WBINVD
PASS: intercept CPUID
PASS: intercept INVD

Test suite: EPT framework
PASS: EPT basic framework
PASS: EPT misconfigurations
PASS: EPT violation - page permission
FAIL: EPT violation - paging structure

Test suite: interrupt
PASS: direct interrupt while running guest
PASS: intercepted interrupt while running guest
PASS: direct interrupt + hlt

Here it just hangs.

* KVM Related Config:

CONFIG_KVM_GUEST=y
CONFIG_KVM_DEBUG_FS=y
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
# CONFIG_KVM_MMU_AUDIT is not set
CONFIG_KVM_DEVICE_ASSIGNMENT=y

* Module parameters:

$ tail /sys/module/kvm/parameters/*
==> /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts <==
N

==> /sys/module/kvm/parameters/ignore_msrs <==
N

==> /sys/module/kvm/parameters/min_timer_period_us <==
500

==> /sys/module/kvm/parameters/tsc_tolerance_ppm <==
250

$ tail /sys/module/kvm_intel/parameters/*
==> /sys/module/kvm_intel/parameters/emulate_invalid_guest_state <==
Y

==> /sys/module/kvm_intel/parameters/enable_apicv <==
Y

==> /sys/module/kvm_intel/parameters/enable_shadow_vmcs <==
Y

==> /sys/module/kvm_intel/parameters/ept <==
Y

==> /sys/module/kvm_intel/parameters/eptad <==
Y

==> /sys/module/kvm_intel/parameters/fasteoi <==
Y

==> /sys/module/kvm_intel/parameters/flexpriority <==
Y

==> /sys/module/kvm_intel/parameters/nested <==
Y

==> /sys/module/kvm_intel/parameters/ple_gap <==
128

==> /sys/module/kvm_intel/parameters/ple_window <==
4096

==> /sys/module/kvm_intel/parameters/unrestricted_guest <==
Y

==> /sys/module/kvm_intel/parameters/vmm_exclusive <==
Y

==> /sys/module/kvm_intel/parameters/vpid <==
Y

Re: kvm-unit-tests failures

[Paolo Bonzini]

Il 04/08/2014 19:02, Chris J Arges ha scritto:
> Paulo,
> Hopefully this is enough to get started; I can file this as a bug if its
> easier to track. Let me know if there is additional information that is
> needed. I'm thinking some may be userspace qemu-system-x86_64 issues
> since three of them can be resolved by adding "-cpu host"; however it
> may be something to add in the test-suite. Let me know and I am happy to
> write a patch depending on what you think will work.

Are you aware of x86/unittests.cfg and run_tests.sh?

Paolo

Re: kvm-unit-tests failures

[Chris J Arges]

On 08/04/2014 01:20 PM, Paolo Bonzini wrote:
> Il 04/08/2014 19:02, Chris J Arges ha scritto:
>> Paulo,
>> Hopefully this is enough to get started; I can file this as a bug if its
>> easier to track. Let me know if there is additional information that is
>> needed. I'm thinking some may be userspace qemu-system-x86_64 issues
>> since three of them can be resolved by adding "-cpu host"; however it
>> may be something to add in the test-suite. Let me know and I am happy to
>> write a patch depending on what you think will work.
> 
> Are you aware of x86/unittests.cfg and run_tests.sh?
> 
> Paolo
> 

Paolo,
Yes this is how I initially ran the script:
./configure && make
sudo ./run_tests.sh -v

Then if any commands that failed, I re-ran the command it output in
order to show more output here for debugging.

--chris j arges

Re: kvm-unit-tests failures

[Paolo Bonzini]

Il 04/08/2014 20:37, Chris J Arges ha scritto:
> Paolo,
> Yes this is how I initially ran the script:
> ./configure && make
> sudo ./run_tests.sh -v
> 
> Then if any commands that failed, I re-ran the command it output in
> order to show more output here for debugging.

Which tests are failing and which require "-cpu host"?

Paolo

Re: kvm-unit-tests failures

[Chris J Arges]

On 08/04/2014 01:40 PM, Paolo Bonzini wrote:
> Il 04/08/2014 20:37, Chris J Arges ha scritto:
>> Paolo,
>> Yes this is how I initially ran the script:
>> ./configure && make
>> sudo ./run_tests.sh -v
>>
>> Then if any commands that failed, I re-ran the command it output in
>> order to show more output here for debugging.
> 
> Which tests are failing and which require "-cpu host"?
> 
> Paolo
> 

Paolo,
I included these results with my original reply:
https://www.mail-archive.com/kvm@vger.kernel.org/msg105950.html
See the "Failures" part.

Thanks,
--chris j arges

Re: kvm-unit-tests failures

[Paolo Bonzini]

Il 04/08/2014 19:02, Chris J Arges ha scritto:
> All tests below are done with the kvm tip (fresh as of today) merged
> with 3.16-rc4.

Sorry, I had missed the list of failures.

> * Failures:
> - pmu
> As suggested in comment 2 on this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=1079796
> Adding the -cpu host allows this test to PASS.

Correct, I thought unittests.cfg already did that.

> - tsc_adjust
> Here again adding "-cpu host" allows this to pass.

Looks like QEMU doesn't know the tsc_adjust flag at all.  CCing Eduardo.

> - pcid
> Here again adding "-cpu host" allows this to pass.

It passes here, and fails with "-cpu host":

PASS: CPUID consistency
FAIL: Test on PCID when enabled
PASS: Test on INVPCID when disabled

Looks like the "Test on PCID when enabled" has problems, and the
-cpu flag is a red herring.

> - msr (HANG)
> ./x86-run x86/msr.flat -smp 1
> 
> $ sudo ./x86-run x86/msr.flat -smp 1qemu-system-x86_64 -enable-kvm
> -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4
> -display none -serial stdio -device pci-testdev -kernel x86/msr.flat -smp 1
> enabling apic
> enabling apic
> enabling apic
> 
> "enabling apic" repeats and adding -cpu host seems to have no effect.
> Interestingly enough on an Ubuntu 3.13 series kernel this PASSes.

The APICBASE register is already tested by the apic testcase, and the
simple-minded tests in msr.flat actually trigger a general protection
fault nowadays because the quality of the emulation improved.  I'm
dropping the APICBASE tests from msr.flat.

> - vmx (HANG)
> Test suite: interrupt
> PASS: direct interrupt while running guest
> PASS: intercepted interrupt while running guest
> PASS: direct interrupt + hlt
> 
> Here it just hangs.

It fails here for me.  It could be related to APICv, can you try with
the enable_apic=0 parameter to kvm-intel.ko?

Paolo

Re: kvm-unit-tests failures

[Eduardo Habkost]

On Tue, Aug 05, 2014 at 12:26:26PM +0200, Paolo Bonzini wrote:
> Il 04/08/2014 19:02, Chris J Arges ha scritto:
> > All tests below are done with the kvm tip (fresh as of today) merged
> > with 3.16-rc4.
> 
> Sorry, I had missed the list of failures.
> 
> > * Failures:
> > - pmu
> > As suggested in comment 2 on this bug:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1079796
> > Adding the -cpu host allows this test to PASS.
> 
> Correct, I thought unittests.cfg already did that.
> 
> > - tsc_adjust
> > Here again adding "-cpu host" allows this to pass.
> 
> Looks like QEMU doesn't know the tsc_adjust flag at all.  CCing Eduardo.

QEMU doesn't know the flag name yet (although we have migration support for the
MSR since 2012).

Note that with the new "migratable=yes" default on QEMU 2.1, an unknown feature
won't be enabled even when using "-cpu host", and will require "-cpu
host,migratable=no".

I see a bug on the unit test code: it is calling cpuid(7) without checking if
(cpuid(0).a >= 7) first, so it is checking meaningless CPUID bits when
cpuid_level < 7.  This causes the following problem:

  [kvm-unit-tests/master=]$ ./x86-run x86/tsc_adjust.flat -smp 2 -cpu qemu64,level=7
  qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/tsc_adjust.flat -smp 2 -cpu qemu64,level=7
  enabling apic
  enabling apic
  success: IA32_TSC_ADJUST feature not enabled
  Return value from qemu: 1
  [kvm-unit-tests/master=]$ ./x86-run x86/tsc_adjust.flat -smp 2 -cpu qemu64,level=4
  qemu-kvm -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/tsc_adjust.flat -smp 2 -cpu qemu64,level=4
  enabling apic
  enabling apic
  failure: IA32_TSC_ADJUST msr read / write incorrect
  failure: TSC did not adjust for IA32_TSC_ADJUST value
  failure: IA32_TSC_ADJUST msr incorrectly adjusted on tsc write
  Return value from qemu: 3

-- 
Eduardo

Re: kvm-unit-tests failures

[Chris J Arges]

On 08/05/2014 05:26 AM, Paolo Bonzini wrote:
> Il 04/08/2014 19:02, Chris J Arges ha scritto:
>> All tests below are done with the kvm tip (fresh as of today) merged
>> with 3.16-rc4.
> 
Updated information with the three new kvm-unit-tests patches you've posted.

> Sorry, I had missed the list of failures.
> 
>> * Failures:
>> - pmu
>> As suggested in comment 2 on this bug:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1079796
>> Adding the -cpu host allows this test to PASS.
> 
> Correct, I thought unittests.cfg already did that.
> 

Ok with your latest patches, the test case runs, but fails on a few
other tests:
⟫ sudo ./x86-run x86/pmu.flat -smp 1 -cpu host | grep FAIL
FAIL: fixed-2
FAIL: all counters

>> - tsc_adjust
>> Here again adding "-cpu host" allows this to pass.
> 
> Looks like QEMU doesn't know the tsc_adjust flag at all.  CCing Eduardo.
> 
>> - pcid
>> Here again adding "-cpu host" allows this to pass.
> 
> It passes here, and fails with "-cpu host":
> 
> PASS: CPUID consistency
> FAIL: Test on PCID when enabled
> PASS: Test on INVPCID when disabled
> 
> Looks like the "Test on PCID when enabled" has problems, and the
> -cpu flag is a red herring.
> 
This test now passes.

>> - msr (HANG)
>> ./x86-run x86/msr.flat -smp 1
>>
>> $ sudo ./x86-run x86/msr.flat -smp 1qemu-system-x86_64 -enable-kvm
>> -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4
>> -display none -serial stdio -device pci-testdev -kernel x86/msr.flat -smp 1
>> enabling apic
>> enabling apic
>> enabling apic
>>
>> "enabling apic" repeats and adding -cpu host seems to have no effect.
>> Interestingly enough on an Ubuntu 3.13 series kernel this PASSes.
> 
> The APICBASE register is already tested by the apic testcase, and the
> simple-minded tests in msr.flat actually trigger a general protection
> fault nowadays because the quality of the emulation improved.  I'm
> dropping the APICBASE tests from msr.flat.
> 
This test now passes.

>> - vmx (HANG)
>> Test suite: interrupt
>> PASS: direct interrupt while running guest
>> PASS: intercepted interrupt while running guest
>> PASS: direct interrupt + hlt
>>
>> Here it just hangs.
> 
> It fails here for me.  It could be related to APICv, can you try with
> the enable_apic=0 parameter to kvm-intel.ko?
> 
> Paolo

With enable_apicv=0, this test case just FAILs for me.

⟫ sudo ./x86-run x86/vmx.flat -smp 1 -cpu host,+vmx | grep FAIL
FAIL: EPT violation - paging structure
FAIL: intercepted interrupt + hlt

I'll try to dig a bit more, but things are looking much better! Thanks
for the help and patches.
--chris j arges