Re: [kvm-unit-tests PATCH] x86: vmx: Break the 'vmx' monolith into multiple tests

[Sean Christopherson <seanjc@google.com>]

On Mon, May 02, 2022, Aaron Lewis wrote:
> Break 'vmx' into multiple tests to reduce the number of tests run at
> once.  This will help get a better signal up front on failure with them
> broken up into categories.  This will also do a better job of not
> disguising new failures if one of the tests is already failing.
> 
> One side effect of breaking this test up is any new test added will
> have to be manually added to a categorized test.  It will not
> automatically be picked up by 'vmx'.
> 
> Signed-off-by: Aaron Lewis <aaronlewis@google.com>
> ---
>  x86/unittests.cfg | 42 +++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 39 insertions(+), 3 deletions(-)
> 
> diff --git a/x86/unittests.cfg b/x86/unittests.cfg
> index b48c98b..0d90413 100644
> --- a/x86/unittests.cfg
> +++ b/x86/unittests.cfg
> @@ -254,13 +254,49 @@ extra_params = -cpu qemu64,+umip
>  file = la57.flat
>  arch = i386
>  
> -[vmx]
> +[vmx_legacy]
>  file = vmx.flat
> -extra_params = -cpu host,+vmx -append "-exit_monitor_from_l2_test -ept_access* -vmx_smp* -vmx_vmcs_shadow_test -atomic_switch_overflow_msrs_test -vmx_init_signal_test -vmx_apic_passthrough_tpr_threshold_test"
> +extra_params = -cpu max,+vmx -append "null vmenter preemption_timer control_field_PAT control_field_EFER CR_shadowing I/O_bitmap instruction_intercept EPT_A/D_disabled EPT_A/D_enabled PML interrupt nmi_hlt debug_controls MSR_switch vmmcall disable_RDTSCP int3 into invalid_msr"
>  arch = x86_64
>  groups = vmx
>  
> -[ept]
> +[vmx_basic]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "v2_null_test v2_multiple_entries_test fixture_test_case1 fixture_test_case2"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_vm_entry]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "vmx_controls_test vmx_host_state_area_test vmx_guest_state_area_test vmentry_movss_shadow_test vmentry_unrestricted_guest_test"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_apic]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "vmx_eoi_bitmap_ioapic_scan_test vmx_hlt_with_rvi_test vmx_apic_passthrough_test vmx_apic_passthrough_thread_test vmx_sipi_signal_test"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_regression]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "vmx_ldtr_test vmx_cr_load_test vmx_cr4_osxsave_test vmx_nm_test vmx_db_test vmx_nmi_window_test vmx_intr_window_test vmx_pending_event_test vmx_pending_event_hlt_test vmx_store_tsc_test"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_preemption_timer]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "vmx_preemption_timer_zero_test vmx_preemption_timer_tf_test vmx_preemption_timer_expiry_test"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_misc]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "invvpid_test atomic_switch_max_msrs_test rdtsc_vmexit_diff_test vmx_mtf_test vmx_mtf_pdpte_test vmx_exception_test"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_ept]
>  file = vmx.flat
>  extra_params = -cpu host,host-phys-bits,+vmx -m 2560 -append "ept_access*"
>  arch = x86_64

hmm, I don't like that there's no common criteria for the buckets.  E.g.
"regression" is a bucket for how a test came to exist, whereas "apic" is a bucket
for what a test exercises.  And the granularity ends up being odd too, e.g. the
preemption timer gets its own bucket, but "regression" has a massive variety of
tests.

My preference would be to separate tests by what they cover, which I think will be
most useful.  E.g. if I'm making changes to EPT/paging related stuff, it would be
nice to be able to run just the paging tests early on.  And when a bucket fails,
it'd also give the user a hint as to want might have gone wrong.  I'm sure there
are tests that straddle multiple buckets, so it won't be perfect, but I think we
can get something useful.

 vmx_cpu 		(don't like the name, want it to be for tests that verify CPU arch behavior)
 vmx_consistency_checks
 vmx_events 		(non-vAPIC APIC tests, timers, NMIs, IRQs, preemption timer, etc...)
 vmx_paging		(ept A/D, PML, invvpid, vmx_pf_exception_test, etc...)
 vmx_vapic		(vAPIC, VID, PI, etc...)
 ept_access 		(because they probably needs their own config, but they aren't the only EPT tests)