Re: [PATCH v1] kvm: x86: implement PV send_IPI method

[Chao Gao <chao.gao@intel.com>]

>> >> >> From: Jason Wang <jasowang@redhat.com>
>> >> >>
>> >> >> We used to have PV version of send_IPI_mask and
>> >> >> send_IPI_mask_allbutself. This patch implements PV send_IPI method to
>> >> >> reduce the number of vmexits.
>> >>
>> >> It won't reduce the number of VM-exits; in fact, it may increase them on CPUs
>> >> that support IPI virtualization.
>> >
>> >Sure, but I wonder if it reduces the vmexits when there's no APICV or
>> >L2 VM. I thought it can reduce the 2 vmexits to 1?
>> 
>> Even without APICv, there is just 1 vmexit due to APIC write (xAPIC mode)
>> or MSR write (x2APIC mode).
>
>xAPIC will have two exits: ICR2 and then ICR.

ah, yes.

>If xAPIC vs. x2APIC is stable when
>kvm_setup_pv_ipi() runs, maybe key off of that?

But the guest doesn't know if APICv is enabled or even IPI virtualization
is enabled.

>
>> >> With IPI virtualization enabled, *unicast* and physical-addressing IPIs won't
>> >> cause a VM-exit.
>> >
>> >Right.
>> >
>> >> Instead, the microcode posts interrupts directly to the target
>> >> vCPU. The PV version always causes a VM-exit.
>> >
>> >Yes, but it applies to all PV IPI I think.
>> 
>> For multi-cast IPIs, a single hypercall (PV IPI) outperforms multiple ICR
>> writes, even when IPI virtualization is enabled.
>
>FWIW, I doubt _all_ multi-cast IPIs outperform IPI virtualization.  My guess is
>there's a threshold in the number of targets where the cost of sending multiple
>virtual IPIs becomes more expensive than the VM-Exit and software processing,
>and I assume/hope that threshold isn't '2'.

Yes. Determining the threshold is tricky, and it's likely not a constant value
across different CPU generations.

>
>> >> >> Signed-off-by: Jason Wang <jasowang@redhat.com>
>> >> >> Tested-by: Cindy Lu <lulu@redhat.com>
>> >> >
>> >> >I think a question here is are we able to see performance improvement
>> >> >in any kind of setup?
>> >>
>> >> It may result in a negative performance impact.
>> >
>> >Userspace can check and enable PV IPI for the case where it suits.
>> 
>> Yeah, we need to identify the cases. One example may be for TDX guests, using
>> a PV approach (TDVMCALL) can avoid the #VE cost.
>
>TDX doesn't need a PV approach.  Or rather, TDX already has an "architectural"
>PV approach.  Make a TDVMCALL to request emulation of WRMSR(ICR).  Don't plumb
>more KVM logic into it.

Agree. It should be an optimization for TDX guests, regardless of the
underlying hypervisor.