Re: [PATCH] x86/hvm: don't expose XENFEAT_hvm_pirqs by default

[Xenia Ragiadakou <xenia.ragiadakou@amd.com>]

Hi Roger,

On 10/1/24 17:21, Roger Pau Monné wrote:
> On Wed, Jan 10, 2024 at 03:47:12PM +0200, Xenia Ragiadakou wrote:
>>
>>
>> On 10/1/24 12:26, Jan Beulich wrote:
>>> On 10.01.2024 10:53, Roger Pau Monne wrote:
>>>> The HVM pirq feature allows routing interrupts from both physical and emulated
>>>> devices over event channels, this was done a performance improvement.  However
>>>> its usage is fully undocumented, and the only reference implementation is in
>>>> Linux.  It defeats the purpose of local APIC hardware virtualization, because
>>>> when using it interrupts avoid the usage of the local APIC altogether.
>>>
>>> So without sufficient APIC acceleration, isn't this arranging for degraded
>>> performance then? IOW should the new default perhaps be dependent on the
>>> degree of APIC acceleration?
>>>
>>>> It has also been reported to not work properly with certain devices, at least
>>>> when using some AMD GPUs Linux attempts to route interrupts over event
>>>> channels, but Xen doesn't correctly detect such routing, which leads to the
>>>> hypervisor complaining with:
>>>>
>>>> (XEN) d15v0: Unsupported MSI delivery mode 7 for Dom15
>>>>
>>>> When MSIs are attempted to be routed over event channels the entry delivery
>>>> mode is set to ExtINT, but Xen doesn't detect such routing and attempts to
>>>> inject the interrupt following the native MSI path, and the ExtINT delivery
>>>> mode is not supported.
>>>
>>> Shouldn't this be properly addressed nevertheless? The way it's described
>>> it sounds as if MSI wouldn't work at all this way; I can't spot why the
>>> issue would only be "with certain devices". Yet that in turn doesn't look
>>> to be very likely - pass-through use cases, in particular SR-IOV ones,
>>> would certainly have noticed.
>>
>> The issue gets triggered when the guest performs save/restore of MSIs,
>> because PHYSDEVOP_map_pirq is not implemented for MSIs, and thus, QEMU
>> cannot remap the MSI to the event channel once unmapped.
> 
> I'm kind of confused by this sentence, PHYSDEVOP_map_pirq does support
> MSIs, see xc_physdev_map_pirq_msi() helper in Xen code base.
> 

Sorry I had to explain it better. For an HVM guest with 
XENFEAT_hvm_pirqs set, physdev_hvm_map_pirq() will be called, that has 
not support for MSI.

>> So, to fix this issue either would be needed to change QEMU to not unmap
>> pirq-emulated MSIs or to implement PHYSDEVOP_map_pirq for MSIs.
>>
>> But still, even when no device has been passed-through, scheduling latencies
>> (of hundreds of ms), were observed in the guest even when running a simple
>> loop application, that disappear once the flag is disabled. We did not have
>> the chance to root cause it further.
> 
> So XENFEAT_hvm_pirqs is causing such latency issues?  That I certainly
> didn't notice.

We 've seen that in a setup with an HVM guest using emulated MSIs. We 
were running an application, like the one below, pinned on an isolated 
and pinned vcpu.

int main()
{
     struct timeval cur_time = {0}, prev_time = {0};
     int diff;

     gettimeofday(&prev_time, NULL);
     do {
         gettimeofday(&cur_time, NULL);

         diff = (((cur_time.tv_sec - prev_time.tv_sec)*1000) + ((cur_time
.tv_usec - prev_time.tv_usec)/1000));
         if (diff>10)
            printf("app scheduled after: %d msec\n",diff);

         gettimeofday(&prev_time, NULL);
     } while(1);
}

And we 're getting values of hundreds of ms like:
app scheduled after: 985 msec

> 
> Regards, Roger.