[QUESTION] Exception print when enabling GICv4

[QUESTION] Exception print when enabling GICv4

[chenxiang (M) via]

Hi,
I encounter a issue related to GICv4 enable on ARM64 platform (kernel 
5.19-rc4, qemu 6.2.0):
We have a accelaration module whose VF has 3 MSI interrupts, and we 
passthrough it to virtual machine with following steps:

echo 0000:79:00.1 > /sys/bus/pci/drivers/hisi_hpre/unbind
echo vfio-pci > 
/sys/devices/pci0000\:78/0000\:78\:00.0/0000\:79\:00.1/driver_override
echo 0000:79:00.1 > /sys/bus/pci/drivers_probe

Then we boot VM with "-device vfio-pci,host=79:00.1,id=net0 \".
When insmod the driver which registers 3 PCI MSI interrupts in VM,  some 
exception print occur as following:

vfio-pci 0000:3a:00.1: irq bypass producer (token 000000008f08224d) 
registration fails: 66311

I find that bit[6:4] of register PCI_MSI_FLAGS is 2 (4 MSI interrupts) 
though we only register 3 PCI MSI interrupt,

and only 3 MSI interrupt is activated at last.
It allocates 4 vectors in function vfio_msi_enable() (qemu)  as it reads 
the register PCI_MSI_FLAGS.
Later it will  call system call VFIO_DEVICE_SET_IRQS to set forwarding 
for those interrupts
using function kvm_vgic_v4_set_forrwarding() as GICv4 is enabled. For 
interrupt 0~2, it success to set forwarding as they are already activated,
but for the 4th interrupt, it is not activated, so ite is not found in 
function vgic_its_resolve_lpi(), so above printk occurs.

It seems that we only allocate and activate 3 MSI interrupts in guest 
while it tried to set forwarding for 4 MSI interrupts in host.
Do you have any idea about this issue?


Best regards,

Xiang Chen

Re: [QUESTION] Exception print when enabling GICv4

[Marc Zyngier]

Hi Xiang,

On Tue, 12 Jul 2022 13:55:16 +0100,
"chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
> 
> Hi,
> I encounter a issue related to GICv4 enable on ARM64 platform (kernel
> 5.19-rc4, qemu 6.2.0):
> We have a accelaration module whose VF has 3 MSI interrupts, and we
> passthrough it to virtual machine with following steps:
> 
> echo 0000:79:00.1 > /sys/bus/pci/drivers/hisi_hpre/unbind
> echo vfio-pci >
> /sys/devices/pci0000\:78/0000\:78\:00.0/0000\:79\:00.1/driver_override
> echo 0000:79:00.1 > /sys/bus/pci/drivers_probe
> 
> Then we boot VM with "-device vfio-pci,host=79:00.1,id=net0 \".
> When insmod the driver which registers 3 PCI MSI interrupts in VM,
> some exception print occur as following:
> 
> vfio-pci 0000:3a:00.1: irq bypass producer (token 000000008f08224d)
> registration fails: 66311
> 
> I find that bit[6:4] of register PCI_MSI_FLAGS is 2 (4 MSI interrupts)
> though we only register 3 PCI MSI interrupt,
>
> and only 3 MSI interrupt is activated at last.
> It allocates 4 vectors in function vfio_msi_enable() (qemu)  as it
> reads the register PCI_MSI_FLAGS.
> Later it will  call system call VFIO_DEVICE_SET_IRQS to set forwarding
> for those interrupts
> using function kvm_vgic_v4_set_forrwarding() as GICv4 is enabled. For
> interrupt 0~2, it success to set forwarding as they are already
> activated,
> but for the 4th interrupt, it is not activated, so ite is not found in
> function vgic_its_resolve_lpi(), so above printk occurs.
> 
> It seems that we only allocate and activate 3 MSI interrupts in guest
> while it tried to set forwarding for 4 MSI interrupts in host.
> Do you have any idea about this issue?

I have a hunch: QEMU cannot know that the guest is only using 3 MSIs
out of the 4 that the device can use, and PCI/Multi-MSI only has a
single enable bit for all MSIs. So it probably iterates over all
possible MSIs and enable the forwarding. Since the guest has only
created 3 mappings in the virtual ITS, the last call fails. I would
expect the guest to still work properly though.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [QUESTION] Exception print when enabling GICv4

[chenxiang (M) via]

Hi Marc,

Thank you for your reply.

在 2022/7/12 23:25, Marc Zyngier 写道:
> Hi Xiang,
>
> On Tue, 12 Jul 2022 13:55:16 +0100,
> "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
>> Hi,
>> I encounter a issue related to GICv4 enable on ARM64 platform (kernel
>> 5.19-rc4, qemu 6.2.0):
>> We have a accelaration module whose VF has 3 MSI interrupts, and we
>> passthrough it to virtual machine with following steps:
>>
>> echo 0000:79:00.1 > /sys/bus/pci/drivers/hisi_hpre/unbind
>> echo vfio-pci >
>> /sys/devices/pci0000\:78/0000\:78\:00.0/0000\:79\:00.1/driver_override
>> echo 0000:79:00.1 > /sys/bus/pci/drivers_probe
>>
>> Then we boot VM with "-device vfio-pci,host=79:00.1,id=net0 \".
>> When insmod the driver which registers 3 PCI MSI interrupts in VM,
>> some exception print occur as following:
>>
>> vfio-pci 0000:3a:00.1: irq bypass producer (token 000000008f08224d)
>> registration fails: 66311
>>
>> I find that bit[6:4] of register PCI_MSI_FLAGS is 2 (4 MSI interrupts)
>> though we only register 3 PCI MSI interrupt,
>>
>> and only 3 MSI interrupt is activated at last.
>> It allocates 4 vectors in function vfio_msi_enable() (qemu)  as it
>> reads the register PCI_MSI_FLAGS.
>> Later it will  call system call VFIO_DEVICE_SET_IRQS to set forwarding
>> for those interrupts
>> using function kvm_vgic_v4_set_forrwarding() as GICv4 is enabled. For
>> interrupt 0~2, it success to set forwarding as they are already
>> activated,
>> but for the 4th interrupt, it is not activated, so ite is not found in
>> function vgic_its_resolve_lpi(), so above printk occurs.
>>
>> It seems that we only allocate and activate 3 MSI interrupts in guest
>> while it tried to set forwarding for 4 MSI interrupts in host.
>> Do you have any idea about this issue?
> I have a hunch: QEMU cannot know that the guest is only using 3 MSIs
> out of the 4 that the device can use, and PCI/Multi-MSI only has a
> single enable bit for all MSIs. So it probably iterates over all
> possible MSIs and enable the forwarding. Since the guest has only
> created 3 mappings in the virtual ITS, the last call fails. I would
> expect the guest to still work properly though.

Yes, that's the reason of exception print.
Is it possible for QEMU to get the exact number of interrupts guest is 
using? It seems not.

>
> Thanks,
>
> 	M.
>

Re: [QUESTION] Exception print when enabling GICv4

[Marc Zyngier]

On Wed, 13 Jul 2022 07:02:10 +0100,
"chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
> 
> Hi Marc,
> 
> Thank you for your reply.
> 
> 在 2022/7/12 23:25, Marc Zyngier 写道:
> > Hi Xiang,
> > 
> > On Tue, 12 Jul 2022 13:55:16 +0100,
> > "chenxiang (M)" <chenxiang66@hisilicon.com> wrote:
> >> Hi,
> >> I encounter a issue related to GICv4 enable on ARM64 platform (kernel
> >> 5.19-rc4, qemu 6.2.0):
> >> We have a accelaration module whose VF has 3 MSI interrupts, and we
> >> passthrough it to virtual machine with following steps:
> >> 
> >> echo 0000:79:00.1 > /sys/bus/pci/drivers/hisi_hpre/unbind
> >> echo vfio-pci >
> >> /sys/devices/pci0000\:78/0000\:78\:00.0/0000\:79\:00.1/driver_override
> >> echo 0000:79:00.1 > /sys/bus/pci/drivers_probe
> >> 
> >> Then we boot VM with "-device vfio-pci,host=79:00.1,id=net0 \".
> >> When insmod the driver which registers 3 PCI MSI interrupts in VM,
> >> some exception print occur as following:
> >> 
> >> vfio-pci 0000:3a:00.1: irq bypass producer (token 000000008f08224d)
> >> registration fails: 66311
> >> 
> >> I find that bit[6:4] of register PCI_MSI_FLAGS is 2 (4 MSI interrupts)
> >> though we only register 3 PCI MSI interrupt,
> >> 
> >> and only 3 MSI interrupt is activated at last.
> >> It allocates 4 vectors in function vfio_msi_enable() (qemu)  as it
> >> reads the register PCI_MSI_FLAGS.
> >> Later it will  call system call VFIO_DEVICE_SET_IRQS to set forwarding
> >> for those interrupts
> >> using function kvm_vgic_v4_set_forrwarding() as GICv4 is enabled. For
> >> interrupt 0~2, it success to set forwarding as they are already
> >> activated,
> >> but for the 4th interrupt, it is not activated, so ite is not found in
> >> function vgic_its_resolve_lpi(), so above printk occurs.
> >> 
> >> It seems that we only allocate and activate 3 MSI interrupts in guest
> >> while it tried to set forwarding for 4 MSI interrupts in host.
> >> Do you have any idea about this issue?
> > I have a hunch: QEMU cannot know that the guest is only using 3 MSIs
> > out of the 4 that the device can use, and PCI/Multi-MSI only has a
> > single enable bit for all MSIs. So it probably iterates over all
> > possible MSIs and enable the forwarding. Since the guest has only
> > created 3 mappings in the virtual ITS, the last call fails. I would
> > expect the guest to still work properly though.
> 
> Yes, that's the reason of exception print.
> Is it possible for QEMU to get the exact number of interrupts guest is
> using? It seems not.

Not really. Or rather, this is a pretty involved process: you'd need
to stop the guest, perform a save operation on the ITS (as if you were
doing a migration), and then introspect the ITS tables to find whether
there is a mapping for each of the possible events generated by the
device. Clearly, that's overkill.

A better approach would be to be able to retrieve an individual
mapping, using a new API that would be similar to KVM_SIGNAL_MSI. It
would take the same kvm_msi structure as input, and retrieving the
{LPI, CPU} pair or an error if there is no mapping.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.