passthroughed msix device

passthroughed msix device

[DuanZhenzhong]

Hi maintainer,

Reprodue an bug on xen-unstable, it's an irq affinity issue for 
passthroughed msix device to uek1 pvhvm(2.6.32 stable).
I passthrough two mptsas devices(0000:0d:0.0, 0000:1f:0.0) to a pvhvm, 
irq affinity can't be changed.
Step to reproduce is as below:
1. xl -f pci-assignable-add 0000:0d:0.0; xl -f pci-assignable-add 
0000:1f:0.0
2.xm cr -c vm.cfg
[root@rac10box2 ~]# cat /proc/interrupts |grep mpt2sas0
 48:     340449          0          0          0          0 0          
0          0          0          0          0 0   PCI-MSI-edge      
mpt2sas0-msix0
[root@rac10box2 ~]# cat /proc/irq/48/smp_affinity
0fff
[root@rac10box2 ~]# echo 2>/proc/irq/48/smp_affinity
[root@rac10box2 ~]# cat /proc/irq/48/smp_affinity
0002
[root@rac10box2 ~]# cat /proc/interrupts |grep mpt
 48:     342051        0          0          0          0 0          
0          0          0          0          0 0   PCI-MSI-edge      
mpt2sas0-msix0
After change affinity to vcpu1, interrupt still happen on vcpu0. And 
sometimes we got "/No irq handler for vector (irq -1)/" and panic.
qemu-dm.log shows error:
pt_msixctrl_reg_write: guest enabling MSI-X, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_disable: Unmap msi with pirq 4f
pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
pt_msixctrl_reg_write: guest enabling MSI-X, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_disable: Unmap msi with pirq 4e
pt_msix_update_one: Update msix entry 0 with pirq 4e gvec 69
pci_msix_writel: Can't update entry 0 since MSI-X is already enabled 
(fee00000 -> fee02000)
pci_msix_writel: Can't update entry 0 since MSI-X is already enabled 
(00004059 -> 00004071)

Can't reproduce with uek2(3.1 stable), but if I disable hvm_pirqs 
support in uek2, reproduce the same.
Test patch below:
diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c
index a22ad4b..2d795d1 100644
--- a/arch/x86/pci/xen.c
+++ b/arch/x86/pci/xen.c
@-400,7 +400,6 @int __init pci_xen_init(void)
.
 int __init pci_xen_hvm_init(void)
 {
-       if (!xen_have_vector_callback || !xen_feature(XENFEAT_hvm_pirqs))
                return 0;
.
 #ifdef CONFIG_ACPI
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index bf28d69..64f6c6c 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@-1536,8 +1536,6 @bool xen_hvm_need_lapic(void)
                return false;
        if (!xen_hvm_domain())
                return false;
-       if (xen_feature(XENFEAT_hvm_pirqs) && xen_have_vector_callback)
-               return false;
        return true;
 }
 EXPORT_SYMBOL_GPL(xen_hvm_need_lapic);

btw: can't reproduce too if disable msi support.
I'll upload what you need, just tell me.

thanks
zduan

Re: passthroughed msix device

[Jan Beulich]

>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote:
> pci_msix_writel: Can't update entry 0 since MSI-X is already enabled 
> (fee00000 -> fee02000)
> pci_msix_writel: Can't update entry 0 since MSI-X is already enabled 
> (00004059 -> 00004071)

If you look at the code issuing this message, the situation is
pretty clear (and I think it as described already in the past,
albeit I have no link at hand): qemu lacks proper emulation of
the mask bit. pci_msix_write() looks at the physical one, yet
when the guest sets the virtual mask bit, nothing is being
done at all to make the hypervisor also mask the physical
entry:

    if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) {
        if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) {
            xen_pt_msix_update_one(s, entry_nr);
        }
    }

There's probably quite a bit of code to be written to make this
work.

Jan

Re: passthroughed msix device

[Zhenzhong Duan]

On 2013-02-26 18:08, Jan Beulich wrote:
>>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote:
>> pci_msix_writel: Can't update entry 0 since MSI-X is already enabled
>> (fee00000 -> fee02000)
>> pci_msix_writel: Can't update entry 0 since MSI-X is already enabled
>> (00004059 -> 00004071)
> If you look at the code issuing this message, the situation is
> pretty clear (and I think it as described already in the past,
> albeit I have no link at hand): qemu lacks proper emulation of
> the mask bit. pci_msix_write() looks at the physical one, yet
> when the guest sets the virtual mask bit, nothing is being
> done at all to make the hypervisor also mask the physical
> entry:
>
>      if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) {
>          if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) {
>              xen_pt_msix_update_one(s, entry_nr);
>          }
>      }
>
> There's probably quite a bit of code to be written to make this
> work.
Is there plan of fixing it?

thanks
zduan

Re: passthroughed msix device

[Jan Beulich]

>>> On 27.02.13 at 06:49, Zhenzhong Duan <zhenzhong.duan@oracle.com> wrote:

> On 2013-02-26 18:08, Jan Beulich wrote:
>>>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote:
>>> pci_msix_writel: Can't update entry 0 since MSI-X is already enabled
>>> (fee00000 -> fee02000)
>>> pci_msix_writel: Can't update entry 0 since MSI-X is already enabled
>>> (00004059 -> 00004071)
>> If you look at the code issuing this message, the situation is
>> pretty clear (and I think it as described already in the past,
>> albeit I have no link at hand): qemu lacks proper emulation of
>> the mask bit. pci_msix_write() looks at the physical one, yet
>> when the guest sets the virtual mask bit, nothing is being
>> done at all to make the hypervisor also mask the physical
>> entry:
>>
>>      if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) {
>>          if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) {
>>              xen_pt_msix_update_one(s, entry_nr);
>>          }
>>      }
>>
>> There's probably quite a bit of code to be written to make this
>> work.
> Is there plan of fixing it?

I'm not aware of anyone working on this, or having planned to.
Want to take a shot?

Jan

Re: passthroughed msix device

[Zhenzhong Duan]

On 2013-02-27 16:36, Jan Beulich wrote:
>>>> On 27.02.13 at 06:49, Zhenzhong Duan <zhenzhong.duan@oracle.com> wrote:
>> On 2013-02-26 18:08, Jan Beulich wrote:
>>>>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote:
>>>> pci_msix_writel: Can't update entry 0 since MSI-X is already enabled
>>>> (fee00000 -> fee02000)
>>>> pci_msix_writel: Can't update entry 0 since MSI-X is already enabled
>>>> (00004059 -> 00004071)
>>> If you look at the code issuing this message, the situation is
>>> pretty clear (and I think it as described already in the past,
>>> albeit I have no link at hand): qemu lacks proper emulation of
>>> the mask bit. pci_msix_write() looks at the physical one, yet
>>> when the guest sets the virtual mask bit, nothing is being
>>> done at all to make the hypervisor also mask the physical
>>> entry:
>>>
>>>       if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) {
>>>           if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) {
>>>               xen_pt_msix_update_one(s, entry_nr);
>>>           }
>>>       }
>>>
>>> There's probably quite a bit of code to be written to make this
>>> work.
>> Is there plan of fixing it?
> I'm not aware of anyone working on this, or having planned to.
> Want to take a shot?
I'm looking at it.
zduan