[report] boot a vm that with PCI only hierarchy devices and with GICv3 , it's failed.

[report] boot a vm that with PCI only hierarchy devices and with GICv3 , it's failed.

[wanghaibin]

Hi, all:

I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt  sections 2.3)

Here is part of qemu cmd:
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2
-device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
-vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4

A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it.  Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge.
Boot the vm, it's failed.

I try to debug this problem, and the info just as follow:
(1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed.
    Of course, I think this commit must be correct surely.

(2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration).  This commit brings in that the
    four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
    (Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count),
    So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.

Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.

Thanks.

Re: [report] boot a vm that with PCI only hierarchy devices and with GICv3 , it's failed.

[Marc Zyngier]

On 18/07/17 05:07, wanghaibin wrote:
> Hi, all:
> 
> I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt  sections 2.3)
> 
> Here is part of qemu cmd:
> -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2
> -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
> -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4
> 
> A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it.  Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge.
> Boot the vm, it's failed.
> 
> I try to debug this problem, and the info just as follow:
> (1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed.
>     Of course, I think this commit must be correct surely.
> 
> (2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration).  This commit brings in that the
>     four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
>     (Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count),
>     So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.
> 
> Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.

+Robin, who is the author of that patch.

Regarding (2), the number of MSIs should be the total number of devices
that are going to generate the same DevID. Since the bridge is
non-transparent, everything behind it aliases with it. So you should
probably see all the virtio devices and the bridges themselves being
counted. If that's not the case, then "we have a bug"(tm).

Can you please post your full qemu cmd line so that we can reproduce it
and investigate the issue?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [report] boot a vm that with PCI only hierarchy devices and with GICv3 , it's failed.

[Robin Murphy]

On 18/07/17 10:15, Marc Zyngier wrote:
> On 18/07/17 05:07, wanghaibin wrote:
>> Hi, all:
>>
>> I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt  sections 2.3)
>>
>> Here is part of qemu cmd:
>> -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2
>> -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
>> -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4
>>
>> A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it.  Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge.
>> Boot the vm, it's failed.

What's the nature of the failure? Does it hit some actual error case in
the GIC code, or does it simply hang up probing the virtio devices
because interrupts never arrive?

>> I try to debug this problem, and the info just as follow:
>> (1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed.
>>     Of course, I think this commit must be correct surely.
>>
>> (2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration).  This commit brings in that the
>>     four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
>>     (Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count),
>>     So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.

Now, at worst that patch *should* result in the same number of vectors
being reserved as before - never fewer. Does anything change with it
reverted?

>> Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.

Note that there are further issues with PCI RID to DevID mappings in the
face of aliases[1], but I think the current code does happen to work out
OK for the PCI-PCIe bridge case already.

> +Robin, who is the author of that patch.
> 
> Regarding (2), the number of MSIs should be the total number of devices
> that are going to generate the same DevID. Since the bridge is
> non-transparent, everything behind it aliases with it. So you should
> probably see all the virtio devices and the bridges themselves being
> counted. If that's not the case, then "we have a bug"(tm).
> 
> Can you please post your full qemu cmd line so that we can reproduce it
> and investigate the issue?

Yes, that would be good.

Robin.

[1] https://patchwork.ozlabs.org/patch/769303/

> Thanks,
> 
> 	M.
> 

Re: [report] boot a vm that with PCI only hierarchy devices and with GICv3 , it's failed.

[wanghaibin]

On 2017/7/18 18:02, Robin Murphy wrote:

> On 18/07/17 10:15, Marc Zyngier wrote:
>> On 18/07/17 05:07, wanghaibin wrote:
>>> Hi, all:
>>>
>>> I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt  sections 2.3)
>>>
>>> Here is part of qemu cmd:
>>> -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2
>>> -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
>>> -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4
>>>
>>> A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it.  Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge.
>>> Boot the vm, it's failed.
> 
> What's the nature of the failure? Does it hit some actual error case in
> the GIC code, or does it simply hang up probing the virtio devices
> because interrupts never arrive?


Qemu cmdline, xml info, qemu version info, guest kernel version info at the bottom of this mail.


Guest hang log:

[  242.740171] INFO: task kworker/u16:4:446 blocked for more than 120 seconds.
[  242.741102]       Not tainted 4.12.0+ #18
[  242.741619] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.742610] kworker/u16:4   D    0   446      2 0x00000000
[  242.743339] Workqueue: scsi_tmf_0 scmd_eh_abort_handler
[  242.744014] Call trace:
[  242.744375] [<ffff0000080856bc>] __switch_to+0x94/0xa8
[  242.745042] [<ffff00000892a3f8>] __schedule+0x1a0/0x5e0
[  242.745716] [<ffff00000892a870>] schedule+0x38/0xa0
[  242.746346] [<ffff00000892da84>] schedule_timeout+0x194/0x2b8
[  242.747092] [<ffff00000892b438>] wait_for_common+0xa0/0x148
[  242.747810] [<ffff00000892b4f4>] wait_for_completion+0x14/0x20
[  242.748595] [<ffff0000085e78d8>] virtscsi_tmf.constprop.15+0x88/0xf0
[  242.749408] [<ffff0000085e79dc>] virtscsi_abort+0x9c/0xb8
[  242.750099] [<ffff0000085db4dc>] scmd_eh_abort_handler+0x5c/0x108
[  242.750887] [<ffff0000080d8594>] process_one_work+0x124/0x2a8
[  242.751618] [<ffff0000080d8774>] worker_thread+0x5c/0x3d8
[  242.752330] [<ffff0000080de7b4>] kthread+0xfc/0x128
[  242.752960] [<ffff000008082ec0>] ret_from_fork+0x10/0x50

But I still doubt the total vector count takes this problem in, I add some log in guest:
In guest boot, pci_dev: 02:04:00(virtio-gpu-pci) first load, and only alloc 4 ITT entries, the log as follow:

[    0.986233] ~~its_pci_msi_prepare:pci dev: 2,32, nvec:3~~
[    0.998952] ~~its_pci_msi_prepare:devid:8,alias count:4~~
[    1.000028] **its_msi_prepare:devid:8, nves:4**
[    1.001001] ##its_create_device:devid: 8, ITT 4 entries, 2 bits, lpi base:8192, nr:32##
[    1.002585] **its_msi_prepare:ITT 4 entries, 2 bits**
[    1.003593] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.004880] ID:0 pID:8192 vID:52
[    1.005529] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.006777] ID:1 pID:8193 vID:53
[    1.007437] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.008718] ID:2 pID:8194 vID:54
[    1.009366] !!msi_domain_alloc_irqs: to active!!
[    1.010281] ^^^SEND mapti: hwirq:8192,event:0^^
[    1.011224] !!msi_domain_alloc_irqs: to active!!
[    1.012161] ^^^SEND mapti: hwirq:8193,event:1^^
[    1.013095] !!msi_domain_alloc_irqs: to active!!
[    1.014013] ^^^SEND mapti: hwirq:8194,event:2^^

and the guest booted continue, when load the pci_dev: 02:03:00 (virtio-scsi), the log shows it shared the same devid
with virtio-gpu-pci, shared ite_dev, reusing ITT. So that, the  virtio-gpu-pci dev only alloc 4 ITT, and the virtio-scsi send
mapti with eventid 5/6, this will be captured by Eric's commit:
guest log:
[    1.057978] !!msi_domain_alloc_irqs: to prepare: nvec:4!!
[    1.072773] ~~its_pci_msi_prepare:devid:8,alias count:5~~
[    1.073943] **its_msi_prepare:devid:8, nves:5**
[    1.074850] **its_msi_prepare:Reusing ITT for devID:8**
[    1.075873] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.077154] ID:3 pID:8195 vID:55
[    1.077813] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.079044] ID:4 pID:8196 vID:56
[    1.079683] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.080947] ID:5 pID:8197 vID:57
[    1.081592] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
[    1.082825] ID:6 pID:8198 vID:58


part of Eric's commit:

@@ -784,6 +788,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
        if (!device)
                return E_ITS_MAPTI_UNMAPPED_DEVICE;

+       if (event_id >= BIT_ULL(device->num_eventid_bits))
+               return E_ITS_MAPTI_ID_OOR;

Thanks!

> 
>>> I try to debug this problem, and the info just as follow:
>>> (1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed.
>>>     Of course, I think this commit must be correct surely.
>>>
>>> (2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration).  This commit brings in that the
>>>     four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
>>>     (Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count),
>>>     So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.
> 
> Now, at worst that patch *should* result in the same number of vectors
> being reserved as before - never fewer. Does anything change with it
> reverted?
> 
>>> Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.
> 
> Note that there are further issues with PCI RID to DevID mappings in the
> face of aliases[1], but I think the current code does happen to work out
> OK for the PCI-PCIe bridge case already.
> 
>> +Robin, who is the author of that patch.
>>
>> Regarding (2), the number of MSIs should be the total number of devices
>> that are going to generate the same DevID. Since the bridge is
>> non-transparent, everything behind it aliases with it. So you should
>> probably see all the virtio devices and the bridges themselves being
>> counted. If that's not the case, then "we have a bug"(tm).
>>
>> Can you please post your full qemu cmd line so that we can reproduce it
>> and investigate the issue?
> 
> Yes, that would be good.



I used the qemu version: 2.9.50,
guest linux version: Linux 4.12

xml :

<domain type='kvm' id='5'>
  <name>abu</name>
  <uuid>76365c65-7ee7-43ff-bb57-f0f80b75323a</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='aarch64' machine='virt-2.9'>hvm</type>
    <kernel>/mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image</kernel>
    <cmdline>console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw</cmdline>
    <boot dev='hd'/>
  </os>
  <features>
    <gic version='3'/>
  </features>
  <cpu mode='host-passthrough'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup' track='guest'/>
    <timer name='hpet' present='no'/>
    <timer name='pit' tickpolicy='delay'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/wanghaibin/vm-res/euler_b500.raw'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ehci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:60:6b:1d'/>
      <source bridge='br0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='c216af51-9088-4bf8-ba04-cf6172cb3753'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <driver name='vhost'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='usb'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </video>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>
  </seclabel>
</domain>

qemu cmdline :

/mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64 -name guest=abu,debug-threads=on -S
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-abu/master-key.aes -machine virt-2.9,accel=kvm,usb=off,gic-version=3
-cpu host -m 8192 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 76365c65-7ee7-43ff-bb57-f0f80b75323a -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-abu/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
-rtc base=utc,clock=vm,driftfix=slew -no-shutdown -boot strict=on -kernel /mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image
-append console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3
-drive file=/mnt/wanghaibin/vm-res/euler_b500.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
-serial pty -device usb-tablet,id=input0 -device usb-kbd,id=input1 -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4 -msg timestamp=on


Thanks

> 
> Robin.
> 
> [1] https://patchwork.ozlabs.org/patch/769303/
> 
>> Thanks,
>>
>> 	M.
>>
> 
> 
> .
> 

Re: [report] boot a vm that with PCI only hierarchy devices and with GICv3 , it's failed.

[Robin Murphy]

On 18/07/17 12:07, wanghaibin wrote:
> On 2017/7/18 18:02, Robin Murphy wrote:
> 
>> On 18/07/17 10:15, Marc Zyngier wrote:
>>> On 18/07/17 05:07, wanghaibin wrote:
>>>> Hi, all:
>>>>
>>>> I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt  sections 2.3)
>>>>
>>>> Here is part of qemu cmd:
>>>> -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2
>>>> -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
>>>> -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4
>>>>
>>>> A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it.  Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge.
>>>> Boot the vm, it's failed.
>>
>> What's the nature of the failure? Does it hit some actual error case in
>> the GIC code, or does it simply hang up probing the virtio devices
>> because interrupts never arrive?
> 
> 
> Qemu cmdline, xml info, qemu version info, guest kernel version info at the bottom of this mail.
> 
> 
> Guest hang log:
> 
> [  242.740171] INFO: task kworker/u16:4:446 blocked for more than 120 seconds.
> [  242.741102]       Not tainted 4.12.0+ #18
> [  242.741619] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [  242.742610] kworker/u16:4   D    0   446      2 0x00000000
> [  242.743339] Workqueue: scsi_tmf_0 scmd_eh_abort_handler
> [  242.744014] Call trace:
> [  242.744375] [<ffff0000080856bc>] __switch_to+0x94/0xa8
> [  242.745042] [<ffff00000892a3f8>] __schedule+0x1a0/0x5e0
> [  242.745716] [<ffff00000892a870>] schedule+0x38/0xa0
> [  242.746346] [<ffff00000892da84>] schedule_timeout+0x194/0x2b8
> [  242.747092] [<ffff00000892b438>] wait_for_common+0xa0/0x148
> [  242.747810] [<ffff00000892b4f4>] wait_for_completion+0x14/0x20
> [  242.748595] [<ffff0000085e78d8>] virtscsi_tmf.constprop.15+0x88/0xf0
> [  242.749408] [<ffff0000085e79dc>] virtscsi_abort+0x9c/0xb8
> [  242.750099] [<ffff0000085db4dc>] scmd_eh_abort_handler+0x5c/0x108
> [  242.750887] [<ffff0000080d8594>] process_one_work+0x124/0x2a8
> [  242.751618] [<ffff0000080d8774>] worker_thread+0x5c/0x3d8
> [  242.752330] [<ffff0000080de7b4>] kthread+0xfc/0x128
> [  242.752960] [<ffff000008082ec0>] ret_from_fork+0x10/0x50
> 
> But I still doubt the total vector count takes this problem in, I add some log in guest:
> In guest boot, pci_dev: 02:04:00(virtio-gpu-pci) first load, and only alloc 4 ITT entries, the log as follow:
> 
> [    0.986233] ~~its_pci_msi_prepare:pci dev: 2,32, nvec:3~~
> [    0.998952] ~~its_pci_msi_prepare:devid:8,alias count:4~~
> [    1.000028] **its_msi_prepare:devid:8, nves:4**
> [    1.001001] ##its_create_device:devid: 8, ITT 4 entries, 2 bits, lpi base:8192, nr:32##
> [    1.002585] **its_msi_prepare:ITT 4 entries, 2 bits**
> [    1.003593] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
> [    1.004880] ID:0 pID:8192 vID:52
> [    1.005529] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
> [    1.006777] ID:1 pID:8193 vID:53
> [    1.007437] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
> [    1.008718] ID:2 pID:8194 vID:54
> [    1.009366] !!msi_domain_alloc_irqs: to active!!
> [    1.010281] ^^^SEND mapti: hwirq:8192,event:0^^
> [    1.011224] !!msi_domain_alloc_irqs: to active!!
> [    1.012161] ^^^SEND mapti: hwirq:8193,event:1^^
> [    1.013095] !!msi_domain_alloc_irqs: to active!!
> [    1.014013] ^^^SEND mapti: hwirq:8194,event:2^^
> 
> and the guest booted continue, when load the pci_dev: 02:03:00 (virtio-scsi), the log shows it shared the same devid
> with virtio-gpu-pci, shared ite_dev, reusing ITT. So that, the  virtio-gpu-pci dev only alloc 4 ITT, and the virtio-scsi send
> mapti with eventid 5/6, this will be captured by Eric's commit:
> guest log:
> [    1.057978] !!msi_domain_alloc_irqs: to prepare: nvec:4!!
> [    1.072773] ~~its_pci_msi_prepare:devid:8,alias count:5~~
> [    1.073943] **its_msi_prepare:devid:8, nves:5**
> [    1.074850] **its_msi_prepare:Reusing ITT for devID:8**
> [    1.075873] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
> [    1.077154] ID:3 pID:8195 vID:55
> [    1.077813] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
> [    1.079044] ID:4 pID:8196 vID:56
> [    1.079683] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
> [    1.080947] ID:5 pID:8197 vID:57
> [    1.081592] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
> [    1.082825] ID:6 pID:8198 vID:58
> 
> 
> part of Eric's commit:
> 
> @@ -784,6 +788,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>         if (!device)
>                 return E_ITS_MAPTI_UNMAPPED_DEVICE;
> 
> +       if (event_id >= BIT_ULL(device->num_eventid_bits))
> +               return E_ITS_MAPTI_ID_OOR;
> 
> Thanks!
> 
>>
>>>> I try to debug this problem, and the info just as follow:
>>>> (1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed.
>>>>     Of course, I think this commit must be correct surely.
>>>>
>>>> (2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration).  This commit brings in that the
>>>>     four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
>>>>     (Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count),
>>>>     So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.
>>
>> Now, at worst that patch *should* result in the same number of vectors
>> being reserved as before - never fewer. Does anything change with it
>> reverted?
>>
>>>> Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.
>>
>> Note that there are further issues with PCI RID to DevID mappings in the
>> face of aliases[1], but I think the current code does happen to work out
>> OK for the PCI-PCIe bridge case already.
>>
>>> +Robin, who is the author of that patch.

Oops, only now do I actually notice - e8137f4f5088 is the *original*
commit trying to do the right thing, but incorrectly. My patch fixing it
is actually 3403b0259d15 ("irqchip/gic-v3-its: Fix MSI alias
accounting"), and isn't in 4.12, which would explain things here. I
expect that cherry-picking 3403b0259d15 should lead to success, since it
looks like this is a case of the exact problem it was intended to solve.

Thanks,
Robin.

>>> Regarding (2), the number of MSIs should be the total number of devices
>>> that are going to generate the same DevID. Since the bridge is
>>> non-transparent, everything behind it aliases with it. So you should
>>> probably see all the virtio devices and the bridges themselves being
>>> counted. If that's not the case, then "we have a bug"(tm).
>>>
>>> Can you please post your full qemu cmd line so that we can reproduce it
>>> and investigate the issue?
>>
>> Yes, that would be good.
> 
> 
> 
> I used the qemu version: 2.9.50,
> guest linux version: Linux 4.12
> 
> xml :
> 
> <domain type='kvm' id='5'>
>   <name>abu</name>
>   <uuid>76365c65-7ee7-43ff-bb57-f0f80b75323a</uuid>
>   <memory unit='KiB'>8388608</memory>
>   <currentMemory unit='KiB'>8388608</currentMemory>
>   <vcpu placement='static'>8</vcpu>
>   <resource>
>     <partition>/machine</partition>
>   </resource>
>   <os>
>     <type arch='aarch64' machine='virt-2.9'>hvm</type>
>     <kernel>/mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image</kernel>
>     <cmdline>console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw</cmdline>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <gic version='3'/>
>   </features>
>   <cpu mode='host-passthrough'/>
>   <clock offset='utc'>
>     <timer name='rtc' tickpolicy='catchup' track='guest'/>
>     <timer name='hpet' present='no'/>
>     <timer name='pit' tickpolicy='delay'/>
>   </clock>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>restart</on_crash>
>   <devices>
>     <emulator>/mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64</emulator>
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='raw' cache='none' io='native'/>
>       <source file='/mnt/wanghaibin/vm-res/euler_b500.raw'/>
>       <backingStore/>
>       <target dev='sda' bus='scsi'/>
>       <alias name='scsi0-0-0-0'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
>     <controller type='usb' index='0' model='ehci'>
>       <alias name='usb'/>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
>     </controller>
>     <controller type='scsi' index='0' model='virtio-scsi'>
>       <alias name='scsi0'/>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
>     </controller>
>     <controller type='pci' index='0' model='pcie-root'>
>       <alias name='pcie.0'/>
>     </controller>
>     <controller type='pci' index='1' model='dmi-to-pci-bridge'>
>       <model name='i82801b11-bridge'/>
>       <alias name='pci.1'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
>     </controller>
>     <controller type='pci' index='2' model='pci-bridge'>
>       <model name='pci-bridge'/>
>       <target chassisNr='2'/>
>       <alias name='pci.2'/>
>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>     </controller>
>     <interface type='bridge'>
>       <mac address='52:54:00:60:6b:1d'/>
>       <source bridge='br0'/>
>       <virtualport type='openvswitch'>
>         <parameters interfaceid='c216af51-9088-4bf8-ba04-cf6172cb3753'/>
>       </virtualport>
>       <target dev='vnet0'/>
>       <model type='virtio'/>
>       <driver name='vhost'/>
>       <alias name='net0'/>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <source path='/dev/pts/2'/>
>       <target port='0'/>
>       <alias name='serial0'/>
>     </serial>
>     <console type='pty' tty='/dev/pts/2'>
>       <source path='/dev/pts/2'/>
>       <target type='serial' port='0'/>
>       <alias name='serial0'/>
>     </console>
>     <input type='tablet' bus='usb'>
>       <alias name='input0'/>
>     </input>
>     <input type='keyboard' bus='usb'>
>       <alias name='input1'/>
>     </input>
>     <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
>       <listen type='address' address='0.0.0.0'/>
>     </graphics>
>     <video>
>       <model type='virtio' heads='1' primary='yes'/>
>       <alias name='video0'/>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
>     </video>
>   </devices>
>   <seclabel type='none' model='none'/>
>   <seclabel type='dynamic' model='dac' relabel='yes'>
>     <label>+0:+0</label>
>     <imagelabel>+0:+0</imagelabel>
>   </seclabel>
> </domain>
> 
> qemu cmdline :
> 
> /mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64 -name guest=abu,debug-threads=on -S
> -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-abu/master-key.aes -machine virt-2.9,accel=kvm,usb=off,gic-version=3
> -cpu host -m 8192 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 76365c65-7ee7-43ff-bb57-f0f80b75323a -no-user-config -nodefaults
> -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-abu/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
> -rtc base=utc,clock=vm,driftfix=slew -no-shutdown -boot strict=on -kernel /mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image
> -append console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1
> -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3
> -drive file=/mnt/wanghaibin/vm-res/euler_b500.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
> -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
> -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
> -serial pty -device usb-tablet,id=input0 -device usb-kbd,id=input1 -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4 -msg timestamp=on
> 
> 
> Thanks
> 
>>
>> Robin.
>>
>> [1] https://patchwork.ozlabs.org/patch/769303/
>>
>>> Thanks,
>>>
>>> 	M.
>>>
>>
>>
>> .
>>
> 
> 
> 

Re: [report] boot a vm that with PCI only hierarchy devices and with GICv3 , it's failed.

[wanghaibin]

On 2017/7/18 19:22, Robin Murphy wrote:

> On 18/07/17 12:07, wanghaibin wrote:
>> On 2017/7/18 18:02, Robin Murphy wrote:
>>
>>> On 18/07/17 10:15, Marc Zyngier wrote:
>>>> On 18/07/17 05:07, wanghaibin wrote:
>>>>> Hi, all:
>>>>>
>>>>> I met a problem, I just try to test PCI only hierarchy devices model (qemu/docs/pcie.txt  sections 2.3)
>>>>>
>>>>> Here is part of qemu cmd:
>>>>> -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2
>>>>> -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
>>>>> -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4
>>>>>
>>>>> A single DMI-PCI Bridge, a single PCI-PCI Bridge attached to it.  Four PCI_DEV legacy devices (usb, virtio-scsi-pci, virtio-gpu-pci, virtio-net-pci)attached to the PCI-PCI Bridge.
>>>>> Boot the vm, it's failed.
>>>
>>> What's the nature of the failure? Does it hit some actual error case in
>>> the GIC code, or does it simply hang up probing the virtio devices
>>> because interrupts never arrive?
>>
>>
>> Qemu cmdline, xml info, qemu version info, guest kernel version info at the bottom of this mail.
>>
>>
>> Guest hang log:
>>
>> [  242.740171] INFO: task kworker/u16:4:446 blocked for more than 120 seconds.
>> [  242.741102]       Not tainted 4.12.0+ #18
>> [  242.741619] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> [  242.742610] kworker/u16:4   D    0   446      2 0x00000000
>> [  242.743339] Workqueue: scsi_tmf_0 scmd_eh_abort_handler
>> [  242.744014] Call trace:
>> [  242.744375] [<ffff0000080856bc>] __switch_to+0x94/0xa8
>> [  242.745042] [<ffff00000892a3f8>] __schedule+0x1a0/0x5e0
>> [  242.745716] [<ffff00000892a870>] schedule+0x38/0xa0
>> [  242.746346] [<ffff00000892da84>] schedule_timeout+0x194/0x2b8
>> [  242.747092] [<ffff00000892b438>] wait_for_common+0xa0/0x148
>> [  242.747810] [<ffff00000892b4f4>] wait_for_completion+0x14/0x20
>> [  242.748595] [<ffff0000085e78d8>] virtscsi_tmf.constprop.15+0x88/0xf0
>> [  242.749408] [<ffff0000085e79dc>] virtscsi_abort+0x9c/0xb8
>> [  242.750099] [<ffff0000085db4dc>] scmd_eh_abort_handler+0x5c/0x108
>> [  242.750887] [<ffff0000080d8594>] process_one_work+0x124/0x2a8
>> [  242.751618] [<ffff0000080d8774>] worker_thread+0x5c/0x3d8
>> [  242.752330] [<ffff0000080de7b4>] kthread+0xfc/0x128
>> [  242.752960] [<ffff000008082ec0>] ret_from_fork+0x10/0x50
>>
>> But I still doubt the total vector count takes this problem in, I add some log in guest:
>> In guest boot, pci_dev: 02:04:00(virtio-gpu-pci) first load, and only alloc 4 ITT entries, the log as follow:
>>
>> [    0.986233] ~~its_pci_msi_prepare:pci dev: 2,32, nvec:3~~
>> [    0.998952] ~~its_pci_msi_prepare:devid:8,alias count:4~~
>> [    1.000028] **its_msi_prepare:devid:8, nves:4**
>> [    1.001001] ##its_create_device:devid: 8, ITT 4 entries, 2 bits, lpi base:8192, nr:32##
>> [    1.002585] **its_msi_prepare:ITT 4 entries, 2 bits**
>> [    1.003593] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
>> [    1.004880] ID:0 pID:8192 vID:52
>> [    1.005529] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
>> [    1.006777] ID:1 pID:8193 vID:53
>> [    1.007437] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
>> [    1.008718] ID:2 pID:8194 vID:54
>> [    1.009366] !!msi_domain_alloc_irqs: to active!!
>> [    1.010281] ^^^SEND mapti: hwirq:8192,event:0^^
>> [    1.011224] !!msi_domain_alloc_irqs: to active!!
>> [    1.012161] ^^^SEND mapti: hwirq:8193,event:1^^
>> [    1.013095] !!msi_domain_alloc_irqs: to active!!
>> [    1.014013] ^^^SEND mapti: hwirq:8194,event:2^^
>>
>> and the guest booted continue, when load the pci_dev: 02:03:00 (virtio-scsi), the log shows it shared the same devid
>> with virtio-gpu-pci, shared ite_dev, reusing ITT. So that, the  virtio-gpu-pci dev only alloc 4 ITT, and the virtio-scsi send
>> mapti with eventid 5/6, this will be captured by Eric's commit:
>> guest log:
>> [    1.057978] !!msi_domain_alloc_irqs: to prepare: nvec:4!!
>> [    1.072773] ~~its_pci_msi_prepare:devid:8,alias count:5~~
>> [    1.073943] **its_msi_prepare:devid:8, nves:5**
>> [    1.074850] **its_msi_prepare:Reusing ITT for devID:8**
>> [    1.075873] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
>> [    1.077154] ID:3 pID:8195 vID:55
>> [    1.077813] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
>> [    1.079044] ID:4 pID:8196 vID:56
>> [    1.079683] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
>> [    1.080947] ID:5 pID:8197 vID:57
>> [    1.081592] !!msi_domain_alloc_irqs: to alloc: desc->nvec_used:1!!
>> [    1.082825] ID:6 pID:8198 vID:58
>>
>>
>> part of Eric's commit:
>>
>> @@ -784,6 +788,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>         if (!device)
>>                 return E_ITS_MAPTI_UNMAPPED_DEVICE;
>>
>> +       if (event_id >= BIT_ULL(device->num_eventid_bits))
>> +               return E_ITS_MAPTI_ID_OOR;
>>
>> Thanks!
>>
>>>
>>>>> I try to debug this problem, and the info just as follow:
>>>>> (1) Since Eric Auger commit (0d44cdb631ef53ea75be056886cf0541311e48df: KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors), This problem has been exposed.
>>>>>     Of course, I think this commit must be correct surely.
>>>>>
>>>>> (2) For guestOS, I notice Marc commit (e8137f4f5088d763ced1db82d3974336b76e1bd2: irqchip: gicv3-its: Iterate over PCI aliases to generate ITS configuration).  This commit brings in that the
>>>>>     four PCI_DEV legacy devices shared the same devID, same its_dev, same ITT tables, but I think here calculate with wrong total msi vector count.
>>>>>     (Currently, It seems the total count is the vector count of virtio-net-pci + PCI-PCI bridge + dmi-pci bridge, maybe here should be the total count of the four PCI_DEV legacy devices vector count),
>>>>>     So that, any pci device using the over bounds eventID and mapti at a certain moment , the abnormal behavior will captured by Eric's commit.
>>>
>>> Now, at worst that patch *should* result in the same number of vectors
>>> being reserved as before - never fewer. Does anything change with it
>>> reverted?
>>>
>>>>> Actually, I don't understand very well about non-transparent bridge, PCI aliases. So just supply these message.
>>>
>>> Note that there are further issues with PCI RID to DevID mappings in the
>>> face of aliases[1], but I think the current code does happen to work out
>>> OK for the PCI-PCIe bridge case already.
>>>
>>>> +Robin, who is the author of that patch.
> 
> Oops, only now do I actually notice - e8137f4f5088 is the *original*
> commit trying to do the right thing, but incorrectly. My patch fixing it
> is actually 3403b0259d15 ("irqchip/gic-v3-its: Fix MSI alias
> accounting"), and isn't in 4.12, which would explain things here. I
> expect that cherry-picking 3403b0259d15 should lead to success, since it
> looks like this is a case of the exact problem it was intended to solve.
> 
> Thanks,
> Robin.


Already tested, everything is OK.

Thanks very much!

> 
>>>> Regarding (2), the number of MSIs should be the total number of devices
>>>> that are going to generate the same DevID. Since the bridge is
>>>> non-transparent, everything behind it aliases with it. So you should
>>>> probably see all the virtio devices and the bridges themselves being
>>>> counted. If that's not the case, then "we have a bug"(tm).
>>>>
>>>> Can you please post your full qemu cmd line so that we can reproduce it
>>>> and investigate the issue?
>>>
>>> Yes, that would be good.
>>
>>
>>
>> I used the qemu version: 2.9.50,
>> guest linux version: Linux 4.12
>>
>> xml :
>>
>> <domain type='kvm' id='5'>
>>   <name>abu</name>
>>   <uuid>76365c65-7ee7-43ff-bb57-f0f80b75323a</uuid>
>>   <memory unit='KiB'>8388608</memory>
>>   <currentMemory unit='KiB'>8388608</currentMemory>
>>   <vcpu placement='static'>8</vcpu>
>>   <resource>
>>     <partition>/machine</partition>
>>   </resource>
>>   <os>
>>     <type arch='aarch64' machine='virt-2.9'>hvm</type>
>>     <kernel>/mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image</kernel>
>>     <cmdline>console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw</cmdline>
>>     <boot dev='hd'/>
>>   </os>
>>   <features>
>>     <gic version='3'/>
>>   </features>
>>   <cpu mode='host-passthrough'/>
>>   <clock offset='utc'>
>>     <timer name='rtc' tickpolicy='catchup' track='guest'/>
>>     <timer name='hpet' present='no'/>
>>     <timer name='pit' tickpolicy='delay'/>
>>   </clock>
>>   <on_poweroff>destroy</on_poweroff>
>>   <on_reboot>restart</on_reboot>
>>   <on_crash>restart</on_crash>
>>   <devices>
>>     <emulator>/mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64</emulator>
>>     <disk type='file' device='disk'>
>>       <driver name='qemu' type='raw' cache='none' io='native'/>
>>       <source file='/mnt/wanghaibin/vm-res/euler_b500.raw'/>
>>       <backingStore/>
>>       <target dev='sda' bus='scsi'/>
>>       <alias name='scsi0-0-0-0'/>
>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>     </disk>
>>     <controller type='usb' index='0' model='ehci'>
>>       <alias name='usb'/>
>>       <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
>>     </controller>
>>     <controller type='scsi' index='0' model='virtio-scsi'>
>>       <alias name='scsi0'/>
>>       <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
>>     </controller>
>>     <controller type='pci' index='0' model='pcie-root'>
>>       <alias name='pcie.0'/>
>>     </controller>
>>     <controller type='pci' index='1' model='dmi-to-pci-bridge'>
>>       <model name='i82801b11-bridge'/>
>>       <alias name='pci.1'/>
>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
>>     </controller>
>>     <controller type='pci' index='2' model='pci-bridge'>
>>       <model name='pci-bridge'/>
>>       <target chassisNr='2'/>
>>       <alias name='pci.2'/>
>>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>>     </controller>
>>     <interface type='bridge'>
>>       <mac address='52:54:00:60:6b:1d'/>
>>       <source bridge='br0'/>
>>       <virtualport type='openvswitch'>
>>         <parameters interfaceid='c216af51-9088-4bf8-ba04-cf6172cb3753'/>
>>       </virtualport>
>>       <target dev='vnet0'/>
>>       <model type='virtio'/>
>>       <driver name='vhost'/>
>>       <alias name='net0'/>
>>       <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
>>     </interface>
>>     <serial type='pty'>
>>       <source path='/dev/pts/2'/>
>>       <target port='0'/>
>>       <alias name='serial0'/>
>>     </serial>
>>     <console type='pty' tty='/dev/pts/2'>
>>       <source path='/dev/pts/2'/>
>>       <target type='serial' port='0'/>
>>       <alias name='serial0'/>
>>     </console>
>>     <input type='tablet' bus='usb'>
>>       <alias name='input0'/>
>>     </input>
>>     <input type='keyboard' bus='usb'>
>>       <alias name='input1'/>
>>     </input>
>>     <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
>>       <listen type='address' address='0.0.0.0'/>
>>     </graphics>
>>     <video>
>>       <model type='virtio' heads='1' primary='yes'/>
>>       <alias name='video0'/>
>>       <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
>>     </video>
>>   </devices>
>>   <seclabel type='none' model='none'/>
>>   <seclabel type='dynamic' model='dac' relabel='yes'>
>>     <label>+0:+0</label>
>>     <imagelabel>+0:+0</imagelabel>
>>   </seclabel>
>> </domain>
>>
>> qemu cmdline :
>>
>> /mnt/wanghaibin/vm-res/src/open-sorce/qemu/aarch64-softmmu/qemu-system-aarch64 -name guest=abu,debug-threads=on -S
>> -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-abu/master-key.aes -machine virt-2.9,accel=kvm,usb=off,gic-version=3
>> -cpu host -m 8192 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 76365c65-7ee7-43ff-bb57-f0f80b75323a -no-user-config -nodefaults
>> -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-abu/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
>> -rtc base=utc,clock=vm,driftfix=slew -no-shutdown -boot strict=on -kernel /mnt/wanghaibin/vm-res/src/open-sorce/linux-stable/arch/arm64/boot/Image
>> -append console=ttyAMA0 root=/dev/sda2 earlyprintk=pl011,0x9000000 rw -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1
>> -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device usb-ehci,id=usb,bus=pci.2,addr=0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x3
>> -drive file=/mnt/wanghaibin/vm-res/euler_b500.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
>> -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
>> -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:60:6b:1d,bus=pci.2,addr=0x1
>> -serial pty -device usb-tablet,id=input0 -device usb-kbd,id=input1 -vnc 0.0.0.0:0 -device virtio-gpu-pci,id=video0,bus=pci.2,addr=0x4 -msg timestamp=on
>>
>>
>> Thanks
>>
>>>
>>> Robin.
>>>
>>> [1] https://patchwork.ozlabs.org/patch/769303/
>>>
>>>> Thanks,
>>>>
>>>> 	M.
>>>>
>>>
>>>
>>> .
>>>
>>
>>
>>
> 
> 
> .
>