* Cannot use more than one MSI interrupts
@ 2019-10-24 14:45 Thanos Makatos
2019-10-24 15:02 ` Alex Williamson
0 siblings, 1 reply; 2+ messages in thread
From: Thanos Makatos @ 2019-10-24 14:45 UTC (permalink / raw)
To: qemu-devel@nongnu.org; +Cc: Alex Williamson, Felipe Franciosi
I have a Ubuntu VM (4.15.0-48-generic) to which I pass through a PCI device,
specifically a virtual NVMe controller. The problem I have is that only one I/O queue
is initialized, while there should be more (e.g. four). I'm using upstream QEMU
v4.1.0 confgiured without any additional options. Most likely there's something
broken with my virtual device implementation but I can't figure out exactly
what, I was hoping to get some debugging directions.
I run QEMU as follows:
~/src/qemu/x86_64-softmmu/qemu-system-x86_64 \
-kernel bionic-server-cloudimg-amd64-vmlinuz-generic \
-smp cores=2,sockets=2 \
-nographic \
-append "console=ttyS0 root=/dev/sda1 single nvme.sgl_threshold=0 nokaslr nvme.io_queue_depth=4" \
-initrd bionic-server-cloudimg-amd64-initrd-generic \
-hda bionic-server-cloudimg-amd64.raw \
-hdb data.raw \
-m 1G \
-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=mem,share=yes,size=1073741824 -numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \
-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000000-0000-0000-0000-000000000000 \
-trace enable=vfio*,file=qemu.trace \
-net none \
-s
This is what QEMU thinks of the devices:
(qemu) info pci
Bus 0, device 0, function 0:
Host bridge: PCI device 8086:1237
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 0:
ISA bridge: PCI device 8086:7000
PCI subsystem 1af4:1100
id ""
Bus 0, device 1, function 1:
IDE controller: PCI device 8086:7010
PCI subsystem 1af4:1100
BAR4: I/O at 0xc000 [0xc00f].
id ""
Bus 0, device 1, function 3:
Bridge: PCI device 8086:7113
PCI subsystem 1af4:1100
IRQ 9.
id ""
Bus 0, device 2, function 0:
VGA controller: PCI device 1234:1111
PCI subsystem 1af4:1100
BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff].
BAR2: 32 bit memory at 0xfebf4000 [0xfebf4fff].
BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
id ""
Bus 0, device 3, function 0:
Class 0264: PCI device 4e58:0001
PCI subsystem 0000:0000
IRQ 11.
BAR0: 32 bit memory at 0xfebf0000 [0xfebf3fff].
id ""
And this is what the guest thinks of the device in question:
root@ubuntu:~# lspci -vvv -s 00:03.0
00:03.0 Non-Volatile memory controller: Device 4e58:0001 (prog-if 02 [NVM Express])
Physical Slot: 3
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 24
NUMA node: 0
Region 0: Memory at febf0000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 0
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/4 Maskable- 64bit+
Address: 00000000fee01004 Data: 4023
Capabilities: [60] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn+ AttnInd- PwrInd- RBE- FLReset- SlotPowerLimit 128.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed unknown, Width x0, ASPM not supported, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
Kernel driver in use: nvme
Kernel modules: nvme
I tried debugging the guest kernel and I think the following if is taken in
pci_msi_domain_check_cap():
if (pci_msi_desc_is_multi_msi(desc) &&
!(info->flags & MSI_FLAG_MULTI_PCI_MSI))
return 1;
because flags is 0x3b (MSI_FLAG_MULTI_PCI_MSI is 0x4). And this I think means
that MSI_FLAG_MULTI_PCI_MSI is not set for that msi_domain_info.
# grep -i msi qemu.trace
1327@1571926064.595365:vfio_msi_setup 00000000-0000-0000-0000-000000000000 PCI MSI CAP @0x48
1334@1571926073.489691:vfio_msi_enable (00000000-0000-0000-0000-000000000000) Enabled 1 MSI vectors
1334@1571926073.501741:vfio_msi_disable (00000000-0000-0000-0000-000000000000)
1334@1571926073.507127:vfio_msi_enable (00000000-0000-0000-0000-000000000000) Enabled 1 MSI vectors
1327@1571926073.520840:vfio_msi_interrupt (00000000-0000-0000-0000-000000000000) vector 0 0xfee01004/0x4023
... more vfio_msi_interrupt ...
How can I further debug this?
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: Cannot use more than one MSI interrupts
2019-10-24 14:45 Cannot use more than one MSI interrupts Thanos Makatos
@ 2019-10-24 15:02 ` Alex Williamson
0 siblings, 0 replies; 2+ messages in thread
From: Alex Williamson @ 2019-10-24 15:02 UTC (permalink / raw)
To: Thanos Makatos; +Cc: qemu-devel@nongnu.org, Felipe Franciosi
On Thu, 24 Oct 2019 14:45:28 +0000
Thanos Makatos <thanos.makatos@nutanix.com> wrote:
> I have a Ubuntu VM (4.15.0-48-generic) to which I pass through a PCI device,
> specifically a virtual NVMe controller. The problem I have is that only one I/O queue
> is initialized, while there should be more (e.g. four). I'm using upstream QEMU
> v4.1.0 confgiured without any additional options. Most likely there's something
> broken with my virtual device implementation but I can't figure out exactly
> what, I was hoping to get some debugging directions.
>
> I run QEMU as follows:
>
> ~/src/qemu/x86_64-softmmu/qemu-system-x86_64 \
> -kernel bionic-server-cloudimg-amd64-vmlinuz-generic \
> -smp cores=2,sockets=2 \
> -nographic \
> -append "console=ttyS0 root=/dev/sda1 single nvme.sgl_threshold=0 nokaslr nvme.io_queue_depth=4" \
> -initrd bionic-server-cloudimg-amd64-initrd-generic \
> -hda bionic-server-cloudimg-amd64.raw \
> -hdb data.raw \
> -m 1G \
> -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=mem,share=yes,size=1073741824 -numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \
> -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000000-0000-0000-0000-000000000000 \
> -trace enable=vfio*,file=qemu.trace \
> -net none \
> -s
>
> This is what QEMU thinks of the devices:
>
> (qemu) info pci
> Bus 0, device 0, function 0:
> Host bridge: PCI device 8086:1237
> PCI subsystem 1af4:1100
> id ""
> Bus 0, device 1, function 0:
> ISA bridge: PCI device 8086:7000
> PCI subsystem 1af4:1100
> id ""
> Bus 0, device 1, function 1:
> IDE controller: PCI device 8086:7010
> PCI subsystem 1af4:1100
> BAR4: I/O at 0xc000 [0xc00f].
> id ""
> Bus 0, device 1, function 3:
> Bridge: PCI device 8086:7113
> PCI subsystem 1af4:1100
> IRQ 9.
> id ""
> Bus 0, device 2, function 0:
> VGA controller: PCI device 1234:1111
> PCI subsystem 1af4:1100
> BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff].
> BAR2: 32 bit memory at 0xfebf4000 [0xfebf4fff].
> BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
> id ""
> Bus 0, device 3, function 0:
> Class 0264: PCI device 4e58:0001
> PCI subsystem 0000:0000
> IRQ 11.
> BAR0: 32 bit memory at 0xfebf0000 [0xfebf3fff].
> id ""
>
> And this is what the guest thinks of the device in question:
>
> root@ubuntu:~# lspci -vvv -s 00:03.0
> 00:03.0 Non-Volatile memory controller: Device 4e58:0001 (prog-if 02 [NVM Express])
> Physical Slot: 3
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0
> Interrupt: pin A routed to IRQ 24
> NUMA node: 0
> Region 0: Memory at febf0000 (32-bit, non-prefetchable) [size=16K]
> Capabilities: [40] Power Management version 0
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: [48] MSI: Enable+ Count=1/4 Maskable- 64bit+
> Address: 00000000fee01004 Data: 4023
> Capabilities: [60] Express (v1) Endpoint, MSI 00
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
> ExtTag- AttnBtn+ AttnInd- PwrInd- RBE- FLReset- SlotPowerLimit 128.000W
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> MaxPayload 128 bytes, MaxReadReq 128 bytes
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
> LnkCap: Port #0, Speed unknown, Width x0, ASPM not supported, Exit Latency L0s <64ns, L1 <1us
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
> Kernel driver in use: nvme
> Kernel modules: nvme
>
> I tried debugging the guest kernel and I think the following if is taken in
> pci_msi_domain_check_cap():
>
> if (pci_msi_desc_is_multi_msi(desc) &&
> !(info->flags & MSI_FLAG_MULTI_PCI_MSI))
> return 1;
>
> because flags is 0x3b (MSI_FLAG_MULTI_PCI_MSI is 0x4). And this I think means
> that MSI_FLAG_MULTI_PCI_MSI is not set for that msi_domain_info.
>
> # grep -i msi qemu.trace
> 1327@1571926064.595365:vfio_msi_setup 00000000-0000-0000-0000-000000000000 PCI MSI CAP @0x48
> 1334@1571926073.489691:vfio_msi_enable (00000000-0000-0000-0000-000000000000) Enabled 1 MSI vectors
> 1334@1571926073.501741:vfio_msi_disable (00000000-0000-0000-0000-000000000000)
> 1334@1571926073.507127:vfio_msi_enable (00000000-0000-0000-0000-000000000000) Enabled 1 MSI vectors
> 1327@1571926073.520840:vfio_msi_interrupt (00000000-0000-0000-0000-000000000000) vector 0 0xfee01004/0x4023
> ... more vfio_msi_interrupt ...
>
> How can I further debug this?
The quick answer is to implement MSI-X in your device, MSI requires a
contiguous block of vectors and there's limited support for actually
making use of more than a single vector. There is almost no real
hardware that doesn't implement MSI-X for multiple vector support.
Thanks,
Alex
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2019-10-24 16:18 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-10-24 14:45 Cannot use more than one MSI interrupts Thanos Makatos
2019-10-24 15:02 ` Alex Williamson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).