Mellanox IB PCI passthrough fails with qemu-kvm>=1.1.2

Mellanox IB PCI passthrough fails with qemu-kvm>=1.1.2

[Vlastimil Holer]

[-- Attachment #1: Type: text/plain, Size: 2149 bytes --]

Hi,

we have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer versions IB device
probe in guest fails leaving firmware to kill whole physical machine.

I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.

On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16

Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue

Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI

Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.

Any idea or help is very appreciated!

Many thanks,
Vlastimil Holer

-- 
Vlastimil Holer                             phone:   +420-549 49 5349
CERIT Scientific Cloud                      e-mail:  holer@ics.muni.cz
Institute of Computer Science MU            twitter: @vholer

[-- Attachment #2: qemu-kvm-passthrough-logs.tar.bz2 --]
[-- Type: application/x-bzip2, Size: 27446 bytes --]