* [Qemu-devel] [Bug 1091766] [NEW] Physical host crash with Mellanox IB PCI passthrough
@ 2012-12-18 17:12 Vlastimil Holer
2012-12-18 22:00 ` [Qemu-devel] [Bug 1091766] " Alex Williamson
` (7 more replies)
0 siblings, 8 replies; 13+ messages in thread
From: Vlastimil Holer @ 2012-12-18 17:12 UTC (permalink / raw)
To: qemu-devel
Public bug reported:
(from http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
** Affects: qemu
Importance: Undecided
Status: New
** Tags: kvm qemu qemu-kvm
** Attachment added: "Host/guest system logs for different qemu-kvm versions"
https://bugs.launchpad.net/bugs/1091766/+attachment/3463136/+files/qemu-kvm-passthrough-logs.tar.bz2
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-18 17:12 [Qemu-devel] [Bug 1091766] [NEW] Physical host crash with Mellanox IB PCI passthrough Vlastimil Holer
@ 2012-12-18 22:00 ` Alex Williamson
2012-12-19 4:01 ` Alex Williamson
` (6 subsequent siblings)
7 siblings, 0 replies; 13+ messages in thread
From: Alex Williamson @ 2012-12-18 22:00 UTC (permalink / raw)
To: qemu-devel
Does the attached patch against qemu 1.3 fix it for pci-assign?
** Attachment added: "pci-assign: Enable MSIX on device to match guest"
https://bugs.launchpad.net/qemu/+bug/1091766/+attachment/3463375/+files/foo
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-18 17:12 [Qemu-devel] [Bug 1091766] [NEW] Physical host crash with Mellanox IB PCI passthrough Vlastimil Holer
2012-12-18 22:00 ` [Qemu-devel] [Bug 1091766] " Alex Williamson
@ 2012-12-19 4:01 ` Alex Williamson
2012-12-19 12:11 ` Vlastimil Holer
2012-12-19 15:31 ` Alex Williamson
` (5 subsequent siblings)
7 siblings, 1 reply; 13+ messages in thread
From: Alex Williamson @ 2012-12-19 4:01 UTC (permalink / raw)
To: qemu-devel
If the above pci-assign patch works, please also try this vfio-pci
version as it requires a slightly different implementation. Also
against qemu 1.3. Thanks.
** Attachment added: "vfio-pci: Make host MSI-X enable track guest"
https://bugs.launchpad.net/qemu/+bug/1091766/+attachment/3463505/+files/bar
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-19 4:01 ` Alex Williamson
@ 2012-12-19 12:11 ` Vlastimil Holer
0 siblings, 0 replies; 13+ messages in thread
From: Vlastimil Holer @ 2012-12-19 12:11 UTC (permalink / raw)
To: qemu-devel
Both patches against qemu 1.3 *works*, first with traditional PCI
pass., second with VFIO. Mellanox IB card in guest works fine again.
Great early Christmas present, thank you!
Just FYI: between both ways I can see little difference on host system
regarding the numbers of IRQs for MSI/MSI-X:
* VFIO:
| vfio_ecap_init: 0000:20:00.0 hiding ecap 0x19@0x18c
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 151 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 151 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 152 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 151 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 152 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 153 for MSI/MSI-X
* old way:
| assign device 0:20:0.0
| pci-stub 0000:20:00.0: restoring config space at offset 0xf (was
0x100, writing 0x10a)
| pci-stub 0000:20:00.0: restoring config space at offset 0x6 (was
0xc, writing 0xdf00000c)
| pci-stub 0000:20:00.0: restoring config space at offset 0x4 (was
0x4, writing 0x91b00004)
| pci-stub 0000:20:00.0: restoring config space at offset 0x3 (was
0x0, writing 0x10)
| pci-stub 0000:20:00.0: restoring config space at offset 0x1 (was
0x100000, writing 0x100042)
| pci-stub 0000:20:00.0: irq 134 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 135 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 136 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 137 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 138 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 139 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 140 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 141 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 142 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 143 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 144 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 145 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 146 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 147 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 148 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 149 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 150 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 151 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 152 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 153 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 154 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 155 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 156 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 157 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 158 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 159 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 160 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 161 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 162 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 163 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 134 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 135 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 136 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 137 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 138 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 139 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 140 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 141 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 142 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 143 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 144 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 145 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 146 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 147 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 148 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 149 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 150 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 151 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 152 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 153 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 154 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 155 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 156 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 157 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 158 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 159 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 160 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 161 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 162 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 163 for MSI/MSI-X
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-18 17:12 [Qemu-devel] [Bug 1091766] [NEW] Physical host crash with Mellanox IB PCI passthrough Vlastimil Holer
2012-12-18 22:00 ` [Qemu-devel] [Bug 1091766] " Alex Williamson
2012-12-19 4:01 ` Alex Williamson
@ 2012-12-19 15:31 ` Alex Williamson
2012-12-19 16:20 ` Vlastimil Holer
2012-12-19 16:46 ` Alex Williamson
` (4 subsequent siblings)
7 siblings, 1 reply; 13+ messages in thread
From: Alex Williamson @ 2012-12-19 15:31 UTC (permalink / raw)
To: qemu-devel
VFIO is doing what I expect, growing the enabled vectors as each is
unmasked. It starts from 1 and has to disable and re-enable each time
it grows, so we have:
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
1
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 151 for MSI/MSI-X
1 -> 2
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 151 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 152 for MSI/MSI-X
2 -> 3
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 151 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 152 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 153 for MSI/MSI-X
3 -> 4
So you likely see through lspci that both the guest and host have 4
vectors enabled.
On legacy assignment we have:
| pci-stub 0000:20:00.0: irq 134 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 135 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 136 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 137 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 138 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 139 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 140 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 141 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 142 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 143 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 144 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 145 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 146 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 147 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 148 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 149 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 150 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 151 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 152 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 153 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 154 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 155 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 156 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 157 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 158 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 159 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 160 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 161 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 162 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 163 for MSI/MSI-X
So you have 30 vectors enabled in the host! wow. Does lspci in host
and guest confirm this? (ie. 30 vectors on the host and likely 4
vectors on the guest) If so, I'll need to look to see if we can apply
something more similar to the vfio solution to legacy assignment,
otherwise we're wasting a lot of vectors in the host. Did qemu-kvm-1.0
consume this many vectors? Thanks.
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-19 15:31 ` Alex Williamson
@ 2012-12-19 16:20 ` Vlastimil Holer
0 siblings, 0 replies; 13+ messages in thread
From: Vlastimil Holer @ 2012-12-19 16:20 UTC (permalink / raw)
To: qemu-devel
On Wed, Dec 19, 2012 at 4:31 PM, Alex Williamson
<alex.l.williamson@gmail.com> wrote:
> So you likely see through lspci that both the guest and host have 4
> vectors enabled.
Maybe you'll have to kick me on how to find this data. If I look on
lspci output, I find only this relevant thing:
Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
Vector table: BAR=0 offset=0007c000
PBA: BAR=0 offset=0007d000
enabled or disabled, no count=x/y with enabled/limit which I can see
for some other devices. I'm attaching full lspci -vvv output for one
Mellanox card on physical host, where it's not used.
> vectors on the guest) If so, I'll need to look to see if we can apply
> something more similar to the vfio solution to legacy assignment,
> otherwise we're wasting a lot of vectors in the host. Did qemu-kvm-1.0
> consume this many vectors? Thanks.
Yes, it did. All host/guest kernel logs for qemu-kvm 1.0.1, 1.1.2 and
1.2.0 were attached to this bug.
** Attachment added: "lspci-mlx.txt"
https://bugs.launchpad.net/bugs/1091766/+attachment/3464010/+files/lspci-mlx.txt
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-18 17:12 [Qemu-devel] [Bug 1091766] [NEW] Physical host crash with Mellanox IB PCI passthrough Vlastimil Holer
` (2 preceding siblings ...)
2012-12-19 15:31 ` Alex Williamson
@ 2012-12-19 16:46 ` Alex Williamson
2012-12-19 22:46 ` Alex Williamson
` (3 subsequent siblings)
7 siblings, 0 replies; 13+ messages in thread
From: Alex Williamson @ 2012-12-19 16:46 UTC (permalink / raw)
To: qemu-devel
You're right, I'm thinking of MSI where lspci reports x/y vectors. The
only way I know to get this is to grep /proc/interrupts on host and
guest. Look for kvm or vfio in the host and likely some device specific
identifier in the guest.
I also see your original log now, so old qemu worked, but wasted lots
and lots of host vectors. Thanks.
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-18 17:12 [Qemu-devel] [Bug 1091766] [NEW] Physical host crash with Mellanox IB PCI passthrough Vlastimil Holer
` (3 preceding siblings ...)
2012-12-19 16:46 ` Alex Williamson
@ 2012-12-19 22:46 ` Alex Williamson
2012-12-20 10:46 ` Vlastimil Holer
2012-12-21 0:19 ` Vlastimil Holer
2013-01-02 16:20 ` Vlastimil Holer
` (2 subsequent siblings)
7 siblings, 2 replies; 13+ messages in thread
From: Alex Williamson @ 2012-12-19 22:46 UTC (permalink / raw)
To: qemu-devel
Here's another version of the legacy pci-assign patch. This should also
only use 4 vectors on the host, like vfio. I'm a little uneasy about
setting up an MSIMessage with unknown data, but I guess we did it for a
long time previously. Please test. Thanks
** Attachment added: "v2: pci-assign: Enable MSIX on device to match guest"
https://bugs.launchpad.net/qemu/+bug/1091766/+attachment/3464302/+files/foo2
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-19 22:46 ` Alex Williamson
@ 2012-12-20 10:46 ` Vlastimil Holer
2012-12-21 0:19 ` Vlastimil Holer
1 sibling, 0 replies; 13+ messages in thread
From: Vlastimil Holer @ 2012-12-20 10:46 UTC (permalink / raw)
To: qemu-devel
On Wed, Dec 19, 2012 at 5:46 PM, Alex Williamson
<alex.l.williamson@gmail.com> wrote:
> You're right, I'm thinking of MSI where lspci reports x/y vectors. The
> only way I know to get this is to grep /proc/interrupts on host and
> guest. Look for kvm or vfio in the host and likely some device specific
> identifier in the guest.
You were absolutely right with vector counts. With VFIO I can see 4
and 4 vectors in host and guest and with legacy PCI assign. (on
qemu-1.0.1) I can see 30 in host and 4 in guest.
On Wed, Dec 19, 2012 at 11:46 PM, Alex Williamson
<alex.l.williamson@gmail.com> wrote:
> Here's another version of the legacy pci-assign patch. This should also
> only use 4 vectors on the host, like vfio. I'm a little uneasy about
> setting up an MSIMessage with unknown data, but I guess we did it for a
> long time previously. Please test. Thanks
I have physical machine in production back again, so it'll take few
days to test your new patch. I'll let you know.
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-19 22:46 ` Alex Williamson
2012-12-20 10:46 ` Vlastimil Holer
@ 2012-12-21 0:19 ` Vlastimil Holer
1 sibling, 0 replies; 13+ messages in thread
From: Vlastimil Holer @ 2012-12-21 0:19 UTC (permalink / raw)
To: qemu-devel
Confirmed, third patch for legacy pci-assign enables only 4 vectors on
host and guest. Mellanox IB works for me fine as well. Thanks!
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-18 17:12 [Qemu-devel] [Bug 1091766] [NEW] Physical host crash with Mellanox IB PCI passthrough Vlastimil Holer
` (4 preceding siblings ...)
2012-12-19 22:46 ` Alex Williamson
@ 2013-01-02 16:20 ` Vlastimil Holer
2013-01-02 16:29 ` Alex Williamson
2013-05-27 15:08 ` Vlastimil Holer
7 siblings, 0 replies; 13+ messages in thread
From: Vlastimil Holer @ 2013-01-02 16:20 UTC (permalink / raw)
To: qemu-devel
Just a silly questions, because I don't know the deveploment process in
QEMU project -- can be your patches commited into project's VCS so that
new stable release contains them and doesn't fail again? Thank you!
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-18 17:12 [Qemu-devel] [Bug 1091766] [NEW] Physical host crash with Mellanox IB PCI passthrough Vlastimil Holer
` (5 preceding siblings ...)
2013-01-02 16:20 ` Vlastimil Holer
@ 2013-01-02 16:29 ` Alex Williamson
2013-05-27 15:08 ` Vlastimil Holer
7 siblings, 0 replies; 13+ messages in thread
From: Alex Williamson @ 2013-01-02 16:29 UTC (permalink / raw)
To: qemu-devel
I'm currently trying to make that happen, starting with the patches in
comments 2 & 3 and moving to something like the patch in comment 8 in
the development branch.
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
New
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough
2012-12-18 17:12 [Qemu-devel] [Bug 1091766] [NEW] Physical host crash with Mellanox IB PCI passthrough Vlastimil Holer
` (6 preceding siblings ...)
2013-01-02 16:29 ` Alex Williamson
@ 2013-05-27 15:08 ` Vlastimil Holer
7 siblings, 0 replies; 13+ messages in thread
From: Vlastimil Holer @ 2013-05-27 15:08 UTC (permalink / raw)
To: qemu-devel
My believe is that this bug can be closed. We have tested QEMU 1.4.x and
1.5.x series (with pci-assign) and at least these works fine.
** Changed in: qemu
Status: New => Fix Released
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766
Title:
Physical host crash with Mellanox IB PCI passthrough
Status in QEMU:
Fix Released
Bug description:
(from
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)
We have been using PCI passthrough with the Mellanox IB interface
(MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
qemu-kvm-1.0 (both from backports). It worked fine until latest
update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
IB device probe in guest fails leaving firmware to kill whole physical machine.
I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
dx360 M4 Server.
Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
new VFIO mechanism and behaves the same way.
On guest the mlx4_core fails to probe device:
| mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
| mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
| mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 51).
| mlx4_core 0000:00:08.0: Trying again without MSI-X.
| mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), aborting.
| mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
| mlx4_core 0000:00:08.0: PCI INT A disabled
| mlx4_core: probe of 0000:00:08.0 failed with error -16
Which immediately results in reset of the whole physical machine:
| Uhhuh. NMI received for unknown reason 3d on CPU 0.
| Do you have a strange power saving mode enabled?
| Dazed and confused, but trying to continue
Followed by events in hardware management module:
| A software NMI has occurred on system "SN# xxxxxxx"
| Fault in slot "All PCI Err" on system "SN# xxxxxxx"
| Fault in slot "PCI 1" on system "SN# xxxxxxx"
| A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
| "Host Power" has been Power Cycled
| System "SN# xxxxxxx" has recovered from an NMI
Kernel logs for both host/guest machines and different qemu-kvm
versions are attached. PCI passthrough for e.g. Intel e1000 works
fine with all tested qemu-kvm versions.
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2013-05-27 15:15 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-12-18 17:12 [Qemu-devel] [Bug 1091766] [NEW] Physical host crash with Mellanox IB PCI passthrough Vlastimil Holer
2012-12-18 22:00 ` [Qemu-devel] [Bug 1091766] " Alex Williamson
2012-12-19 4:01 ` Alex Williamson
2012-12-19 12:11 ` Vlastimil Holer
2012-12-19 15:31 ` Alex Williamson
2012-12-19 16:20 ` Vlastimil Holer
2012-12-19 16:46 ` Alex Williamson
2012-12-19 22:46 ` Alex Williamson
2012-12-20 10:46 ` Vlastimil Holer
2012-12-21 0:19 ` Vlastimil Holer
2013-01-02 16:20 ` Vlastimil Holer
2013-01-02 16:29 ` Alex Williamson
2013-05-27 15:08 ` Vlastimil Holer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).