PCI passthrough = NMIs

PCI passthrough = NMIs

[Zwane Mwaikambo]

Hi,
	I've been trying to get a Linux guest working with PCI passthrough 
of an ethernet card using the vtd branches. The device detection works and 
the guest reports a link, however as soon as i try and ping the guest it 
receives an NMI (i'm guessing this is PCI DMA related). Interrupt delivery 
to the guest looks fine (count increases at a low rate) and isn't shared 
with anything else on the host.

Thanks for any hints.

	Zwane

Re: PCI passthrough = NMIs

[Muli Ben-Yehuda]

On Sun, Oct 12, 2008 at 11:09:14PM -0700, Zwane Mwaikambo wrote:
> Hi,
> 	I've been trying to get a Linux guest working with PCI passthrough 
> of an ethernet card using the vtd branches. The device detection works and 
> the guest reports a link, however as soon as i try and ping the guest it 
> receives an NMI (i'm guessing this is PCI DMA related). Interrupt delivery 
> to the guest looks fine (count increases at a low rate) and isn't shared 
> with anything else on the host.
> 
> Thanks for any hints.

Anything in the host dmesg? Are you using VT-d?

Cheers,
Muli
-- 
The First Workshop on I/O Virtualization (WIOV '08)
Dec 2008, San Diego, CA, http://www.usenix.org/wiov08/
                       <->
SYSTOR 2009---The Israeli Experimental Systems Conference
http://www.haifa.il.ibm.com/conferences/systor2009/

Re: PCI passthrough = NMIs

[Zwane Mwaikambo]

On Mon, 13 Oct 2008, Muli Ben-Yehuda wrote:

> On Sun, Oct 12, 2008 at 11:09:14PM -0700, Zwane Mwaikambo wrote:
> > Hi,
> > 	I've been trying to get a Linux guest working with PCI passthrough 
> > of an ethernet card using the vtd branches. The device detection works and 
> > the guest reports a link, however as soon as i try and ping the guest it 
> > receives an NMI (i'm guessing this is PCI DMA related). Interrupt delivery 
> > to the guest looks fine (count increases at a low rate) and isn't shared 
> > with anything else on the host.
> > 
> > Thanks for any hints.
> 
> Anything in the host dmesg? Are you using VT-d?

Yep using VT-d on an intel Q35, i get the following in order;

pci 0000:11:0a.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
DMAR:[DMA Read] Request device [00:1e.0] fault addr 7841000 
DMAR:[fault reason 06] PTE Read access is not set
Uhhuh. NMI received for unknown reason a1.
You have some hardware problem, likely on the PCI bus.
Dazed and confused, but trying to continue
DMAR:[DMA Write] Request device [00:1e.0] fault addr 0 
DMAR:[fault reason 05] PTE Write access is not set
DMAR:[DMA Write] Request device [00:1e.0] fault addr 0 
DMAR:[fault reason 05] PTE Write access is not set

The DMAR write fault printouts continue as long as you ping the IP address 
of the network card so it appears to be receiving packets.

00:1e.0 is the PCI bridge;

00:00.0 Host bridge: Intel Corporation 82Q35 Express DRAM Controller (rev 02)
00:02.0 VGA compatible controller: Intel Corporation 82Q35 Express Integrated Graphics Controller (rev 02)
00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit Network Connection (rev 02)
00:1a.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02)
00:1a.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 02)
00:1a.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 02)
00:1a.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02)
00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5 (rev 02)
00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IO (ICH9DO) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA AHCI Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
00:1f.6 Signal processing controller: Intel Corporation 82801I (ICH9 Family) Thermal Subsystem (rev 02)
0d:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
11:04.0 IDE interface: Integrated Technology Express, Inc. Unknown device 8213
11:09.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 10)
11:0a.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)

Re: PCI passthrough = NMIs

[Sheng Yang]

On Monday 13 October 2008 23:04:34 Zwane Mwaikambo wrote:
> On Mon, 13 Oct 2008, Muli Ben-Yehuda wrote:
> > On Sun, Oct 12, 2008 at 11:09:14PM -0700, Zwane Mwaikambo wrote:
> > > Hi,
> > > 	I've been trying to get a Linux guest working with PCI passthrough
> > > of an ethernet card using the vtd branches. The device detection works
> > > and the guest reports a link, however as soon as i try and ping the
> > > guest it receives an NMI (i'm guessing this is PCI DMA related).
> > > Interrupt delivery to the guest looks fine (count increases at a low
> > > rate) and isn't shared with anything else on the host.
> > >
> > > Thanks for any hints.
> >
> > Anything in the host dmesg? Are you using VT-d?
>
> Yep using VT-d on an intel Q35, i get the following in order;
>
> pci 0000:11:0a.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
> DMAR:[DMA Read] Request device [00:1e.0] fault addr 7841000
> DMAR:[fault reason 06] PTE Read access is not set
> Uhhuh. NMI received for unknown reason a1.
> You have some hardware problem, likely on the PCI bus.
> Dazed and confused, but trying to continue
> DMAR:[DMA Write] Request device [00:1e.0] fault addr 0
> DMAR:[fault reason 05] PTE Write access is not set
> DMAR:[DMA Write] Request device [00:1e.0] fault addr 0
> DMAR:[fault reason 05] PTE Write access is not set

Do you have VT-d enabled? Also reinstalled host kernel with KVM upstream(we 
modified host kernel to support VT-d in KVM)? Look like VT-d table is not set 
up... You can check what you got with "dmesg|grep DMAR".

--
regards
Yang, Sheng
>
> The DMAR write fault printouts continue as long as you ping the IP address
> of the network card so it appears to be receiving packets.
>
> 00:1e.0 is the PCI bridge;
>
> 00:00.0 Host bridge: Intel Corporation 82Q35 Express DRAM Controller (rev
> 02) 00:02.0 VGA compatible controller: Intel Corporation 82Q35 Express
> Integrated Graphics Controller (rev 02) 00:19.0 Ethernet controller: Intel
> Corporation 82566DM-2 Gigabit Network Connection (rev 02) 00:1a.0 USB
> Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4
> (rev 02) 00:1a.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB
> UHCI Controller #5 (rev 02) 00:1a.2 USB Controller: Intel Corporation
> 82801I (ICH9 Family) USB UHCI Controller #6 (rev 02) 00:1a.7 USB
> Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2
> (rev 02) 00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD
> Audio Controller (rev 02) 00:1c.0 PCI bridge: Intel Corporation 82801I
> (ICH9 Family) PCI Express Port 1 (rev 02) 00:1c.4 PCI bridge: Intel
> Corporation 82801I (ICH9 Family) PCI Express Port 5 (rev 02) 00:1d.0 USB
> Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
> (rev 02) 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB
> UHCI Controller #2 (rev 02) 00:1d.2 USB Controller: Intel Corporation
> 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02) 00:1d.7 USB
> Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
> (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
> 00:1f.0 ISA bridge: Intel Corporation 82801IO (ICH9DO) LPC Interface
> Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation
> 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA AHCI Controller (rev 02) 00:1f.3
> SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
> 00:1f.6 Signal processing controller: Intel Corporation 82801I (ICH9
> Family) Thermal Subsystem (rev 02) 0d:00.0 Ethernet controller: Intel
> Corporation 82573L Gigabit Ethernet Controller 11:04.0 IDE interface:
> Integrated Technology Express, Inc. Unknown device 8213 11:09.0 Ethernet
> controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 10)
> 11:0a.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet
> Controller (rev 05) --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: PCI passthrough = NMIs

[Zwane Mwaikambo]

On Tue, 14 Oct 2008, Sheng Yang wrote:

> On Monday 13 October 2008 23:04:34 Zwane Mwaikambo wrote:
> > On Mon, 13 Oct 2008, Muli Ben-Yehuda wrote:
> > > On Sun, Oct 12, 2008 at 11:09:14PM -0700, Zwane Mwaikambo wrote:
> > > > Hi,
> > > > 	I've been trying to get a Linux guest working with PCI passthrough
> > > > of an ethernet card using the vtd branches. The device detection works
> > > > and the guest reports a link, however as soon as i try and ping the
> > > > guest it receives an NMI (i'm guessing this is PCI DMA related).
> > > > Interrupt delivery to the guest looks fine (count increases at a low
> > > > rate) and isn't shared with anything else on the host.
> > > >
> > > > Thanks for any hints.
> > >
> > > Anything in the host dmesg? Are you using VT-d?
> >
> > Yep using VT-d on an intel Q35, i get the following in order;
> >
> > pci 0000:11:0a.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
> > DMAR:[DMA Read] Request device [00:1e.0] fault addr 7841000
> > DMAR:[fault reason 06] PTE Read access is not set
> > Uhhuh. NMI received for unknown reason a1.
> > You have some hardware problem, likely on the PCI bus.
> > Dazed and confused, but trying to continue
> > DMAR:[DMA Write] Request device [00:1e.0] fault addr 0
> > DMAR:[fault reason 05] PTE Write access is not set
> > DMAR:[DMA Write] Request device [00:1e.0] fault addr 0
> > DMAR:[fault reason 05] PTE Write access is not set
> 
> Do you have VT-d enabled? Also reinstalled host kernel with KVM upstream(we 
> modified host kernel to support VT-d in KVM)? Look like VT-d table is not set 
> up... You can check what you got with "dmesg|grep DMAR".

Yep, i have VT-d enabled in the BIOS, i also had to disable legacy USB in 
order to get Linux to boot. This is what i get on the host kernel;

ACPI: DMAR 7F487CA8, 00C8 (r1 Intel  OEMDMAR   6040000 LOHR        1)
DMAR:Host address width 36
DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
DMAR:RMRR base: 0x000000007f4e4000 end: 0x000000007f4f0000

Although legacy USB not working would suggest that there are table entries 
already missing in the BIOS?

Thanks,
	Zwane

Re: PCI passthrough = NMIs

[Sheng Yang]

On Tuesday 14 October 2008 12:39:52 Zwane Mwaikambo wrote:
> On Tue, 14 Oct 2008, Sheng Yang wrote:
> > On Monday 13 October 2008 23:04:34 Zwane Mwaikambo wrote:
> > > On Mon, 13 Oct 2008, Muli Ben-Yehuda wrote:
> > > > On Sun, Oct 12, 2008 at 11:09:14PM -0700, Zwane Mwaikambo wrote:
> > > > > Hi,
> > > > > 	I've been trying to get a Linux guest working with PCI passthrough
> > > > > of an ethernet card using the vtd branches. The device detection
> > > > > works and the guest reports a link, however as soon as i try and
> > > > > ping the guest it receives an NMI (i'm guessing this is PCI DMA
> > > > > related). Interrupt delivery to the guest looks fine (count
> > > > > increases at a low rate) and isn't shared with anything else on the
> > > > > host.
> > > > >
> > > > > Thanks for any hints.
> > > >
> > > > Anything in the host dmesg? Are you using VT-d?
> > >
> > > Yep using VT-d on an intel Q35, i get the following in order;
> > >
> > > pci 0000:11:0a.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
> > > DMAR:[DMA Read] Request device [00:1e.0] fault addr 7841000
> > > DMAR:[fault reason 06] PTE Read access is not set
> > > Uhhuh. NMI received for unknown reason a1.
> > > You have some hardware problem, likely on the PCI bus.
> > > Dazed and confused, but trying to continue
> > > DMAR:[DMA Write] Request device [00:1e.0] fault addr 0
> > > DMAR:[fault reason 05] PTE Write access is not set
> > > DMAR:[DMA Write] Request device [00:1e.0] fault addr 0
> > > DMAR:[fault reason 05] PTE Write access is not set
> >
> > Do you have VT-d enabled? Also reinstalled host kernel with KVM
> > upstream(we modified host kernel to support VT-d in KVM)? Look like VT-d
> > table is not set up... You can check what you got with "dmesg|grep DMAR".
>
> Yep, i have VT-d enabled in the BIOS, i also had to disable legacy USB in
> order to get Linux to boot. This is what i get on the host kernel;
>
> ACPI: DMAR 7F487CA8, 00C8 (r1 Intel  OEMDMAR   6040000 LOHR        1)
> DMAR:Host address width 36
> DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000
> DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed91000
> DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000
> DMAR:RMRR base: 0x000000007f4e4000 end: 0x000000007f4f0000
>
> Although legacy USB not working would suggest that there are table entries
> already missing in the BIOS?

The VT-d table should be set up by KVM, but it's very strange that if you 
enable VT-d for native, USB can't work. If you sure you are using kernel 
compiled from KVM upstream, I think it maybe a BIOS issue...

--
regards
Yang, Sheng

RE: PCI passthrough = NMIs

[Han, Weidong]

Zwane Mwaikambo wrote:
> Hi,
> 	I've been trying to get a Linux guest working with PCI
passthrough
> of an ethernet card using the vtd branches. The device detection
> works and the guest reports a link, however as soon as i try and ping
> the guest it receives an NMI (i'm guessing this is PCI DMA related).
> Interrupt delivery to the guest looks fine (count increases at a low
> rate) and isn't shared with anything else on the host.
> 
> Thanks for any hints.
> 
> 	Zwane

What do you mean "vtd branches"? Is it the vtd branch of Amit's tree?

VT-d patches are in kvm.git, as long as you apply amit's userspace patch
(you can find it on mailing list), you should can assign device to
guest. Make sure you enable VT-d in BIOS and set CONFIG_DMAR in config,
and remove device driver before you assign device. NIC and USB
controller assignment works fine for me.

Regards,
Weidong

RE: PCI passthrough = NMIs

[Zwane Mwaikambo]

On Tue, 14 Oct 2008, Han, Weidong wrote:

> Zwane Mwaikambo wrote:
> > Hi,
> > 	I've been trying to get a Linux guest working with PCI
> passthrough
> > of an ethernet card using the vtd branches. The device detection
> > works and the guest reports a link, however as soon as i try and ping
> > the guest it receives an NMI (i'm guessing this is PCI DMA related).
> > Interrupt delivery to the guest looks fine (count increases at a low
> > rate) and isn't shared with anything else on the host.
> > 
> > Thanks for any hints.
> > 
> > 	Zwane
> 
> What do you mean "vtd branches"? Is it the vtd branch of Amit's tree?
> 
> VT-d patches are in kvm.git, as long as you apply amit's userspace patch
> (you can find it on mailing list), you should can assign device to
> guest. Make sure you enable VT-d in BIOS and set CONFIG_DMAR in config,
> and remove device driver before you assign device. NIC and USB
> controller assignment works fine for me.

I'm using the following repositories without any additional patches;

git://git.kernel.org/pub/scm/linux/kernel/git/amit/kvm.git vtd
git://git.kernel.org/pub/scm/linux/kernel/git/amit/kvm-userspace.git vtd

The host kernel only has the e100 network driver enabled, the guest kernel 
only has the e1000 network driver enabled. I have enabled VT-d in the BIOS 
and enabled CONFIG_DMAR in the host kernel config. I'm really beginning to 
suspect the BIOS unfortunately :(

Thanks,
	Zwane

Re: PCI passthrough = NMIs

[Amit Shah]

* On Tuesday 14 Oct 2008 22:46:37 Zwane Mwaikambo wrote:
> On Tue, 14 Oct 2008, Han, Weidong wrote:
> > Zwane Mwaikambo wrote:
> > > Hi,
> > > 	I've been trying to get a Linux guest working with PCI
> >
> > passthrough
> >
> > > of an ethernet card using the vtd branches. The device detection
> > > works and the guest reports a link, however as soon as i try and ping
> > > the guest it receives an NMI (i'm guessing this is PCI DMA related).
> > > Interrupt delivery to the guest looks fine (count increases at a low
> > > rate) and isn't shared with anything else on the host.
> > >
> > > Thanks for any hints.
> > >
> > > 	Zwane
> >
> > What do you mean "vtd branches"? Is it the vtd branch of Amit's tree?
> >
> > VT-d patches are in kvm.git, as long as you apply amit's userspace patch
> > (you can find it on mailing list), you should can assign device to
> > guest. Make sure you enable VT-d in BIOS and set CONFIG_DMAR in config,
> > and remove device driver before you assign device. NIC and USB
> > controller assignment works fine for me.
>
> I'm using the following repositories without any additional patches;
>
> git://git.kernel.org/pub/scm/linux/kernel/git/amit/kvm.git vtd
> git://git.kernel.org/pub/scm/linux/kernel/git/amit/kvm-userspace.git vtd

Can you try using Avi's kvm.git tree and the userspace tree same as above 
(it's been updated now and you'll have to use +vtd as the branch)?

That's the newest code; though I'm not sure if it will solve your problem.

> The host kernel only has the e100 network driver enabled, the guest kernel
> only has the e1000 network driver enabled. I have enabled VT-d in the BIOS
> and enabled CONFIG_DMAR in the host kernel config. I'm really beginning to
> suspect the BIOS unfortunately :(

Amiit.