From: Joerg Roedel <joerg.roedel@amd.com>
To: Jeff Kirsher <jeffrey.t.kirsher@intel.com>,
Jesse Brandeburg <jesse.brandeburg@intel.com>,
Bruce Allan <bruce.w.allan@intel.com>,
Carolyn Wyborny <carolyn.wyborny@intel.com>,
Don Skidmore <donald.c.skidmore@intel.com>,
Greg Rose <gregory.v.rose@intel.com>,
Peter P Waskiewicz Jr <peter.p.waskiewicz.jr@intel.com>,
Alex Duyck <alexander.h.duyck@intel.com>,
John Ronciak <john.ronciak@intel.com>
Cc: <e1000-devel@lists.sourceforge.net>, <linux-kernel@vger.kernel.org>
Subject: IO_PAGE_FAULTS with igb or igbvf on AMD IOMMU system
Date: Tue, 19 Jun 2012 12:20:41 +0200 [thread overview]
Message-ID: <20120619102041.GI2624@amd.com> (raw)
Hi,
I am trying to use an Intel 82576 NIC on an AMD IOMMU system with
SR-IOV. When I load the igb module with max_vfs=1 to enable a virtual
function I get IO_PAGE_FAULTS from the virtual functions. The relevant
part of dmesg is:
[ 45.788134] igb: Intel(R) Gigabit Ethernet Network Driver - version 3.4.7-k
[ 45.795090] igb: Copyright (c) 2007-2012 Intel Corporation.
[ 45.801049] igb 0000:02:00.0: irq 80 for MSI/MSI-X
[ 45.801056] igb 0000:02:00.0: irq 81 for MSI/MSI-X
[ 45.801061] igb 0000:02:00.0: irq 82 for MSI/MSI-X
[ 45.801067] igb 0000:02:00.0: irq 83 for MSI/MSI-X
[ 45.901445] pci 0000:02:10.0: [8086:10ca] type 00 class 0x020000
[ 45.901585] AMD-Vi: New device 0000:02:10.0
[ 45.906486] igb 0000:02:00.0: 1 VFs allocated
[ 45.937918] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.0.1-k
[ 45.945751] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 46.071749] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 46.078605] igb 0000:02:00.0: eth5: (PCIe:2.5Gb/s:Width x4) 00:1b:21:49:2e:cc
[ 46.085804] igb 0000:02:00.0: eth5: PBA No: E43709-003
[ 46.090946] igb 0000:02:00.0: Using MSI-X interrupts. 2 rx queue(s), 1 tx queue(s)
[ 46.098870] igb 0000:02:00.1: irq 84 for MSI/MSI-X
[ 46.098876] igb 0000:02:00.1: irq 85 for MSI/MSI-X
[ 46.098881] igb 0000:02:00.1: irq 86 for MSI/MSI-X
[ 46.098886] igb 0000:02:00.1: irq 87 for MSI/MSI-X
[ 46.104262] AMD-Vi: Using protection domain 23 for device 0000:02:00.0
[ 46.172988] IPv6: ADDRCONF(NETDEV_UP): eth5: link is not ready
[ 46.202875] pci 0000:02:10.1: [8086:10ca] type 00 class 0x020000
[ 46.203013] AMD-Vi: New device 0000:02:10.1
[ 46.207935] igb 0000:02:00.1: 1 VFs allocated
[ 46.373149] igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 46.380019] igb 0000:02:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 00:1b:21:49:2e:cd
[ 46.387213] igb 0000:02:00.1: eth6: PBA No: E43709-003
[ 46.392347] igb 0000:02:00.1: Using MSI-X interrupts. 2 rx queue(s), 1 tx queue(s)
[ 46.400072] igbvf 0000:02:10.0: enabling device (0000 -> 0002)
[ 46.405977] igbvf 0000:02:10.0: irq 88 for MSI/MSI-X
[ 46.405983] igbvf 0000:02:10.0: irq 89 for MSI/MSI-X
[ 46.405988] igbvf 0000:02:10.0: irq 90 for MSI/MSI-X
[ 46.411492] AMD-Vi: Using protection domain 24 for device 0000:02:00.1
[ 46.480625] IPv6: ADDRCONF(NETDEV_UP): eth6: link is not ready
[ 46.486980] igbvf 0000:02:10.0: Intel(R) 82576 Virtual Function
[ 46.492895] igbvf 0000:02:10.0: Address: ce:5e:41:2f:36:ce
[ 46.498510] igbvf 0000:02:10.1: enabling device (0000 -> 0002)
[ 46.504394] igbvf 0000:02:10.1: irq 91 for MSI/MSI-X
[ 46.504400] igbvf 0000:02:10.1: irq 92 for MSI/MSI-X
[ 46.504405] igbvf 0000:02:10.1: irq 93 for MSI/MSI-X
[ 46.527012] igbvf 0000:02:10.1: Intel(R) 82576 Virtual Function
[ 46.532931] igbvf 0000:02:10.1: Address: 52:3e:8f:47:60:da
[ 46.573209] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000 address=0x000000021e170000 flags=0x0050]
[ 46.575620] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 46.589607] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000 address=0x000000021e170040 flags=0x0050]
[ 46.600186] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000 address=0x000000021e170080 flags=0x0050]
[ 46.610763] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000 address=0x000000021e1700c0 flags=0x0050]
[ 46.669940] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
The devices (physical and virtual) are not operational after this. I
think this is a problem in the igb or igbvf driver. The addresses
reported in the IO_PAGE_FAULTS are system-ram addresses are were not
handed out by the AMD IOMMU driver (the driver only hands out DMA
handles below 4GB). Also the reported domain is 0 which means that the
driver for that device has not yet issued _any_ call to the DMA-API. But
the device is doing an DMA write request (as seen in the flags). Any
ideas? Also let me know if you need any additional information.
Thanks,
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
next reply other threads:[~2012-06-19 10:21 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-06-19 10:20 Joerg Roedel [this message]
2012-06-19 18:19 ` IO_PAGE_FAULTS with igb or igbvf on AMD IOMMU system Alexander Duyck
2012-06-20 9:48 ` Joerg Roedel
2012-06-20 16:51 ` Rose, Gregory V
2012-06-20 22:48 ` Alexander Duyck
2012-06-25 11:20 ` Joerg Roedel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120619102041.GI2624@amd.com \
--to=joerg.roedel@amd.com \
--cc=alexander.h.duyck@intel.com \
--cc=bruce.w.allan@intel.com \
--cc=carolyn.wyborny@intel.com \
--cc=donald.c.skidmore@intel.com \
--cc=e1000-devel@lists.sourceforge.net \
--cc=gregory.v.rose@intel.com \
--cc=jeffrey.t.kirsher@intel.com \
--cc=jesse.brandeburg@intel.com \
--cc=john.ronciak@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=peter.p.waskiewicz.jr@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).