Re: IXGBE, IOMMU DMAR DRHD handling fault issue

["Burakov, Anatoly" <anatoly.burakov@intel.com>]

On 23-Jan-18 5:25 PM, Ravi Kerur wrote:
> Hi,
> 
> I am running into an issue when DPDK is started with iommu on via GRUB
> command. Problem is not seen with regular kernel driver, error messages
> show when DPDK is started and happens for both PF and VF interfaces.
> 
> I am using DPDK 17.05 so the patch proposed in the following link is
> available
> http://dpdk.org/ml/archives/dev/2017-February/057048.html
> 
> Workaround is to use "iommu=pt" but I want iommu enabled in my setup. I
> checked BIOS for reserved memory(DMA RMRR for IXGBE) didn't get any details
> on it.
> 
> Kindly let me know how to resolve this issue.
> 
> Following are the details
> 
> (1) Linux kernel 4.9
> (2) DPDK 17.05
> 
> (3) IXGBE details
> ethtool -i enp4s0f0  (PF driver)
> driver: ixgbe
> version: 5.3.3
> firmware-version: 0x800007b8, 1.1018.0
> bus-info: 0000:04:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
> 
> ethtool -i enp4s16f2 (VF driver)
> driver: ixgbevf
> version: 4.3.2
> firmware-version:
> bus-info: 0000:04:10.2
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: no
> supports-register-dump: yes
> supports-priv-flags: no
> 
> Bus info          Device       Class          Description
> =========================================================
> pci@0000:01:00.0  ens11f0      network        82599ES 10-Gigabit SFI/SFP+
> Network Connection
> pci@0000:01:00.1  ens11f1      network        82599ES 10-Gigabit SFI/SFP+
> Network Connection
> pci@0000:04:00.0  enp4s0f0     network        82599ES 10-Gigabit SFI/SFP+
> Network Connection
> pci@0000:04:00.1  enp4s0f1     network        82599ES 10-Gigabit SFI/SFP+
> Network Connection
> pci@0000:04:10.0  enp4s16      network        Illegal Vendor ID
> pci@0000:04:10.2  enp4s16f2    network        Illegal Vendor ID
> 
> (4) DPDK bind interfaces
> 
> # dpdk-devbind -s
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> drv=igb_uio unused=vfio-pci
> 0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=igb_uio
> unused=vfio-pci
> 
> Network devices using kernel driver
> ===================================
> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> if=ens11f1 drv=ixgbe unused=igb_uio,vfio-pci
> 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> if=enp4s0f0 drv=ixgbe unused=igb_uio,vfio-pci
> 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> if=enp4s0f1 drv=ixgbe unused=igb_uio,vfio-pci
> 0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=enp4s16
> drv=ixgbevf unused=igb_uio,vfio-pci
> 0000:06:00.0 'I210 Gigabit Network Connection 1533' if=eno1 drv=igb
> unused=igb_uio,vfio-pci *Active*
> 
> Other Network devices
> =====================
> <none>
> 
> ...
> 
> (5) Kernel dmesg
> 
> # dmesg | grep -e DMAR
> [    0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I
> 00000001 INTL 20091013)
> [    0.000000] DMAR: IOMMU enabled
> [    0.518747] DMAR: Host address width 46
> [    0.526616] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
> [    0.537447] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap
> d2078c106f0466 ecap f020df
> [    0.553620] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
> [    0.564445] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap
> d2078c106f0466 ecap f020df
> [    0.580611] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff
> [    0.593344] DMAR: ATSR flags: 0x0
> [    0.600178] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0
> [    0.612905] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1
> [    0.625632] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000 IOMMU 0
> [    0.638522] DMAR-IR: IOAPIC id 1 under DRHD base  0xc7ffc000 IOMMU 1
> [    0.651426] DMAR-IR: IOAPIC id 2 under DRHD base  0xc7ffc000 IOMMU 1
> [    0.664324] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
> [    0.675326] DMAR-IR: Queued invalidation will be enabled to support
> x2apic and Intr-remapping.
> [    0.693805] DMAR-IR: Enabled IRQ remapping in x2apic mode
> [    9.395170] DMAR: dmar1: Using Queued invalidation
> [    9.405011] DMAR: Setting RMRR:
> [    9.412006] DMAR: Setting identity map for device 0000:00:1d.0
> [0x7bbc6000 - 0x7bbd4fff]
> [    9.428569] DMAR: Prepare 0-16MiB unity mapping for LPC
> [    9.439712] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
> 0xffffff]
> [    9.454684] DMAR: Intel(R) Virtualization Technology for Directed I/O
> [  287.023068] DMAR: DRHD: handling fault status reg 2
> [  287.023073] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 18a260a000 [fault reason 06] PTE Read access is not set
> [  287.023180] DMAR: DRHD: handling fault status reg 102
> [  287.023183] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 18a3010000 [fault reason 06] PTE Read access is not set
> [  287.038250] DMAR: DRHD: handling fault status reg 202
> [  287.038252] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 18a3010000 [fault reason 06] PTE Read access is not set
> [  288.170165] DMAR: DRHD: handling fault status reg 302
> [  288.170170] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [  288.694496] DMAR: DRHD: handling fault status reg 402
> [  288.694499] DMAR: [DMA Read] Request device [04:10.2] fault addr
> 189069c000 [fault reason 06] PTE Read access is not set
> [  289.927113] DMAR: DRHD: handling fault status reg 502
> [  289.927116] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [  290.174275] DMAR: DRHD: handling fault status reg 602
> [  290.174279] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [  292.174247] DMAR: DRHD: handling fault status reg 702
> [  292.174251] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [  294.174227] DMAR: DRHD: handling fault status reg 2
> [  294.174230] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [  296.174216] DMAR: DRHD: handling fault status reg 102
> [  296.174219] DMAR: [DMA Read] Request device [01:00.0] fault addr
> 1890754000 [fault reason 06] PTE Read access is not set
> [root@infradev-comp006.naw02.infradev.viasat.io ~]
> #
> 
> Thanks.
> 

Hi Ravi,

The "iommu=pt" workaround applies only when you want to use igb_uio 
driver. VFIO driver is able to fully utilize IOMMU without the need for 
pass-through mode. From your log i can see that some devices are bound 
to igb_uio while others are bound to vfio-pci. Just bind all of the 
devices you want to use with DPDK to vfio-pci and these errors should go 
away.

-- 
Thanks,
Anatoly