From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Lamparter Subject: mlx4 w/ IOMMU broken, kernel 3.12.6 Date: Wed, 8 Jan 2014 14:49:17 +0100 Message-ID: <20140108134917.GC351037@jupiter.n2.diac24.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: netdev@vger.kernel.org To: Amir Vadai Return-path: Received: from spaceboyz.net ([87.106.131.203]:40604 "EHLO spaceboyz.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756524AbaAHORy (ORCPT ); Wed, 8 Jan 2014 09:17:54 -0500 Content-Disposition: inline Sender: netdev-owner@vger.kernel.org List-ID: Hi, mlx4 is currently broken when an IOMMU is enabled, the driver does not seem to set up regions correctly (and also, the card seems to access things before its driver is loaded): (This is the in-kernel driver. The separately distributed 1.5.10 Mellanox driver does not seem to build against kernel 3.12.6.) [ 1.897271] IOMMU 0 0xfbffc000: using Queued invalidation [ 1.897569] IOMMU: Setting RMRR: [ 1.897872] IOMMU: Setting identity map for device 0000:00:1d.0 [0x7dea2000 - 0x7deaefff] [ 1.898415] IOMMU: Setting identity map for device 0000:00:1a.0 [0x7dea2000 - 0x7deaefff] [ 1.898930] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 1.899217] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 1.899726] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 1.902086] dmar: DRHD: handling fault status reg 2 [ 1.902362] dmar: DMAR:[DMA Read] Request device [06:00.0] fault addr 72469000 DMAR:[fault reason 01] Present bit in root entry is clear (...) [ 3.299985] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) [ 3.300247] mlx4_core: Initializing 0000:06:00.0 [ 4.306091] dmar: DRHD: handling fault status reg 102 [ 4.306359] dmar: DMAR:[DMA Read] Request device [06:00.0] fault addr 7236f000 DMAR:[fault reason 06] PTE Read access is not set [ 14.309294] mlx4_core 0000:06:00.0: command 0x4 timed out (go bit not cleared) [ 14.309759] mlx4_core 0000:06:00.0: QUERY_FW command failed, aborting. [ 15.313896] mlx4_core: probe of 0000:06:00.0 failed with error -110 (PCI/IOMMU Host is an Intel Xeon E5-2630v2) Since the driver hasn't been touched in net-next, I assume this issue hasn't been fixed in the time since 3.12 was released; my apologies if this is incorrect. Card information: # lspci -vvnns 06:00.0 06:00.0 Ethernet controller [0200]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003] Subsystem: Mellanox Technologies Device [15b3:0077] Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-