From mboxrd@z Thu Jan  1 00:00:00 1970
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Subject: Re: AMD_IOV: IO_PAGE_FALT trying to pass through
	Mellanox ConnectX HCA (debian testing)
Date: Fri, 28 Jan 2011 14:27:42 -0500
Message-ID: <20110128192742.GB23494@dumpdata.com>
References: <20110128185809.GA24465@countzero.vandewege.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Return-path: <xen-devel-bounces@lists.xensource.com>
Content-Disposition: inline
In-Reply-To: <20110128185809.GA24465@countzero.vandewege.net>
List-Unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xensource.com>
List-Help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-Subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
Sender: xen-devel-bounces@lists.xensource.com
Errors-To: xen-devel-bounces@lists.xensource.com
To: Ward Vandewege <ward@gnu.org>
Cc: xen-devel@lists.xensource.com
List-Id: xen-devel@lists.xenproject.org

> The mlx4_core driver in the domU is not happy:
> 
> [    0.411867] mlx4_core: Mellanox ConnectX core driver v0.01 (May 1, 2007)
> [    0.411879] mlx4_core: Initializing 0000:00:00.0
> [    0.412027] mlx4_core 0000:00:00.0: enabling device (0000 -> 0002)
> [    0.412027] mlx4_core 0000:00:00.0: Xen PCI enabling IRQ: 19
> [    1.417477] mlx4_core 0000:00:00.0: Installed FW has unsupported command
> interface revision 0.
> [    1.417509] mlx4_core 0000:00:00.0: (Installed FW version is 0.0.000)
> [    1.417527] mlx4_core 0000:00:00.0: This driver version supports only
> revisions 2 to 3.
> [    1.417549] mlx4_core 0000:00:00.0: QUERY_FW command failed, aborting.
> 
> When trying to boot a PV domU with kernel options iommu=soft and
> swiotlb=force, the output is slightly different. The full bootlog is attached

Don't use swiotlb=force unless neccessary.

> (testsqueeze.bootlog.txt). Here's the relevant excerpt:

That is b/c you are missing iommu=pv on Xen hypervisor line, and
you might need to make sure your driver is using the VM_IO flag.

There was some discussion on LKML about this and they proposed
a patch that wasn't neccessary. Don't remember the details but I can
look that up next week.
> 
> [    0.441684] mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.2
> (August 4, 2010)
> [    0.441696] mlx4_core: Initializing 0000:00:00.0
> [    0.442044] mlx4_core 0000:00:00.0: enabling device (0000 -> 0002)
> [    0.442741] mlx4_core 0000:00:00.0: Xen PCI enabling IRQ: 19
> [    2.752125] mlx4_core 0000:00:00.0: NOP command failed to generate MSI-X
> interrupt IRQ 54).
> [    2.752158] mlx4_core 0000:00:00.0: Trying again without MSI-X.
> [    2.884105] mlx4_core 0000:00:00.0: NOP command failed to generate
> interrupt (IRQ 54), aborting.
> [    2.884138] mlx4_core 0000:00:00.0: BIOS or ACPI interrupt routing