Sun V880 + Infiniband?

Sun V880 + Infiniband?

[Patrick Finnegan]

So, I'm trying to get a Mellanox IB card to work on my Sun V880, and I'm 
getting this error output to dmesg: 

[   99.529658] ib_mthca: Mellanox InfiniBand HCA driver v1.0 (April 4, 
2008)
[   99.609988] ib_mthca: Initializing 0003:01:00.0
[   99.664180] PCI: Enabling device: (0003:01:00.0), cmd 2
[   99.664193] ib_mthca 0003:01:00.0: Missing DCS, aborting.

I'm using a kernel compiled off of Linus's git tree as of a few days ago 
(to fix other SPARC issues).  From what I've seen from google searches, 
this means that the first BAR isn't mapped or visible to the driver, so 
maybe this is a SPARC related thing?

lspci -v output for the device:

0003:00:01.0 PCI bridge: Mellanox Technologies MT23108 PCI Bridge (rev 
a1) (prog-if 00 [Normal decode])
        Flags: bus master, 66MHz, slow devsel, latency 64
        [virtual] Memory at fffff80500000000 (32-bit, non-prefetchable) 
[size=1]
        [virtual] Memory at fffff80500000000 (32-bit, non-prefetchable) 
[size=1]
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        Memory behind bridge: 00100000-0fffffff
        [virtual] Expansion ROM at fffff80500000000 [disabled] [size=1]
        Capabilities: [70] PCI-X bridge device

0003:01:00.0 InfiniBand: Mellanox Technologies MT23108 InfiniHost (rev 
a1)
        Subsystem: Mellanox Technologies MT23108 InfiniHost
        Flags: 66MHz, slow devsel, IRQ 30
        Memory at fffff80500000000 (64-bit, non-prefetchable) [size=1]
        Memory at fffff80500000000 (64-bit, prefetchable) [size=1]
        Memory at fffff80500000000 (64-bit, prefetchable) [size=1]
        [virtual] Expansion ROM at fffff80500000000 [disabled] [size=1]
        Capabilities: [40] MSI-X: Enable- Mask- TabSize=32
        Capabilities: [50] Vital Product Data <?>
        Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ 
Queue=0/5 Enable-
        Capabilities: [70] PCI-X non-bridge device
        Kernel modules: ib_mthca


Pat
-- 
Purdue University Research Computing ---  http://www.rcac.purdue.edu/
The Computer Refuge                  ---  http://computer-refuge.org

Re: Sun V880 + Infiniband?

[Roland Dreier]

 > [   99.664193] ib_mthca 0003:01:00.0: Missing DCS, aborting.
 > 
 > I'm using a kernel compiled off of Linus's git tree as of a few days ago 
 > (to fix other SPARC issues).  From what I've seen from google searches, 
 > this means that the first BAR isn't mapped or visible to the driver, so 
 > maybe this is a SPARC related thing?

I think the problem is related to:

 >         Memory at fffff80500000000 (64-bit, non-prefetchable) [size=1]
 >         Memory at fffff80500000000 (64-bit, prefetchable) [size=1]
 >         Memory at fffff80500000000 (64-bit, prefetchable) [size=1]

So it says you have 3 BARs at the same address, all with size 1 (?!)
which means the PCI setup or probing is messed up.  The mthca driver
checks that the first BAR has size 1 megabyte as it should, and it's
bailing out because the kernel is telling it that it's the wrong size.

 - R.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Sun V880 + Infiniband?

[David Miller]

From: Roland Dreier <rdreier@cisco.com>
Date: Wed, 02 Dec 2009 09:55:19 -0800

> 
>  > [   99.664193] ib_mthca 0003:01:00.0: Missing DCS, aborting.
>  > 
>  > I'm using a kernel compiled off of Linus's git tree as of a few days ago 
>  > (to fix other SPARC issues).  From what I've seen from google searches, 
>  > this means that the first BAR isn't mapped or visible to the driver, so 
>  > maybe this is a SPARC related thing?
> 
> I think the problem is related to:
> 
>  >         Memory at fffff80500000000 (64-bit, non-prefetchable) [size=1]
>  >         Memory at fffff80500000000 (64-bit, prefetchable) [size=1]
>  >         Memory at fffff80500000000 (64-bit, prefetchable) [size=1]
> 
> So it says you have 3 BARs at the same address, all with size 1 (?!)
> which means the PCI setup or probing is messed up.  The mthca driver
> checks that the first BAR has size 1 megabyte as it should, and it's
> bailing out because the kernel is telling it that it's the wrong size.

These BARs are allocated and setup by the boot firmware long before
Linus boots up.  Not being able to handle 64-bit BARs properly
wouldn't surprise me.

But there could also be a Linux bug in decoding the openfirmware
property values as well, so let's investigate that.

Patrick, can you post a new "prtconf -pv" dump under Linux with this
card in the machine?  Also, please bootup with:

	of_debug=1 ofpci_debug=1

added to the kernel command line and post the resulting "dmesg".

If there's something Linux isn't doing right, those dumps will help
me spot it.

Thanks!

Re: Sun V880 + Infiniband?

[Patrick Finnegan]

On Wednesday 02 December 2009, David Miller wrote:
> From: Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
> Date: Wed, 02 Dec 2009 09:55:19 -0800
> 
> >  > [   99.664193] ib_mthca 0003:01:00.0: Missing DCS, aborting.
> >  >
> >  > I'm using a kernel compiled off of Linus's git tree as of a few
> >  > days ago (to fix other SPARC issues).  From what I've seen from
> >  > google searches, this means that the first BAR isn't mapped or
> >  > visible to the driver, so maybe this is a SPARC related thing?
> >
> > I think the problem is related to:
> >  >         Memory at fffff80500000000 (64-bit, non-prefetchable)
> >  > [size=1] Memory at fffff80500000000 (64-bit, prefetchable)
> >  > [size=1] Memory at fffff80500000000 (64-bit, prefetchable)
> >  > [size=1]
> >
> > So it says you have 3 BARs at the same address, all with size 1
> > (?!) which means the PCI setup or probing is messed up.  The mthca
> > driver checks that the first BAR has size 1 megabyte as it should,
> > and it's bailing out because the kernel is telling it that it's the
> > wrong size.
> 
> These BARs are allocated and setup by the boot firmware long before
> Linus boots up.  Not being able to handle 64-bit BARs properly
> wouldn't surprise me.
> 
> But there could also be a Linux bug in decoding the openfirmware
> property values as well, so let's investigate that.
> 
> Patrick, can you post a new "prtconf -pv" dump under Linux with this
> card in the machine?  Also, please bootup with:
> 
> 	of_debug=1 ofpci_debug=1
> 
> added to the kernel command line and post the resulting "dmesg".

I have placed an updated prtconf here:

http://ned.cc.purdue.edu/prtconf-v880-8cpu

The last two entries on it (PCI bridge and pci15b3,5a44) are from the 
card.

Based upon my best guess as to what the fields indicate, it looks like 
the OF tree may have the proper BAR sizes in it at least (I believe that 
they're supposed to be 1MB, 8MB and 128MB respectively), so I think it's 
a Linux thing not an OpenBoot thing.

The console output after booting with the debugging options is here:

http://ned.cc.purdue.edu/v880-of_debug-dmesg

I see that the output claims that the 3 BARs start and end at address 
"0" which doesn't look right.

> If there's something Linux isn't doing right, those dumps will help
> me spot it.

Thanks again!

Pat
-- 
Purdue University Research Computing ---  http://www.rcac.purdue.edu/
The Computer Refuge                  ---  http://computer-refuge.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Sun V880 + Infiniband?

[David Miller]

Please don't post in HTML, the list server rejects all such postings.
Use plain ASCII text only.

Thank you.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Sun V880 + Infiniband?

[David Miller]

From: Patrick Finnegan <pat@computer-refuge.org>
Date: Thu, 3 Dec 2009 01:44:18 -0500

> On Wednesday 02 December 2009, David Miller wrote:
>> If there's something Linux isn't doing right, those dumps will help
>> me spot it.
> 
> Thanks again!

Please give this patch a try:

sparc64: Fix overly strict range type matching for PCI devices.

When we are trying to see if a range property entry applies
to a given address, we are overly strict about the type.

We should only allow I/O ranges for I/O addresses, and only allow
CONFIG space ranges for CONFIG space address.

However for MEM ranges, they come in 32-bit and 64-bit flavors.
And a lack of an exact match is OK if the range is 32-bit and
the address is 64-bit.  We can assign a 64-bit address properly
into a 32-bit parent range just fine.

So allow it.

Reported-by: Patrick Finnegan <pat@computer-refuge.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
---
 arch/sparc/kernel/of_device_64.c |   14 ++++++++++++--
 1 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/arch/sparc/kernel/of_device_64.c b/arch/sparc/kernel/of_device_64.c
index 881947e..0a6f2d1 100644
--- a/arch/sparc/kernel/of_device_64.c
+++ b/arch/sparc/kernel/of_device_64.c
@@ -104,9 +104,19 @@ static int of_bus_pci_map(u32 *addr, const u32 *range,
 	int i;
 
 	/* Check address type match */
-	if ((addr[0] ^ range[0]) & 0x03000000)
-		return -EINVAL;
+	if (!((addr[0] ^ range[0]) & 0x03000000))
+		goto type_match;
+
+	/* Special exception, we can map a 64-bit address into
+	 * a 32-bit range.
+	 */
+	if ((addr[0] & 0x03000000) == 0x03000000 &&
+	    (range[0] & 0x03000000) == 0x02000000)
+		goto type_match;
+
+	return -EINVAL;
 
+type_match:
 	if (of_out_of_range(addr + 1, range + 1, range + na + pna,
 			    na - 1, ns))
 		return -EINVAL;
-- 
1.6.5

Re: Sun V880 + Infiniband?

[Patrick Finnegan]

On Wednesday 09 December 2009, David Miller wrote:
> Please give this patch a try:
> 
> sparc64: Fix overly strict range type matching for PCI devices.

This appears to work for me.

Thanks!

Pat
-- 
Purdue University Research Computing ---  http://www.rcac.purdue.edu/
The Computer Refuge                  ---  http://computer-refuge.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Sun V880 + Infiniband?

[David Miller]

From: Patrick Finnegan <pat-bdq14YP6qtQCo4FBHZViTkB+6BGkLq7r@public.gmane.org>
Date: Wed, 9 Dec 2009 14:29:56 -0500

> On Wednesday 09 December 2009, David Miller wrote:
>> Please give this patch a try:
>> 
>> sparc64: Fix overly strict range type matching for PCI devices.
> 
> This appears to work for me.
> 
> Thanks!

Thanks for your report and testing the fix.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html