Re: [PATCH 0/1] mthca: Modify to support embedded PowerPC platforms

["J.L. Burr" <jlburr-vna1KIf7WgpBDgjK7y7TUQ@public.gmane.org>]

I found the problem:

"We have met the enemy and he is us."

> OK, that's very interesting information.  The effect of that flag is to
> have the driver write firmware commands into the doorbell region (BAR 2)
> instead of the command region (BAR 0).  And it seems writes to the
> doorbell region are getting lost on your system (that might also explain
> why there were no packets received without the fw_cmd_doorbell flag;
> receives are posted to the HCA by doorbell writes too).  I suspect if we
> could get fw_cmd_doorbell=1 working on your system then that would fix
> everything else too.

Your insight that BAR 2 was not working while BAR 0 was working was 
extremely helpful!

BAR 2 uses the prefetchable attribute and is the only prefetchable 
memory in my system.  Then I remembered that we use a quirk routine to 
allocate resources for a custom device.  This is needed because the 
custom device has a huge BAR (petabyte in size), much larger than can be 
mapped to the 440.

The quirk routine forcibly changes the parent bridges' prefetchable 
ranges to accommodate the custom device's petabyte BAR address value. 
This is because only prefetchable ranges in the bridges accommodate 
64-bit PCIe addresses.  But the quirk routine does this change without 
regard to any prefetchable ranges already established by Linux 
enumeration in the parent bridges.  That is, it implicitly assumes there 
are no other prefetchable devices in the system.

Therefore, the quirk routine left the system with no viable forwarding 
path through the PCIe hierarchy to the Infiniband card's BAR 2.

And (sheepishly), *I* am the quirk routine author.  The mirror reveals 
the culprit!

I've temporarily disabled the custom device from my system, which 
prevents the quirk routine from running.

Everything now works!

I'm still loading mthca with fw_cmd_doorbell=1.  Exactly as you 
surmised, once the system could work in that mode, the whole thing just 
works.  I've got both card LEDs on now and can configure IPoIB links OK. 
  The subnet manager log looks good on the host workstation.

I now need to work on an improved quirk routine as I need the custom 
device and Infiniband device to work together.  I've also got to try to 
get some of the Infiniband tools compiled for the PowerPC.  The subnet 
manager cross-compiled with a bit of coaxing, some of the other tools 
look like a fair bit of work may be involved.

I'm *very* sorry this turned out to be a "user error".  I really 
appreciate your patience in helping me as I swam in my "fetid backwater" 
as mentioned above.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html