From mboxrd@z Thu Jan 1 00:00:00 1970 From: "J.L. Burr" Subject: Re: [PATCH 0/1] mthca: Modify to support embedded PowerPC platforms Date: Fri, 7 Jan 2011 04:24:34 -0500 Message-ID: References: Return-path: Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: linux-rdma@vger.kernel.org > Unfortunately it's very difficult to debug problems with the denx > variant of an 18-month-old kernel. I realize that I'm not only swimming in a backwater (embedded PowerPC), but a rather fetid one at that (2.6.26). I'm trying to evaluate how Infiniband would work for us, using an existing product base so cannot easily change levels. I *greatly* appreciate your forbearance and patience! > It might be instructive to watch /proc/interrupts on the ppc system and > see if it goes up when the x86 subnet manager does a sweep. > Also, do you have lspci on the ppc system? What does "lspci -vvvxx" show? I've captured this information but let me mention some more about my environment first. (1) There's no support for MSI / MSI-X interrupts on this platform at the 2.6.26 level. I'm using conventional IRQs with mthca, and loading it with the module parameter msi_x=0 just to be sure. (2) I've adapted and applied two "reachahead" mthca patches: I looked at all changes to mthca since 2.6.26 and found only one that looked interesting (DMA coherency related). The other one looked good from a performance standpoint. These are both from you: commit e8bb4beb2b1f90d499134f2849727ed04c3bedc4 Author: Roland Dreier Date: Tue Jul 22 14:20:05 2008 -0700 IB/mthca: Keep free count for MTT buddy allocator commit 99987bea474ceca8ec6fb05f81d7d188634cdffd Author: Roland Dreier Date: Mon Jun 22 23:04:13 2009 -0700 IB/mthca: Replace dma_sync_single() use with proper functions These patches had no effect on the problem and were present in the system when I captured the information below. (3) I've added a few printk's of my own as well as enabling all code in the driver which looked to be debugging-related. There were a number "if (0)" conditionals for debugging which I have changed to "if (1)". Unfortunately, I don't know anything about the Mellanox hardware so wasn't able to learn anything useful from this. I'm hoping this info might provide additional insight. I think the *big* change since 2.6.19 was the retirement of the old "ppc" Linux tree and its replacement with the "powerpc" tree. I settled on using 2.6.26 two years ago because this transition from ppc to powerpc had been completed. I think it was the merger of the 64-bit PowerPC with the 32-bit ppc that led to things like the phys_addr_t type, etc. There must be some other aspect new to the powerpc tree which requires adjustment by the driver. I've had some experience with this, in getting some of the other drivers to work in this environment (e.g. a FC driver and some work with USB drivers). But I haven't been able to figure out what's wrong here. I suppose it is possible that the problem is not in the driver, but elsewhere in the 2.6.26 kernel. I've reviewed the I/O mappings and 440 core definitions but have not found anything wrong. I'm including two attachments: (A) ib-console.txt This is the Linux console showing the Infiniband driver load. The card is using IRQ 18 and I have displayed /proc/interrupts before loading the IB drivers, after loading the IB drivers, and after starting the subnet manager on the Intel host. While a number of interrupts are seen during the driver loading, *no* interrupts occur as a result of starting the subnet manager on the Intel host. I assume the use of conventional IRQ is OK, as the driver did its NOP test and it passed. But I take it that the incoming messages from the subnet manager should have caused additional IRQs to occur (but did not). I've also displayed /proc/iomem to show the I/O mappings before and after loading the drivers. That's to show that the phys_addr_t changes are working and doing the right thing (I think they are). I'm loading all IB drivers in my universe, which is overkill at this point, I know. I've also tried just loading the three that would seem minimally required to get the subnet manager on the other host happy (ib_core, ib_mad, and ib_mthca). This does not change the symptoms. To be clear: I did not have the subnet manager running at the time of loading the drivers. I waited until they had come up and then started the subnet manager. I have marked this point by manually invoking the logger utility to write a message to the messages file: logger -t JLB about to start subnet manager (B) ib-messages.txt This is the 440SPe Linux messages file, showing the drivers being loaded as well as the driver debugging messages. Note that *nothing* happens after the "about to start subnet manager" entry. Should I look at trying to dump some part of the Mellanox card memory areas? In December, I had used GDB against the live system using /proc/kcore. I used this to display and format the driver's struct mthca_dev. I was hoping something might "leap out at me" (but nothing did!). I just don't know enough about the Mellanox hardware to know what to check. For example, if there's a queue showing pending work, etc. But I'll stop here, as I fear this note may already be too big! Thanks again for your help. John Burr begin 666 ib-console.txt M?B C(&-A=" O<')O8R]I;G1E$!D,C P,# P,# *?B C"GX@(PI^(",*?B C("]W9F)N9G,O;F-C<"]S M8W)I<'1S+VEB+FEN:2XS"E-E='1I;F<@86-T:6]N('1O.B @;&]A9 I!8W1I M;VXZ("!L;V%D"D-R96%T:6YG("]V87(O8V%C:&4O;W!E;G-M(&1I0I#;W!Y:6YG("]E=&,O=61E=B]R=6QE6EN9R O;&EB+VQI8F]S;79E;F1O M"TR+C8N,C8O9')I=F5R"TR+C8N,C8O9')I=F5R"TR+C8N,C8O9')I=F5R"TR+C8N,C8O9')I=F5R"!) M;F9I;FE"86YD($A#02!DFEN9R P,# P.C$V.C P+C *:6)?;71H8V$@,# P M,#HQ-CHP,"XP.B!E;F%B;&EN9R!D979I8V4@*# P,# @+3X@,# P,BD*"TR+C8N M,C8O9')I=F5R$!D,C P,# P,# *"GX@ M(R!C870@+W!R;V,O:6YT97)R=7!T'AX3H@, H@(" @(" @($EN=&5RF4].$U="B @(" @(" @ M0V%P86)I;&ET:65S.B!;-#!=(%!O=V5R($UA;F%G96UE;G0@=F5R'!R97-S("AV,2D@16YD<&]I;G0L($U322 P, H@ M(" @(" @(" @(" @(" @1&5V0V%P.B!-87A087EL;V%D(#$R."!B>71E'148671E#@L($%34$T@3#!S+"!,871E;F-Y M($PP('5N;&EM:71E9"P@3#$@=6YL:6UI=&5D"B @(" @(" @(" @(" @(" @ M(" @(" @($-L;V-K4$TM(%-U71E($E"32!)24,*0D%$.B @(" @(" @(" P"@I^(",* ` end begin 666 ib-messages.txt M2F%N(" W(# R.C4X.C4R($Y#0U @=7-E6EN9R O971C+W5D978O64N:V\*2F%N(" W(# R M.C4X.C4R($Y#0U @=7-E"TR+C8N,C8O9')I=F5R"TR+C8N,C8O9')I=F5R"TR M+C8N,C8O9')I=F5R"TR+C8N,C8O9')I=F5R M"TR+C8N,C8O9')I=F5R"TR M+C8N,C8O9')I=F5R2 H2!S:7IE.B R-38*2F%N(" W(# R.C4X M.C4U($Y#0U @=7-E2!S:7IE.B S,@I*86X@(#<@,#(Z-3@Z-34@3D-#4"!UF4Z(#8T"DIA;B @-R P M,CHU.#HU-2!.0T-0('5S97(N9&5B=6<@:V5R;F5L.B!I8E]M=&AC82 P,# P M.C$V.C P+C Z($UA>"!%47,Z(#8T+"!R97-EF4Z(#8T"DIA;B @-R P,CHU.#HU-2!.0T-0('5S97(N9&5B=6<@:V5R M;F5L.B!I8E]M=&AC82 P,# P.C$V.C P+C Z(')E"!01',Z M(#@S.#@V,#@L(')E"!#445S.B Q,S$P-S(L M(&UA>"!7445S.B Q-C,X-"P@;6%X(%-242!7445S.B Q-C,X- I*86X@(#<@ M,#(Z-3@Z-34@3D-#4"!U" T,# P,# P*0I* M86X@(#<@,#(Z-3@Z-34@3D-#4"!U" @(" @ M(" @,C0P,# P,# @*'-I>F4@,'@@,3 P,# P,"D*2F%N(" W(# R.C4X.C4U M($Y#0U @=7-E" @.# P,# P M*0I*86X@(#<@,#(Z-3@Z-34@3D-#4"!U" @ M(" @(" @,C8P,# P,# @*'-I>F4@,'@@(#0P,# P,"D*2F%N(" W(# R.C4X M.C4U($Y#0U @=7-E" @(#0P M,# P*0I*86X@(#<@,#(Z-3@Z-34@3D-#4"!U" @(" @(" @,C8V-# P,# @*'-I>F4@,'@@(" Q,# P,"D*2F%N(" W(# R M.C4X.C4U($Y#0U @=7-E" @ M(" Q,# P*0I*86X@(#<@,#(Z-3@Z-34@3D-#4"!U" @(" @(" @,C8V-3DP,# @*'-I>F4@,'@@(" @,3 P,"D*2F%N(" W M(# R.C4X.C4U($Y#0U @=7-E" @(" Q,# P*0I*86X@(#<@,#(Z-3@Z-34@3D-#4"!U2X*2F%N(" W(# R.C4X.C4U($Y#0U @=7-E2!K97D@=&AR;W5G:'!U="!O<'1I M;6EZ871I;VX@86-T:79A=&5D+@I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U2 P,# P,3 P,#H*2F%N(" W(# R.C4X.C4V($Y#0U @=7-E M2 P,# P,3$P M,#H*2F%N(" W(# R.C4X.C4V($Y#0U @=7-E7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I* M86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@ M/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z M-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E M+3YO=VYE7!E(#T@,$$O M,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N M97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E M+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@ M(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P M02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@ M3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO M=VYE7!E(#T@,$$O,# [ M(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@ M/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U M8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@ M,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P M,#L@97%E+3YO=VYE7!E M(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-# M4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE M7!E(#T@,$$O,# [(&5Q M92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P M, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y M<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z M-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@ M97%E+3YO=VYE7!E(#T@ M,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U M7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^ M;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I* M86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@ M/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z M-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E M+3YO=VYE7!E(#T@,$$O M,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N M97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E M+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@ M(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P M02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@ M3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO M=VYE7!E(#T@,$$O,# [ M(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@ M/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U M8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@ M,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P M,#L@97%E+3YO=VYE7!E M(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-# M4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE M7!E(#T@,$$O,# [(&5Q M92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P M, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y M<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z M-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@ M97%E+3YO=VYE7!E(#T@ M,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U M7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^ M;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I* M86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@ M/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z M-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E M+3YO=VYE7!E(#T@,$$O M,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N M97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E M+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@ M(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P M02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@ M3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO M=VYE7!E(#T@,$$O,# [ M(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@ M/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U M8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@ M,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P M,#L@97%E+3YO=VYE7!E M(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-# M4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE M7!E(#T@,$$O,# [(&5Q M92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P M, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y M<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z M-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@ M97%E+3YO=VYE7!E(#T@ M,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U M7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^ M;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O M,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@ M(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P M02\P,#L@97%E+3YO=VYE7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@ M/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U M8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z M-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@ M97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@ M/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U M8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^ M;W=N97(@/2 P, I*86X@(#<@,#(Z-3@Z-38@3D-#4"!U7!E+W-U8G1Y<&4@/2 P02\P,#L@97%E+3YO=VYE7!E(#T@,$$O,# [(&5Q92T^;W=N97(@/2 P, I*86X@(#<@,#(Z M-3@Z-38@3D-#4"!U2 P,# P,C,P,#H*2F%N(" W(# R M.C4X.C4V($Y#0U @=7-E