* [PATCH] management: adding mad_dump_fields to libibmad
From: Mike Heinz @ 2010-05-06 18:27 UTC (permalink / raw)
To: Sasha Khapyorsky,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
ewg-G2znmakfqn7U1rindQTSdQ@public.gmane.org
[-- Attachment #1: Type: text/plain, Size: 1825 bytes --]
Sasha asked that I re-submit the patches for perfquery in a slightly different format. This is the first of 3 patches.
This patch adds a function to libibmad that allows the caller to dump a configurable range of MAD attributes. Basically, this provides an external interface to the internal function _dump_fields.
Signed Off: Michael Heinz
---------------- snip --------------
diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h
index 02ef551..0478c2b 100644
--- a/libibmad/include/infiniband/mad.h
+++ b/libibmad/include/infiniband/mad.h
@@ -1031,6 +1031,9 @@ MAD_EXPORT ib_mad_dump_fn
mad_dump_perfcounters_xmt_disc, mad_dump_perfcounters_rcv_err,
mad_dump_portsamples_control;
+MAD_EXPORT void mad_dump_fields(char *buf, int bufsz, void *val, int valsz,
+ int start, int end);
+
MAD_EXPORT int ibdebug;
#if __BYTE_ORDER == __LITTLE_ENDIAN
diff --git a/libibmad/src/dump.c b/libibmad/src/dump.c
index 335e190..cc9c10f 100644
--- a/libibmad/src/dump.c
+++ b/libibmad/src/dump.c
@@ -671,6 +671,11 @@ static int _dump_fields(char *buf, int bufsz, void *data, int start, int end)
return (int)(s - buf);
}
+void mad_dump_fields(char *buf, int bufsz, void *val, int valsz, int start, int end)
+{
+ return _dump_fields(buf, bufsz, val, start, end);
+}
+
void mad_dump_nodedesc(char *buf, int bufsz, void *val, int valsz)
{
strncpy(buf, val, bufsz);
diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map
index e2d0b05..5778e3e 100644
--- a/libibmad/src/libibmad.map
+++ b/libibmad/src/libibmad.map
@@ -20,6 +20,7 @@ IBMAD_1.3 {
mad_dump_nodedesc;
mad_dump_nodeinfo;
mad_dump_opervls;
+ mad_dump_fields;
mad_dump_perfcounters;
mad_dump_perfcounters_ext;
mad_dump_perfcounters_xmt_sl;
[-- Attachment #2: dump_fields.patch --]
[-- Type: application/octet-stream, Size: 1395 bytes --]
diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h
index 02ef551..0478c2b 100644
--- a/libibmad/include/infiniband/mad.h
+++ b/libibmad/include/infiniband/mad.h
@@ -1031,6 +1031,9 @@ MAD_EXPORT ib_mad_dump_fn
mad_dump_perfcounters_xmt_disc, mad_dump_perfcounters_rcv_err,
mad_dump_portsamples_control;
+MAD_EXPORT void mad_dump_fields(char *buf, int bufsz, void *val, int valsz,
+ int start, int end);
+
MAD_EXPORT int ibdebug;
#if __BYTE_ORDER == __LITTLE_ENDIAN
diff --git a/libibmad/src/dump.c b/libibmad/src/dump.c
index 335e190..cc9c10f 100644
--- a/libibmad/src/dump.c
+++ b/libibmad/src/dump.c
@@ -671,6 +671,11 @@ static int _dump_fields(char *buf, int bufsz, void *data, int start, int end)
return (int)(s - buf);
}
+void mad_dump_fields(char *buf, int bufsz, void *val, int valsz, int start, int end)
+{
+ return _dump_fields(buf, bufsz, val, start, end);
+}
+
void mad_dump_nodedesc(char *buf, int bufsz, void *val, int valsz)
{
strncpy(buf, val, bufsz);
diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map
index e2d0b05..5778e3e 100644
--- a/libibmad/src/libibmad.map
+++ b/libibmad/src/libibmad.map
@@ -20,6 +20,7 @@ IBMAD_1.3 {
mad_dump_nodedesc;
mad_dump_nodeinfo;
mad_dump_opervls;
+ mad_dump_fields;
mad_dump_perfcounters;
mad_dump_perfcounters_ext;
mad_dump_perfcounters_xmt_sl;
^ permalink raw reply related
* RE: [PATCH] management: adding mad_dump_fields to libibmad
From: Mike Heinz @ 2010-05-06 18:30 UTC (permalink / raw)
To: Mike Heinz, Sasha Khapyorsky,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
ewg-G2znmakfqn7U1rindQTSdQ@public.gmane.org
In-Reply-To: <4C2744E8AD2982428C5BFE523DF8CDCB49A4740CD8-amwN6d8PyQWXx9kJd3VG2h2eb7JE58TQ@public.gmane.org>
Sasha asked that I re-submit the patches for perfquery in a slightly different format. This is the second of 3 patches.
This patch uses the new mad_dump_fields function to suppress the display of extended attributes when querying switches that do not support them.
Signed off: Michael Heinz
---------- snip ----------
diff --git a/infiniband-diags/src/perfquery.c b/infiniband-diags/src/perfquery.c
index 00ebfff..07a9226 100644
--- a/infiniband-diags/src/perfquery.c
+++ b/infiniband-diags/src/perfquery.c
@@ -302,7 +302,10 @@ static void dump_perfcounters(int extended, int timeout, uint16_t cap_mask,
if (aggregate)
aggregate_perfcounters();
else
- mad_dump_perfcounters(buf, sizeof buf, pc, sizeof pc);
+ mad_dump_fields(buf, sizeof buf, pc, sizeof pc,
+ IB_PC_FIRST_F,
+ (cap_mask & 0x1000)?IB_PC_LAST_F:(IB_PC_RCV_PKTS_F+1));
+
} else {
if (!(cap_mask & 0x200)) /* 1.2 errata: bit 9 is extended counter support */
IBWARN
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* RE: [PATCH] management: adding mad_dump_fields to libibmad
From: Mike Heinz @ 2010-05-06 18:34 UTC (permalink / raw)
To: Mike Heinz, Sasha Khapyorsky,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
ewg-G2znmakfqn7U1rindQTSdQ@public.gmane.org
In-Reply-To: <4C2744E8AD2982428C5BFE523DF8CDCB49A4740CD8-amwN6d8PyQWXx9kJd3VG2h2eb7JE58TQ@public.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]
Sasha asked that I re-submit the patches for perfquery in a slightly different format. This is the third of 3 patches.
This patch corrects the AllPortSelect error message that is generated by ibcheckerrors when used against switches that do not support that attribute.
Signed-off-by: Michael Heinz
--------- snip -------
diff --git a/infiniband-diags/scripts/ibcheckerrs.in b/infiniband-diags/scripts/ibcheckerrs.in
index 305379a..15bfd4a 100644
--- a/infiniband-diags/scripts/ibcheckerrs.in
+++ b/infiniband-diags/scripts/ibcheckerrs.in
@@ -155,6 +155,14 @@ nodename=`$IBPATH/smpquery $ca_info nodedesc $lid | sed -e "s/^Node Description:
text="`eval $IBPATH/perfquery $ca_info $lid $portnum`"
rv=$?
+if echo $text | grep -q 'AllPortSelect not supported'; then
+ if [ "$verbose" = "yes" ]; then
+ echo -n "Error check on lid $lid ($nodename) port $portname: "
+ green "AllPortSelect not supported"
+ fi
+ exit 0
+fi
+
if echo "$text" | awk -v mono=$bw -v brief=$brief -F '[.:]*' '
function blue(s)
{
[-- Attachment #2: allportselect.patch --]
[-- Type: application/octet-stream, Size: 689 bytes --]
diff --git a/infiniband-diags/scripts/ibcheckerrs.in b/infiniband-diags/scripts/ibcheckerrs.in
index 305379a..15bfd4a 100644
--- a/infiniband-diags/scripts/ibcheckerrs.in
+++ b/infiniband-diags/scripts/ibcheckerrs.in
@@ -155,6 +155,14 @@ nodename=`$IBPATH/smpquery $ca_info nodedesc $lid | sed -e "s/^Node Description:
text="`eval $IBPATH/perfquery $ca_info $lid $portnum`"
rv=$?
+if echo $text | grep -q 'AllPortSelect not supported'; then
+ if [ "$verbose" = "yes" ]; then
+ echo -n "Error check on lid $lid ($nodename) port $portname: "
+ green "AllPortSelect not supported"
+ fi
+ exit 0
+fi
+
if echo "$text" | awk -v mono=$bw -v brief=$brief -F '[.:]*' '
function blue(s)
{
^ permalink raw reply related
* business marketing database
From: Vance Cornell @ 2010-05-06 19:20 UTC (permalink / raw)
To: rollercoasterflight-Re5JQEeQqe8AvxtiuMwx3w, 32353rbn-j9N7ZwQwUs0,
info-T/zw26LcSym7ZbX1PVObng,
webmaster-P0NvWEq3wAmshxzGDNidGixXY32XiHfO,
paulbiz57-/E1597aS9LQAvxtiuMwx3w
Here's some of the healthcare lists we have:
Pharmaceutical Companies - Email only list 47,000 emails of pharma company employees
Physical Therapists - 125,460 total records with 5,483 emails and 4,405 fax numbers
Oncology Doctors - 2,200 records all with emails
Theres many more too, just send me an email here for additional info/samples: Vivian.Temple-k2xzaFxgLweHQwpFp6j/jMXXUOn6P5/W@public.gmane.org
To invoke no further correspondence status please send an email to disappear-k2xzaFxgLweHQwpFp6j/jMXXUOn6P5/W@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] libibverbs: Add huge page support to ibv_madvise_range()
From: Roland Dreier @ 2010-05-06 20:51 UTC (permalink / raw)
To: alexv-smomgflXvOZWk0Htik3J/w
Cc: roland, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
alexr-smomgflXvOZWk0Htik3J/w
In-Reply-To: <4BCFFC48.4060401-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> In order to avoid adding additional dependency to libibverbs, maybe
> we should just to enhance the get_huge_page_size() so it will support
> multiple huge page sizes?
I think that does make sense. However see my reply to Alexander
Schmidt -- maybe there is something clever we can do with /proc/*/maps
to figure out when we need to use this?
- R.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [RFC] libibverbs: ibv_fork_init() and libhugetlbfs
From: Roland Dreier @ 2010-05-06 20:55 UTC (permalink / raw)
To: Alexander Schmidt
Cc: of-ewg, Linux RDMA, Hoang-Nam Nguyen, Stefan Roscher,
Joachim Fenkes, Christoph Raisch, Alex Vainman
In-Reply-To: <20100506093949.55916ab0@alex-laptop>
> When fork support is enabled in libibverbs, madvise() is called for every
> memory page that is registered as a memory region. Memory ranges that
> are passed to madvise() must be page aligned and the size must be a
> multiple of the page size. libibverbs uses sysconf(_SC_PAGESIZE) to find
> out the system page size and rounds all ranges passed to reg_mr() according
> to this page size. When memory from libhugetlbfs is passed to reg_mr(), this
> does not work as the page size for this memory range might be different
> (e.g. 16Mb). So libibverbs would have to use the huge page size to
> calculate a page aligned range for madvise.
Yes, Alex Vainman reaised this same issue a while ago.
> The patch below demonstrates a possible solution for this. It parses the
> /proc/PID/maps file when registering a memory region and decides if the
> memory that is to be registered is part of a libhugetlbfs range or not. If so,
> a page size of 16Mb is used to align the memory range passed to madvise().
>
> We see two problems with this: it is not a very elegant solution to parse the
> procfs file and the 16Mb are hardcoded currently. The latter point could be
> solved by calling gethugepagesize() from libhugetlbfs, which would add a new
> dependency to libibverbs.
I think that we cannot assume huge pages only come from libhugetlbfs --
we should support an application directly enabling huge pages (possibly
via another library too, so we can't assume that an application knows
the page size for a memory range it is about to register).
And also the 16 MB page size constant is of course not feasible -- with
all due respect, the x86 page size of 2 MB is much more likely in
practice :) (Although perhaps the much slower PowerPC TLB refill makes
users more likely to try and use hugetlb pages ;)
Alex suggested parsing files in the same way as libhugetlbfs does to get
the page size, and that seems to be the best solution, since I don't
think the libhugetlbfs license is compatible with the BSD license for
libibverbs.
But your trick of using /proc/*/maps looks nice. Does that only work
for libhugetlbfs or can we recognize direct mmap of hugetlb pages?
- R.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] management: adding mad_dump_fields to libibmad
From: Sasha Khapyorsky @ 2010-05-06 21:02 UTC (permalink / raw)
To: Mike Heinz
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
ewg-G2znmakfqn7U1rindQTSdQ@public.gmane.org
In-Reply-To: <4C2744E8AD2982428C5BFE523DF8CDCB49A4740CD8-amwN6d8PyQWXx9kJd3VG2h2eb7JE58TQ@public.gmane.org>
On 13:27 Thu 06 May , Mike Heinz wrote:
> Sasha asked that I re-submit the patches for perfquery in a slightly different format. This is the first of 3 patches.
I just asked to try to follow the normal patch submission format
described in details there:
http://git.kernel.org/?p=linux/kernel/git/roland/infiniband.git;a=blob;f=Documentation/SubmittingPatches
So each patch will have its own subject line, commit message, etc.
> This patch adds a function to libibmad that allows the caller to dump a configurable range of MAD attributes. Basically, this provides an external interface to the internal function _dump_fields.
>
> Signed Off: Michael Heinz
All three applied. Thanks.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [ewg] ibcheckerrors "Port All FAILED" reported
From: Ira Weiny @ 2010-05-06 21:08 UTC (permalink / raw)
To: Sasha Khapyorsky
Cc: Woodruff, Robert J,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, EWG,
tziporet-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org
In-Reply-To: <20100506211124.GH7099-o14lFNPAa+WKTadZzrrH2Q@public.gmane.org>
On Thu, 6 May 2010 14:11:24 -0700
Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
> On 18:09 Wed 05 May , Ira Weiny wrote:
> >
> > 14:29:03 > ./perfquery 40 255
> > ./perfquery: iberror: failed: AllPortSelect not supported
> >
> > It seems there is an issue with the CapabilityMask value...
> >
> > 14:43:32 > ./perfquery 40 255
> > cap_mask 0x400 <=== my debug output
> > ./perfquery: iberror: failed: AllPortSelect not supported
> >
> > 14:43:38 > ./saquery CPI 40
> > SA ClassPortInfo:
> > ...
> > Capability mask..........0x2602
> > ...
> >
> > Those don't match because... perfquery has a bug...
> >
> > perfquery is issuing a PMA query when it should be issuing a SA query.
>
> I'm not following. How should it be related to each other SA and PM
> ClassPortInfo(s)?
It's not, I was confused... :-D
Ira
>
> Sasha
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: ibcheckerrors "Port All FAILED" reported
From: Sasha Khapyorsky @ 2010-05-06 21:11 UTC (permalink / raw)
To: Ira Weiny; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, EWG
In-Reply-To: <20100505180943.a9bbb74e.weiny2-i2BcT+NCU+M@public.gmane.org>
On 18:09 Wed 05 May , Ira Weiny wrote:
>
> 14:29:03 > ./perfquery 40 255
> ./perfquery: iberror: failed: AllPortSelect not supported
>
> It seems there is an issue with the CapabilityMask value...
>
> 14:43:32 > ./perfquery 40 255
> cap_mask 0x400 <=== my debug output
> ./perfquery: iberror: failed: AllPortSelect not supported
>
> 14:43:38 > ./saquery CPI 40
> SA ClassPortInfo:
> ...
> Capability mask..........0x2602
> ...
>
> Those don't match because... perfquery has a bug...
>
> perfquery is issuing a PMA query when it should be issuing a SA query.
I'm not following. How should it be related to each other SA and PM
ClassPortInfo(s)?
Sasha
^ permalink raw reply
* Re: [PATCH] infiniband-diags/scripts/ibcheckerrs.in: emulate all ports if necessary.
From: Sasha Khapyorsky @ 2010-05-06 21:13 UTC (permalink / raw)
To: Ira Weiny
Cc: Robert Woodruff,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Michael Heinz,
Openfabrics-ewg-0P3JtQMG0aQdnm+yROfE0A
In-Reply-To: <20100506091829.dd275edf.weiny2-i2BcT+NCU+M@public.gmane.org>
On 09:18 Thu 06 May , Ira Weiny wrote:
> Upon thinking about this a bit more, and seeing Mikes patch. I think that the
> patch which Mike sent some time ago is a better fix. This will work fine for
> ibcheckerrs. However ibcheckerrors will run AllPortSelect and then go on to
> query all the ports individually. The patch below will cause a double read
> for each port which will kill ibcheckerrors performance on a large cluster.
>
> Sasha, what is the status of Mikes patch?
Applied now.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] management: adding mad_dump_fields to libibmad
From: Mike Heinz @ 2010-05-06 21:19 UTC (permalink / raw)
To: Sasha Khapyorsky
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
ewg-G2znmakfqn7U1rindQTSdQ@public.gmane.org
In-Reply-To: <20100506210248.GG7099-o14lFNPAa+WKTadZzrrH2Q@public.gmane.org>
Sasha, thanks for sending me that.
Despite asking several times over the past couple of years, you're the first person to actually point me to a document on how to submit patches to the group.
I will be sure to adhere to that format in the future.
-----Original Message-----
From: Sasha Khapyorsky [mailto:sashakvolt-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org] On Behalf Of Sasha Khapyorsky
Sent: Thursday, May 06, 2010 5:03 PM
To: Mike Heinz
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; ewg-G2znmakfqn7U1rindQTSdQ@public.gmane.org
Subject: Re: [PATCH] management: adding mad_dump_fields to libibmad
On 13:27 Thu 06 May , Mike Heinz wrote:
> Sasha asked that I re-submit the patches for perfquery in a slightly different format. This is the first of 3 patches.
I just asked to try to follow the normal patch submission format
described in details there:
http://git.kernel.org/?p=linux/kernel/git/roland/infiniband.git;a=blob;f=Documentation/SubmittingPatches
So each patch will have its own subject line, commit message, etc.
> This patch adds a function to libibmad that allows the caller to dump a configurable range of MAD attributes. Basically, this provides an external interface to the internal function _dump_fields.
>
> Signed Off: Michael Heinz
All three applied. Thanks.
Sasha
^ permalink raw reply
* Re: [RESEND] [PATCH/RFC] opensm: toggle sweeping
From: Sasha Khapyorsky @ 2010-05-06 22:14 UTC (permalink / raw)
To: Arthur Kepner; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100428223321.GO8749-sJ/iWh9BUns@public.gmane.org>
Hi Arthur,
On 15:33 Wed 28 Apr , Arthur Kepner wrote:
>
> One of our customers recently merged some new systems into a
> large, existing cluster. They requested a mechanism to prevent
> opensm from sweeping while the new equipment was being added to
> the IB fabric, and then resume sweeping once they felt confident
> that the newly added (sub)fabric was correctly cabled, and fully
> functional. They used the following patch.
You want to disable sweep. So why do you need "toggling"?
Run OpenSM with sweep disabled, config parameters:
sweep_interval 0
sweep_on_trap FALSE
And when you are comfortable with doing resweep just rerun OpenSM
without such options (or may be sending SIGHUP).
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] libibverbs: add path record definitions to sa.h
From: Sean Hefty @ 2010-05-06 22:22 UTC (permalink / raw)
To: Roland Dreier, linux-rdma
Add definitions for path record wire definition. This
will be used by the librdmacm and ib_acm service, and is
exchanged with the kernel using the newer set and query
route functionality.
Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
I went with the simpler definition, rather than defining bitfields.
Once a definition is accepted into the libibverbs git tree, I will
request that the same definition be pulled into the OFED 1.5.2 codebase.
include/infiniband/sa.h | 37 +++++++++++++++++++++++++++++++++++++
1 files changed, 37 insertions(+), 0 deletions(-)
diff --git a/include/infiniband/sa.h b/include/infiniband/sa.h
index ec90f14..f475c66 100644
--- a/include/infiniband/sa.h
+++ b/include/infiniband/sa.h
@@ -97,4 +97,41 @@ struct ibv_sa_service_rec {
uint64_t data64[2];
};
+#define IB_PATH_RECORD_REVERSIBLE 0x80
+
+struct ib_path_record
+{
+ uint64_t service_id;
+ union ibv_gid dgid;
+ union ibv_gid sgid;
+ uint16_t dlid;
+ uint16_t slid;
+ uint32_t flowlabel_hoplimit; /* resv-31:28 flow label-27:8 hop limit-7:0*/
+ uint8_t tclass;
+ uint8_t reversible_numpath; /* reversible-7:7 num path-6:0 */
+ uint16_t pkey;
+ uint16_t qosclass_sl; /* qos class-15:4 sl-3:0 */
+ uint8_t mtu; /* mtu selector-7:6 mtu-5:0 */
+ uint8_t rate; /* rate selector-7:6 rate-5:0 */
+ uint8_t packetlifetime; /* lifetime selector-7:6 lifetime-5:0 */
+ uint8_t preference;
+ uint8_t reserved[6];
+};
+
+#define IB_PATH_FLAG_GMP (1<<0)
+#define IB_PATH_FLAG_PRIMARY (1<<1)
+#define IB_PATH_FLAG_ALTERNATE (1<<2)
+#define IB_PATH_FLAG_OUTBOUND (1<<3)
+#define IB_PATH_FLAG_INBOUND (1<<4)
+#define IB_PATH_FLAG_INBOUND_REVERSE (1<<5)
+#define IB_PATH_FLAG_BIDIRECTIONAL (IB_PATH_FLAG_OUTBOUND | \
+ IB_PATH_FLAG_INBOUND_REVERSE)
+
+struct ib_path_data
+{
+ uint32_t flags;
+ uint32_t reserved;
+ struct ib_path_record path;
+};
+
#endif /* INFINIBAND_SA_H */
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: libibnetdisc: Add grouping for Voltaire's ISR4700 switch
From: Sasha Khapyorsky @ 2010-05-06 22:27 UTC (permalink / raw)
To: sebastien dugue; +Cc: Yair Goldel, OF EWG, linux-rdma
In-Reply-To: <20100427143529.75d64f9f@frecb007965>
On 14:35 Tue 27 Apr , sebastien dugue wrote:
>
> The ISR4700 features 3 kind of boards:
>
> - sLB-4018 line board with a single 36 port asic
> - sFB-4700 fabric board with a single 36 port asic
> - sFB-4700X2 double density fabric board with 2 36 port asics
>
> The double density fabric board (sFB-4700X2) features external 12X
> connectors that are only an aggregation of 3 4X ports, therefore
> ext_portnum is set to match the number printed on the faceplate.
>
> Signed-off-by: Sebastien Dugue <sebastien.dugue-6ktuUTfB/bM@public.gmane.org>
Applied. Thanks.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] libibverbs: add path record definitions to sa.h
From: Roland Dreier @ 2010-05-06 22:33 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma
In-Reply-To: <2420730B531C4665BD842A7F46007DA2-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
> +#define IB_PATH_RECORD_REVERSIBLE 0x80
> +
> +struct ib_path_record
Can we use IBV_ / ibv_ prefixes here to match the rest of libibverbs?
Also having struct ibv_sa_path_rec and struct ibv_path_record is sort of
unfortunate. Not sure if I can think of a better name here though --
ibv_raw_path_record? ibv_packed_path_record? Oh well...
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 1/7] RDMA/cxgb4: Make ord/ird max for T4 match T3.
From: Steve Wise @ 2010-05-06 23:06 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Until I implement the MPA extensions Internet Draft, make the ord/ird
max values for T4 match those of T3 (8). Also add a module option to
allow upping the value for pure T4 environments.
Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
---
drivers/infiniband/hw/cxgb4/cm.c | 13 +++++++++++--
drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 1 +
drivers/infiniband/hw/cxgb4/provider.c | 4 ++--
drivers/infiniband/hw/cxgb4/qp.c | 4 ++--
drivers/infiniband/hw/cxgb4/t4.h | 1 -
5 files changed, 16 insertions(+), 7 deletions(-)
diff --git a/drivers/infiniband/hw/cxgb4/cm.c b/drivers/infiniband/hw/cxgb4/cm.c
index 07b068b..cf6dbf4 100644
--- a/drivers/infiniband/hw/cxgb4/cm.c
+++ b/drivers/infiniband/hw/cxgb4/cm.c
@@ -61,6 +61,10 @@ static char *states[] = {
NULL,
};
+int c4iw_max_read_depth = 8;
+module_param(c4iw_max_read_depth, int, 0644);
+MODULE_PARM_DESC(c4iw_max_read_depth, "Per-connection max ORD/IRD (default=8)");
+
static int enable_tcp_timestamps;
module_param(enable_tcp_timestamps, int, 0644);
MODULE_PARM_DESC(enable_tcp_timestamps, "Enable tcp timestamps (default=0)");
@@ -1904,8 +1908,8 @@ int c4iw_accept_cr(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param)
BUG_ON(state_read(&ep->com) != MPA_REQ_RCVD);
BUG_ON(!qp);
- if ((conn_param->ord > T4_MAX_READ_DEPTH) ||
- (conn_param->ird > T4_MAX_READ_DEPTH)) {
+ if ((conn_param->ord > c4iw_max_read_depth) ||
+ (conn_param->ird > c4iw_max_read_depth)) {
abort_connection(ep, NULL, GFP_KERNEL);
err = -EINVAL;
goto err;
@@ -1968,6 +1972,11 @@ int c4iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param)
struct net_device *pdev;
int step;
+ if ((conn_param->ord > c4iw_max_read_depth) ||
+ (conn_param->ird > c4iw_max_read_depth)) {
+ err = -EINVAL;
+ goto out;
+ }
ep = alloc_ep(sizeof(*ep), GFP_KERNEL);
if (!ep) {
printk(KERN_ERR MOD "%s - cannot alloc ep.\n", __func__);
diff --git a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
index ccce6fe..c3ea5a2 100644
--- a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
+++ b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
@@ -739,5 +739,6 @@ void c4iw_ev_dispatch(struct c4iw_dev *dev, struct t4_cqe *err_cqe);
extern struct cxgb4_client t4c_client;
extern c4iw_handler_func c4iw_handlers[NUM_CPL_CMDS];
+extern int c4iw_max_read_depth;
#endif
diff --git a/drivers/infiniband/hw/cxgb4/provider.c b/drivers/infiniband/hw/cxgb4/provider.c
index 3cb50af..dfc4902 100644
--- a/drivers/infiniband/hw/cxgb4/provider.c
+++ b/drivers/infiniband/hw/cxgb4/provider.c
@@ -267,8 +267,8 @@ static int c4iw_query_device(struct ib_device *ibdev,
props->max_qp_wr = T4_MAX_QP_DEPTH;
props->max_sge = T4_MAX_RECV_SGE;
props->max_sge_rd = 1;
- props->max_qp_rd_atom = T4_MAX_READ_DEPTH;
- props->max_qp_init_rd_atom = T4_MAX_READ_DEPTH;
+ props->max_qp_rd_atom = c4iw_max_read_depth;
+ props->max_qp_init_rd_atom = c4iw_max_read_depth;
props->max_cq = T4_MAX_NUM_CQ;
props->max_cqe = T4_MAX_CQ_DEPTH;
props->max_mr = c4iw_num_stags(&dev->rdev);
diff --git a/drivers/infiniband/hw/cxgb4/qp.c b/drivers/infiniband/hw/cxgb4/qp.c
index bd56c84..7ff6aea 100644
--- a/drivers/infiniband/hw/cxgb4/qp.c
+++ b/drivers/infiniband/hw/cxgb4/qp.c
@@ -1130,14 +1130,14 @@ int c4iw_modify_qp(struct c4iw_dev *rhp, struct c4iw_qp *qhp,
if (mask & C4IW_QP_ATTR_ENABLE_RDMA_BIND)
newattr.enable_bind = attrs->enable_bind;
if (mask & C4IW_QP_ATTR_MAX_ORD) {
- if (attrs->max_ord > T4_MAX_READ_DEPTH) {
+ if (attrs->max_ord > c4iw_max_read_depth) {
ret = -EINVAL;
goto out;
}
newattr.max_ord = attrs->max_ord;
}
if (mask & C4IW_QP_ATTR_MAX_IRD) {
- if (attrs->max_ird > T4_MAX_READ_DEPTH) {
+ if (attrs->max_ird > c4iw_max_read_depth) {
ret = -EINVAL;
goto out;
}
diff --git a/drivers/infiniband/hw/cxgb4/t4.h b/drivers/infiniband/hw/cxgb4/t4.h
index 3f0d217..aeeb005 100644
--- a/drivers/infiniband/hw/cxgb4/t4.h
+++ b/drivers/infiniband/hw/cxgb4/t4.h
@@ -36,7 +36,6 @@
#include "t4_msg.h"
#include "t4fw_ri_api.h"
-#define T4_MAX_READ_DEPTH 16
#define T4_QID_BASE 1024
#define T4_MAX_QIDS 256
#define T4_MAX_NUM_QP (1<<16)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 2/7] RDMA/cxgb4: shrink .text with compile-time init of handlers arrays
From: Steve Wise @ 2010-05-06 23:06 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100506230619.25362.97591.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
---
drivers/infiniband/hw/cxgb4/cm.c | 189 +++++++++++++++++++-------------------
1 files changed, 96 insertions(+), 93 deletions(-)
diff --git a/drivers/infiniband/hw/cxgb4/cm.c b/drivers/infiniband/hw/cxgb4/cm.c
index cf6dbf4..85418f3 100644
--- a/drivers/infiniband/hw/cxgb4/cm.c
+++ b/drivers/infiniband/hw/cxgb4/cm.c
@@ -117,13 +117,9 @@ static int snd_win = 32 * 1024;
module_param(snd_win, int, 0644);
MODULE_PARM_DESC(snd_win, "TCP send window in bytes (default=32KB)");
-static void process_work(struct work_struct *work);
static struct workqueue_struct *workq;
-static DECLARE_WORK(skb_work, process_work);
static struct sk_buff_head rxq;
-static c4iw_handler_func work_handlers[NUM_CPL_CMDS];
-c4iw_handler_func c4iw_handlers[NUM_CPL_CMDS];
static struct sk_buff *get_skb(struct sk_buff *skb, int len, gfp_t gfp);
static void ep_timeout(unsigned long arg);
@@ -275,26 +271,6 @@ static void release_ep_resources(struct c4iw_ep *ep)
c4iw_put_ep(&ep->com);
}
-static void process_work(struct work_struct *work)
-{
- struct sk_buff *skb = NULL;
- struct c4iw_dev *dev;
- struct cpl_act_establish *rpl = cplhdr(skb);
- unsigned int opcode;
- int ret;
-
- while ((skb = skb_dequeue(&rxq))) {
- rpl = cplhdr(skb);
- dev = *((struct c4iw_dev **) (skb->cb + sizeof(void *)));
- opcode = rpl->ot.opcode;
-
- BUG_ON(!work_handlers[opcode]);
- ret = work_handlers[opcode](dev, skb);
- if (!ret)
- kfree_skb(skb);
- }
-}
-
static int status2errno(int status)
{
switch (status) {
@@ -1799,36 +1775,6 @@ static int fw4_ack(struct c4iw_dev *dev, struct sk_buff *skb)
return 0;
}
-static int fw6_msg(struct c4iw_dev *dev, struct sk_buff *skb)
-{
- struct cpl_fw6_msg *rpl = cplhdr(skb);
- struct c4iw_wr_wait *wr_waitp;
- int ret;
-
- PDBG("%s type %u\n", __func__, rpl->type);
-
- switch (rpl->type) {
- case 1:
- ret = (int)((be64_to_cpu(rpl->data[0]) >> 8) & 0xff);
- wr_waitp = (__force struct c4iw_wr_wait *)rpl->data[1];
- PDBG("%s wr_waitp %p ret %u\n", __func__, wr_waitp, ret);
- if (wr_waitp) {
- wr_waitp->ret = ret;
- wr_waitp->done = 1;
- wake_up(&wr_waitp->wait);
- }
- break;
- case 2:
- c4iw_ev_dispatch(dev, (struct t4_cqe *)&rpl->data[0]);
- break;
- default:
- printk(KERN_ERR MOD "%s unexpected fw6 msg type %u\n", __func__,
- rpl->type);
- break;
- }
- return 0;
-}
-
static void ep_timeout(unsigned long arg)
{
struct c4iw_ep *ep = (struct c4iw_ep *)arg;
@@ -2253,6 +2199,49 @@ int c4iw_ep_disconnect(struct c4iw_ep *ep, int abrupt, gfp_t gfp)
}
/*
+ * These are the real handlers that are called from a
+ * work queue.
+ */
+static c4iw_handler_func work_handlers[NUM_CPL_CMDS] = {
+ [CPL_ACT_ESTABLISH] = act_establish,
+ [CPL_ACT_OPEN_RPL] = act_open_rpl,
+ [CPL_RX_DATA] = rx_data,
+ [CPL_ABORT_RPL_RSS] = abort_rpl,
+ [CPL_ABORT_RPL] = abort_rpl,
+ [CPL_PASS_OPEN_RPL] = pass_open_rpl,
+ [CPL_CLOSE_LISTSRV_RPL] = close_listsrv_rpl,
+ [CPL_PASS_ACCEPT_REQ] = pass_accept_req,
+ [CPL_PASS_ESTABLISH] = pass_establish,
+ [CPL_PEER_CLOSE] = peer_close,
+ [CPL_ABORT_REQ_RSS] = peer_abort,
+ [CPL_CLOSE_CON_RPL] = close_con_rpl,
+ [CPL_RDMA_TERMINATE] = terminate,
+ [CPL_FW4_ACK] = fw4_ack
+};
+
+static void process_work(struct work_struct *work)
+{
+ struct sk_buff *skb = NULL;
+ struct c4iw_dev *dev;
+ struct cpl_act_establish *rpl = cplhdr(skb);
+ unsigned int opcode;
+ int ret;
+
+ while ((skb = skb_dequeue(&rxq))) {
+ rpl = cplhdr(skb);
+ dev = *((struct c4iw_dev **) (skb->cb + sizeof(void *)));
+ opcode = rpl->ot.opcode;
+
+ BUG_ON(!work_handlers[opcode]);
+ ret = work_handlers[opcode](dev, skb);
+ if (!ret)
+ kfree_skb(skb);
+ }
+}
+
+static DECLARE_WORK(skb_work, process_work);
+
+/*
* All the CM events are handled on a work queue to have a safe context.
*/
static int sched(struct c4iw_dev *dev, struct sk_buff *skb)
@@ -2282,6 +2271,59 @@ static int set_tcb_rpl(struct c4iw_dev *dev, struct sk_buff *skb)
return 0;
}
+static int fw6_msg(struct c4iw_dev *dev, struct sk_buff *skb)
+{
+ struct cpl_fw6_msg *rpl = cplhdr(skb);
+ struct c4iw_wr_wait *wr_waitp;
+ int ret;
+
+ PDBG("%s type %u\n", __func__, rpl->type);
+
+ switch (rpl->type) {
+ case 1:
+ ret = (int)((be64_to_cpu(rpl->data[0]) >> 8) & 0xff);
+ wr_waitp = (__force struct c4iw_wr_wait *)rpl->data[1];
+ PDBG("%s wr_waitp %p ret %u\n", __func__, wr_waitp, ret);
+ if (wr_waitp) {
+ wr_waitp->ret = ret;
+ wr_waitp->done = 1;
+ wake_up(&wr_waitp->wait);
+ }
+ break;
+ case 2:
+ c4iw_ev_dispatch(dev, (struct t4_cqe *)&rpl->data[0]);
+ break;
+ default:
+ printk(KERN_ERR MOD "%s unexpected fw6 msg type %u\n", __func__,
+ rpl->type);
+ break;
+ }
+ return 0;
+}
+
+/*
+ * Most upcalls from the T4 Core go to sched() to
+ * schedule the processing on a work queue.
+ */
+c4iw_handler_func c4iw_handlers[NUM_CPL_CMDS] = {
+ [CPL_ACT_ESTABLISH] = sched,
+ [CPL_ACT_OPEN_RPL] = sched,
+ [CPL_RX_DATA] = sched,
+ [CPL_ABORT_RPL_RSS] = sched,
+ [CPL_ABORT_RPL] = sched,
+ [CPL_PASS_OPEN_RPL] = sched,
+ [CPL_CLOSE_LISTSRV_RPL] = sched,
+ [CPL_PASS_ACCEPT_REQ] = sched,
+ [CPL_PASS_ESTABLISH] = sched,
+ [CPL_PEER_CLOSE] = sched,
+ [CPL_CLOSE_CON_RPL] = sched,
+ [CPL_ABORT_REQ_RSS] = sched,
+ [CPL_RDMA_TERMINATE] = sched,
+ [CPL_FW4_ACK] = sched,
+ [CPL_SET_TCB_RPL] = set_tcb_rpl,
+ [CPL_FW6_MSG] = fw6_msg
+};
+
int __init c4iw_cm_init(void)
{
skb_queue_head_init(&rxq);
@@ -2290,45 +2332,6 @@ int __init c4iw_cm_init(void)
if (!workq)
return -ENOMEM;
- /*
- * Most upcalls from the T4 Core go to sched() to
- * schedule the processing on a work queue.
- */
- c4iw_handlers[CPL_ACT_ESTABLISH] = sched;
- c4iw_handlers[CPL_ACT_OPEN_RPL] = sched;
- c4iw_handlers[CPL_RX_DATA] = sched;
- c4iw_handlers[CPL_ABORT_RPL_RSS] = sched;
- c4iw_handlers[CPL_ABORT_RPL] = sched;
- c4iw_handlers[CPL_PASS_OPEN_RPL] = sched;
- c4iw_handlers[CPL_CLOSE_LISTSRV_RPL] = sched;
- c4iw_handlers[CPL_PASS_ACCEPT_REQ] = sched;
- c4iw_handlers[CPL_PASS_ESTABLISH] = sched;
- c4iw_handlers[CPL_PEER_CLOSE] = sched;
- c4iw_handlers[CPL_CLOSE_CON_RPL] = sched;
- c4iw_handlers[CPL_ABORT_REQ_RSS] = sched;
- c4iw_handlers[CPL_RDMA_TERMINATE] = sched;
- c4iw_handlers[CPL_FW4_ACK] = sched;
- c4iw_handlers[CPL_SET_TCB_RPL] = set_tcb_rpl;
- c4iw_handlers[CPL_FW6_MSG] = fw6_msg;
-
- /*
- * These are the real handlers that are called from a
- * work queue.
- */
- work_handlers[CPL_ACT_ESTABLISH] = act_establish;
- work_handlers[CPL_ACT_OPEN_RPL] = act_open_rpl;
- work_handlers[CPL_RX_DATA] = rx_data;
- work_handlers[CPL_ABORT_RPL_RSS] = abort_rpl;
- work_handlers[CPL_ABORT_RPL] = abort_rpl;
- work_handlers[CPL_PASS_OPEN_RPL] = pass_open_rpl;
- work_handlers[CPL_CLOSE_LISTSRV_RPL] = close_listsrv_rpl;
- work_handlers[CPL_PASS_ACCEPT_REQ] = pass_accept_req;
- work_handlers[CPL_PASS_ESTABLISH] = pass_establish;
- work_handlers[CPL_PEER_CLOSE] = peer_close;
- work_handlers[CPL_ABORT_REQ_RSS] = peer_abort;
- work_handlers[CPL_CLOSE_CON_RPL] = close_con_rpl;
- work_handlers[CPL_RDMA_TERMINATE] = terminate;
- work_handlers[CPL_FW4_ACK] = fw4_ack;
return 0;
}
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 3/7] RDMA/cxgb4: process ep timeouts in safe context.
From: Steve Wise @ 2010-05-06 23:06 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100506230619.25362.97591.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
From: root <root-m5ktC/4JIOOIE2W2IvP7LA@public.gmane.org>
Schedule the workq handler to process timed out endpoints rather than
try and process them on the timeout interrupt.
Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
---
drivers/infiniband/hw/cxgb4/cm.c | 112 +++++++++++++++++++++-----------
drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 1
2 files changed, 73 insertions(+), 40 deletions(-)
diff --git a/drivers/infiniband/hw/cxgb4/cm.c b/drivers/infiniband/hw/cxgb4/cm.c
index 85418f3..d146639 100644
--- a/drivers/infiniband/hw/cxgb4/cm.c
+++ b/drivers/infiniband/hw/cxgb4/cm.c
@@ -125,6 +125,9 @@ static struct sk_buff *get_skb(struct sk_buff *skb, int len, gfp_t gfp);
static void ep_timeout(unsigned long arg);
static void connect_reply_upcall(struct c4iw_ep *ep, int status);
+static LIST_HEAD(timeout_list);
+static spinlock_t timeout_lock;
+
static void start_ep_timer(struct c4iw_ep *ep)
{
PDBG("%s ep %p\n", __func__, ep);
@@ -1775,46 +1778,6 @@ static int fw4_ack(struct c4iw_dev *dev, struct sk_buff *skb)
return 0;
}
-static void ep_timeout(unsigned long arg)
-{
- struct c4iw_ep *ep = (struct c4iw_ep *)arg;
- struct c4iw_qp_attributes attrs;
- unsigned long flags;
- int abort = 1;
-
- spin_lock_irqsave(&ep->com.lock, flags);
- PDBG("%s ep %p tid %u state %d\n", __func__, ep, ep->hwtid,
- ep->com.state);
- switch (ep->com.state) {
- case MPA_REQ_SENT:
- __state_set(&ep->com, ABORTING);
- connect_reply_upcall(ep, -ETIMEDOUT);
- break;
- case MPA_REQ_WAIT:
- __state_set(&ep->com, ABORTING);
- break;
- case CLOSING:
- case MORIBUND:
- if (ep->com.cm_id && ep->com.qp) {
- attrs.next_state = C4IW_QP_STATE_ERROR;
- c4iw_modify_qp(ep->com.qp->rhp,
- ep->com.qp, C4IW_QP_ATTR_NEXT_STATE,
- &attrs, 1);
- }
- __state_set(&ep->com, ABORTING);
- break;
- default:
- printk(KERN_ERR "%s unexpected state ep %p tid %u state %u\n",
- __func__, ep, ep->hwtid, ep->com.state);
- WARN_ON(1);
- abort = 0;
- }
- spin_unlock_irqrestore(&ep->com.lock, flags);
- if (abort)
- abort_connection(ep, NULL, GFP_ATOMIC);
- c4iw_put_ep(&ep->com);
-}
-
int c4iw_reject_cr(struct iw_cm_id *cm_id, const void *pdata, u8 pdata_len)
{
int err;
@@ -2219,6 +2182,62 @@ static c4iw_handler_func work_handlers[NUM_CPL_CMDS] = {
[CPL_FW4_ACK] = fw4_ack
};
+static void process_timeout(struct c4iw_ep *ep)
+{
+ struct c4iw_qp_attributes attrs;
+ int abort = 1;
+
+ spin_lock_irq(&ep->com.lock);
+ PDBG("%s ep %p tid %u state %d\n", __func__, ep, ep->hwtid,
+ ep->com.state);
+ switch (ep->com.state) {
+ case MPA_REQ_SENT:
+ __state_set(&ep->com, ABORTING);
+ connect_reply_upcall(ep, -ETIMEDOUT);
+ break;
+ case MPA_REQ_WAIT:
+ __state_set(&ep->com, ABORTING);
+ break;
+ case CLOSING:
+ case MORIBUND:
+ if (ep->com.cm_id && ep->com.qp) {
+ attrs.next_state = C4IW_QP_STATE_ERROR;
+ c4iw_modify_qp(ep->com.qp->rhp,
+ ep->com.qp, C4IW_QP_ATTR_NEXT_STATE,
+ &attrs, 1);
+ }
+ __state_set(&ep->com, ABORTING);
+ break;
+ default:
+ printk(KERN_ERR "%s unexpected state ep %p tid %u state %u\n",
+ __func__, ep, ep->hwtid, ep->com.state);
+ WARN_ON(1);
+ abort = 0;
+ }
+ spin_unlock_irq(&ep->com.lock);
+ if (abort)
+ abort_connection(ep, NULL, GFP_KERNEL);
+ c4iw_put_ep(&ep->com);
+}
+
+static void process_timedout_eps(void)
+{
+ struct c4iw_ep *ep;
+
+ spin_lock_irq(&timeout_lock);
+ while (!list_empty(&timeout_list)) {
+ struct list_head *tmp;
+
+ tmp = timeout_list.next;
+ list_del(tmp);
+ spin_unlock_irq(&timeout_lock);
+ ep = list_entry(tmp, struct c4iw_ep, entry);
+ process_timeout(ep);
+ spin_lock_irq(&timeout_lock);
+ }
+ spin_unlock_irq(&timeout_lock);
+}
+
static void process_work(struct work_struct *work)
{
struct sk_buff *skb = NULL;
@@ -2237,10 +2256,21 @@ static void process_work(struct work_struct *work)
if (!ret)
kfree_skb(skb);
}
+ process_timedout_eps();
}
static DECLARE_WORK(skb_work, process_work);
+static void ep_timeout(unsigned long arg)
+{
+ struct c4iw_ep *ep = (struct c4iw_ep *)arg;
+
+ spin_lock(&timeout_lock);
+ list_add_tail(&ep->entry, &timeout_list);
+ spin_unlock(&timeout_lock);
+ queue_work(workq, &skb_work);
+}
+
/*
* All the CM events are handled on a work queue to have a safe context.
*/
@@ -2326,6 +2356,7 @@ c4iw_handler_func c4iw_handlers[NUM_CPL_CMDS] = {
int __init c4iw_cm_init(void)
{
+ spin_lock_init(&timeout_lock);
skb_queue_head_init(&rxq);
workq = create_singlethread_workqueue("iw_cxgb4");
@@ -2337,6 +2368,7 @@ int __init c4iw_cm_init(void)
void __exit c4iw_cm_term(void)
{
+ WARN_ON(!list_empty(&timeout_list));
flush_workqueue(workq);
destroy_workqueue(workq);
}
diff --git a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
index c3ea5a2..a626998 100644
--- a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
+++ b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
@@ -597,6 +597,7 @@ struct c4iw_ep {
struct c4iw_ep_common com;
struct c4iw_ep *parent_ep;
struct timer_list timer;
+ struct list_head entry;
unsigned int atid;
u32 hwtid;
u32 snd_seq;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 4/7] RDMA/cxgb4: Use proper gfp_t values based on thread context
From: Steve Wise @ 2010-05-06 23:06 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100506230619.25362.97591.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
From: root <root-m5ktC/4JIOOIE2W2IvP7LA@public.gmane.org>
Calls from post_qp_event() are in the interrupt context, so use GFP_ATOMIC
as necessary.
Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
---
drivers/infiniband/hw/cxgb4/ev.c | 2 +-
drivers/infiniband/hw/cxgb4/qp.c | 24 ++++++++++++------------
2 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/drivers/infiniband/hw/cxgb4/ev.c b/drivers/infiniband/hw/cxgb4/ev.c
index 1bd6a3e..ca05b5b 100644
--- a/drivers/infiniband/hw/cxgb4/ev.c
+++ b/drivers/infiniband/hw/cxgb4/ev.c
@@ -60,7 +60,7 @@ static void post_qp_event(struct c4iw_dev *dev, struct c4iw_cq *chp,
if (qhp->attr.state == C4IW_QP_STATE_RTS) {
attrs.next_state = C4IW_QP_STATE_TERMINATE;
c4iw_modify_qp(qhp->rhp, qhp, C4IW_QP_ATTR_NEXT_STATE,
- &attrs, 0);
+ &attrs, 1);
}
event.event = ib_event;
diff --git a/drivers/infiniband/hw/cxgb4/qp.c b/drivers/infiniband/hw/cxgb4/qp.c
index 7ff6aea..83a01dc 100644
--- a/drivers/infiniband/hw/cxgb4/qp.c
+++ b/drivers/infiniband/hw/cxgb4/qp.c
@@ -856,7 +856,8 @@ int c4iw_post_zb_read(struct c4iw_qp *qhp)
return c4iw_ofld_send(&qhp->rhp->rdev, skb);
}
-int c4iw_post_terminate(struct c4iw_qp *qhp, struct t4_cqe *err_cqe)
+static void post_terminate(struct c4iw_qp *qhp, struct t4_cqe *err_cqe,
+ gfp_t gfp)
{
struct fw_ri_wr *wqe;
struct sk_buff *skb;
@@ -865,9 +866,9 @@ int c4iw_post_terminate(struct c4iw_qp *qhp, struct t4_cqe *err_cqe)
PDBG("%s qhp %p qid 0x%x tid %u\n", __func__, qhp, qhp->wq.sq.qid,
qhp->ep->hwtid);
- skb = alloc_skb(sizeof *wqe, GFP_KERNEL | __GFP_NOFAIL);
+ skb = alloc_skb(sizeof *wqe, gfp);
if (!skb)
- return -ENOMEM;
+ return;
set_wr_txq(skb, CPL_PRIORITY_DATA, qhp->ep->txq_idx);
wqe = (struct fw_ri_wr *)__skb_put(skb, sizeof(*wqe));
@@ -881,7 +882,7 @@ int c4iw_post_terminate(struct c4iw_qp *qhp, struct t4_cqe *err_cqe)
wqe->u.terminate.immdlen = cpu_to_be32(sizeof *term);
term = (struct terminate_message *)wqe->u.terminate.termmsg;
build_term_codes(err_cqe, &term->layer_etype, &term->ecode);
- return c4iw_ofld_send(&qhp->rhp->rdev, skb);
+ c4iw_ofld_send(&qhp->rhp->rdev, skb);
}
/*
@@ -1215,12 +1216,10 @@ int c4iw_modify_qp(struct c4iw_dev *rhp, struct c4iw_qp *qhp,
qhp->attr.state = C4IW_QP_STATE_TERMINATE;
if (qhp->ibqp.uobject)
t4_set_wq_in_error(&qhp->wq);
- if (!internal) {
- ep = qhp->ep;
- c4iw_get_ep(&ep->com);
- terminate = 1;
- disconnect = 1;
- }
+ ep = qhp->ep;
+ c4iw_get_ep(&ep->com);
+ terminate = 1;
+ disconnect = 1;
break;
case C4IW_QP_STATE_ERROR:
qhp->attr.state = C4IW_QP_STATE_ERROR;
@@ -1301,7 +1300,7 @@ out:
spin_unlock_irqrestore(&qhp->lock, flag);
if (terminate)
- c4iw_post_terminate(qhp, NULL);
+ post_terminate(qhp, NULL, internal ? GFP_ATOMIC : GFP_KERNEL);
/*
* If disconnect is 1, then we need to initiate a disconnect
@@ -1309,7 +1308,8 @@ out:
* an abnormal close (RTS/CLOSING->ERROR).
*/
if (disconnect) {
- c4iw_ep_disconnect(ep, abort, GFP_KERNEL);
+ c4iw_ep_disconnect(ep, abort, internal ? GFP_ATOMIC :
+ GFP_KERNEL);
c4iw_put_ep(&ep->com);
}
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 5/7] RDMA/cxgb4: clean up a few printks.
From: Steve Wise @ 2010-05-06 23:06 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100506230619.25362.97591.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
---
drivers/infiniband/hw/cxgb4/cm.c | 2 +-
drivers/infiniband/hw/cxgb4/ev.c | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/hw/cxgb4/cm.c b/drivers/infiniband/hw/cxgb4/cm.c
index d146639..30ce0a8 100644
--- a/drivers/infiniband/hw/cxgb4/cm.c
+++ b/drivers/infiniband/hw/cxgb4/cm.c
@@ -2033,7 +2033,7 @@ int c4iw_create_listen(struct iw_cm_id *cm_id, int backlog)
*/
ep->stid = cxgb4_alloc_stid(dev->rdev.lldi.tids, PF_INET, ep);
if (ep->stid == -1) {
- printk(KERN_ERR MOD "%s - cannot alloc atid.\n", __func__);
+ printk(KERN_ERR MOD "%s - cannot alloc stid.\n", __func__);
err = -ENOMEM;
goto fail2;
}
diff --git a/drivers/infiniband/hw/cxgb4/ev.c b/drivers/infiniband/hw/cxgb4/ev.c
index ca05b5b..491e76a 100644
--- a/drivers/infiniband/hw/cxgb4/ev.c
+++ b/drivers/infiniband/hw/cxgb4/ev.c
@@ -51,8 +51,8 @@ static void post_qp_event(struct c4iw_dev *dev, struct c4iw_cq *chp,
return;
}
- printk(KERN_ERR "%s - AE qpid 0x%x opcode %d status 0x%x "
- "type %d wrid.hi 0x%x wrid.lo 0x%x\n", __func__,
+ printk(KERN_ERR MOD "AE qpid 0x%x opcode %d status 0x%x "
+ "type %d wrid.hi 0x%x wrid.lo 0x%x\n",
CQE_QPID(err_cqe), CQE_OPCODE(err_cqe),
CQE_STATUS(err_cqe), CQE_TYPE(err_cqe),
CQE_WRID_HI(err_cqe), CQE_WRID_LOW(err_cqe));
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 6/7] RDMA/cxgb4: Avoid CQ arm overflows.
From: Steve Wise @ 2010-05-06 23:06 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100506230619.25362.97591.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
There are 2 limits that need to be taken into account when arming the CQ.
1) the GTS register limits the delta idx to <= 0xfff.
2) T4 HW limits it to < cq size.
Update t4_arm_cq() to account for these limits.
Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
---
drivers/infiniband/hw/cxgb4/t4.h | 24 +++++++++++++++++++-----
1 files changed, 19 insertions(+), 5 deletions(-)
diff --git a/drivers/infiniband/hw/cxgb4/t4.h b/drivers/infiniband/hw/cxgb4/t4.h
index aeeb005..7f04dc5 100644
--- a/drivers/infiniband/hw/cxgb4/t4.h
+++ b/drivers/infiniband/hw/cxgb4/t4.h
@@ -449,11 +449,25 @@ struct t4_cq {
static inline int t4_arm_cq(struct t4_cq *cq, int se)
{
u32 val;
-
- val = SEINTARM(se) | CIDXINC(cq->cidx_inc) | TIMERREG(6) |
- INGRESSQID(cq->cqid);
- cq->cidx_inc = 0;
- writel(val, cq->gts);
+ u16 inc;
+
+ do {
+ /*
+ * inc must be less the both the max update value -and-
+ * the size of the CQ.
+ */
+ inc = cq->cidx_inc <= CIDXINC_MASK ? cq->cidx_inc :
+ CIDXINC_MASK;
+ inc = inc <= (cq->size - 1) ? inc : (cq->size - 1);
+ if (inc == cq->cidx_inc)
+ val = SEINTARM(se) | CIDXINC(inc) | TIMERREG(6) |
+ INGRESSQID(cq->cqid);
+ else
+ val = SEINTARM(0) | CIDXINC(inc) | TIMERREG(7) |
+ INGRESSQID(cq->cqid);
+ cq->cidx_inc -= inc;
+ writel(val, cq->gts);
+ } while (cq->cidx_inc);
return 0;
}
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 7/7] CQ overflow detection giving false positives
From: Steve Wise @ 2010-05-06 23:06 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100506230619.25362.97591.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
CQ overflow detection needs to read the gen bit and the timestamp in
one read operation. Otherwise false overflows can result.
Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
---
drivers/infiniband/hw/cxgb4/t4.h | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/hw/cxgb4/t4.h b/drivers/infiniband/hw/cxgb4/t4.h
index 7f04dc5..d0e8af3 100644
--- a/drivers/infiniband/hw/cxgb4/t4.h
+++ b/drivers/infiniband/hw/cxgb4/t4.h
@@ -502,11 +502,12 @@ static inline int t4_valid_cqe(struct t4_cq *cq, struct t4_cqe *cqe)
static inline int t4_next_hw_cqe(struct t4_cq *cq, struct t4_cqe **cqe)
{
int ret = 0;
+ u64 bits_type_ts = be64_to_cpu(cq->queue[cq->cidx].bits_type_ts);
- if (t4_valid_cqe(cq, &cq->queue[cq->cidx])) {
+ if (G_CQE_GENBIT(bits_type_ts) == cq->gen) {
*cqe = &cq->queue[cq->cidx];
- cq->timestamp = CQE_TS(*cqe);
- } else if (CQE_TS(&cq->queue[cq->cidx]) > cq->timestamp)
+ cq->timestamp = G_CQE_TS(bits_type_ts);
+ } else if (G_CQE_TS(bits_type_ts) > cq->timestamp)
ret = -EOVERFLOW;
else
ret = -ENODATA;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH v2] libibverbs: add path record definitions to sa.h
From: Sean Hefty @ 2010-05-06 23:20 UTC (permalink / raw)
To: 'Roland Dreier'; +Cc: linux-rdma
In-Reply-To: <adapr18zmmj.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
Add definitions for path record wire definition. This
will be used by the librdmacm and ib_acm service, and is
exchanged with the kernel using the newer set and query
route functionality.
Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Change from v1:
Used ibv/IBV prefix to match rest of libibverbs.
I used/kept the name 'path_record' to match the attribute name listed
in the spec. I just avoided the StudlyCaps. I don't have a strong
opinion on this. And, FWIW, the IB management code uses ib_path_rec.
include/infiniband/sa.h | 37 +++++++++++++++++++++++++++++++++++++
1 files changed, 37 insertions(+), 0 deletions(-)
diff --git a/include/infiniband/sa.h b/include/infiniband/sa.h
index ec90f14..5a5f6e3 100644
--- a/include/infiniband/sa.h
+++ b/include/infiniband/sa.h
@@ -97,4 +97,41 @@ struct ibv_sa_service_rec {
uint64_t data64[2];
};
+#define IBV_PATH_RECORD_REVERSIBLE 0x80
+
+struct ibv_path_record
+{
+ uint64_t service_id;
+ union ibv_gid dgid;
+ union ibv_gid sgid;
+ uint16_t dlid;
+ uint16_t slid;
+ uint32_t flowlabel_hoplimit; /* resv-31:28 flow label-27:8 hop limit-7:0*/
+ uint8_t tclass;
+ uint8_t reversible_numpath; /* reversible-7:7 num path-6:0 */
+ uint16_t pkey;
+ uint16_t qosclass_sl; /* qos class-15:4 sl-3:0 */
+ uint8_t mtu; /* mtu selector-7:6 mtu-5:0 */
+ uint8_t rate; /* rate selector-7:6 rate-5:0 */
+ uint8_t packetlifetime; /* lifetime selector-7:6 lifetime-5:0 */
+ uint8_t preference;
+ uint8_t reserved[6];
+};
+
+#define IBV_PATH_FLAG_GMP (1<<0)
+#define IBV_PATH_FLAG_PRIMARY (1<<1)
+#define IBV_PATH_FLAG_ALTERNATE (1<<2)
+#define IBV_PATH_FLAG_OUTBOUND (1<<3)
+#define IBV_PATH_FLAG_INBOUND (1<<4)
+#define IBV_PATH_FLAG_INBOUND_REVERSE (1<<5)
+#define IBV_PATH_FLAG_BIDIRECTIONAL (IBV_PATH_FLAG_OUTBOUND | \
+ IBV_PATH_FLAG_INBOUND_REVERSE)
+
+struct ibv_path_data
+{
+ uint32_t flags;
+ uint32_t reserved;
+ struct ibv_path_record path;
+};
+
#endif /* INFINIBAND_SA_H */
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 0/52] IB/qib: add
From: Ralph Campbell @ 2010-05-06 23:58 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
The following patches introduce an updated and renamed version of
the ipath HCA driver which supports the QLogic PCIe QLE SDR, DDR,
and QDR series of HCAs.
Rather than try to patch the ipath driver to include support for QDR,
multiple ports, bug fixes, and many other structual changes, the
ib_qib driver replaces the ib_ipath driver.
Changes in v3:
Added .xxx = yyy to qib_dma_mapping_ops initializer.
Added ib_post_send(IB_WR_FAST_REG_MR) support.
Changed ib_register_device() to pass a function for creating the
port sysfs files.
Changed /sys/class/infiniband/qib0/stats to
/sys/class/infiniband/qib0/ports/1/diag_counters/*
Moved /sys/class/infiniband/qib0/ports/1/linkcontrol/qsfp
to ipathfs/<N>/qsfp<P>
Removed debug printks until they can be replaced with event tracing.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH v3 01/52] IB/qib: Add Kconfig
From: Ralph Campbell @ 2010-05-06 23:58 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100506235849.3441.85930.stgit-/vjeY7uYZjrPXfVEPVhPGq6RkeBMCJyt@public.gmane.org>
This creates the Kconfig file.
Signed-off-by: Ralph Campbell <ralph.campbell-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org>
---
drivers/infiniband/hw/qib/Kconfig | 11 +++++++++++
1 files changed, 11 insertions(+), 0 deletions(-)
create mode 100644 drivers/infiniband/hw/qib/Kconfig
diff --git a/drivers/infiniband/hw/qib/Kconfig b/drivers/infiniband/hw/qib/Kconfig
new file mode 100644
index 0000000..1acfb54
--- /dev/null
+++ b/drivers/infiniband/hw/qib/Kconfig
@@ -0,0 +1,11 @@
+config INFINIBAND_QIB
+ tristate "QLogic PCIe HCA support"
+ depends on 64BIT && NET
+ ---help---
+ This is a driver for QLogic PCIe QLE IB host channel adapters,
+ including InfiniBand verbs support. This driver allows these
+ devices to be used with both kernel upper level protocols such
+ as IP-over-InfiniBand as well as with userspace applications
+ (in conjunction with InfiniBand userspace access).
+ This driver does not support the QLogic HyperTransport card
+ (model QHT7140).
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox