Re: [PATCH 0/2] Using multi-smps on the wire in libibnetdisc

[Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org>]

On Thu, 4 Feb 2010 16:13:25 -0800
Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:

> On Thu, 4 Feb 2010 15:01:32 -0500
> Hal Rosenstock <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> > On Thu, Feb 4, 2010 at 1:00 PM, Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:
> > > On Thu, 4 Feb 2010 09:19:39 -0500
> > > Hal Rosenstock <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >
> > >> On Tue, Feb 2, 2010 at 7:45 PM, Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:
> > >> > Sasha,
> > >> >
> 
> [snip]

[snip]

> > >>
> > >> Is there a speedup with 4 rather than 2 ?
> > >
> > > There is a bit of a speed up (~0.5 to 1.0 sec).  But my main reason to want to
> > > go to 4 is that if there are issues on the fabric, unresponsive nodes etc.; 4
> > > will give us better parallelism to get around these issues.  I have not had
> > > the chance to test this condition with the new algorithm but the original
> > > ibnetdiscover would slow way down when there are nodes which have unresponsive
> > > SMA's.  If there are only 2 outstanding this will not give us much speed up.
> > > This was the main motivation I had for improving the library in this way.

Ok, I found a fabric with just 2 nodes which were unresponsive...  A quick
test shows...

Original ibnetdiscover:

18:12:29 > time ./ibnetdiscover > foo
ibwarn: [26993] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,11,9)
src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,11,9) failed, skipping port
ibwarn: [26993] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,24,18,7,6)
src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,24,18,7,6) failed, skipping port

real    0m9.073s
user    0m0.137s
sys     0m0.172s

18:12:43 > time ./ibnetdiscover > foo
ibwarn: [31111] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,11,9)
src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,11,9) failed, skipping port
ibwarn: [31111] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,24,18,7,6)
src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,24,18,7,6) failed, skipping port

real    0m9.103s
user    0m0.046s
sys     0m0.046s


*New* ibnetdiscover with different outstanding SMP's.

18:12:14 > time ./ibnetdiscover -o 2 > foo
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,11,9 Attr 0x11:0) bad status 110; Connection timed out
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,13,7,7,6 Attr 0x11:0) bad status 110; Connection timed out

real    0m9.746s
user    0m6.559s
sys     0m3.156s

18:13:00 > time ./ibnetdiscover -o 4 > foo
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,11,9 Attr 0x11:0) bad status 110; Connection timed out
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,13,7,7,6 Attr 0x11:0) bad status 110; Connection timed out

real    0m4.668s
user    0m3.043s
sys     0m1.601s

18:13:10 > time ./ibnetdiscover -o 8 > foo
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,11,9 Attr 0x11:0) bad status 110; Connection timed out
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,13,7,7,6 Attr 0x11:0) bad status 110; Connection timed out

real    0m4.360s
user    0m2.891s
sys     0m1.451s


Note that 2 does not give much speed up, where 4 does.  Obviously this could
have to do with the fact there were 2 nodes which were bad (so if you had
100's of nodes unresponsive a higher value might be worth using) but as a
default compromise I think 4 is good.

Ira

> > >
> > > Also, I think you are correct that we should increase OpenSM's default from 4
> > > to 8.  For the same reason as above.  Some of our clusters have worked better
> > > with 8 when we are having issues.  But right now we are still running with 4.
> > 
> > I'm concerned about just increasing ibnetdiscover to 4 rather than 2.
> > I've seen a number of clusters with SMP dropping with the current
> > lower defaults.
> 
> So OpenSM is seeing dropped packets?  With 4 SMP's on the wire?  I do see some
> VL15Dropped errors (maybe 2-3 a day) but I did not think that would be an
> issue.  What kind of rate are you seeing?
> 
> The other question is; do people regularly run the tools which are using
> libibnetdisc (ibqueryerrors, iblinkinfo, ibnetdiscover)?  We do.  If others
> are not then I would say this change would have less impact as they would want
> the diags to have some priority for debugging.  The other option is to change
> the patch to be a default of 2 and allow user to change it depending on what
> they are trying to do.  If you think that is best I will change the patch.
> 
> Ira
> 
> > 
> > -- Hal
> > 
> > > Ira
> > >
> > >>
> > >> -- Hal
> > >>
> > >> >
> > >> > The first patch converts the algorithm and the second adds the ibnd_set_max_smps_on_wire call.
> > >> >
> > >> > Let me know what you think.  Because the algorithm changed so much testing this is a bit difficult because the order of the node discovery is different.  However, I have done some extensive diffing of the output of ibnetdiscover and things look good.
> > >> >
> > >> > Ira
> > >> >
> > >> > --
> > >> > Ira Weiny
> > >> > Math Programmer/Computer Scientist
> > >> > Lawrence Livermore National Lab
> > >> > 925-423-8008
> > >> > weiny2-i2BcT+NCU+M@public.gmane.org
> > >> > --
> > >> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > >> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > >> > More majordomo info at  http://**vger.kernel.org/majordomo-info.html
> > >> >
> > >> --
> > >> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > >> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > >> More majordomo info at  http://**vger.kernel.org/majordomo-info.html
> > >>
> > >
> > >
> > > --
> > > Ira Weiny
> > > Math Programmer/Computer Scientist
> > > Lawrence Livermore National Lab
> > > 925-423-8008
> > > weiny2-i2BcT+NCU+M@public.gmane.org
> > >
> > 
> 
> 
> -- 
> Ira Weiny
> Math Programmer/Computer Scientist
> Lawrence Livermore National Lab
> 925-423-8008
> weiny2-i2BcT+NCU+M@public.gmane.org


-- 
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html