Re: InfiniBand HCA loopback on a single host (subnet manager needed?)

[Konstantin Boyanov <konstantin.boyanov-T5F83Mi6MZE@public.gmane.org>]

Hello,

Thanks for the advices! I have gotten my hands on an QSFP loopback plug, 
and yestrday inserted it in the machine (sinlge slot IB card).

Unfortunately I am having problems when starting the Subnet Manager.I 
believe I have installed and loaded all the necessary kernel modules 
needed.

*# lsmod | grep ib
ib_ipoib               78893  0
ib_ucm                 12567  0
ib_uverbs              31293  6 rdma_ucm,ib_ucm
ib_umad                12147  4
ib_cm                  36419  3 ib_ipoib,ib_ucm,rdma_cm
ib_addr                 6089  1 rdma_cm
ib_sa                  22820  4 ib_ipoib,rdma_ucm,rdma_cm,ib_cm
mlx4_ib                52866  1
ib_mad                 40542  4 ib_umad,ib_cm,ib_sa,mlx4_ib
ib_core                66295  11 
ib_ipoib,rdma_ucm,ib_ucm,ib_uverbs,ib_umad,rdma_cm,ib_cm,iw_cm,ib_sa,mlx4_ib,ib_mad 

ipv6                  321509  72 ib_ipoib,ib_addr
mlx4_core              93453  2 mlx4_ib,mlx4_en*


But when I start the opensm via:

*# /etc/init.d/opensm start*

I see a lot of error messages at the end of /var/log/opensm.log:

*Mar 30 12:50:05 622171 [1795B700] 0x80 -> SM port is down
Mar 30 12:50:05 622184 [1795B700] 0x01 -> sm_state_mgr_signal_error: ERR 
3207: Invalid signal OSM_SM_SIGNAL_DISCOVER in state DISCOVERING
SM port is down

Mar 30 12:50:15 622345 [1795B700] 0x80 -> SM port is down
Mar 30 12:50:15 622356 [1795B700] 0x01 -> sm_state_mgr_signal_error: ERR 
3207: Invalid signal OSM_SM_SIGNAL_DISCOVER in state DISCOVERING
Errors on subnet. Duplicate GUID found by link from a port to itself. 
See verbose opensm.log for more details

Mar 30 12:50:25 622645 [1C963700] 0x80 -> Errors on subnet. Duplicate 
GUID found by link from a port to itself. See verbose opensm.log for 
more details

After that, the port state is changed to PORT_INIT, but non of my test 
programs for the loopback (as well as thous in the OFED examples) can 
find a valid LID and oeprate properly.

*# ibv_devinfo
hca_id: mlx4_0
       transport:                      InfiniBand (0)
       fw_ver:                         2.7.626
       node_guid:                      0002:c903:000b:e242
       sys_image_guid:                 0002:c903:000b:e245
       vendor_id:                      0x02c9
       vendor_part_id:                 26428
       hw_ver:                         0xB0
       board_id:                       MT_0D90110009
       phys_port_cnt:                  1
               port:   1
                       state:                  PORT_INIT (2)
                       max_mtu:                2048 (4)
                       active_mtu:             2048 (4)
                       sm_lid:                 0
                       port_lid:               0
                       port_lmc:               0x00

*I am using OFED drivers version 1.4 and the machine is as follows:

*# uname -a
Linux myhost.domain.de 2.6.32-71.18.2.el6.x86_64 #1 SMP Tue Mar 8 
15:00:52 CST 2011 x86_64 x86_64 x86_64 GNU/Linux*

It seems to me that the loopback connector is somehow tricking the 
openSM to think that there is something wrong with the ports. Am I right?

Another thing: If I try to force bring the port to the ACTIVE state with 
ibportstate I get the following error:

# ibportstate -G 0x0002c903000be243 1 enable
ibwarn: [4824] mad_rpc_open_port: can't open UMAD port ((null):0)
ibportstate: iberror: failed: Failed to open '(null)' port '0'


I am really a greenehorn to all this InfiniBand stuff, so please can 
someone decrypt the above error messages in the opensm.log? What should 
I do in order to have a running openSM and a port configured the right 
way, so I can loopback messages? Is there any documentation out there 
which describes the set up of an loopback on a single port, or at least 
the initial setup of an InfiniBand network?

Thnaks in advance for your time and sorry if I am bothering you too much 
with my lame questions.

Best regards,
Konstantin Boyanov


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html