Re: bug report: opensm 3.3.15 crash (with traces)

Florent Parent
2014-04-02T21:43:37Z

Hi,

We experienced constant crashing from opensm 3.3.15 (3.3.15-1.el6.cq5)
after a recent upgrade. We compiled and installed 3.3.17 and problem
went away.

OpenSM server: CentOS 6.5 w/ stock RDMA. OpenSM 3.3.15 was from the
CentOS repository.

A behaviour that may help diagnose this: Unusual large amount messages
were filling up the opensm.log file:

Mar 13 09:50:04 909147 [4FAFC700] 0x01 -> log_rcv_cb_error: ERR 3111:
Received MAD with error status = 0x1C
SubnGetResp(SwitchInfo), attr_mod 0x0, TID 0x73c86e46
Initial path: 0,1,33,30,28 Return path: 0,10,32,13,28

80 of these messages occur periodically. smpquery on the paths shows
that these all point to the Sun QNEM switches (80 I4 chips).
"use_mfttop FALSE" eliminated these messages.

Florent

*** glibc detected *** /usr/sbin/opensm: malloc(): smallbin double
linked list corrupted: 0x00007f9b3c4352a0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x76166)[0x7f9b56279166]
/lib64/libc.so.6(+0x79f1f)[0x7f9b5627cf1f]
/lib64/libc.so.6(__libc_malloc+0x71)[0x7f9b5627d991]
/usr/sbin/opensm[0x4216f3]
/usr/sbin/opensm(osm_pkey_mgr_process+0x467)[0x422187]
/usr/sbin/opensm[0x446efb]
/usr/sbin/opensm(osm_state_mgr_process+0x1f8)[0x448538]
/usr/sbin/opensm[0x4422bb]
/usr/lib64/libosmcomp.so.3(+0x85fe)[0x7f9b56ddb5fe]
/lib64/libpthread.so.0(+0x79d1)[0x7f9b5659e9d1]
/lib64/libc.so.6(clone+0x6d)[0x7f9b562ebb6d]

*** glibc detected *** /usr/sbin/opensm: double free or corruption
(out): 0x00007fe2f42e1830 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x76166)[0x7fe30ec9d166]
/lib64/libc.so.6(+0x78c93)[0x7fe30ec9fc93]
/usr/sbin/opensm[0x449cf6]
/usr/sbin/opensm(osm_subn_rescan_conf_files+0x194)[0x44af14]
/usr/sbin/opensm[0x447260]
/usr/sbin/opensm(osm_state_mgr_process+0x1f8)[0x448538]
/usr/sbin/opensm[0x4422bb]
/usr/lib64/libosmcomp.so.3(+0x85fe)[0x7fe30f7ff5fe]
/lib64/libpthread.so.0(+0x79d1)[0x7fe30efc29d1]
/lib64/libc.so.6(clone+0x6d)[0x7fe30ed0fb6d]

*** glibc detected *** /usr/sbin/opensm: malloc(): smallbin double
linked list corrupted: 0x00007f200838ede0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x76166)[0x7f2025131166]
/lib64/libc.so.6(+0x79f1f)[0x7f2025134f1f]
/lib64/libc.so.6(__libc_malloc+0x71)[0x7f2025135991]
/usr/sbin/opensm[0x4216f3]
/usr/sbin/opensm(osm_pkey_mgr_process+0x467)[0x422187]
/usr/sbin/opensm[0x446efb]
/usr/sbin/opensm(osm_state_mgr_process+0x1f8)[0x448538]
/usr/sbin/opensm[0x4422bb]
/usr/lib64/libosmcomp.so.3(+0x85fe)[0x7f2025c935fe]
/lib64/libpthread.so.0(+0x79d1)[0x7f20254569d1]
/lib64/libc.so.6(clone+0x6d)[0x7f20251a3b6d]

*** glibc detected *** /usr/sbin/opensm: malloc(): smallbin double
linked list corrupted: 0x00007f8464013df0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x76166)[0x7f847ec95166]
/lib64/libc.so.6(+0x79f1f)[0x7f847ec98f1f]
/lib64/libc.so.6(__libc_malloc+0x71)[0x7f847ec99991]
/usr/sbin/opensm[0x4216f3]
/usr/sbin/opensm(osm_pkey_mgr_process+0x467)[0x422187]
/usr/sbin/opensm[0x446efb]
/usr/sbin/opensm(osm_state_mgr_process+0x1f8)[0x448538]
/usr/sbin/opensm[0x4422bb]
/usr/lib64/libosmcomp.so.3(+0x85fe)[0x7f847f7f75fe]
/lib64/libpthread.so.0(+0x79d1)[0x7f847efba9d1]
/lib64/libc.so.6(clone+0x6d)[0x7f847ed07b6d]