dapltest-server segfault seen on recent OFED-1.5.4 daily build

dapltest-server segfault seen on recent OFED-1.5.4 daily build

[Kumar Sanghvi]

Hi All,

I am trying to debug a segfault observed on dapltest-server with OFED-1.5.4.
I am using the daily-build OFED-1.5.4-20111116-0600 for this test.
The test setup involves 4 machines connected via switch.
1 machine acts as dapltest-server while rest 3 machines act as dapltest clients.

We are running several different kinds of RDMA read/write tests on dapl in continuous
loop using a script. The test runs fine for around 2 hours or so. And after that, the
dapltest-server segfaults with below stack trace:

-----------
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6e30710 (LWP 2397)]
dapl_llist_remove_entry (head=0x636960, entry=0x7ffff0004bf8) at
dapl/common/dapl_llist.c:272
272     dapl/common/dapl_llist.c: No such file or directory.
        in dapl/common/dapl_llist.c
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.7.el6.x86_64
libgcc-4.4.4-13.el6.x86_64
(gdb) bt
#0  dapl_llist_remove_entry (head=0x636960, entry=0x7ffff0004bf8) at
dapl/common/dapl_llist.c:272
#1  0x00007ffff799fb09 in dapl_sp_remove_cr (sp_ptr=0x6368c0,
cr_ptr=0x7ffff0004be0) at dapl/common/dapl_sp_util.c:229
#2  0x00007ffff7998148 in dapli_connection_request (ib_cm_handle=<value
optimized out>, sp_ptr=0x6368c0, prd_ptr=<value optimized out>, 
    private_data_size=<value optimized out>, evd_ptr=0x633fb0) at
dapl/common/dapl_cr_callback.c:424
#3  0x00007ffff799851e in dapls_cr_callback (ib_cm_handle=0x7ffff0004880,
ib_cm_event=IB_CME_CONNECTION_REQUEST_PENDING, 
    private_data_ptr=0x0, private_data_size=0, context=0x6368c0) at
dapl/common/dapl_cr_callback.c:178
#4  0x00007ffff79a4c33 in dapli_cm_passive_cb () at dapl/openib_cma/cm.c:524
#5  dapli_cma_event_cb () at dapl/openib_cma/cm.c:1207
#6  0x00007ffff79a6657 in dapli_thread (arg=<value optimized out>) at
dapl/openib_cma/device.c:692
#7  0x00007ffff79971d1 in dapli_thread_init (thread_draft=0x630320) at
dapl/udapl/linux/dapl_osd.c:590
#8  0x0000003b156077e1 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003b14ee153d in clone () from /lib64/libc.so.6
(gdb) p
The history is empty.
(gdb) info args
head = 0x636960
entry = 0x7ffff0004bf8
(gdb) p *head
$1 = (DAPL_LLIST_HEAD) 0x7ffff00107d8
(gdb) p *entry
$2 = {flink = 0x0, blink = 0x7ffff0003cf8, data = 0x7ffff0004be0, list_head =
0x0}
(gdb) info thread
  950 Thread 0x7ffff7fef710 (LWP 3924)  0x0000003b1560b43c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  949 Thread 0x7ffff5f76710 (LWP 3923)  0x0000003b1560eced in nanosleep () from
/lib64/libpthread.so.0
* 2 Thread 0x7ffff6e30710 (LWP 2397)  dapl_llist_remove_entry (head=0x636960,
entry=0x7ffff0004bf8) at dapl/common/dapl_llist.c:272
  1 Thread 0x7ffff7bb0700 (LWP 2394)  0x0000003b14ed7e33 in poll () from
/lib64/libc.so.6
(gdb) p head
$3 = (DAPL_LLIST_HEAD *) 0x636960
(gdb) p entry
$4 = (DAPL_LLIST_ENTRY *) 0x7ffff0004bf8
(gdb) p *entry
$5 = {flink = 0x0, blink = 0x7ffff0003cf8, data = 0x7ffff0004be0, list_head =
0x0}
(gdb) p *head
$6 = (DAPL_LLIST_HEAD) 0x7ffff00107d8
(gdb) 
-----------


The problematic line in dapl source code is:
-------------
File dapl/common/dapl_llist.c#dapl_llist_remove_entry function:
....
        dapl_os_assert(entry->list_head == head);
        entry->list_head = NULL;

        entry->flink->blink = entry->blink; <===== Problem line. flink is NULL
        entry->blink->flink = entry->flink;
....
--------------

Now, it seems that some time back, a new release of dapl (dapl-2.0.34-1.src.rpm) was
introduced in OFED-1.5.4. So, I am just wondering if this is a regression in the new
release of dapl?
Or if anyone is aware of this issue and what could possibly lead to this
dapltest-server segfault then, it would be helpful if someone can shed some light.


Thanks,
Kumar.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] dapltest-server segfault seen on recent OFED-1.5.4 daily build

[Davis, Arlin R]

> #0  dapl_llist_remove_entry (head=0x636960, entry=0x7ffff0004bf8) at
> dapl/common/dapl_llist.c:272
> #1  0x00007ffff799fb09 in dapl_sp_remove_cr (sp_ptr=0x6368c0,
> cr_ptr=0x7ffff0004be0) at dapl/common/dapl_sp_util.c:229
> #2  0x00007ffff7998148 in dapli_connection_request (ib_cm_handle=<value
> optimized out>, sp_ptr=0x6368c0, prd_ptr=<value optimized out>,
>     private_data_size=<value optimized out>, evd_ptr=0x633fb0) at
> dapl/common/dapl_cr_callback.c:424
>
> ...
>
> Now, it seems that some time back, a new release of dapl (dapl-2.0.34-
> 1.src.rpm) was
> introduced in OFED-1.5.4. So, I am just wondering if this is a
> regression in the new
> release of dapl?
> Or if anyone is aware of this issue and what could possibly lead to
> this
> dapltest-server segfault then, it would be helpful if someone can shed
> some light.
> 
 
You should have seen a message like "WARNING: overflow event on EVD".

It appears that the default dapltest server allocates too small of a CR EVD for many client test configurations. When it hits the overflow queue case, the CR callback incorrectly frees the CR before it is removed from SP list. In your case, I am guessing that another CR came in on another thread and this memory was reallocated with flink ptr reinitialized. 

Please try the following patches. 

---------
Common: CR EVD overflow causes segfault.

The CR is freed up incorrectly before unlinking with SP.

Signed-off-by: Arlin Davis <arlin.r.davis-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>


diff --git a/dapl/common/dapl_cr_callback.c b/dapl/common/dapl_cr_callback.c
index 3997b38..c58444b 100644
--- a/dapl/common/dapl_cr_callback.c
+++ b/dapl/common/dapl_cr_callback.c
@@ -414,7 +414,6 @@ dapli_connection_request(IN dp_ib_cm_handle_t ib_cm_handle,
                                                     (DAT_CR_HANDLE) cr_ptr);
 
        if (dat_status != DAT_SUCCESS) {
-               dapls_cr_free(cr_ptr);
                (void)dapls_ib_reject_connection(ib_cm_handle,
                                                 DAT_CONNECTION_EVENT_BROKEN,
                                                 0, NULL);
@@ -423,6 +422,7 @@ dapli_connection_request(IN dp_ib_cm_handle_t ib_cm_handle,
                dapl_os_lock(&sp_ptr->header.lock);
                dapl_sp_remove_cr(sp_ptr, cr_ptr);
                dapl_os_unlock(&sp_ptr->header.lock);
+               dapls_cr_free(cr_ptr);
                return DAT_INSUFFICIENT_RESOURCES;
        }


----------
dapltest: server CR EVD is too small for multi-client configurations.

Increase default size from 8 to 32.

Signed-off-by: Arlin Davis <arlin.r.davis-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

diff --git a/test/dapltest/test/dapl_server.c b/test/dapltest/test/dapl_server.c
index 443425c..92e0d21 100644
--- a/test/dapltest/test/dapl_server.c
+++ b/test/dapltest/test/dapl_server.c
@@ -34,7 +34,7 @@
 #undef DFLT_QLEN
 #endif
 
-#define DFLT_QLEN 8            /* default event queue length */
+#define DFLT_QLEN 32           /* default event queue length */
 
 int send_control_data(DT_Tdep_Print_Head * phead,
                      unsigned char *buffp,






--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] dapltest-server segfault seen on recent OFED-1.5.4 daily build

[Kumar Sanghvi]

Hi,

On 11/19/2011 05:18 AM, Davis, Arlin R wrote:
>
>> #0  dapl_llist_remove_entry (head=0x636960, entry=0x7ffff0004bf8) at
[...]
>>
>
> You should have seen a message like "WARNING: overflow event on EVD".
>
> It appears that the default dapltest server allocates too small of a CR EVD for many client test configurations. When it hits the overflow queue case, the CR callback incorrectly frees the CR before it is removed from SP list. In your case, I am guessing that another CR came in on another thread and this memory was reallocated with flink ptr reinitialized.
>
> Please try the following patches.
>
> ---------
> Common: CR EVD overflow causes segfault.
>
> The CR is freed up incorrectly before unlinking with SP.
>
> Signed-off-by: Arlin Davis<arlin.r.davis-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
>
> diff --git a/dapl/common/dapl_cr_callback.c b/dapl/common/dapl_cr_callback.c
> index 3997b38..c58444b 100644
> --- a/dapl/common/dapl_cr_callback.c
> +++ b/dapl/common/dapl_cr_callback.c
> @@ -414,7 +414,6 @@ dapli_connection_request(IN dp_ib_cm_handle_t ib_cm_handle,
>                                                       (DAT_CR_HANDLE) cr_ptr);
>
>          if (dat_status != DAT_SUCCESS) {
> -               dapls_cr_free(cr_ptr);
>                  (void)dapls_ib_reject_connection(ib_cm_handle,
>                                                   DAT_CONNECTION_EVENT_BROKEN,
>                                                   0, NULL);
> @@ -423,6 +422,7 @@ dapli_connection_request(IN dp_ib_cm_handle_t ib_cm_handle,
>                  dapl_os_lock(&sp_ptr->header.lock);
>                  dapl_sp_remove_cr(sp_ptr, cr_ptr);
>                  dapl_os_unlock(&sp_ptr->header.lock);
> +               dapls_cr_free(cr_ptr);
>                  return DAT_INSUFFICIENT_RESOURCES;
>          }
>
>
> ----------
> dapltest: server CR EVD is too small for multi-client configurations.
>
> Increase default size from 8 to 32.
>
> Signed-off-by: Arlin Davis<arlin.r.davis-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
> diff --git a/test/dapltest/test/dapl_server.c b/test/dapltest/test/dapl_server.c
> index 443425c..92e0d21 100644
> --- a/test/dapltest/test/dapl_server.c
> +++ b/test/dapltest/test/dapl_server.c
> @@ -34,7 +34,7 @@
>   #undef DFLT_QLEN
>   #endif
>
> -#define DFLT_QLEN 8            /* default event queue length */
> +#define DFLT_QLEN 32           /* default event queue length */
>
>   int send_control_data(DT_Tdep_Print_Head * phead,
>                        unsigned char *buffp,
>
>
Thank you for the two patches. I tried the two patches and now, I have 
not seen a segfault till now on dapl-server at least.
However, after about 2 hours of test, some of dapl-client throws below 
error on console:
----
Server Name: 3.4.5.1
Server Net Address: 3.4.5.1
DT_cs_Client: Starting Test ...
FAIL: 16 Server test connections did not report ready.
FAIL: 16 Server test connections did not report ready.
----

dapl-client is stalled at this stage, and needs to be manually killed by 
Ctrl+C.
And below errors are seen on dapl-server console:
----
Test Error: Client_Mem_Info_Send-reaping DTO problem, status = FAILURE
Test Error: Client_Mem_Info_Send-reaping DTO problem, status = FAILURE
Test[b368]: Warning: dat_ep_disconnect (abrupt) #2 error 
DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED
Test[b368]: dat_evd_free (creq) error: DAT_INVALID_STATE 
DAT_INVALID_STATE_EVD_IN_USE
Test[b368]: Warning: dat_ep_disconnect (abrupt) #3 error 
DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED
Test[b368]: dat_evd_free (creq) error: DAT_INVALID_STATE 
DAT_INVALID_STATE_EVD_IN_USE
...
----

No message is seen in dmesg on either dapl-server or dapl-client machine.

If I manually kill the dapl-client, and restart it then, test again 
starts fine and runs for about 2 hours or so.


Thanks,
Kumar.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] dapltest-server segfault seen on recent OFED-1.5.4 daily build

[Davis, Arlin R]

> Thank you for the two patches. I tried the two patches and now, I have
> not seen a segfault till now on dapl-server at least.
> However, after about 2 hours of test, some of dapl-client throws below
> error on console:
> ----
> Server Name: 3.4.5.1
> Server Net Address: 3.4.5.1
> DT_cs_Client: Starting Test ...
> FAIL: 16 Server test connections did not report ready.
> FAIL: 16 Server test connections did not report ready.
> ----
> 
> dapl-client is stalled at this stage, and needs to be manually killed
> by
> Ctrl+C.
> And below errors are seen on dapl-server console:
> ----
> Test Error: Client_Mem_Info_Send-reaping DTO problem, status = FAILURE
> Test Error: Client_Mem_Info_Send-reaping DTO problem, status = FAILURE
> Test[b368]: Warning: dat_ep_disconnect (abrupt) #2 error
> DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED
> Test[b368]: dat_evd_free (creq) error: DAT_INVALID_STATE
> DAT_INVALID_STATE_EVD_IN_USE
> Test[b368]: Warning: dat_ep_disconnect (abrupt) #3 error
> DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED
> Test[b368]: dat_evd_free (creq) error: DAT_INVALID_STATE
> DAT_INVALID_STATE_EVD_IN_USE
> ...
> ----

Can you send me your scripts so I can attempt to duplicate?
Did this test setup run successfully before OFED 1.5.4 RC3 release?

Thanks,

Arlin


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] dapltest-server segfault seen on recent OFED-1.5.4 daily build

[Kumar Sanghvi]

[-- Attachment #1: Type: text/plain, Size: 393 bytes --]

Hi Arlin,

On 11/22/2011 01:51 AM, Davis, Arlin R wrote:
> Can you send me your scripts so I can attempt to duplicate?
Find attached the script with this mail.

> Did this test setup run successfully before OFED 1.5.4 RC3 release?
I am not sure when it stopped working.
However, we have been running this script on dapl for quite some time, 
and it used to work fine earlier.


Thanks,
Kumar.

[-- Attachment #2: dapltest_stress_bug.txt --]
[-- Type: text/plain, Size: 2098 bytes --]

#!/bin/sh

host=$1
device=$2

while [ 1 ]
do
i=$(( i+1 ))
#====================================================================
#test1
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 1 -w 1 \
	client SR 256 					   \
	server SR 256

#====================================================================
#test2
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 1 -w 1 \
	client SR 256 					   \
	server RW 4096 					   \
	server SR 256

#====================================================================
#test3
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 1 -w 1 \
	client SR 256 					   \
	server RR 4096 					   \
	server SR 256

#====================================================================
#test4
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 1 -w 1 \
	client SR 256 					   \
	server RW 4096 					   \
	server SR 256 					   \
	client SR 256 					   \
	server RW 4096 					   \
	server SR 256 					   \
	client SR 4096 					   \
	server SR 256
 
#====================================================================
#test5
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 1 -w 8 \
	client SR 256 					   \
	server RW 4096 					   \
	server SR 256 					   \
	client SR 256 					   \
	server RW 4096 					   \
	server SR 256 					   \
	client SR 4096 					   \
	server SR 256
 
#====================================================================
#test6
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 4 -w 8 \
	client SR 256 					   \
	server RW 4096 					   \
	server SR 256 					   \
	client SR 256 					   \
	server RW 4096 					   \
	server SR 256 					   \
	client SR 4096 					   \
	server SR 256

wait
echo "Iteration $i"
sleep 10
done

Re: [PATCH] dapltest-server segfault seen on recent OFED-1.5.4 daily build

[Kumar Sanghvi]

Hi Arlin,

Did you get a chance to look into this further?

Also, will you be releasing a new version of dapl with the two patches 
provided earlier to OFED-1.5.4 ?


Thanks,
Kumar.

On 11/22/2011 12:41 PM, Kumar Sanghvi wrote:
> Hi Arlin,
>
> On 11/22/2011 01:51 AM, Davis, Arlin R wrote:
>> Can you send me your scripts so I can attempt to duplicate?
> Find attached the script with this mail.
>
>> Did this test setup run successfully before OFED 1.5.4 RC3 release?
> I am not sure when it stopped working.
> However, we have been running this script on dapl for quite some time,
> and it used to work fine earlier.
>
>
> Thanks,
> Kumar.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html