NULL pointer dereference in rdma_ucm

NULL pointer dereference in rdma_ucm

[Josh England]

Hi,
I'm experimenting with an rdma_cm application to push data around
between nodes on an ~1000 node cluster (CentOS-5.3 with 2.6.18-128.el5
and OFED-1.4.2).  Under heavy load, I'm seeing several nodes per day
kernel panic due to a NULL pointer dereference.  It may be that the
in-kernel field cm_id_priv has a NULL ->alt_av.port , causing the
Oops, but I don't know for sure.  Any ideas on how to debug this?

Unable to handle kernel NULL pointer dereference at 0000000000000078
RIP:
 [<ffffffff88047be5>] :ib_cm:ib_cm_init_qp_attr+0x23b/0x27a
PGD 3e420e067 PUD 342b3b067 PMD 0
Oops: 0000 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 5
Modules linked in: panfs(PFU) rdma_ucm(FU) rdma_cm(FU) iw_cm(FU)
ib_addr(FU) ib_uverbs(FU) ib_umad(FU) dm_mirror dm_log dm_multipath
scsi_dh dm_mod video hwmon backlight sbs i2c_ec button battery asus_acpi
acpi_memhotplug ac parport_pc lp parport joydev shpchp e1000e i2c_i801
i2c_core pcspkr ehci_hcd ata_piix libata scsi_mod uhci_hcd nfs(FU)
nfs_acl(FU) lockd(FU) sunrpc(FU) mlx4_ib(FU) mlx4_core(FU) ib_ipoib(FU)
ipv6 xfrm_nalgo crypto_api ib_cm(FU) ib_sa(FU) ib_mad(FU) ib_core(FU)
ipoib_helper(FU)
Pid: 28666, comm: pprod Tainted: PF     2.6.18-128.el5 #1
RIP: 0010:[<ffffffff88047be5>]
[<ffffffff88047be5>] :ib_cm:ib_cm_init_qp_attr+0x23b/0x27a
RSP: 0018:ffff8104a9af3d28  EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff81031f80d400 RCX: 0000000000000008
RDX: 0000000000000246 RSI: ffff81031f80d508 RDI: ffff8104a9af3e70
RBP: ffff8104a9af3e18 R08: 000000030000003a R09: 0000000000000000
R10: ffff8104a9af3e18 R11: 0000000000000088 R12: ffff8104a9af3d88
R13: 0000000000000000 R14: ffff81031f80d470 R15: 00007fff79f62940
FS:  00002ab130b6bca0(0000) GS:ffff81069524dd40(0000)
knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000078 CR3: 000000049626c000 CR4: 00000000000006e0
Process pprod (pid: 28666, threadinfo ffff8104a9af2000, task
ffff81051fe737a0)
Stack:  ffff810666cad000 ffff8104a9af3d88 ffff8104a9af3e18
ffff8106952a1800
 0000000013a210c0 ffffffff88407850 000000000000003a ffff810466e06540
 ffff810c3dadeb40 ffff810466e06540 ffff8104a9af3e18 ffffffff8841725e
Call Trace:
 [<ffffffff88407850>] :rdma_cm:rdma_init_qp_attr+0xed/0x13f
 [<ffffffff8841725e>] :rdma_ucm:ucma_init_qp_attr+0x97/0xe4
 [<ffffffff8008a461>] default_wake_function+0x0/0xe
 [<ffffffff8008a461>] default_wake_function+0x0/0xe
 [<ffffffff800d66d2>] shmem_file_write+0x23f/0x251
 [<ffffffff88416326>] :rdma_ucm:ucma_write+0x73/0x91
 [<ffffffff8001659e>] vfs_write+0xce/0x174
 [<ffffffff80016e6b>] sys_write+0x45/0x6e
 [<ffffffff8005d116>] system_call+0x7e/0x83


Code: 8a 40 78 88 85 85 00 00 00 8b 83 28 01 00 00 66 89 45 7a 8a
RIP  [<ffffffff88047be5>] :ib_cm:ib_cm_init_qp_attr+0x23b/0x27a
 RSP <ffff8104a9af3d28>
CR2: 0000000000000078
 <0>Kernel panic - not syncing: Fatal exception

-JE
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NULL pointer dereference in rdma_ucm

[Or Gerlitz]

Josh England wrote:
> It may be that the in-kernel field cm_id_priv has a NULL ->alt_av.port , causing the Oops, but I don't know for sure.  Any ideas on how to debug this?
seems like this was reported in the past but remained unsolved,
http://lists.openfabrics.org/pipermail/general/2009-August/thread.html#61522

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NULL pointer dereference in rdma_ucm

[Josh England]

Do you think upgrading to OFED-1.5.1 would help at all?

-JE

On Mon, Jul 19, 2010 at 11:40 PM, Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
> Josh England wrote:
>>
>> It may be that the in-kernel field cm_id_priv has a NULL ->alt_av.port ,
>> causing the Oops, but I don't know for sure.  Any ideas on how to debug
>> this?
>
> seems like this was reported in the past but remained unsolved,
> http://lists.openfabrics.org/pipermail/general/2009-August/thread.html#61522
>
> Or.
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NULL pointer dereference in rdma_ucm

[Roland Dreier]

 > I'm experimenting with an rdma_cm application to push data around
 > between nodes on an ~1000 node cluster (CentOS-5.3 with 2.6.18-128.el5
 > and OFED-1.4.2).  Under heavy load, I'm seeing several nodes per day
 > kernel panic due to a NULL pointer dereference.  It may be that the
 > in-kernel field cm_id_priv has a NULL ->alt_av.port , causing the
 > Oops, but I don't know for sure.  Any ideas on how to debug this?

You have a pretty unsupportable combination of ancient kernel and old
OFED stack.  Is there any way you can test this with a recent mainline
kernel?

If I were debugging this I guess I would try to find out for sure where
the NULL dereference occurs -- I guess ib_cm_init_qp_attr() has all the
leaf functions (cm_init_qp_init_attr etc) inlined, so the first step
would be to figure out which one of those functions you're crashing in
(and also confirm that it's always the same one).  You could do that by
marking them noinline, or just put a WARN_ON(!<ptr>) before every
pointer dereference (does 2.6.18 even have WARN_ON?).

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NULL pointer dereference in rdma_ucm

[Josh England]

My kernel selections are limited to which versions I can get a panfs
module for.  The most recent kernel I see support for is a 2.6.31
variant from FC12.  I could ask them for a custom port for a newer
mainline kernel but the turn-around will likely be several weeks.
I'll go ahead and ask for one I guess.

It looks like 2.6.18 WARN_ON doesn't really do much:
#define WARN_ON(condition) do { if (condition) ; } while (0)

-JE

On Tue, Jul 20, 2010 at 1:52 PM, Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
>  > I'm experimenting with an rdma_cm application to push data around
>  > between nodes on an ~1000 node cluster (CentOS-5.3 with 2.6.18-128.el5
>  > and OFED-1.4.2).  Under heavy load, I'm seeing several nodes per day
>  > kernel panic due to a NULL pointer dereference.  It may be that the
>  > in-kernel field cm_id_priv has a NULL ->alt_av.port , causing the
>  > Oops, but I don't know for sure.  Any ideas on how to debug this?
>
> You have a pretty unsupportable combination of ancient kernel and old
> OFED stack.  Is there any way you can test this with a recent mainline
> kernel?
>
> If I were debugging this I guess I would try to find out for sure where
> the NULL dereference occurs -- I guess ib_cm_init_qp_attr() has all the
> leaf functions (cm_init_qp_init_attr etc) inlined, so the first step
> would be to figure out which one of those functions you're crashing in
> (and also confirm that it's always the same one).  You could do that by
> marking them noinline, or just put a WARN_ON(!<ptr>) before every
> pointer dereference (does 2.6.18 even have WARN_ON?).
>
>  - R.
> --
> Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/index.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NULL pointer dereference in rdma_ucm

[Or Gerlitz]

Josh England wrote:
> Do you think upgrading to OFED-1.5.1 would help at all?

it might help you to diagnose the problem better, if you read through the
thread I pointed on (its very short, four messages, let then two minutes),
you would see that Arthur is reporting on the lap_state and Sean is suggesting to use the IB CM sysfs counter to further debug this. I don't know if these counters exist on the IB stack used for the ofed drop you're using, but they should be in 1.5.x

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NULL pointer dereference in rdma_ucm

[Josh England]

I do have the sysfs counters in
/sys/class/infiniband_cm/<device>/<port_num>/cm_tx_msgs/.  Could you
point me to a reference for what they all mean?  There are a few
patches I've had to throw into 1.4.2 so I'll need to check whether
they are still needed in 1.5.1, but I'll work on that today.

Now, I'm not sure how relevant it is, but under heavy load I also see
a lot of these:
kernel: ib_cm: calculated mra timeout 67584 > 8192, decreasing use timeout_ms

-JE

On Wed, Jul 21, 2010 at 6:09 AM, Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
> Josh England wrote:
>> Do you think upgrading to OFED-1.5.1 would help at all?
>
> it might help you to diagnose the problem better, if you read through the
> thread I pointed on (its very short, four messages, let then two minutes),
> you would see that Arthur is reporting on the lap_state and Sean is suggesting to use the IB CM sysfs counter to further debug this. I don't know if these counters exist on the IB stack used for the ofed drop you're using, but they should be in 1.5.x
>
> Or.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: NULL pointer dereference in rdma_ucm

[Hefty, Sean]

> I do have the sysfs counters in
> /sys/class/infiniband_cm/<device>/<port_num>/cm_tx_msgs/.  Could you
> point me to a reference for what they all mean?

These are counting the number of CM messages sent for each type.  You would need to refer to the Infiniband specification to understand the CM protocol and messages.  In short, IB uses a 3 way handshake to connect:

REQ (request) ->
	<- REP (reply)
RTU (ready to use) ->

MRA (message received) can also appear to indicate that a message has been received, but its processing will be delayed.

> Now, I'm not sure how relevant it is, but under heavy load I also see
> a lot of these:
> kernel: ib_cm: calculated mra timeout 67584 > 8192, decreasing use
> timeout_ms

This is likely not an issue.  The CM has received an MRA, but is using a shorter timeout than what was specified in the MRA message.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: NULL pointer dereference in rdma_ucm

[Hefty, Sean]

>  [<ffffffff88407850>] :rdma_cm:rdma_init_qp_attr+0xed/0x13f
>  [<ffffffff8841725e>] :rdma_ucm:ucma_init_qp_attr+0x97/0xe4
>  [<ffffffff8008a461>] default_wake_function+0x0/0xe
>  [<ffffffff8008a461>] default_wake_function+0x0/0xe
>  [<ffffffff800d66d2>] shmem_file_write+0x23f/0x251
>  [<ffffffff88416326>] :rdma_ucm:ucma_write+0x73/0x91
>  [<ffffffff8001659e>] vfs_write+0xce/0x174
>  [<ffffffff80016e6b>] sys_write+0x45/0x6e
>  [<ffffffff8005d116>] system_call+0x7e/0x83

Here's a guess at what may be happening:

The rdma_cm receives a REQ, in cma_req_handler() we have:

	ret = conn_id->id.event_handler(&conn_id->id, &event);
	if (!ret) {
		/*
		 * Acquire mutex to prevent user executing rdma_destroy_id()
		 * while we're accessing the cm_id.
		 */
		mutex_lock(&lock);
		if (cma_comp(conn_id, CMA_CONNECT) &&
		    !cma_is_ud_ps(conn_id->id.ps))
			ib_send_cm_mra(cm_id, CMA_CM_MRA_SETTING, NULL, 0);

Note that the call to ib_send_cm_mra() is after the event has been reported to the user.  I'm guessing that the connection is getting established before the call to ib_send_cm_mra() is invoked.  ib_send_cm_mra() does this:

	case IB_CM_ESTABLISHED:
		cm_state = cm_id->state;
		lap_state = IB_CM_MRA_LAP_SENT;
		msg_response = CM_MSG_RESPONSE_OTHER;
		break;
	...

	cm_id->state = cm_state;
	cm_id->lap_state = lap_state;
	cm_id_priv->service_timeout = service_timeout;

If the cm_id state is established when ib_send_cm_mra() is called, the lap_state is changed.  This would result in cm_init_qp_rts_attr() incorrectly falling into:

		if (cm_id_priv->id.lap_state == IB_CM_LAP_UNINIT) {
			...
		} else {
---> here

where the alt_av.port would not be set.

If this is what is happening, then removing the call to ib_send_cm_mra() from cma_req_handler() should eliminate the crash.  The drawback is that connections may begin timing out under load, but it may be worth trying.  I'll see if I can come up with a better fix.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NULL pointer dereference in rdma_ucm

[Josh England]

On Wed, Jul 21, 2010 at 11:13 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> If this is what is happening, then removing the call to ib_send_cm_mra() from cma_req_handler() should eliminate the crash.  The drawback is that connections may begin timing out under load, but it may be worth trying.  I'll see if I can come up with a better fix.

Timed out connections might be something that can be compensated for
in the app.  It is definitely preferable to a kernel panic.  Still,
I'll work on making OFED-1.5.1 happy before playing around with
removing the ib_send_cm_mra() call.

-JE
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: NULL pointer dereference in rdma_ucm

[Hefty, Sean]

> Timed out connections might be something that can be compensated for
> in the app.  It is definitely preferable to a kernel panic.  Still,
> I'll work on making OFED-1.5.1 happy before playing around with
> removing the ib_send_cm_mra() call.

FYI - the possible issue I'm describing is in the upstream kernel, so it would also be in 1.5.1.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NULL pointer dereference in rdma_ucm

[Josh England]

Upgrading 1.5.1 is the way to go for me.  I have other dependencies
tying me down to the CentOS kernel for the time being.  Hopefully any
patch to mainstream should apply fairly cleanly to 1.5.1.

-JE

On Wed, Jul 21, 2010 at 1:51 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> Timed out connections might be something that can be compensated for
>> in the app.  It is definitely preferable to a kernel panic.  Still,
>> I'll work on making OFED-1.5.1 happy before playing around with
>> removing the ib_send_cm_mra() call.
>
> FYI - the possible issue I'm describing is in the upstream kernel, so it would also be in 1.5.1.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] rdma/ib_cm: check LAP state before sending an MRA

[Hefty, Sean]

This problem was originally reported by Arthur Kepner <akepner-sJ/iWh9BUns@public.gmane.org>:

We have a customer who has repeatedly had system panics with 
the following signature:

Unable to handle kernel NULL pointer dereference at 0000000000000010 RIP:
<ffffffff882c2c5c>{:ib_cm:ib_cm_init_qp_attr+580}
PGD 3a2db6067 PUD 0
Oops: 0000 [1] SMP
last sysfs file: /class/infiniband/mlx4_0/node_guid
CPU 4
Modules linked in: i2c_dev sg sd_mod crc32c libcrc32c iscsi_tcp libiscsi
scsi_transport_iscsi rdma_ucm rdma_cm
iw_cm ib_addr ib_ipoib ib_cm ib_sa ipv6 ib_uverbs ib_umad iw_cxgb3 cxgb3
firmware_class mlx4_ib ib_mthca ib_mad
 ib_core loop numatools xpmem worm mlx4_core libata i2c_i801 scsi_mod i2c_core
shpchp pci_hotplug nfs lockd nfs
_acl af_packet sunrpc e1000
Pid: 3256, comm: star Tainted: G     U 2.6.16.60-0.34-smp #1
RIP: 0010:[<ffffffff882c2c5c>]
<ffffffff882c2c5c>{:ib_cm:ib_cm_init_qp_attr+580}
RSP: 0018:ffff810369d09d38  EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff810419678c00 RCX: 0000000000000008
RDX: 0000000000000246 RSI: ffff810419678d18 RDI: ffff810369d09e70
RBP: ffff810369d09e18 R08: 000000030000003d R09: 0000000000000000
R10: ffff810369d09e18 R11: 0000000000000088 R12: ffff810369d09d88
R13: 0000000000000000 R14: ffff810419678c80 R15: 00000000403500b0
FS:  0000000040354940(0063) GS:ffff810420ffbbc0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000010 CR3: 000000039f0c4000 CR4: 00000000000006e0
Process star (pid: 3256, threadinfo ffff810369d08000, task ffff8103b81b5830)
Stack: ffff810419678a00 ffff810369d09d88 ffff810369d09e18 ffff810369d09e18
       0000000040143430 ffffffff882fb6d5 ffff810376261540 ffff81040bea4740
       ffff810376261540 ffffffff88309285
Call Trace: <ffffffff882fb6d5>{:rdma_cm:rdma_init_qp_attr+209}
       <ffffffff88309285>{:rdma_ucm:ucma_init_qp_attr+160}
       <ffffffff802ea55a>{thread_return+0}
<ffffffff8830832e>{:rdma_ucm:ucma_write+115}
       <ffffffff80186662>{vfs_write+215} <ffffffff80186c2b>{sys_write+69}
      <ffffffff8010adba>{system_call+126}

Code: 8a 40 10 88 85 85 00 00 00 8b 83 38 01 00 00 66 89 45 7a 8a
RIP <ffffffff882c2c5c>{:ib_cm:ib_cm_init_qp_attr+580} RSP <ffff810369d09d38>


>From a crash dump, I determined that we died in cm_init_qp_rts_attr() 
(it's inline, so it doesn't show up in the traceback) on the line 
labeled below:

static int cm_init_qp_rts_attr(struct cm_id_private *cm_id_priv,
                               struct ib_qp_attr *qp_attr,
                               int *qp_attr_mask)
{
        ........
        if (cm_id_priv->id.lap_state == IB_CM_LAP_UNINIT) {
                .....
        } else {
               *qp_attr_mask = IB_QP_ALT_PATH | IB_QP_PATH_MIG_STATE;
               qp_attr->alt_port_num = cm_id_priv->alt_av.port->port_num; <-die


A similar problem was reported by Josh England <jjengla-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>.

The problem is that the rdma_cm can call ib_send_cm_mra() after a
connection has been established.  The ib_cm incorrectly assumes that the
MRA is in response to a LAP (load alternate path) message, even though no
LAP message has been received.  The ib_cm needs to check the lap_state
before sending an MRA if the cm_id state is established.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Josh or Arthur, can either of you confirm if this patch fixes the crashes that
you've seen?

 drivers/infiniband/core/cm.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
index ad63b79..64e0903 100644
--- a/drivers/infiniband/core/cm.c
+++ b/drivers/infiniband/core/cm.c
@@ -2409,10 +2409,12 @@ int ib_send_cm_mra(struct ib_cm_id *cm_id,
 		msg_response = CM_MSG_RESPONSE_REP;
 		break;
 	case IB_CM_ESTABLISHED:
-		cm_state = cm_id->state;
-		lap_state = IB_CM_MRA_LAP_SENT;
-		msg_response = CM_MSG_RESPONSE_OTHER;
-		break;
+		if (cm_id->lap_state == IB_CM_LAP_RCVD) {
+			cm_state = cm_id->state;
+			lap_state = IB_CM_MRA_LAP_SENT;
+			msg_response = CM_MSG_RESPONSE_OTHER;
+			break;
+		}
 	default:
 		ret = -EINVAL;
 		goto error1;


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rdma/ib_cm: check LAP state before sending an MRA

[Arthur Kepner]

On Wed, Jul 21, 2010 at 04:36:52PM -0700, Hefty, Sean wrote:
> ...
> Josh or Arthur, can either of you confirm if this patch fixes the 
> crashes that you've seen?
> 

I can't. It's been practically impossible for us to reproduce.
(Only our customer seems to have the magic recipe.) 

-- 
Arthur
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rdma/ib_cm: check LAP state before sending an MRA

[Roland Dreier]

thanks, applied.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NULL pointer dereference in rdma_ucm

[Josh England]

Sorry to revive a stale thread, but I wanted to post an update and see
about getting this rolling again.

I tried the suggestion of removing the call to ib_send_cm_mra().
Unfortunately, doing this wedges the stack 100% of the time.  Since
the original post, I've upgraded to CentOS-5.5 with OFED-1.5.1.  The
NULL pointer reference now occurs far less frequently but still with
some regularity.  I have full memory dumps from a crashed kernel, but
don't really know how to debug it.

-JE

On Wed, Jul 21, 2010 at 11:13 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>>  [<ffffffff88407850>] :rdma_cm:rdma_init_qp_attr+0xed/0x13f
>>  [<ffffffff8841725e>] :rdma_ucm:ucma_init_qp_attr+0x97/0xe4
>>  [<ffffffff8008a461>] default_wake_function+0x0/0xe
>>  [<ffffffff8008a461>] default_wake_function+0x0/0xe
>>  [<ffffffff800d66d2>] shmem_file_write+0x23f/0x251
>>  [<ffffffff88416326>] :rdma_ucm:ucma_write+0x73/0x91
>>  [<ffffffff8001659e>] vfs_write+0xce/0x174
>>  [<ffffffff80016e6b>] sys_write+0x45/0x6e
>>  [<ffffffff8005d116>] system_call+0x7e/0x83
>
> Here's a guess at what may be happening:
>
> The rdma_cm receives a REQ, in cma_req_handler() we have:
>
>        ret = conn_id->id.event_handler(&conn_id->id, &event);
>        if (!ret) {
>                /*
>                 * Acquire mutex to prevent user executing rdma_destroy_id()
>                 * while we're accessing the cm_id.
>                 */
>                mutex_lock(&lock);
>                if (cma_comp(conn_id, CMA_CONNECT) &&
>                    !cma_is_ud_ps(conn_id->id.ps))
>                        ib_send_cm_mra(cm_id, CMA_CM_MRA_SETTING, NULL, 0);
>
> Note that the call to ib_send_cm_mra() is after the event has been reported to the user.  I'm guessing that the connection is getting established before the call to ib_send_cm_mra() is invoked.  ib_send_cm_mra() does this:
>
>        case IB_CM_ESTABLISHED:
>                cm_state = cm_id->state;
>                lap_state = IB_CM_MRA_LAP_SENT;
>                msg_response = CM_MSG_RESPONSE_OTHER;
>                break;
>        ...
>
>        cm_id->state = cm_state;
>        cm_id->lap_state = lap_state;
>        cm_id_priv->service_timeout = service_timeout;
>
> If the cm_id state is established when ib_send_cm_mra() is called, the lap_state is changed.  This would result in cm_init_qp_rts_attr() incorrectly falling into:
>
>                if (cm_id_priv->id.lap_state == IB_CM_LAP_UNINIT) {
>                        ...
>                } else {
> ---> here
>
> where the alt_av.port would not be set.
>
> If this is what is happening, then removing the call to ib_send_cm_mra() from cma_req_handler() should eliminate the crash.  The drawback is that connections may begin timing out under load, but it may be worth trying.  I'll see if I can come up with a better fix.
>
> - Sean
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: NULL pointer dereference in rdma_ucm

[Hefty, Sean]

> I tried the suggestion of removing the call to ib_send_cm_mra().
> Unfortunately, doing this wedges the stack 100% of the time.  Since
> the original post, I've upgraded to CentOS-5.5 with OFED-1.5.1.  The
> NULL pointer reference now occurs far less frequently but still with
> some regularity.  I have full memory dumps from a crashed kernel, but
> don't really know how to debug it.

upstream commit 50a025c69ee749d822c301f9bf63dee13c113680 should correct this issue.  I'm not sure when OFED will pick up this patch.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NULL pointer dereference in rdma_ucm

[Josh England]

Good news!  I'll apply this and test right away.

-JE

On Wed, Oct 6, 2010 at 12:04 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> I tried the suggestion of removing the call to ib_send_cm_mra().
>> Unfortunately, doing this wedges the stack 100% of the time.  Since
>> the original post, I've upgraded to CentOS-5.5 with OFED-1.5.1.  The
>> NULL pointer reference now occurs far less frequently but still with
>> some regularity.  I have full memory dumps from a crashed kernel, but
>> don't really know how to debug it.
>
> upstream commit 50a025c69ee749d822c301f9bf63dee13c113680 should correct this issue.  I'm not sure when OFED will pick up this patch.
>
> - Sean
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html