[PATCH AUTOSEL 5.10 07/22] scsi: iscsi: Fix race condition between login and sync thread

[PATCH AUTOSEL 5.10 07/22] scsi: iscsi: Fix race condition between login and sync thread

[Sasha Levin]

From: Gulam Mohamed <gulam.mohamed@oracle.com>

[ Upstream commit 9e67600ed6b8565da4b85698ec659b5879a6c1c6 ]

A kernel panic was observed due to a timing issue between the sync thread
and the initiator processing a login response from the target. The session
reopen can be invoked both from the session sync thread when iscsid
restarts and from iscsid through the error handler. Before the initiator
receives the response to a login, another reopen request can be sent from
the error handler/sync session. When the initial login response is
subsequently processed, the connection has been closed and the socket has
been released.

To fix this a new connection state, ISCSI_CONN_BOUND, is added:

 - Set the connection state value to ISCSI_CONN_DOWN upon
   iscsi_if_ep_disconnect() and iscsi_if_stop_conn()

 - Set the connection state to the newly created value ISCSI_CONN_BOUND
   after bind connection (transport->bind_conn())

 - In iscsi_set_param(), return -ENOTCONN if the connection state is not
   either ISCSI_CONN_BOUND or ISCSI_CONN_UP

Link: https://lore.kernel.org/r/20210325093248.284678-1-gulam.mohamed@oracle.com
Reviewed-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Gulam Mohamed <gulam.mohamed@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

index 91074fd97f64..f4bf62b007a0 100644

Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 drivers/scsi/scsi_transport_iscsi.c | 14 +++++++++++++-
 include/scsi/scsi_transport_iscsi.h |  1 +
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/scsi_transport_iscsi.c b/drivers/scsi/scsi_transport_iscsi.c
index c53c3f9fa526..f648452d8d66 100644
--- a/drivers/scsi/scsi_transport_iscsi.c
+++ b/drivers/scsi/scsi_transport_iscsi.c
@@ -2478,6 +2478,7 @@ static void iscsi_if_stop_conn(struct iscsi_cls_conn *conn, int flag)
 	 */
 	mutex_lock(&conn_mutex);
 	conn->transport->stop_conn(conn, flag);
+	conn->state = ISCSI_CONN_DOWN;
 	mutex_unlock(&conn_mutex);
 
 }
@@ -2904,6 +2905,13 @@ iscsi_set_param(struct iscsi_transport *transport, struct iscsi_uevent *ev)
 	default:
 		err = transport->set_param(conn, ev->u.set_param.param,
 					   data, ev->u.set_param.len);
+		if ((conn->state == ISCSI_CONN_BOUND) ||
+			(conn->state == ISCSI_CONN_UP)) {
+			err = transport->set_param(conn, ev->u.set_param.param,
+					data, ev->u.set_param.len);
+		} else {
+			return -ENOTCONN;
+		}
 	}
 
 	return err;
@@ -2963,6 +2971,7 @@ static int iscsi_if_ep_disconnect(struct iscsi_transport *transport,
 		mutex_lock(&conn->ep_mutex);
 		conn->ep = NULL;
 		mutex_unlock(&conn->ep_mutex);
+		conn->state = ISCSI_CONN_DOWN;
 	}
 
 	transport->ep_disconnect(ep);
@@ -3730,6 +3739,8 @@ iscsi_if_recv_msg(struct sk_buff *skb, struct nlmsghdr *nlh, uint32_t *group)
 		ev->r.retcode =	transport->bind_conn(session, conn,
 						ev->u.b_conn.transport_eph,
 						ev->u.b_conn.is_leading);
+		if (!ev->r.retcode)
+			conn->state = ISCSI_CONN_BOUND;
 		mutex_unlock(&conn_mutex);
 
 		if (ev->r.retcode || !transport->ep_connect)
@@ -3969,7 +3980,8 @@ iscsi_conn_attr(local_ipaddr, ISCSI_PARAM_LOCAL_IPADDR);
 static const char *const connection_state_names[] = {
 	[ISCSI_CONN_UP] = "up",
 	[ISCSI_CONN_DOWN] = "down",
-	[ISCSI_CONN_FAILED] = "failed"
+	[ISCSI_CONN_FAILED] = "failed",
+	[ISCSI_CONN_BOUND] = "bound"
 };
 
 static ssize_t show_conn_state(struct device *dev,
diff --git a/include/scsi/scsi_transport_iscsi.h b/include/scsi/scsi_transport_iscsi.h
index 8a26a2ffa952..fc5a39839b4b 100644
--- a/include/scsi/scsi_transport_iscsi.h
+++ b/include/scsi/scsi_transport_iscsi.h
@@ -193,6 +193,7 @@ enum iscsi_connection_state {
 	ISCSI_CONN_UP = 0,
 	ISCSI_CONN_DOWN,
 	ISCSI_CONN_FAILED,
+	ISCSI_CONN_BOUND,
 };
 
 struct iscsi_cls_conn {
-- 
2.30.2

Re: [PATCH AUTOSEL 5.10 07/22] scsi: iscsi: Fix race condition between login and sync thread

[Mike Christie]

On 4/5/21 11:04 AM, Sasha Levin wrote:
> From: Gulam Mohamed <gulam.mohamed@oracle.com>
> 
> [ Upstream commit 9e67600ed6b8565da4b85698ec659b5879a6c1c6 ]
> 
> A kernel panic was observed due to a timing issue between the sync thread
> and the initiator processing a login response from the target. The session
> reopen can be invoked both from the session sync thread when iscsid
> restarts and from iscsid through the error handler. Before the initiator
> receives the response to a login, another reopen request can be sent from
> the error handler/sync session. When the initial login response is
> subsequently processed, the connection has been closed and the socket has
> been released.
> 
> To fix this a new connection state, ISCSI_CONN_BOUND, is added:
> 
>  - Set the connection state value to ISCSI_CONN_DOWN upon
>    iscsi_if_ep_disconnect() and iscsi_if_stop_conn()
> 
>  - Set the connection state to the newly created value ISCSI_CONN_BOUND
>    after bind connection (transport->bind_conn())
> 
>  - In iscsi_set_param(), return -ENOTCONN if the connection state is not
>    either ISCSI_CONN_BOUND or ISCSI_CONN_UP
> 
> Link: https://urldefense.com/v3/__https://lore.kernel.org/r/20210325093248.284678-1-gulam.mohamed@oracle.com__;!!GqivPVa7Brio!Jiqrc6pu3EgrquzpG-KpNQkNebwKUgctkE0MN1MloQ2y5Y4OVOkKN0yCr2_W_CX2oRet$ 
> Reviewed-by: Mike Christie <michael.christie@oracle.com>


There was a mistake in my review of this patch. It will also require
this "[PATCH 1/1] scsi: iscsi: fix iscsi cls conn state":

https://lore.kernel.org/linux-scsi/20210406171746.5016-1-michael.christie@oracle.com/T/#u

Re: [PATCH AUTOSEL 5.10 07/22] scsi: iscsi: Fix race condition between login and sync thread

[Greg KH]

On Tue, Apr 06, 2021 at 12:24:32PM -0500, Mike Christie wrote:
> On 4/5/21 11:04 AM, Sasha Levin wrote:
> > From: Gulam Mohamed <gulam.mohamed@oracle.com>
> > 
> > [ Upstream commit 9e67600ed6b8565da4b85698ec659b5879a6c1c6 ]
> > 
> > A kernel panic was observed due to a timing issue between the sync thread
> > and the initiator processing a login response from the target. The session
> > reopen can be invoked both from the session sync thread when iscsid
> > restarts and from iscsid through the error handler. Before the initiator
> > receives the response to a login, another reopen request can be sent from
> > the error handler/sync session. When the initial login response is
> > subsequently processed, the connection has been closed and the socket has
> > been released.
> > 
> > To fix this a new connection state, ISCSI_CONN_BOUND, is added:
> > 
> >  - Set the connection state value to ISCSI_CONN_DOWN upon
> >    iscsi_if_ep_disconnect() and iscsi_if_stop_conn()
> > 
> >  - Set the connection state to the newly created value ISCSI_CONN_BOUND
> >    after bind connection (transport->bind_conn())
> > 
> >  - In iscsi_set_param(), return -ENOTCONN if the connection state is not
> >    either ISCSI_CONN_BOUND or ISCSI_CONN_UP
> > 
> > Link: https://urldefense.com/v3/__https://lore.kernel.org/r/20210325093248.284678-1-gulam.mohamed@oracle.com__;!!GqivPVa7Brio!Jiqrc6pu3EgrquzpG-KpNQkNebwKUgctkE0MN1MloQ2y5Y4OVOkKN0yCr2_W_CX2oRet$ 
> > Reviewed-by: Mike Christie <michael.christie@oracle.com>
> 
> 
> There was a mistake in my review of this patch. It will also require
> this "[PATCH 1/1] scsi: iscsi: fix iscsi cls conn state":
> 
> https://lore.kernel.org/linux-scsi/20210406171746.5016-1-michael.christie@oracle.com/T/#u
> 
> 

I don't see this in Linus's tree yet, so we can't take it until then :(

thanks,

greg k-h

Re: [PATCH AUTOSEL 5.10 07/22] scsi: iscsi: Fix race condition between login and sync thread

[Sasha Levin]

On Tue, Apr 06, 2021 at 12:24:32PM -0500, Mike Christie wrote:
>On 4/5/21 11:04 AM, Sasha Levin wrote:
>> From: Gulam Mohamed <gulam.mohamed@oracle.com>
>>
>> [ Upstream commit 9e67600ed6b8565da4b85698ec659b5879a6c1c6 ]
>>
>> A kernel panic was observed due to a timing issue between the sync thread
>> and the initiator processing a login response from the target. The session
>> reopen can be invoked both from the session sync thread when iscsid
>> restarts and from iscsid through the error handler. Before the initiator
>> receives the response to a login, another reopen request can be sent from
>> the error handler/sync session. When the initial login response is
>> subsequently processed, the connection has been closed and the socket has
>> been released.
>>
>> To fix this a new connection state, ISCSI_CONN_BOUND, is added:
>>
>>  - Set the connection state value to ISCSI_CONN_DOWN upon
>>    iscsi_if_ep_disconnect() and iscsi_if_stop_conn()
>>
>>  - Set the connection state to the newly created value ISCSI_CONN_BOUND
>>    after bind connection (transport->bind_conn())
>>
>>  - In iscsi_set_param(), return -ENOTCONN if the connection state is not
>>    either ISCSI_CONN_BOUND or ISCSI_CONN_UP
>>
>> Link: https://urldefense.com/v3/__https://lore.kernel.org/r/20210325093248.284678-1-gulam.mohamed@oracle.com__;!!GqivPVa7Brio!Jiqrc6pu3EgrquzpG-KpNQkNebwKUgctkE0MN1MloQ2y5Y4OVOkKN0yCr2_W_CX2oRet$
>> Reviewed-by: Mike Christie <michael.christie@oracle.com>
>
>
>There was a mistake in my review of this patch. It will also require
>this "[PATCH 1/1] scsi: iscsi: fix iscsi cls conn state":
>
>https://lore.kernel.org/linux-scsi/20210406171746.5016-1-michael.christie@oracle.com/T/#u

As the fix isn't upstream yet, I'll drop 9e67600ed6b for now and
re-queue it for the next round. Thanks!

-- 
Thanks,
Sasha