[PATCH 0/9] smb: client/smbdirect: connect bug fixes/cleanups and smbdirect_socket.status_wait

[PATCH 0/9] smb: client/smbdirect: connect bug fixes/cleanups and smbdirect_socket.status_wait

[Stefan Metzmacher]

Hi,

this is the next step towards a common smbdirect layer
between cifs.ko and ksmbd.ko, with the aim to provide
a socket layer for userspace usage at the end of the road.

This patchset focuses on the client side today.

The first one is a fix for very long timeouts against
unreachable servers.

The others prepare the use of a single wait_queue for state
changes. This removes a lot of special handling during
the connect and negotiate phases.

The last two move the state_wait queue into the common
smbdirect_socket.status_wait.

For the server I have only a single patch that also
uses smbdirect_socket.status_wait, but I'm skipping
the server patches today.

I plan a lot more progress on the server side tomorrow
and hopefully finish the moving everything from
struct smb_direct_transport into struct smbdirect_socket.

I used the following xfstests as regression tests:
cifs/001 generic/001 generic/002 generic/005 generic/006 generic/007 generic/010 generic/011

Between cifs.ko against ksmbd.ko via siw.ko.

This is on top of the patches for the client I posted yesterday...

Stefan Metzmacher (9):
  smb: client: return an error if rdma_connect does not return within 5
    seconds
  smb: client: improve logging in smbd_conn_upcall()
  smb: client: don't call init_waitqueue_head(&info->conn_wait) twice in
    _smbd_get_connection
  smb: client: only use a single wait_queue to monitor smbdirect
    connection status
  smb: client/smbdirect: replace SMBDIRECT_SOCKET_CONNECTING with more
    detailed states
  smb: client: use status_wait and SMBDIRECT_SOCKET_NEGOTIATE_RUNNING
    for completion
  smb: client: use status_wait and
    SMBDIRECT_SOCKET_RESOLVE_{ADDR,ROUTE}_RUNNING for completion
  smb: smbdirect: introduce smbdirect_socket.status_wait
  smb: client: make use of smbdirect_socket.status_wait

 fs/smb/client/smbdirect.c                  | 137 ++++++++++++++-------
 fs/smb/client/smbdirect.h                  |   8 --
 fs/smb/common/smbdirect/smbdirect_socket.h |  15 ++-
 3 files changed, 105 insertions(+), 55 deletions(-)

-- 
2.43.0

[PATCH 1/9] smb: client: return an error if rdma_connect does not return within 5 seconds

[Stefan Metzmacher]

This matches the timeout for tcp connections.

Cc: Steve French <smfrench@gmail.com>
Cc: Tom Talpey <tom@talpey.com>
Cc: Long Li <longli@microsoft.com>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Fixes: f198186aa9bb ("CIFS: SMBD: Establish SMB Direct connection")
Signed-off-by: Stefan Metzmacher <metze@samba.org>
---
 fs/smb/client/smbdirect.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c
index 6c2af00be44c..181349eda7a3 100644
--- a/fs/smb/client/smbdirect.c
+++ b/fs/smb/client/smbdirect.c
@@ -1653,8 +1653,10 @@ static struct smbd_connection *_smbd_get_connection(
 		goto rdma_connect_failed;
 	}
 
-	wait_event_interruptible(
-		info->conn_wait, sc->status != SMBDIRECT_SOCKET_CONNECTING);
+	wait_event_interruptible_timeout(
+		info->conn_wait,
+		sc->status != SMBDIRECT_SOCKET_CONNECTING,
+		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 
 	if (sc->status != SMBDIRECT_SOCKET_CONNECTED) {
 		log_rdma_event(ERR, "rdma_connect failed port=%d\n", port);
-- 
2.43.0

[PATCH 2/9] smb: client: improve logging in smbd_conn_upcall()

[Stefan Metzmacher]

Cc: Steve French <smfrench@gmail.com>
Cc: Tom Talpey <tom@talpey.com>
Cc: Long Li <longli@microsoft.com>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Signed-off-by: Stefan Metzmacher <metze@samba.org>
---
 fs/smb/client/smbdirect.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c
index 181349eda7a3..8ed4ab6f1d3a 100644
--- a/fs/smb/client/smbdirect.c
+++ b/fs/smb/client/smbdirect.c
@@ -178,9 +178,10 @@ static int smbd_conn_upcall(
 {
 	struct smbd_connection *info = id->context;
 	struct smbdirect_socket *sc = &info->socket;
+	const char *event_name = rdma_event_msg(event->event);
 
-	log_rdma_event(INFO, "event=%d status=%d\n",
-		event->event, event->status);
+	log_rdma_event(INFO, "event=%s status=%d\n",
+		event_name, event->status);
 
 	switch (event->event) {
 	case RDMA_CM_EVENT_ADDR_RESOLVED:
@@ -190,17 +191,19 @@ static int smbd_conn_upcall(
 		break;
 
 	case RDMA_CM_EVENT_ADDR_ERROR:
+		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		info->ri_rc = -EHOSTUNREACH;
 		complete(&info->ri_done);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_ERROR:
+		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		info->ri_rc = -ENETUNREACH;
 		complete(&info->ri_done);
 		break;
 
 	case RDMA_CM_EVENT_ESTABLISHED:
-		log_rdma_event(INFO, "connected event=%d\n", event->event);
+		log_rdma_event(INFO, "connected event=%s\n", event_name);
 		sc->status = SMBDIRECT_SOCKET_CONNECTED;
 		wake_up_interruptible(&info->conn_wait);
 		break;
@@ -208,7 +211,7 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_CONNECT_ERROR:
 	case RDMA_CM_EVENT_UNREACHABLE:
 	case RDMA_CM_EVENT_REJECTED:
-		log_rdma_event(INFO, "connecting failed event=%d\n", event->event);
+		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
 		wake_up_interruptible(&info->conn_wait);
 		break;
@@ -217,6 +220,7 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_DISCONNECTED:
 		/* This happens when we fail the negotiation */
 		if (sc->status == SMBDIRECT_SOCKET_NEGOTIATE_FAILED) {
+			log_rdma_event(ERR, "event=%s during negotiation\n", event_name);
 			sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
 			wake_up(&info->conn_wait);
 			break;
@@ -229,6 +233,8 @@ static int smbd_conn_upcall(
 		break;
 
 	default:
+		log_rdma_event(ERR, "unexpected event=%s status=%d\n",
+			       event_name, event->status);
 		break;
 	}
 
-- 
2.43.0

[PATCH 3/9] smb: client: don't call init_waitqueue_head(&info->conn_wait) twice in _smbd_get_connection

[Stefan Metzmacher]

It is already called long before we may hit this cleanup code path.

Cc: Steve French <smfrench@gmail.com>
Cc: Tom Talpey <tom@talpey.com>
Cc: Long Li <longli@microsoft.com>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Signed-off-by: Stefan Metzmacher <metze@samba.org>
---
 fs/smb/client/smbdirect.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c
index 8ed4ab6f1d3a..c819cc6dcc4f 100644
--- a/fs/smb/client/smbdirect.c
+++ b/fs/smb/client/smbdirect.c
@@ -1716,7 +1716,6 @@ static struct smbd_connection *_smbd_get_connection(
 	cancel_delayed_work_sync(&info->idle_timer_work);
 	destroy_caches_and_workqueue(info);
 	sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
-	init_waitqueue_head(&info->conn_wait);
 	rdma_disconnect(sc->rdma.cm_id);
 	wait_event(info->conn_wait,
 		sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
-- 
2.43.0

[PATCH 4/9] smb: client: only use a single wait_queue to monitor smbdirect connection status

[Stefan Metzmacher]

There's no need for separate conn_wait and disconn_wait queues.

This will simplify the move to common code, the server code
already a single wait_queue for this.

Cc: Steve French <smfrench@gmail.com>
Cc: Tom Talpey <tom@talpey.com>
Cc: Long Li <longli@microsoft.com>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Signed-off-by: Stefan Metzmacher <metze@samba.org>
---
 fs/smb/client/smbdirect.c | 17 ++++++++---------
 fs/smb/client/smbdirect.h |  3 +--
 2 files changed, 9 insertions(+), 11 deletions(-)

diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c
index c819cc6dcc4f..c628e91c328b 100644
--- a/fs/smb/client/smbdirect.c
+++ b/fs/smb/client/smbdirect.c
@@ -205,7 +205,7 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_ESTABLISHED:
 		log_rdma_event(INFO, "connected event=%s\n", event_name);
 		sc->status = SMBDIRECT_SOCKET_CONNECTED;
-		wake_up_interruptible(&info->conn_wait);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_CONNECT_ERROR:
@@ -213,7 +213,7 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_REJECTED:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-		wake_up_interruptible(&info->conn_wait);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_DEVICE_REMOVAL:
@@ -222,12 +222,12 @@ static int smbd_conn_upcall(
 		if (sc->status == SMBDIRECT_SOCKET_NEGOTIATE_FAILED) {
 			log_rdma_event(ERR, "event=%s during negotiation\n", event_name);
 			sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-			wake_up(&info->conn_wait);
+			wake_up(&info->status_wait);
 			break;
 		}
 
 		sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-		wake_up_interruptible(&info->disconn_wait);
+		wake_up_interruptible(&info->status_wait);
 		wake_up_interruptible(&sc->recv_io.reassembly.wait_queue);
 		wake_up_interruptible_all(&info->wait_send_queue);
 		break;
@@ -1325,7 +1325,7 @@ void smbd_destroy(struct TCP_Server_Info *server)
 		rdma_disconnect(sc->rdma.cm_id);
 		log_rdma_event(INFO, "wait for transport being disconnected\n");
 		wait_event_interruptible(
-			info->disconn_wait,
+			info->status_wait,
 			sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
 	}
 
@@ -1650,8 +1650,7 @@ static struct smbd_connection *_smbd_get_connection(
 	log_rdma_event(INFO, "connecting to IP %pI4 port %d\n",
 		&addr_in->sin_addr, port);
 
-	init_waitqueue_head(&info->conn_wait);
-	init_waitqueue_head(&info->disconn_wait);
+	init_waitqueue_head(&info->status_wait);
 	init_waitqueue_head(&sc->recv_io.reassembly.wait_queue);
 	rc = rdma_connect(sc->rdma.cm_id, &conn_param);
 	if (rc) {
@@ -1660,7 +1659,7 @@ static struct smbd_connection *_smbd_get_connection(
 	}
 
 	wait_event_interruptible_timeout(
-		info->conn_wait,
+		info->status_wait,
 		sc->status != SMBDIRECT_SOCKET_CONNECTING,
 		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 
@@ -1717,7 +1716,7 @@ static struct smbd_connection *_smbd_get_connection(
 	destroy_caches_and_workqueue(info);
 	sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
 	rdma_disconnect(sc->rdma.cm_id);
-	wait_event(info->conn_wait,
+	wait_event(info->status_wait,
 		sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
 
 allocate_cache_failed:
diff --git a/fs/smb/client/smbdirect.h b/fs/smb/client/smbdirect.h
index 0d4d45428c85..e45aa9ddd71d 100644
--- a/fs/smb/client/smbdirect.h
+++ b/fs/smb/client/smbdirect.h
@@ -47,8 +47,7 @@ struct smbd_connection {
 
 	int ri_rc;
 	struct completion ri_done;
-	wait_queue_head_t conn_wait;
-	wait_queue_head_t disconn_wait;
+	wait_queue_head_t status_wait;
 
 	struct completion negotiate_completion;
 	bool negotiate_done;
-- 
2.43.0

[PATCH 5/9] smb: client/smbdirect: replace SMBDIRECT_SOCKET_CONNECTING with more detailed states

[Stefan Metzmacher]

The process of reaching a functional connection regresented by
SMBDIRECT_SOCKET_CONNECTED, is more complex than using a single
SMBDIRECT_SOCKET_CONNECTING state.

This will allow us to remove a lot of special variables and
completions in the following commits.

Cc: Steve French <smfrench@gmail.com>
Cc: Tom Talpey <tom@talpey.com>
Cc: Long Li <longli@microsoft.com>
Cc: Namjae Jeon <linkinjeon@kernel.org>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Signed-off-by: Stefan Metzmacher <metze@samba.org>
---
 fs/smb/client/smbdirect.c                  | 44 +++++++++++++++++++---
 fs/smb/common/smbdirect/smbdirect_socket.h | 14 ++++++-
 2 files changed, 51 insertions(+), 7 deletions(-)

diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c
index c628e91c328b..f95ecc153249 100644
--- a/fs/smb/client/smbdirect.c
+++ b/fs/smb/client/smbdirect.c
@@ -185,26 +185,39 @@ static int smbd_conn_upcall(
 
 	switch (event->event) {
 	case RDMA_CM_EVENT_ADDR_RESOLVED:
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED;
+		info->ri_rc = 0;
+		complete(&info->ri_done);
+		break;
+
 	case RDMA_CM_EVENT_ROUTE_RESOLVED:
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED;
 		info->ri_rc = 0;
 		complete(&info->ri_done);
 		break;
 
 	case RDMA_CM_EVENT_ADDR_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_FAILED;
 		info->ri_rc = -EHOSTUNREACH;
 		complete(&info->ri_done);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_FAILED;
 		info->ri_rc = -ENETUNREACH;
 		complete(&info->ri_done);
 		break;
 
 	case RDMA_CM_EVENT_ESTABLISHED:
 		log_rdma_event(INFO, "connected event=%s\n", event_name);
-		sc->status = SMBDIRECT_SOCKET_CONNECTED;
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_NEGOTIATE_NEEDED;
 		wake_up_interruptible(&info->status_wait);
 		break;
 
@@ -212,7 +225,8 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_UNREACHABLE:
 	case RDMA_CM_EVENT_REJECTED:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
-		sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_FAILED;
 		wake_up_interruptible(&info->status_wait);
 		break;
 
@@ -481,6 +495,12 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
 		info->negotiate_done =
 			process_negotiation_response(response, wc->byte_len);
 		put_receive_buffer(info, response);
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_NEGOTIATE_RUNNING);
+		if (!info->negotiate_done) {
+			sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
+		} else {
+			sc->status = SMBDIRECT_SOCKET_CONNECTED;
+		}
 		complete(&info->negotiate_completion);
 		return;
 
@@ -556,6 +576,7 @@ static struct rdma_cm_id *smbd_create_id(
 		struct smbd_connection *info,
 		struct sockaddr *dstaddr, int port)
 {
+	struct smbdirect_socket *sc = &info->socket;
 	struct rdma_cm_id *id;
 	int rc;
 	__be16 *sport;
@@ -578,6 +599,8 @@ static struct rdma_cm_id *smbd_create_id(
 	init_completion(&info->ri_done);
 	info->ri_rc = -ETIMEDOUT;
 
+	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_NEEDED);
+	sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING;
 	rc = rdma_resolve_addr(id, NULL, (struct sockaddr *)dstaddr,
 		RDMA_RESOLVE_TIMEOUT);
 	if (rc) {
@@ -598,6 +621,8 @@ static struct rdma_cm_id *smbd_create_id(
 	}
 
 	info->ri_rc = -ETIMEDOUT;
+	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED);
+	sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING;
 	rc = rdma_resolve_route(id, RDMA_RESOLVE_TIMEOUT);
 	if (rc) {
 		log_rdma_event(ERR, "rdma_resolve_route() failed %i\n", rc);
@@ -644,6 +669,9 @@ static int smbd_ia_open(
 	struct smbdirect_socket *sc = &info->socket;
 	int rc;
 
+	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_CREATED);
+	sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_NEEDED;
+
 	sc->rdma.cm_id = smbd_create_id(info, dstaddr, port);
 	if (IS_ERR(sc->rdma.cm_id)) {
 		rc = PTR_ERR(sc->rdma.cm_id);
@@ -1085,6 +1113,9 @@ static int smbd_negotiate(struct smbd_connection *info)
 	int rc;
 	struct smbdirect_recv_io *response = get_receive_buffer(info);
 
+	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_NEGOTIATE_NEEDED);
+	sc->status = SMBDIRECT_SOCKET_NEGOTIATE_RUNNING;
+
 	sc->recv_io.expected = SMBDIRECT_EXPECT_NEGOTIATE_REP;
 	rc = smbd_post_recv(info, response);
 	log_rdma_event(INFO, "smbd_post_recv rc=%d iov.addr=0x%llx iov.length=%u iov.lkey=0x%x\n",
@@ -1540,7 +1571,7 @@ static struct smbd_connection *_smbd_get_connection(
 	sc = &info->socket;
 	sp = &sc->parameters;
 
-	sc->status = SMBDIRECT_SOCKET_CONNECTING;
+	sc->status = SMBDIRECT_SOCKET_CREATED;
 	rc = smbd_ia_open(info, dstaddr, port);
 	if (rc) {
 		log_rdma_event(INFO, "smbd_ia_open rc=%d\n", rc);
@@ -1652,6 +1683,9 @@ static struct smbd_connection *_smbd_get_connection(
 
 	init_waitqueue_head(&info->status_wait);
 	init_waitqueue_head(&sc->recv_io.reassembly.wait_queue);
+
+	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED);
+	sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING;
 	rc = rdma_connect(sc->rdma.cm_id, &conn_param);
 	if (rc) {
 		log_rdma_event(ERR, "rdma_connect() failed with %i\n", rc);
@@ -1660,10 +1694,10 @@ static struct smbd_connection *_smbd_get_connection(
 
 	wait_event_interruptible_timeout(
 		info->status_wait,
-		sc->status != SMBDIRECT_SOCKET_CONNECTING,
+		sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING,
 		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 
-	if (sc->status != SMBDIRECT_SOCKET_CONNECTED) {
+	if (sc->status != SMBDIRECT_SOCKET_NEGOTIATE_NEEDED) {
 		log_rdma_event(ERR, "rdma_connect failed port=%d\n", port);
 		goto rdma_connect_failed;
 	}
diff --git a/fs/smb/common/smbdirect/smbdirect_socket.h b/fs/smb/common/smbdirect/smbdirect_socket.h
index 3c4a8d627aa3..f43eabdd413d 100644
--- a/fs/smb/common/smbdirect/smbdirect_socket.h
+++ b/fs/smb/common/smbdirect/smbdirect_socket.h
@@ -8,9 +8,19 @@
 
 enum smbdirect_socket_status {
 	SMBDIRECT_SOCKET_CREATED,
-	SMBDIRECT_SOCKET_CONNECTING,
-	SMBDIRECT_SOCKET_CONNECTED,
+	SMBDIRECT_SOCKET_RESOLVE_ADDR_NEEDED,
+	SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING,
+	SMBDIRECT_SOCKET_RESOLVE_ADDR_FAILED,
+	SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED,
+	SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING,
+	SMBDIRECT_SOCKET_RESOLVE_ROUTE_FAILED,
+	SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED,
+	SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING,
+	SMBDIRECT_SOCKET_RDMA_CONNECT_FAILED,
+	SMBDIRECT_SOCKET_NEGOTIATE_NEEDED,
+	SMBDIRECT_SOCKET_NEGOTIATE_RUNNING,
 	SMBDIRECT_SOCKET_NEGOTIATE_FAILED,
+	SMBDIRECT_SOCKET_CONNECTED,
 	SMBDIRECT_SOCKET_DISCONNECTING,
 	SMBDIRECT_SOCKET_DISCONNECTED,
 	SMBDIRECT_SOCKET_DESTROYED
-- 
2.43.0

[PATCH 6/9] smb: client: use status_wait and SMBDIRECT_SOCKET_NEGOTIATE_RUNNING for completion

[Stefan Metzmacher]

We can use the state change from SMBDIRECT_SOCKET_NEGOTIATE_RUNNING to
SMBDIRECT_SOCKET_CONNECTED or SMBDIRECT_SOCKET_NEGOTIATE_FAILED in order
to notify the caller if the negotiation is over.

Cc: Steve French <smfrench@gmail.com>
Cc: Tom Talpey <tom@talpey.com>
Cc: Long Li <longli@microsoft.com>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Signed-off-by: Stefan Metzmacher <metze@samba.org>
---
 fs/smb/client/smbdirect.c | 19 ++++++++++---------
 fs/smb/client/smbdirect.h |  3 ---
 2 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c
index f95ecc153249..bdc4600d03e3 100644
--- a/fs/smb/client/smbdirect.c
+++ b/fs/smb/client/smbdirect.c
@@ -470,6 +470,7 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
 	struct smbd_connection *info =
 		container_of(sc, struct smbd_connection, socket);
 	int data_length = 0;
+	bool negotiate_done = false;
 
 	log_rdma_recv(INFO, "response=0x%p type=%d wc status=%d wc opcode %d byte_len=%d pkey_index=%u\n",
 		      response, sc->recv_io.expected, wc->status, wc->opcode,
@@ -492,16 +493,16 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
 	case SMBDIRECT_EXPECT_NEGOTIATE_REP:
 		dump_smbdirect_negotiate_resp(smbdirect_recv_io_payload(response));
 		sc->recv_io.reassembly.full_packet_received = true;
-		info->negotiate_done =
+		negotiate_done =
 			process_negotiation_response(response, wc->byte_len);
 		put_receive_buffer(info, response);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_NEGOTIATE_RUNNING);
-		if (!info->negotiate_done) {
+		if (!negotiate_done) {
 			sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
 		} else {
 			sc->status = SMBDIRECT_SOCKET_CONNECTED;
 		}
-		complete(&info->negotiate_completion);
+		wake_up_interruptible(&info->status_wait);
 		return;
 
 	/* SMBD data transfer packet */
@@ -1124,17 +1125,17 @@ static int smbd_negotiate(struct smbd_connection *info)
 	if (rc)
 		return rc;
 
-	init_completion(&info->negotiate_completion);
-	info->negotiate_done = false;
 	rc = smbd_post_send_negotiate_req(info);
 	if (rc)
 		return rc;
 
-	rc = wait_for_completion_interruptible_timeout(
-		&info->negotiate_completion, SMBD_NEGOTIATE_TIMEOUT * HZ);
-	log_rdma_event(INFO, "wait_for_completion_timeout rc=%d\n", rc);
+	rc = wait_event_interruptible_timeout(
+		info->status_wait,
+		sc->status != SMBDIRECT_SOCKET_NEGOTIATE_RUNNING,
+		secs_to_jiffies(SMBD_NEGOTIATE_TIMEOUT));
+	log_rdma_event(INFO, "wait_event_interruptible_timeout rc=%d\n", rc);
 
-	if (info->negotiate_done)
+	if (sc->status == SMBDIRECT_SOCKET_CONNECTED)
 		return 0;
 
 	if (rc == 0)
diff --git a/fs/smb/client/smbdirect.h b/fs/smb/client/smbdirect.h
index e45aa9ddd71d..82505df4a751 100644
--- a/fs/smb/client/smbdirect.h
+++ b/fs/smb/client/smbdirect.h
@@ -49,9 +49,6 @@ struct smbd_connection {
 	struct completion ri_done;
 	wait_queue_head_t status_wait;
 
-	struct completion negotiate_completion;
-	bool negotiate_done;
-
 	struct work_struct disconnect_work;
 	struct work_struct post_send_credits_work;
 
-- 
2.43.0

[PATCH 7/9] smb: client: use status_wait and SMBDIRECT_SOCKET_RESOLVE_{ADDR,ROUTE}_RUNNING for completion

[Stefan Metzmacher]

We can use the state change from SMBDIRECT_SOCKET_RESOLVE_{ADDR,ROUTE}_RUNNING
to the next state in order to wake the caller to do the next step.

Cc: Steve French <smfrench@gmail.com>
Cc: Tom Talpey <tom@talpey.com>
Cc: Long Li <longli@microsoft.com>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Signed-off-by: Stefan Metzmacher <metze@samba.org>
---
 fs/smb/client/smbdirect.c | 46 ++++++++++++++++++++++-----------------
 fs/smb/client/smbdirect.h |  2 --
 2 files changed, 26 insertions(+), 22 deletions(-)

diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c
index bdc4600d03e3..ab5b7ae04032 100644
--- a/fs/smb/client/smbdirect.c
+++ b/fs/smb/client/smbdirect.c
@@ -187,31 +187,27 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_ADDR_RESOLVED:
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED;
-		info->ri_rc = 0;
-		complete(&info->ri_done);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_RESOLVED:
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED;
-		info->ri_rc = 0;
-		complete(&info->ri_done);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ADDR_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_FAILED;
-		info->ri_rc = -EHOSTUNREACH;
-		complete(&info->ri_done);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_FAILED;
-		info->ri_rc = -ENETUNREACH;
-		complete(&info->ri_done);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ESTABLISHED:
@@ -597,9 +593,6 @@ static struct rdma_cm_id *smbd_create_id(
 
 	*sport = htons(port);
 
-	init_completion(&info->ri_done);
-	info->ri_rc = -ETIMEDOUT;
-
 	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_NEEDED);
 	sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING;
 	rc = rdma_resolve_addr(id, NULL, (struct sockaddr *)dstaddr,
@@ -608,20 +601,26 @@ static struct rdma_cm_id *smbd_create_id(
 		log_rdma_event(ERR, "rdma_resolve_addr() failed %i\n", rc);
 		goto out;
 	}
-	rc = wait_for_completion_interruptible_timeout(
-		&info->ri_done, msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
+	rc = wait_event_interruptible_timeout(
+		info->status_wait,
+		sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING,
+		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 	/* e.g. if interrupted returns -ERESTARTSYS */
 	if (rc < 0) {
 		log_rdma_event(ERR, "rdma_resolve_addr timeout rc: %i\n", rc);
 		goto out;
 	}
-	rc = info->ri_rc;
-	if (rc) {
+	if (sc->status == SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING) {
+		rc = -ETIMEDOUT;
+		log_rdma_event(ERR, "rdma_resolve_addr() completed %i\n", rc);
+		goto out;
+	}
+	if (sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED) {
+		rc = -EHOSTUNREACH;
 		log_rdma_event(ERR, "rdma_resolve_addr() completed %i\n", rc);
 		goto out;
 	}
 
-	info->ri_rc = -ETIMEDOUT;
 	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED);
 	sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING;
 	rc = rdma_resolve_route(id, RDMA_RESOLVE_TIMEOUT);
@@ -629,15 +628,22 @@ static struct rdma_cm_id *smbd_create_id(
 		log_rdma_event(ERR, "rdma_resolve_route() failed %i\n", rc);
 		goto out;
 	}
-	rc = wait_for_completion_interruptible_timeout(
-		&info->ri_done, msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
+	rc = wait_event_interruptible_timeout(
+		info->status_wait,
+		sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING,
+		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 	/* e.g. if interrupted returns -ERESTARTSYS */
 	if (rc < 0)  {
 		log_rdma_event(ERR, "rdma_resolve_addr timeout rc: %i\n", rc);
 		goto out;
 	}
-	rc = info->ri_rc;
-	if (rc) {
+	if (sc->status == SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING) {
+		rc = -ETIMEDOUT;
+		log_rdma_event(ERR, "rdma_resolve_route() completed %i\n", rc);
+		goto out;
+	}
+	if (sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED) {
+		rc = -ENETUNREACH;
 		log_rdma_event(ERR, "rdma_resolve_route() completed %i\n", rc);
 		goto out;
 	}
diff --git a/fs/smb/client/smbdirect.h b/fs/smb/client/smbdirect.h
index 82505df4a751..62458a8fd109 100644
--- a/fs/smb/client/smbdirect.h
+++ b/fs/smb/client/smbdirect.h
@@ -45,8 +45,6 @@ enum keep_alive_status {
 struct smbd_connection {
 	struct smbdirect_socket socket;
 
-	int ri_rc;
-	struct completion ri_done;
 	wait_queue_head_t status_wait;
 
 	struct work_struct disconnect_work;
-- 
2.43.0

[PATCH 8/9] smb: smbdirect: introduce smbdirect_socket.status_wait

[Stefan Metzmacher]

This will be used by server and client soon.

Cc: Steve French <smfrench@gmail.com>
Cc: Tom Talpey <tom@talpey.com>
Cc: Long Li <longli@microsoft.com>
Cc: Namjae Jeon <linkinjeon@kernel.org>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Signed-off-by: Stefan Metzmacher <metze@samba.org>
---
 fs/smb/common/smbdirect/smbdirect_socket.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/fs/smb/common/smbdirect/smbdirect_socket.h b/fs/smb/common/smbdirect/smbdirect_socket.h
index f43eabdd413d..aac4b040606b 100644
--- a/fs/smb/common/smbdirect/smbdirect_socket.h
+++ b/fs/smb/common/smbdirect/smbdirect_socket.h
@@ -28,6 +28,7 @@ enum smbdirect_socket_status {
 
 struct smbdirect_socket {
 	enum smbdirect_socket_status status;
+	wait_queue_head_t status_wait;
 
 	/* RDMA related */
 	struct {
-- 
2.43.0

[PATCH 9/9] smb: client: make use of smbdirect_socket.status_wait

[Stefan Metzmacher]

This will allow us to have common helper functions soon.

Cc: Steve French <smfrench@gmail.com>
Cc: Tom Talpey <tom@talpey.com>
Cc: Long Li <longli@microsoft.com>
Cc: linux-cifs@vger.kernel.org
Cc: samba-technical@lists.samba.org
Signed-off-by: Stefan Metzmacher <metze@samba.org>
---
 fs/smb/client/smbdirect.c | 32 ++++++++++++++++----------------
 fs/smb/client/smbdirect.h |  2 --
 2 files changed, 16 insertions(+), 18 deletions(-)

diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c
index ab5b7ae04032..f36226e0331b 100644
--- a/fs/smb/client/smbdirect.c
+++ b/fs/smb/client/smbdirect.c
@@ -187,34 +187,34 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_ADDR_RESOLVED:
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_RESOLVED:
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ADDR_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_FAILED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_FAILED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ESTABLISHED:
 		log_rdma_event(INFO, "connected event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_NEGOTIATE_NEEDED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_CONNECT_ERROR:
@@ -223,7 +223,7 @@ static int smbd_conn_upcall(
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_FAILED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_DEVICE_REMOVAL:
@@ -232,12 +232,12 @@ static int smbd_conn_upcall(
 		if (sc->status == SMBDIRECT_SOCKET_NEGOTIATE_FAILED) {
 			log_rdma_event(ERR, "event=%s during negotiation\n", event_name);
 			sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-			wake_up(&info->status_wait);
+			wake_up(&sc->status_wait);
 			break;
 		}
 
 		sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		wake_up_interruptible(&sc->recv_io.reassembly.wait_queue);
 		wake_up_interruptible_all(&info->wait_send_queue);
 		break;
@@ -498,7 +498,7 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
 		} else {
 			sc->status = SMBDIRECT_SOCKET_CONNECTED;
 		}
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		return;
 
 	/* SMBD data transfer packet */
@@ -602,7 +602,7 @@ static struct rdma_cm_id *smbd_create_id(
 		goto out;
 	}
 	rc = wait_event_interruptible_timeout(
-		info->status_wait,
+		sc->status_wait,
 		sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING,
 		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 	/* e.g. if interrupted returns -ERESTARTSYS */
@@ -629,7 +629,7 @@ static struct rdma_cm_id *smbd_create_id(
 		goto out;
 	}
 	rc = wait_event_interruptible_timeout(
-		info->status_wait,
+		sc->status_wait,
 		sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING,
 		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 	/* e.g. if interrupted returns -ERESTARTSYS */
@@ -1136,7 +1136,7 @@ static int smbd_negotiate(struct smbd_connection *info)
 		return rc;
 
 	rc = wait_event_interruptible_timeout(
-		info->status_wait,
+		sc->status_wait,
 		sc->status != SMBDIRECT_SOCKET_NEGOTIATE_RUNNING,
 		secs_to_jiffies(SMBD_NEGOTIATE_TIMEOUT));
 	log_rdma_event(INFO, "wait_event_interruptible_timeout rc=%d\n", rc);
@@ -1363,7 +1363,7 @@ void smbd_destroy(struct TCP_Server_Info *server)
 		rdma_disconnect(sc->rdma.cm_id);
 		log_rdma_event(INFO, "wait for transport being disconnected\n");
 		wait_event_interruptible(
-			info->status_wait,
+			sc->status_wait,
 			sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
 	}
 
@@ -1688,7 +1688,7 @@ static struct smbd_connection *_smbd_get_connection(
 	log_rdma_event(INFO, "connecting to IP %pI4 port %d\n",
 		&addr_in->sin_addr, port);
 
-	init_waitqueue_head(&info->status_wait);
+	init_waitqueue_head(&sc->status_wait);
 	init_waitqueue_head(&sc->recv_io.reassembly.wait_queue);
 
 	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED);
@@ -1700,7 +1700,7 @@ static struct smbd_connection *_smbd_get_connection(
 	}
 
 	wait_event_interruptible_timeout(
-		info->status_wait,
+		sc->status_wait,
 		sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING,
 		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 
@@ -1757,7 +1757,7 @@ static struct smbd_connection *_smbd_get_connection(
 	destroy_caches_and_workqueue(info);
 	sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
 	rdma_disconnect(sc->rdma.cm_id);
-	wait_event(info->status_wait,
+	wait_event(sc->status_wait,
 		sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
 
 allocate_cache_failed:
diff --git a/fs/smb/client/smbdirect.h b/fs/smb/client/smbdirect.h
index 62458a8fd109..79ab43b7ac19 100644
--- a/fs/smb/client/smbdirect.h
+++ b/fs/smb/client/smbdirect.h
@@ -45,8 +45,6 @@ enum keep_alive_status {
 struct smbd_connection {
 	struct smbdirect_socket socket;
 
-	wait_queue_head_t status_wait;
-
 	struct work_struct disconnect_work;
 	struct work_struct post_send_credits_work;
 
-- 
2.43.0

Re: [PATCH 0/9] smb: client/smbdirect: connect bug fixes/cleanups and smbdirect_socket.status_wait

[Steve French]

Tentatively merged into cifs-2.6.git  for-next pending additional review/testing

I did fix up minor checkpatch warnings in patches 5 and 6 (unneeded {
and } in if ... else)

On Thu, Aug 7, 2025 at 11:12 AM Stefan Metzmacher <metze@samba.org> wrote:
>
> Hi,
>
> this is the next step towards a common smbdirect layer
> between cifs.ko and ksmbd.ko, with the aim to provide
> a socket layer for userspace usage at the end of the road.
>
> This patchset focuses on the client side today.
>
> The first one is a fix for very long timeouts against
> unreachable servers.
>
> The others prepare the use of a single wait_queue for state
> changes. This removes a lot of special handling during
> the connect and negotiate phases.
>
> The last two move the state_wait queue into the common
> smbdirect_socket.status_wait.
>
> For the server I have only a single patch that also
> uses smbdirect_socket.status_wait, but I'm skipping
> the server patches today.
>
> I plan a lot more progress on the server side tomorrow
> and hopefully finish the moving everything from
> struct smb_direct_transport into struct smbdirect_socket.
>
> I used the following xfstests as regression tests:
> cifs/001 generic/001 generic/002 generic/005 generic/006 generic/007 generic/010 generic/011
>
> Between cifs.ko against ksmbd.ko via siw.ko.
>
> This is on top of the patches for the client I posted yesterday...
>
> Stefan Metzmacher (9):
>   smb: client: return an error if rdma_connect does not return within 5
>     seconds
>   smb: client: improve logging in smbd_conn_upcall()
>   smb: client: don't call init_waitqueue_head(&info->conn_wait) twice in
>     _smbd_get_connection
>   smb: client: only use a single wait_queue to monitor smbdirect
>     connection status
>   smb: client/smbdirect: replace SMBDIRECT_SOCKET_CONNECTING with more
>     detailed states
>   smb: client: use status_wait and SMBDIRECT_SOCKET_NEGOTIATE_RUNNING
>     for completion
>   smb: client: use status_wait and
>     SMBDIRECT_SOCKET_RESOLVE_{ADDR,ROUTE}_RUNNING for completion
>   smb: smbdirect: introduce smbdirect_socket.status_wait
>   smb: client: make use of smbdirect_socket.status_wait
>
>  fs/smb/client/smbdirect.c                  | 137 ++++++++++++++-------
>  fs/smb/client/smbdirect.h                  |   8 --
>  fs/smb/common/smbdirect/smbdirect_socket.h |  15 ++-
>  3 files changed, 105 insertions(+), 55 deletions(-)
>
> --
> 2.43.0
>


-- 
Thanks,

Steve

Re: [PATCH 9/9] smb: client: make use of smbdirect_socket.status_wait

[Meetakshi Setiya]

[-- Attachment #1: Type: text/plain, Size: 8195 bytes --]

Hi Metze,

&sc -> status_wait is initialized in _smbd_get_connection() in line
1691 but it is being used by smbd_ia_open() -> smbd_create_id() before
that. This is giving an OOPS on RDMA mount (attached).
Could you please check?

Thanks
Meetakshi

On Thu, Aug 7, 2025 at 9:46 PM Stefan Metzmacher <metze@samba.org> wrote:
>
> This will allow us to have common helper functions soon.
>
> Cc: Steve French <smfrench@gmail.com>
> Cc: Tom Talpey <tom@talpey.com>
> Cc: Long Li <longli@microsoft.com>
> Cc: linux-cifs@vger.kernel.org
> Cc: samba-technical@lists.samba.org
> Signed-off-by: Stefan Metzmacher <metze@samba.org>
> ---
>  fs/smb/client/smbdirect.c | 32 ++++++++++++++++----------------
>  fs/smb/client/smbdirect.h |  2 --
>  2 files changed, 16 insertions(+), 18 deletions(-)
>
> diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c
> index ab5b7ae04032..f36226e0331b 100644
> --- a/fs/smb/client/smbdirect.c
> +++ b/fs/smb/client/smbdirect.c
> @@ -187,34 +187,34 @@ static int smbd_conn_upcall(
>         case RDMA_CM_EVENT_ADDR_RESOLVED:
>                 WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
>                 sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED;
> -               wake_up_interruptible(&info->status_wait);
> +               wake_up_interruptible(&sc->status_wait);
>                 break;
>
>         case RDMA_CM_EVENT_ROUTE_RESOLVED:
>                 WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
>                 sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED;
> -               wake_up_interruptible(&info->status_wait);
> +               wake_up_interruptible(&sc->status_wait);
>                 break;
>
>         case RDMA_CM_EVENT_ADDR_ERROR:
>                 log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
>                 WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
>                 sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_FAILED;
> -               wake_up_interruptible(&info->status_wait);
> +               wake_up_interruptible(&sc->status_wait);
>                 break;
>
>         case RDMA_CM_EVENT_ROUTE_ERROR:
>                 log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
>                 WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
>                 sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_FAILED;
> -               wake_up_interruptible(&info->status_wait);
> +               wake_up_interruptible(&sc->status_wait);
>                 break;
>
>         case RDMA_CM_EVENT_ESTABLISHED:
>                 log_rdma_event(INFO, "connected event=%s\n", event_name);
>                 WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
>                 sc->status = SMBDIRECT_SOCKET_NEGOTIATE_NEEDED;
> -               wake_up_interruptible(&info->status_wait);
> +               wake_up_interruptible(&sc->status_wait);
>                 break;
>
>         case RDMA_CM_EVENT_CONNECT_ERROR:
> @@ -223,7 +223,7 @@ static int smbd_conn_upcall(
>                 log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
>                 WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
>                 sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_FAILED;
> -               wake_up_interruptible(&info->status_wait);
> +               wake_up_interruptible(&sc->status_wait);
>                 break;
>
>         case RDMA_CM_EVENT_DEVICE_REMOVAL:
> @@ -232,12 +232,12 @@ static int smbd_conn_upcall(
>                 if (sc->status == SMBDIRECT_SOCKET_NEGOTIATE_FAILED) {
>                         log_rdma_event(ERR, "event=%s during negotiation\n", event_name);
>                         sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
> -                       wake_up(&info->status_wait);
> +                       wake_up(&sc->status_wait);
>                         break;
>                 }
>
>                 sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
> -               wake_up_interruptible(&info->status_wait);
> +               wake_up_interruptible(&sc->status_wait);
>                 wake_up_interruptible(&sc->recv_io.reassembly.wait_queue);
>                 wake_up_interruptible_all(&info->wait_send_queue);
>                 break;
> @@ -498,7 +498,7 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
>                 } else {
>                         sc->status = SMBDIRECT_SOCKET_CONNECTED;
>                 }
> -               wake_up_interruptible(&info->status_wait);
> +               wake_up_interruptible(&sc->status_wait);
>                 return;
>
>         /* SMBD data transfer packet */
> @@ -602,7 +602,7 @@ static struct rdma_cm_id *smbd_create_id(
>                 goto out;
>         }
>         rc = wait_event_interruptible_timeout(
> -               info->status_wait,
> +               sc->status_wait,
>                 sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING,
>                 msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
>         /* e.g. if interrupted returns -ERESTARTSYS */
> @@ -629,7 +629,7 @@ static struct rdma_cm_id *smbd_create_id(
>                 goto out;
>         }
>         rc = wait_event_interruptible_timeout(
> -               info->status_wait,
> +               sc->status_wait,
>                 sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING,
>                 msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
>         /* e.g. if interrupted returns -ERESTARTSYS */
> @@ -1136,7 +1136,7 @@ static int smbd_negotiate(struct smbd_connection *info)
>                 return rc;
>
>         rc = wait_event_interruptible_timeout(
> -               info->status_wait,
> +               sc->status_wait,
>                 sc->status != SMBDIRECT_SOCKET_NEGOTIATE_RUNNING,
>                 secs_to_jiffies(SMBD_NEGOTIATE_TIMEOUT));
>         log_rdma_event(INFO, "wait_event_interruptible_timeout rc=%d\n", rc);
> @@ -1363,7 +1363,7 @@ void smbd_destroy(struct TCP_Server_Info *server)
>                 rdma_disconnect(sc->rdma.cm_id);
>                 log_rdma_event(INFO, "wait for transport being disconnected\n");
>                 wait_event_interruptible(
> -                       info->status_wait,
> +                       sc->status_wait,
>                         sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
>         }
>
> @@ -1688,7 +1688,7 @@ static struct smbd_connection *_smbd_get_connection(
>         log_rdma_event(INFO, "connecting to IP %pI4 port %d\n",
>                 &addr_in->sin_addr, port);
>
> -       init_waitqueue_head(&info->status_wait);
> +       init_waitqueue_head(&sc->status_wait);
>         init_waitqueue_head(&sc->recv_io.reassembly.wait_queue);
>
>         WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED);
> @@ -1700,7 +1700,7 @@ static struct smbd_connection *_smbd_get_connection(
>         }
>
>         wait_event_interruptible_timeout(
> -               info->status_wait,
> +               sc->status_wait,
>                 sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING,
>                 msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
>
> @@ -1757,7 +1757,7 @@ static struct smbd_connection *_smbd_get_connection(
>         destroy_caches_and_workqueue(info);
>         sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
>         rdma_disconnect(sc->rdma.cm_id);
> -       wait_event(info->status_wait,
> +       wait_event(sc->status_wait,
>                 sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
>
>  allocate_cache_failed:
> diff --git a/fs/smb/client/smbdirect.h b/fs/smb/client/smbdirect.h
> index 62458a8fd109..79ab43b7ac19 100644
> --- a/fs/smb/client/smbdirect.h
> +++ b/fs/smb/client/smbdirect.h
> @@ -45,8 +45,6 @@ enum keep_alive_status {
>  struct smbd_connection {
>         struct smbdirect_socket socket;
>
> -       wait_queue_head_t status_wait;
> -
>         struct work_struct disconnect_work;
>         struct work_struct post_send_credits_work;
>
> --
> 2.43.0
>
>

[-- Attachment #2: rdma_oops.txt --]
[-- Type: text/plain, Size: 6208 bytes --]

[ 2972.050661] CIFS: Attempting to mount //10.5.0.13/MyShare
[ 2972.050704] BUG: unable to handle page fault for address: ffffffffffffffe8
[ 2972.054229] #PF: supervisor read access in kernel mode
[ 2972.056242] #PF: error_code(0x0000) - not-present page
[ 2972.058357] PGD a3163f067 P4D a3163f067 PUD a31641067 PMD 0
[ 2972.060597] Oops: Oops: 0000 [#1] SMP NOPTI
[ 2972.062256] CPU: 12 UID: 0 PID: 208040 Comm: mount.cifs Tainted: G W OE 6.16.06.17+ #1 VOLUNTARY
[ 2972.066177] Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 2972.068729] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 03/08/2024
[ 2972.073001] RIP: 0010:prepare_to_wait_event+0x10e/0x130
[ 2972.075024] Code: 08 4d 8d 44 24 08 48 8d 57 e8 48 89 fe 4c 39 c7 74 1d 4c 89 c7 eb 13 48 8b 42 18 48 89 f7 48 8d 50 e8 49 39 c0 74 08 48 89 c6 <f6> 02 10 75 e8 48 8b 07 48 89 48 08 49 89 47 18 49 89 7f 20 48 89
[ 2972.082356] RSP: 0018:ffffd1da0e71b850 EFLAGS: 00010007
[ 2972.084435] RAX: ffffd1da0e71b8f0 RBX: 0000000000000001 RCX: ffffd1da0e71b8f0
[ 2972.087239] RDX: ffffffffffffffe8 RSI: 0000000000000000 RDI: ffff8f1ec3a92810
[ 2972.090116] RBP: ffffd1da0e71b878 R08: ffff8f1ec3a92810 R09: 0000000000000000
[ 2972.092970] R10: ffffd1da0e71b888 R11: 0000000000000002 R12: ffff8f1ec3a92808
[ 2972.095815] R13: ffff8f1ed864a8c0 R14: 0000000000000246 R15: ffffd1da0e71b8d8
[ 2972.098768] FS: 000070cd1d31b780(0000) GS:ffff8f271bce3000(0000) knlGS:0000000000000000
[ 2972.101955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2972.104323] CR2: ffffffffffffffe8 CR3: 000000014197d000 CR4: 0000000000350ef0
[ 2972.107135] Call Trace:
[ 2972.108265] <TASK>
[ 2972.109145] _smbd_get_connection+0xa43/0x1970 [cifs]
[ 2972.111310] ? srso_return_thunk+0x5/0x5f
[ 2972.112940] ? __pfx_autoremove_wake_function+0x10/0x10
[ 2972.115013] smbd_get_connection+0x26/0x50 [cifs]
[ 2972.116957] cifs_get_tcp_session+0x6a8/0x970 [cifs]
[ 2972.119152] cifs_mount_get_session+0x48/0x1c0 [cifs]
[ 2972.121358] dfs_mount_share+0x7b/0xb10 [cifs]
[ 2972.123161] ? srso_return_thunk+0x5/0x5f
[ 2972.124767] ? __wake_up_klogd+0x56/0x70
[ 2972.126407] ? srso_return_thunk+0x5/0x5f
[ 2972.128179] ? vprintk_emit+0x15d/0x370
[ 2972.129846] ? srso_return_thunk+0x5/0x5f
[ 2972.131461] ? __kmalloc_node_track_caller_noprof+0x172/0x520
[ 2972.133813] cifs_mount+0x5b/0x2e0 [cifs]
[ 2972.135549] cifs_smb3_do_mount+0x180/0x7e0 [cifs]
[ 2972.137626] ? vprintk+0x18/0x40
[ 2972.138997] ? srso_return_thunk+0x5/0x5f
[ 2972.140692] ? _printk+0x5f/0x80
[ 2972.142163] smb3_get_tree+0x16d/0x360 [cifs]
[ 2972.144015] vfs_get_tree+0x2a/0xf0
[ 2972.145411] path_mount+0x500/0xb00
[ 2972.146839] ? srso_return_thunk+0x5/0x5f
[ 2972.148479] __x64_sys_mount+0x116/0x150
[ 2972.150493] x64_sys_call+0x2082/0x20d0
[ 2972.152667] do_syscall_64+0x7b/0x4f0
[ 2972.154649] ? srso_return_thunk+0x5/0x5f
[ 2972.156815] ? __handle_mm_fault+0xa9c/0x1090
[ 2972.159232] ? __put_user_8+0xd/0x20
[ 2972.161060] ? srso_return_thunk+0x5/0x5f
[ 2972.163109] ? count_memcg_events+0xba/0x1a0
[ 2972.165247] ? srso_return_thunk+0x5/0x5f
[ 2972.167192] ? handle_mm_fault+0xbc/0x300
[ 2972.169118] ? srso_return_thunk+0x5/0x5f
[ 2972.171012] ? do_user_addr_fault+0x1b9/0x860
[ 2972.173152] ? srso_return_thunk+0x5/0x5f
[ 2972.175090] ? irqentry_exit_to_user_mode+0x2e/0x240
[ 2972.177519] ? srso_return_thunk+0x5/0x5f
[ 2972.179467] ? irqentry_exit+0x1d/0x30
[ 2972.181473] ? srso_return_thunk+0x5/0x5f
[ 2972.183554] ? exc_page_fault+0x84/0x150
[ 2972.185402] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 2972.187708] RIP: 0033:0x70cd1d12af0e
[ 2972.189706] Code: 48 8b 0d 0d 7f 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d da 7e 0d 00 f7 d8 64 89 01 48
[ 2972.197683] RSP: 002b:00007ffecb09f028 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
[ 2972.201254] RAX: ffffffffffffffda RBX: 000000000000000a RCX: 000070cd1d12af0e
[ 2972.204410] RDX: 000060acc558a47e RSI: 000060acc558a4e4 RDI: 00007ffecb0a0608
[ 2972.207687] RBP: 00007ffecb09f0e0 R08: 000060acea83eeb0 R09: 0000000000000000
[ 2972.210865] R10: 0000000000000000 R11: 0000000000000246 R12: 000060acc558a047
[ 2972.214110] R13: 000060acea83ff20 R14: 00007ffecb0a0608 R15: 000070cd1d32a000
[ 2972.217378] </TASK>
[ 2972.218656] Modules linked in: rpcrdma sunrpc rdma_ucm ib_iser libiscsi scsi_transport_iscsi siw ib_uverbs cifs(OE) nls_utf8 cifs_md4 netfs tls qrtr cfg80211 8021q garp mrp stp llc xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_owner xt_tcpudp nft_compat nf_tables x_tables binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common joydev mac_hid hyperv_drm polyval_clmulni serio_raw hid_generic ghash_clmulni_intel hid_hyperv aesni_intel hv_netvsc hyperv_fb hid hyperv_keyboard vmgenid ksmbd sch_fq_codel cifs_arc4 nls_ucs2_utils rdma_cm iw_cm ib_cm ib_core dm_multipath msr nvme_fabrics efi_pstore nfnetlink autofs4 [last unloaded: cifs]
[ 2972.244156] CR2: ffffffffffffffe8
[ 2972.246119] ---[ end trace 0000000000000000 ]---
[ 2972.345267] RIP: 0010:prepare_to_wait_event+0x10e/0x130
[ 2972.347983] Code: 08 4d 8d 44 24 08 48 8d 57 e8 48 89 fe 4c 39 c7 74 1d 4c 89 c7 eb 13 48 8b 42 18 48 89 f7 48 8d 50 e8 49 39 c0 74 08 48 89 c6 <f6> 02 10 75 e8 48 8b 07 48 89 48 08 49 89 47 18 49 89 7f 20 48 89
[ 2972.356513] RSP: 0018:ffffd1da0e71b850 EFLAGS: 00010007
[ 2972.359150] RAX: ffffd1da0e71b8f0 RBX: 0000000000000001 RCX: ffffd1da0e71b8f0
[ 2972.362453] RDX: ffffffffffffffe8 RSI: 0000000000000000 RDI: ffff8f1ec3a92810
[ 2972.365782] RBP: ffffd1da0e71b878 R08: ffff8f1ec3a92810 R09: 0000000000000000
[ 2972.369295] R10: ffffd1da0e71b888 R11: 0000000000000002 R12: ffff8f1ec3a92808
[ 2972.372907] R13: ffff8f1ed864a8c0 R14: 0000000000000246 R15: ffffd1da0e71b8d8
[ 2972.376455] FS: 000070cd1d31b780(0000) GS:ffff8f271bce3000(0000) knlGS:0000000000000000
[ 2972.380271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2972.383161] CR2: ffffffffffffffe8 CR3: 000000014197d000 CR4: 0000000000350ef0
[ 2972.386551] note: mount.cifs[208040] exited with irqs disabled

Re: [PATCH 9/9] smb: client: make use of smbdirect_socket.status_wait

[Stefan Metzmacher]

Hi Meetakshi,

> &sc -> status_wait is initialized in _smbd_get_connection() in line
> 1691 but it is being used by smbd_ia_open() -> smbd_create_id() before
> that. This is giving an OOPS on RDMA mount (attached).
> Could you please check?

The problem comes with this patch:
[PATCH 7/9] smb: client: use status_wait and SMBDIRECT_SOCKET_RESOLVE_{ADDR, ROUTE}_RUNNING for completion

And yesterday I was thinking the whole day about in which patch
I move init_waitqueue_head(&info->status_wait); to the top and then forgot it.

The strange thing is that all tests I made yesterday evening worked just fine
something between 3-5 times.

And today I got the crash immediately and was about to debug it.

Thanks that you found it! And sorry for introducing the stupid problem.

I'll post v2 soon.

Thanks!
metze