[PATCH net v3 0/2] fix unexpected SMC_CLC_DECL_ERR_REGRMB error

[PATCH net v3 0/2] fix unexpected SMC_CLC_DECL_ERR_REGRMB error

[D. Wythe]

From: "D. Wythe" <alibuda@linux.alibaba.com>

We can easily trigger the SMC_CLC_DECL_ERR_REGRMB exception within
following script:

server: smc_run nginx
client: smc_run  ./wrk -c 2000 -t 8 -d 20 http://smc-server

And we can clearly see that this error is also divided into two types:

1. 0x09990003
2. 0x05000000/0x09990003

Which has the same root causes, but the immediate causes vary.

The root cause of this issues is that remove connections from link group
is not synchronous with add/delete rtoken entry,  which means that even
the number of connections is less that SMC_RMBS_PER_LGR_MAX, it does not
mean that the connection can register rtoken successfully later. In
other words, the rtoken entry may released, This will cause an
unexpected SMC_CLC_DECL_ERR_REGRMB to be reported, and then this SMC
connections have to fallback to TCP.

This patch set handles two types of SMC_CLC_DECL_ERR_REGRMB exceptions
from different perspectives.

Patch 1: fix the 0x05000000/0x09990003 error.
Patch 2: fix the 0x09990003 error.

After those patches, there is no SMC_CLC_DECL_ERR_REGRMB exceptions in
my
test case any more.

v1 -> v2:
- add bugfix patch for SMC_CLC_DECL_ERR_REGRMB cause by server side 
v2 -> v3:
- fix incorrect mail thread 

D. Wythe (2):
  net/smc: fix unexpected SMC_CLC_DECL_ERR_REGRMB error generated by
    client
  net/smc: fix unexpected SMC_CLC_DECL_ERR_REGRMB error cause by server

 net/smc/smc_core.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

-- 
1.8.3.1

[PATCH net v3 1/2] net/smc: fix unexpected SMC_CLC_DECL_ERR_REGRMB error generated by client

[D. Wythe]

From: "D. Wythe" <alibuda@linux.alibaba.com>

The main reason for this unexpected SMC_CLC_DECL_ERR_REGRMB in client
dues to following execution sequence:

Server Conn A:           Server Conn B:			Client Conn B:

smc_lgr_unregister_conn
                        smc_lgr_register_conn
                        smc_clc_send_accept     ->
                                                        smc_rtoken_add
smcr_buf_unuse
		->		Client Conn A:
				smc_rtoken_delete

smc_lgr_unregister_conn() makes current link available to assigned to new
incoming connection, while smcr_buf_unuse() has not executed yet, which
means that smc_rtoken_add may fail because of insufficient rtoken_entry,
reversing their execution order will avoid this problem.

Fixes: 3e034725c0d8 ("net/smc: common functions for RMBs and send buffers")
Signed-off-by: D. Wythe <alibuda@linux.alibaba.com>
---
 net/smc/smc_core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c
index 29525d0..f8c9675 100644
--- a/net/smc/smc_core.c
+++ b/net/smc/smc_core.c
@@ -1161,8 +1161,8 @@ void smc_conn_free(struct smc_connection *conn)
 			cancel_work_sync(&conn->abort_work);
 	}
 	if (!list_empty(&lgr->list)) {
-		smc_lgr_unregister_conn(conn);
 		smc_buf_unuse(conn, lgr); /* allow buffer reuse */
+		smc_lgr_unregister_conn(conn);
 	}
 
 	if (!lgr->conns_num)
-- 
1.8.3.1

[PATCH net v3 2/2] net/smc: fix unexpected SMC_CLC_DECL_ERR_REGRMB error cause by server

[D. Wythe]

From: "D. Wythe" <alibuda@linux.alibaba.com>

The problem of SMC_CLC_DECL_ERR_REGRMB on the server is very clear.
Based on the fact that whether a new SMC connection can be accepted or
not depends on not only the limit of conn nums, but also the available
entries of rtoken. Since the rtoken release is trigger by peer, while
the conn nums is decrease by local, tons of thing can happen in this
time difference.

This only thing that needs to be mentioned is that now all connection
creations are completely protected by smc_server_lgr_pending lock, it's
enough to check only the available entries in rtokens_used_mask.

Fixes: cd6851f30386 ("smc: remote memory buffers (RMBs)")
Signed-off-by: D. Wythe <alibuda@linux.alibaba.com>
---
 net/smc/smc_core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c
index f8c9675..be7d704 100644
--- a/net/smc/smc_core.c
+++ b/net/smc/smc_core.c
@@ -1864,7 +1864,8 @@ int smc_conn_create(struct smc_sock *smc, struct smc_init_info *ini)
 		    (ini->smcd_version == SMC_V2 ||
 		     lgr->vlan_id == ini->vlan_id) &&
 		    (role == SMC_CLNT || ini->is_smcd ||
-		     lgr->conns_num < SMC_RMBS_PER_LGR_MAX)) {
+		    (lgr->conns_num < SMC_RMBS_PER_LGR_MAX &&
+		      !bitmap_full(lgr->rtokens_used_mask, SMC_RMBS_PER_LGR_MAX)))) {
 			/* link group found */
 			ini->first_contact_local = 0;
 			conn->lgr = lgr;
-- 
1.8.3.1