[PATCH Version 4 00/11] NFSV4.1,2 session trunking

[PATCH Version 4 00/11] NFSV4.1,2 session trunking

[andros]

From: Andy Adamson <andros@netapp.com>

This patch set enables session trunking configured two ways.

The NFSv4 minorversion > 0 pseudo-fs is probed
with a GETATTR(fs_locations) to retrieve a replicas list.
Each replica address is tested for session trunking. Those
that pass are added as aliases to the mount rpc client.

Try all multipath addresses for a data server. The first address that
successfully connects and creates a session is the mount address.
All subsequent addresses are tested for session trunking and
added as aliases.

TODO:

- Periodically probe the pseudo-fs with the GETATTR(fs_locations)
to pick up server multipath changes. This is why the
GETATTR(fs_locations) is a stand alone call.

- Call BIND_CONN_TO_SESSION for session trunking addresses
established under SP4_MACH_CRED or SP4_SSV. SP4_NONE established
addresses do not require BIND_CONN_TO_SESSION.

TESTING:

The pseudo-fs GETATTR(fs_locations) probe session trunking
was tested against a Linux server with a pseudo-fs
export stanza (e.g. a stanza with the fsid=0 or fsid=root
export option) and a replicas= export option
(replicas=<path1>@<server1>:<path2>@<server2>..)
Note that this configuration is for testing only. A future
patchset will add the replicas= configuration to the
NFSEXP_V4ROOT nfsd and mountd processing.

The pNFS DS multipath session trunking was tested against a
pynfs server which provides DS multipath configuration.

-->Andy

Andy Adamson (11):
  NFS default callback ops
  NFS refactor nfs4_match_clientids
  NFS refactor nfs4_check_serverowner_major_id
  NFS detect session trunking
  NFS refactor _nfs4_proc_exchange_id for session trunking
  SUNRPC add flag to rpc_task_release_client
  NFS probe pseudo-fs for replicas
  squash into 7
  NFS test and add multipaths for session trunking
  NFS test pnfs data server multipath session trunking aliases
  NFS add multiaddr to nfs_show_nfsv4_options

 fs/nfs/internal.h           |   2 +
 fs/nfs/nfs4_fs.h            |   9 ++
 fs/nfs/nfs4client.c         | 102 +++++++++++++++++---
 fs/nfs/nfs4getroot.c        |   3 +
 fs/nfs/nfs4proc.c           | 224 ++++++++++++++++++++++++++++++++++++++++++--
 fs/nfs/nfs4xdr.c            |  14 ++-
 fs/nfs/pnfs_nfs.c           |  51 +++++++---
 fs/nfs/super.c              |  26 +++++
 include/linux/nfs_xdr.h     |   4 +-
 include/linux/sunrpc/clnt.h |   2 +-
 net/sunrpc/clnt.c           |   6 +-
 net/sunrpc/sched.c          |   2 +-
 net/sunrpc/xprtmultipath.c  |   4 +
 13 files changed, 406 insertions(+), 43 deletions(-)

-- 
1.8.3.1

[PATCH Version 4 01/10] NFS default callback ops

[andros]

From: Andy Adamson <andros@netapp.com>

Also use with refactored exchange_id for session trunking

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4proc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 327b8c3..da2aa2e 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -6784,12 +6784,12 @@ nfs41_same_server_scope(struct nfs41_server_scope *a,
 }
 
 static void
-nfs4_bind_one_conn_to_session_done(struct rpc_task *task, void *calldata)
+nfs4_proc_default_done(struct rpc_task *task, void *calldata)
 {
 }
 
-static const struct rpc_call_ops nfs4_bind_one_conn_to_session_ops = {
-	.rpc_call_done =  &nfs4_bind_one_conn_to_session_done,
+static const struct rpc_call_ops nfs4_proc_default_ops = {
+	.rpc_call_done =  &nfs4_proc_default_done,
 };
 
 /*
@@ -6820,7 +6820,7 @@ int nfs4_proc_bind_one_conn_to_session(struct rpc_clnt *clnt,
 	struct rpc_task_setup task_setup_data = {
 		.rpc_client = clnt,
 		.rpc_xprt = xprt,
-		.callback_ops = &nfs4_bind_one_conn_to_session_ops,
+		.callback_ops = &nfs4_proc_default_ops,
 		.rpc_message = &msg,
 		.flags = RPC_TASK_TIMEOUT,
 	};
-- 
1.8.3.1

[PATCH Version 4 02/10] NFS refactor nfs4_match_clientids

[andros]

From: Andy Adamson <andros@netapp.com>

For session trunking, to compare nfs41_exchange_id_res with
exiting nfs_client.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4client.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index 10410e8..c9177a4 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -562,15 +562,15 @@ out:
 /*
  * Returns true if the client IDs match
  */
-static bool nfs4_match_clientids(struct nfs_client *a, struct nfs_client *b)
+static bool nfs4_match_clientids(u64 a, u64 b)
 {
-	if (a->cl_clientid != b->cl_clientid) {
+	if (a != b) {
 		dprintk("NFS: --> %s client ID %llx does not match %llx\n",
-			__func__, a->cl_clientid, b->cl_clientid);
+			__func__, a, b);
 		return false;
 	}
 	dprintk("NFS: --> %s client ID %llx matches %llx\n",
-		__func__, a->cl_clientid, b->cl_clientid);
+		__func__, a, b);
 	return true;
 }
 
@@ -650,7 +650,7 @@ int nfs41_walk_client_list(struct nfs_client *new,
 		if (pos->cl_cons_state != NFS_CS_READY)
 			continue;
 
-		if (!nfs4_match_clientids(pos, new))
+		if (!nfs4_match_clientids(pos->cl_clientid, new->cl_clientid))
 			continue;
 
 		/*
-- 
1.8.3.1

[PATCH Version 4 03/10] NFS refactor nfs4_check_serverowner_major_id

[andros]

From: Andy Adamson <andros@netapp.com>

For session trunking, to compare nfs41_exchange_id_res with
existing nfs_client

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4client.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index c9177a4..5a14fda 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -578,11 +578,9 @@ static bool nfs4_match_clientids(u64 a, u64 b)
  * Returns true if the server major ids match
  */
 static bool
-nfs4_check_clientid_trunking(struct nfs_client *a, struct nfs_client *b)
+nfs4_check_serverowner_major_id(struct nfs41_server_owner *o1,
+				struct nfs41_server_owner *o2)
 {
-	struct nfs41_server_owner *o1 = a->cl_serverowner;
-	struct nfs41_server_owner *o2 = b->cl_serverowner;
-
 	if (o1->major_id_sz != o2->major_id_sz)
 		goto out_major_mismatch;
 	if (memcmp(o1->major_id, o2->major_id, o1->major_id_sz) != 0)
@@ -658,7 +656,8 @@ int nfs41_walk_client_list(struct nfs_client *new,
 		 * client id trunking. In either case, we want to fall back
 		 * to using the existing nfs_client.
 		 */
-		if (!nfs4_check_clientid_trunking(pos, new))
+		if (!nfs4_check_serverowner_major_id(pos->cl_serverowner,
+						     new->cl_serverowner))
 			continue;
 
 		/* Unlike NFSv4.0, we know that NFSv4.1 always uses the
-- 
1.8.3.1

[PATCH Version 4 04/10] NFS detect session trunking

[andros]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4client.c | 83 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 82 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index 5a14fda..2121c1f 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -586,7 +586,7 @@ nfs4_check_serverowner_major_id(struct nfs41_server_owner *o1,
 	if (memcmp(o1->major_id, o2->major_id, o1->major_id_sz) != 0)
 		goto out_major_mismatch;
 
-	dprintk("NFS: --> %s server owners match\n", __func__);
+	dprintk("NFS: --> %s server owner major IDs match\n", __func__);
 	return true;
 
 out_major_mismatch:
@@ -595,6 +595,87 @@ out_major_mismatch:
 	return false;
 }
 
+/*
+ * Returns true if server minor ids match
+ */
+static bool
+nfs4_check_serverowner_minor_id(struct nfs41_server_owner *o1,
+				struct nfs41_server_owner *o2)
+{
+	/* Check eir_server_owner so_minor_id */
+	if (o1->minor_id != o2->minor_id)
+		goto out_minor_mismatch;
+
+	dprintk("NFS: --> %s server owner minor IDs s match\n", __func__);
+	return true;
+
+out_minor_mismatch:
+	dprintk("NFS: --> %s server owner minor IDs do not match\n", __func__);
+	return false;
+}
+
+/*
+ * Returns true if the server scopes match
+ */
+static bool
+nfs4_check_server_scope(struct nfs41_server_scope *s1,
+			struct nfs41_server_scope *s2)
+{
+	if (s1->server_scope_sz != s2->server_scope_sz)
+		goto out_scope_mismatch;
+	if (memcmp(s1->server_scope, s2->server_scope,
+		   s1->server_scope_sz) != 0)
+		goto out_scope_mismatch;
+
+	dprintk("NFS: --> %s server scopes match\n", __func__);
+	return true;
+
+out_scope_mismatch:
+	dprintk("NFS: --> %s server scopes do not match\n",
+		__func__);
+	return false;
+}
+
+/**
+ * nfs4_detect_session_trunking - Checks for ssession trunking called
+ * after a successuful EXCHANGE_ID testing a multi-addr connection to be
+ * potentially added as a session trunk
+ *
+ * @clp:    original mount nfs_client
+ * @res:    result structure from an exchange_id using the original mount
+ *          nfs_client with a new multi_addr transport
+ *
+ * Returns zero on success, otherwise -EINVAL
+ *
+ * Note: since the exchange_id for the new multi_addr transport uses the
+ * same nfs_client from the original mount, the cl_owner_id is reused,
+ * so eir_clientowner is the same.
+ */
+int nfs4_detect_session_trunking(struct nfs_client *clp,
+				 struct nfs41_exchange_id_res *res)
+{
+
+	/* Check eir_clientid */
+	if (!nfs4_match_clientids(clp->cl_clientid, res->clientid))
+		return -EINVAL;
+
+	/* Check eir_server_owner so_major_id */
+	if (!nfs4_check_serverowner_major_id(clp->cl_serverowner,
+					     res->server_owner))
+		return -EINVAL;
+
+	/* Check eir_server_owner so_minor_id */
+	if (!nfs4_check_serverowner_minor_id(clp->cl_serverowner,
+					     res->server_owner))
+		return -EINVAL;
+
+	/* Check eir_server_scope */
+	if (!nfs4_check_server_scope(clp->cl_serverscope, res->server_scope))
+		return -EINVAL;
+
+	return 0;
+}
+
 /**
  * nfs41_walk_client_list - Find nfs_client that matches a client/server owner
  *
-- 
1.8.3.1

[PATCH Version 4 05/10] NFS refactor _nfs4_proc_exchange_id for session trunking

[andros]

From: Andy Adamson <andros@netapp.com>

Add an rpc_xprt parameter to test it's connection for session trunking

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4_fs.h  |  2 ++
 fs/nfs/nfs4proc.c | 34 ++++++++++++++++++++++++++++------
 2 files changed, 30 insertions(+), 6 deletions(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 4afdee4..ff8cfcd 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -276,6 +276,8 @@ extern int nfs4_proc_get_lease_time(struct nfs_client *clp,
 		struct nfs_fsinfo *fsinfo);
 extern int nfs4_proc_layoutcommit(struct nfs4_layoutcommit_data *data,
 				  bool sync);
+extern int nfs4_detect_session_trunking(struct nfs_client *clp,
+		struct nfs41_exchange_id_res *res);
 
 static inline bool
 is_ds_only_client(struct nfs_client *clp)
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index da2aa2e..70537f0 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -7024,7 +7024,7 @@ static int nfs4_sp4_select_mode(struct nfs_client *clp,
  * Wrapper for EXCHANGE_ID operation.
  */
 static int _nfs4_proc_exchange_id(struct nfs_client *clp, struct rpc_cred *cred,
-	u32 sp4_how)
+	u32 sp4_how, struct rpc_xprt *xprt)
 {
 	nfs4_verifier verifier;
 	struct nfs41_exchange_id_args args = {
@@ -7049,6 +7049,18 @@ static int _nfs4_proc_exchange_id(struct nfs_client *clp, struct rpc_cred *cred,
 		.rpc_resp = &res,
 		.rpc_cred = cred,
 	};
+	struct rpc_task_setup task_setup_data = {
+		.rpc_client = clp->cl_rpcclient,
+		.rpc_xprt = xprt,
+		.callback_ops = &nfs4_proc_default_ops,
+		.rpc_message = &msg,
+		.flags = RPC_TASK_TIMEOUT,
+	};
+	struct rpc_task *task;
+
+	/* Do not run exchange_id against the established mount connection */
+	if (xprt && xprt == rcu_access_pointer(clp->cl_rpcclient->cl_xprt))
+		return 1;
 
 	nfs4_init_boot_verifier(clp, &verifier);
 
@@ -7096,11 +7108,17 @@ static int _nfs4_proc_exchange_id(struct nfs_client *clp, struct rpc_cred *cred,
 		goto out_impl_id;
 	}
 
-	status = rpc_call_sync(clp->cl_rpcclient, &msg, RPC_TASK_TIMEOUT);
+	task = rpc_run_task(&task_setup_data);
+	if (!IS_ERR(task)) {
+		status = task->tk_status;
+		rpc_put_task(task);
+	} else
+		status = PTR_ERR(task);
 	trace_nfs4_exchange_id(clp, status);
 	if (status == 0)
 		status = nfs4_check_cl_exchange_flags(res.flags);
-
+	if (xprt)
+		goto session_trunk;
 	if (status == 0)
 		status = nfs4_sp4_select_mode(clp, &res.state_protect);
 
@@ -7138,7 +7156,6 @@ static int _nfs4_proc_exchange_id(struct nfs_client *clp, struct rpc_cred *cred,
 			res.server_scope = NULL;
 		}
 	}
-
 out_impl_id:
 	kfree(res.impl_id);
 out_server_scope:
@@ -7154,6 +7171,11 @@ out:
 			clp->cl_implid->date.nseconds);
 	dprintk("NFS reply exchange_id: %d\n", status);
 	return status;
+
+session_trunk:
+	if (status == 0)
+		status = nfs4_detect_session_trunking(clp, &res);
+	goto out_impl_id;
 }
 
 /*
@@ -7176,13 +7198,13 @@ int nfs4_proc_exchange_id(struct nfs_client *clp, struct rpc_cred *cred)
 	/* try SP4_MACH_CRED if krb5i/p	*/
 	if (authflavor == RPC_AUTH_GSS_KRB5I ||
 	    authflavor == RPC_AUTH_GSS_KRB5P) {
-		status = _nfs4_proc_exchange_id(clp, cred, SP4_MACH_CRED);
+		status = _nfs4_proc_exchange_id(clp, cred, SP4_MACH_CRED, NULL);
 		if (!status)
 			return 0;
 	}
 
 	/* try SP4_NONE */
-	return _nfs4_proc_exchange_id(clp, cred, SP4_NONE);
+	return _nfs4_proc_exchange_id(clp, cred, SP4_NONE, NULL);
 }
 
 static int _nfs4_proc_destroy_clientid(struct nfs_client *clp,
-- 
1.8.3.1

[PATCH Version 4 06/10] SUNRPC add flag to rpc_task_release_client

[andros]

From: Andy Adamson <andros@netapp.com>

Want to specify which rpc_xprt to use in rpc_run_task.

Don't pass in an rpc_xprt in rpc_init_task just to have it not used as it
is removed in rpc_task_release_client.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 include/linux/sunrpc/clnt.h | 2 +-
 net/sunrpc/clnt.c           | 6 +++---
 net/sunrpc/sched.c          | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/include/linux/sunrpc/clnt.h b/include/linux/sunrpc/clnt.h
index 9a7ddba..b2c5b75 100644
--- a/include/linux/sunrpc/clnt.h
+++ b/include/linux/sunrpc/clnt.h
@@ -150,7 +150,7 @@ int		rpc_switch_client_transport(struct rpc_clnt *,
 
 void		rpc_shutdown_client(struct rpc_clnt *);
 void		rpc_release_client(struct rpc_clnt *);
-void		rpc_task_release_client(struct rpc_task *);
+void		rpc_task_release_client(struct rpc_task *, int);
 
 int		rpcb_create_local(struct net *);
 void		rpcb_put_local(struct net *);
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 7e0c9bf..e157d09 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -942,7 +942,7 @@ out:
 }
 EXPORT_SYMBOL_GPL(rpc_bind_new_program);
 
-void rpc_task_release_client(struct rpc_task *task)
+void rpc_task_release_client(struct rpc_task *task, int rm_xprt)
 {
 	struct rpc_clnt *clnt = task->tk_client;
 	struct rpc_xprt *xprt = task->tk_xprt;
@@ -957,7 +957,7 @@ void rpc_task_release_client(struct rpc_task *task)
 		rpc_release_client(clnt);
 	}
 
-	if (xprt != NULL) {
+	if (rm_xprt && xprt) {
 		task->tk_xprt = NULL;
 
 		xprt_put(xprt);
@@ -969,7 +969,7 @@ void rpc_task_set_client(struct rpc_task *task, struct rpc_clnt *clnt)
 {
 
 	if (clnt != NULL) {
-		rpc_task_release_client(task);
+		rpc_task_release_client(task, 0);
 		if (task->tk_xprt == NULL)
 			task->tk_xprt = xprt_iter_get_next(&clnt->cl_xpi);
 		task->tk_client = clnt;
diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index fcfd48d..0dacd9c 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -993,7 +993,7 @@ static void rpc_release_resources_task(struct rpc_task *task)
 		put_rpccred(task->tk_msg.rpc_cred);
 		task->tk_msg.rpc_cred = NULL;
 	}
-	rpc_task_release_client(task);
+	rpc_task_release_client(task, 1);
 }
 
 static void rpc_final_put_task(struct rpc_task *task,
-- 
1.8.3.1

[PATCH Version 4 07/10] NFS probe pseudo-fs for replicas

[andros]

From: Andy Adamson <andros@netapp.com>

Probe an NFSv4 Pseudo-fs for fs_locations replicas. Replicas on a Pseudo-fs
returns multipath addresses for the server which will be tested
for session trunking.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/internal.h       |  2 ++
 fs/nfs/nfs4getroot.c    |  3 ++
 fs/nfs/nfs4proc.c       | 84 +++++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/nfs4xdr.c        | 14 ++++++++-
 include/linux/nfs_xdr.h |  4 +--
 5 files changed, 104 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index f1d1d2c..4a65288 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -528,6 +528,8 @@ extern int nfs40_walk_client_list(struct nfs_client *clp,
 extern int nfs41_walk_client_list(struct nfs_client *clp,
 				struct nfs_client **result,
 				struct rpc_cred *cred);
+extern void nfs4_get_pseudofs_replicas(struct nfs_server *server,
+				struct nfs_fh *mntfh);
 
 static inline struct inode *nfs_igrab_and_active(struct inode *inode)
 {
diff --git a/fs/nfs/nfs4getroot.c b/fs/nfs/nfs4getroot.c
index 039b3eb..4c1f5e4 100644
--- a/fs/nfs/nfs4getroot.c
+++ b/fs/nfs/nfs4getroot.c
@@ -36,6 +36,9 @@ int nfs4_get_rootfh(struct nfs_server *server, struct nfs_fh *mntfh, bool auth_p
 	}
 
 	memcpy(&server->fsid, &fsinfo.fattr->fsid, sizeof(server->fsid));
+
+	nfs4_get_pseudofs_replicas(server, mntfh);
+
 out:
 	nfs_free_fattr(fsinfo.fattr);
 	dprintk("<-- nfs4_get_rootfh() = %d\n", ret);
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 70537f0..8263ee8 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -3253,6 +3253,90 @@ static int nfs4_do_find_root_sec(struct nfs_server *server,
 	return nfs_v4_minor_ops[mv]->find_root_sec(server, fhandle, info);
 }
 
+static int _nfs4_proc_fs_locations_probe(struct nfs_server *server,
+					 struct nfs_fh *fhandle,
+					 struct nfs4_fs_locations *fs_locations,
+					 struct page *page)
+{
+	u32 bitmask[3] = {
+		[0] = FATTR4_WORD0_FS_LOCATIONS,
+	};
+	struct nfs4_fs_locations_arg args = {
+		.fh = fhandle,
+		.page = page,
+		.bitmask = bitmask,
+		.replicas = 1,
+	};
+	struct nfs4_fs_locations_res res = {
+		.fs_locations = fs_locations,
+		.replicas = 1,
+	};
+	struct rpc_message msg = {
+		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_FS_LOCATIONS],
+		.rpc_argp = &args,
+		.rpc_resp = &res,
+	};
+	struct rpc_clnt *clnt = server->nfs_client->cl_rpcclient;
+	int status = 0;
+
+	nfs_fattr_init(&fs_locations->fattr);
+	fs_locations->nlocations = 0;
+	fs_locations->server = server;
+	status = nfs4_call_sync(clnt, server, &msg, &args.seq_args,
+				&res.seq_res, 0);
+	return status;
+}
+
+static int nfs4_proc_fs_locations_probe(struct nfs_server *server,
+					struct nfs_fh *fhandle,
+					struct nfs4_fs_locations *fs_locations,
+					struct page *page)
+{
+	struct nfs4_exception exception = { };
+	int err;
+
+	do {
+		err = _nfs4_proc_fs_locations_probe(server, fhandle,
+						fs_locations, page);
+		err = nfs4_handle_exception(server, err, &exception);
+	} while (exception.retry);
+	return err;
+}
+
+
+/**
+ * Probe the pseudo filesystem for an fs_locations replicas list.
+ * Note: The replicas list on a pseudofs is a list of multipath
+ * addresses
+ */
+void nfs4_get_pseudofs_replicas(struct nfs_server *server,
+				       struct nfs_fh *mntfh)
+{
+	struct page *page = NULL;
+	struct nfs4_fs_locations *locations = NULL;
+	int status;
+
+	if (server->nfs_client->cl_minorversion == 0)
+		return;
+
+	page = alloc_page(GFP_KERNEL);
+	if (page == NULL)
+		goto out;
+	locations = kmalloc(sizeof(struct nfs4_fs_locations), GFP_KERNEL);
+	if (locations == NULL)
+		goto out;
+
+	status = nfs4_proc_fs_locations_probe(server, mntfh, locations, page);
+	if (status != 0)
+		goto out;
+
+	/* test replicas for session trunking here */
+out:
+	if (page)
+		__free_page(page);
+	kfree(locations);
+}
+
 /**
  * nfs4_proc_get_rootfh - get file handle for server's pseudoroot
  * @server: initialized nfs_server handle
diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
index 88474a4..4dfd136 100644
--- a/fs/nfs/nfs4xdr.c
+++ b/fs/nfs/nfs4xdr.c
@@ -2685,7 +2685,11 @@ static void nfs4_xdr_enc_fs_locations(struct rpc_rqst *req,
 
 	encode_compound_hdr(xdr, req, &hdr);
 	encode_sequence(xdr, &args->seq_args, &hdr);
-	if (args->migration) {
+	if (args->replicas) {
+		encode_putfh(xdr, args->fh, &hdr);
+		replen = hdr.replen;
+		encode_fs_locations(xdr, args->bitmask, &hdr);
+	} else if (args->migration) {
 		encode_putfh(xdr, args->fh, &hdr);
 		replen = hdr.replen;
 		encode_fs_locations(xdr, args->bitmask, &hdr);
@@ -6916,6 +6920,14 @@ static int nfs4_xdr_dec_fs_locations(struct rpc_rqst *req,
 	status = decode_putfh(xdr);
 	if (status)
 		goto out;
+	if (res->replicas) {
+		xdr_enter_page(xdr, PAGE_SIZE);
+		status = decode_getfattr_generic(xdr,
+					&res->fs_locations->fattr,
+					 NULL, res->fs_locations,
+					 NULL, res->fs_locations->server);
+		goto out;
+	}
 	if (res->migration) {
 		xdr_enter_page(xdr, PAGE_SIZE);
 		status = decode_getfattr_generic(xdr,
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index d320906..ab9a08a2 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1120,13 +1120,13 @@ struct nfs4_fs_locations_arg {
 	struct page *page;
 	const u32 *bitmask;
 	clientid4 clientid;
-	unsigned char migration:1, renew:1;
+	unsigned char migration:1, renew:1, replicas:1;
 };
 
 struct nfs4_fs_locations_res {
 	struct nfs4_sequence_res	seq_res;
 	struct nfs4_fs_locations       *fs_locations;
-	unsigned char			migration:1, renew:1;
+	unsigned char			migration:1, renew:1, replicas:1;
 };
 
 struct nfs4_secinfo4 {
-- 
1.8.3.1

[PATCH Version 4 08/10] NFS test and add multipaths for session trunking

[andros]

From: Andy Adamson <andros@netapp.com>

Test the multipath addresses returned by nfs4_get_pseudofs_replicas
for session trunking.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4_fs.h           |   7 ++++
 fs/nfs/nfs4proc.c          | 100 ++++++++++++++++++++++++++++++++++++++++++++-
 net/sunrpc/xprtmultipath.c |   3 ++
 3 files changed, 109 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index ff8cfcd..e886aa7 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -59,6 +59,8 @@ struct nfs4_minor_version_ops {
 			struct nfs4_lock_state *);
 	struct nfs_seqid *
 		(*alloc_seqid)(struct nfs_seqid_counter *, gfp_t);
+	int	(*session_trunk)(struct rpc_clnt *, struct rpc_xprt_switch *,
+			struct rpc_xprt *, void *);
 	const struct rpc_call_ops *call_sync_ops;
 	const struct nfs4_state_recovery_ops *reboot_recovery_ops;
 	const struct nfs4_state_recovery_ops *nograce_recovery_ops;
@@ -214,6 +216,11 @@ struct nfs4_mig_recovery_ops {
 	int (*fsid_present)(struct inode *, struct rpc_cred *);
 };
 
+struct nfs4_add_xprt_data {
+	struct nfs_client	*clp;
+	struct rpc_cred		*cred;
+};
+
 extern const struct dentry_operations nfs4_dentry_operations;
 
 /* dir.c */
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 8263ee8..827ae51 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -44,6 +44,7 @@
 #include <linux/printk.h>
 #include <linux/slab.h>
 #include <linux/sunrpc/clnt.h>
+#include <linux/sunrpc/addr.h>
 #include <linux/nfs.h>
 #include <linux/nfs4.h>
 #include <linux/nfs_fs.h>
@@ -3303,6 +3304,59 @@ static int nfs4_proc_fs_locations_probe(struct nfs_server *server,
 	return err;
 }
 
+/**
+ * Test the multipath addresses returned by nfs4_get_pseudofs_replicas
+ * for session trunking.
+ *
+ * Add session trunking aliases to the cl_rpcclient
+ */
+void nfs4_test_multipath(struct nfs4_fs_locations *locations,
+			struct nfs_client *clp)
+{
+	struct nfs4_add_xprt_data xprtdata = {
+		.clp = clp,
+	};
+	int i;
+
+	if (!clp->cl_mvops->session_trunk || locations == NULL ||
+	    locations->nlocations <= 0)
+		return;
+
+	xprtdata.cred = nfs4_get_clid_cred(clp);
+
+	for (i = 0; i < locations->nlocations; i++) {
+		struct nfs4_fs_location *loc = &locations->locations[i];
+		int i;
+
+		for (i = 0; i < loc->nservers; i++) {
+			struct nfs4_string *server = &loc->servers[i];
+			struct sockaddr_storage addr;
+			size_t addrlen;
+			struct xprt_create xprt_args = {
+				.ident = XPRT_TRANSPORT_TCP,
+				.net = clp->cl_net,
+			};
+
+			addrlen = rpc_pton(clp->cl_net, server->data,
+					server->len, (struct sockaddr *)&addr,
+					sizeof(addr));
+
+			xprt_args.dstaddr = (struct sockaddr *)&addr;
+			xprt_args.addrlen = addrlen;
+			xprt_args.servername = server->data;
+
+			/**
+			 * Check for session trunking. Add this address as
+			 * an alias if session trunking is permitted.
+			 */
+			rpc_clnt_add_xprt(clp->cl_rpcclient, &xprt_args,
+					clp->cl_mvops->session_trunk,
+					&xprtdata);
+		}
+	}
+	if (xprtdata.cred)
+		put_rpccred(xprtdata.cred);
+}
 
 /**
  * Probe the pseudo filesystem for an fs_locations replicas list.
@@ -3330,7 +3384,8 @@ void nfs4_get_pseudofs_replicas(struct nfs_server *server,
 	if (status != 0)
 		goto out;
 
-	/* test replicas for session trunking here */
+	/* test replicas for session trunking */
+	nfs4_test_multipath(locations, server->nfs_client);
 out:
 	if (page)
 		__free_page(page);
@@ -7291,6 +7346,47 @@ int nfs4_proc_exchange_id(struct nfs_client *clp, struct rpc_cred *cred)
 	return _nfs4_proc_exchange_id(clp, cred, SP4_NONE, NULL);
 }
 
+/**
+ * nfs4_test_session_trunk - Test for session trunking with a
+ * synchronous exchange_id call. Upon success, add a new transport
+ * to the rpc_clnt
+ *
+ * @clnt: struct rpc_clnt to get new transport
+ * @xps:  the rpc_xprt_switch to hold the new transport
+ * @xprt: the rpc_xprt to test
+ * @data: call data for _nfs4_proc_exchange_id.
+ *
+ */
+int nfs4_test_session_trunk(struct rpc_clnt *clnt, struct rpc_xprt_switch *xps,
+			    struct rpc_xprt *xprt, void *data)
+{
+	struct nfs4_add_xprt_data *xdata = (struct nfs4_add_xprt_data *)data;
+	u32 sp4_how;
+	int status;
+
+	sp4_how = (xdata->clp->cl_sp4_flags == 0 ? SP4_NONE : SP4_MACH_CRED);
+
+	/* Ensure these stick around for the rpc call */
+	xps = xprt_switch_get(xps);
+	xprt = xprt_get(xprt);
+
+	/* Sync call */
+	status = _nfs4_proc_exchange_id(xdata->clp, xdata->cred, sp4_how, xprt);
+
+	xprt_put(xprt);
+	xprt_switch_put(xps);
+
+	if (status)
+		pr_info("NFS:   %s: Session trunking failed for %s status %d\n",
+			xdata->clp->cl_hostname,
+			xprt->address_strings[RPC_DISPLAY_ADDR], status);
+	else
+		pr_info("NFS:   %s: Session trunking succeeded for %s\n",
+			xdata->clp->cl_hostname,
+			xprt->address_strings[RPC_DISPLAY_ADDR]);
+	return status;
+}
+
 static int _nfs4_proc_destroy_clientid(struct nfs_client *clp,
 		struct rpc_cred *cred)
 {
@@ -8882,6 +8978,7 @@ static const struct nfs4_minor_version_ops nfs_v4_1_minor_ops = {
 	.find_root_sec = nfs41_find_root_sec,
 	.free_lock_state = nfs41_free_lock_state,
 	.alloc_seqid = nfs_alloc_no_seqid,
+	.session_trunk = nfs4_test_session_trunk,
 	.call_sync_ops = &nfs41_call_sync_ops,
 	.reboot_recovery_ops = &nfs41_reboot_recovery_ops,
 	.nograce_recovery_ops = &nfs41_nograce_recovery_ops,
@@ -8910,6 +9007,7 @@ static const struct nfs4_minor_version_ops nfs_v4_2_minor_ops = {
 	.free_lock_state = nfs41_free_lock_state,
 	.call_sync_ops = &nfs41_call_sync_ops,
 	.alloc_seqid = nfs_alloc_no_seqid,
+	.session_trunk = nfs4_test_session_trunk,
 	.reboot_recovery_ops = &nfs41_reboot_recovery_ops,
 	.nograce_recovery_ops = &nfs41_nograce_recovery_ops,
 	.state_renewal_ops = &nfs41_state_renewal_ops,
diff --git a/net/sunrpc/xprtmultipath.c b/net/sunrpc/xprtmultipath.c
index e7fd769..360f64c 100644
--- a/net/sunrpc/xprtmultipath.c
+++ b/net/sunrpc/xprtmultipath.c
@@ -53,6 +53,7 @@ void rpc_xprt_switch_add_xprt(struct rpc_xprt_switch *xps,
 		xprt_switch_add_xprt_locked(xps, xprt);
 	spin_unlock(&xps->xps_lock);
 }
+EXPORT_SYMBOL_GPL(rpc_xprt_switch_add_xprt);
 
 static void xprt_switch_remove_xprt_locked(struct rpc_xprt_switch *xps,
 		struct rpc_xprt *xprt)
@@ -145,6 +146,7 @@ struct rpc_xprt_switch *xprt_switch_get(struct rpc_xprt_switch *xps)
 		return xps;
 	return NULL;
 }
+EXPORT_SYMBOL_GPL(xprt_switch_get);
 
 /**
  * xprt_switch_put - Release a reference to a rpc_xprt_switch
@@ -157,6 +159,7 @@ void xprt_switch_put(struct rpc_xprt_switch *xps)
 	if (xps != NULL)
 		kref_put(&xps->xps_kref, xprt_switch_free);
 }
+EXPORT_SYMBOL_GPL(xprt_switch_put);
 
 /**
  * rpc_xprt_switch_set_roundrobin - Set a round-robin policy on rpc_xprt_switch
-- 
1.8.3.1

[PATCH Version 4 09/10] NFS test pnfs data server multipath session trunking aliases

[andros]

From: Andy Adamson <andros@netapp.com>

Try all multipath addresses for a data server. The first address that
successfully connects and creates a session is the mount address.
All subsequent addresses are tested for session trunking and
added as aliases.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/pnfs_nfs.c | 51 +++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 37 insertions(+), 14 deletions(-)

diff --git a/fs/nfs/pnfs_nfs.c b/fs/nfs/pnfs_nfs.c
index 4aaed89..b1e2b94 100644
--- a/fs/nfs/pnfs_nfs.c
+++ b/fs/nfs/pnfs_nfs.c
@@ -654,13 +654,43 @@ static int _nfs4_pnfs_v4_ds_connect(struct nfs_server *mds_srv,
 		dprintk("%s: DS %s: trying address %s\n",
 			__func__, ds->ds_remotestr, da->da_remotestr);
 
-		clp = nfs4_set_ds_client(mds_srv->nfs_client,
-					(struct sockaddr *)&da->da_addr,
-					da->da_addrlen, IPPROTO_TCP,
-					timeo, retrans, minor_version,
-					au_flavor);
-		if (!IS_ERR(clp))
-			break;
+		if (!IS_ERR(clp) && clp->cl_mvops->session_trunk) {
+			struct xprt_create xprt_args = {
+				.ident = XPRT_TRANSPORT_TCP,
+				.net = clp->cl_net,
+				.dstaddr = (struct sockaddr *)&da->da_addr,
+				.addrlen = da->da_addrlen,
+				.servername = clp->cl_hostname,
+			};
+			struct nfs4_add_xprt_data xprtdata = {
+				.clp = clp,
+			};
+			xprtdata.cred = nfs4_get_clid_cred(clp);
+
+			/**
+			 * Test this address for session trunking and
+			 * add as an alias
+			 */
+			rpc_clnt_add_xprt(clp->cl_rpcclient, &xprt_args,
+					  clp->cl_mvops->session_trunk,
+					  &xprtdata);
+		} else {
+			clp = nfs4_set_ds_client(mds_srv->nfs_client,
+						(struct sockaddr *)&da->da_addr,
+						da->da_addrlen, IPPROTO_TCP,
+						timeo, retrans, minor_version,
+						au_flavor);
+			if (IS_ERR(clp))
+				continue;
+
+			status = nfs4_init_ds_session(clp,
+					mds_srv->nfs_client->cl_lease_time);
+			if (status) {
+				nfs_put_client(clp);
+				clp = ERR_PTR(-EIO);
+				continue;
+			}
+		}
 	}
 
 	if (IS_ERR(clp)) {
@@ -668,18 +698,11 @@ static int _nfs4_pnfs_v4_ds_connect(struct nfs_server *mds_srv,
 		goto out;
 	}
 
-	status = nfs4_init_ds_session(clp, mds_srv->nfs_client->cl_lease_time);
-	if (status)
-		goto out_put;
-
 	smp_wmb();
 	ds->ds_clp = clp;
 	dprintk("%s [new] addr: %s\n", __func__, ds->ds_remotestr);
 out:
 	return status;
-out_put:
-	nfs_put_client(clp);
-	goto out;
 }
 
 /*
-- 
1.8.3.1

[PATCH Version 4 10/10] NFS add multiaddr to nfs_show_nfsv4_options

[andros]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/super.c             | 26 ++++++++++++++++++++++++++
 net/sunrpc/xprtmultipath.c |  1 +
 2 files changed, 27 insertions(+)

diff --git a/fs/nfs/super.c b/fs/nfs/super.c
index f126828..8d96025 100644
--- a/fs/nfs/super.c
+++ b/fs/nfs/super.c
@@ -603,8 +603,34 @@ static void nfs_show_nfsv4_options(struct seq_file *m, struct nfs_server *nfss,
 				    int showdefaults)
 {
 	struct nfs_client *clp = nfss->nfs_client;
+	struct rpc_clnt *clnt = clp->cl_rpcclient;
+	struct rpc_xprt_switch *xps;
+	struct rpc_xprt *pos;
+	unsigned int nxprts;
 
 	seq_printf(m, ",clientaddr=%s", clp->cl_ipaddr);
+	rcu_read_lock();
+	xps = xprt_switch_get(rcu_dereference(clnt->cl_xpi.xpi_xpswitch));
+	pos = xprt_iter_xprt(&clnt->cl_xpi);
+	if (xps == NULL || pos == NULL) {
+		rcu_read_unlock();
+		return;
+	}
+	nxprts = xps->xps_nxprts;
+	list_for_each_entry_rcu(pos, &xps->xps_xprt_list, xprt_switch) {
+		if (nxprts == 0)
+			break;
+		if (pos->address_strings[RPC_DISPLAY_ADDR] == NULL)
+			break;
+		/* Do not display mount xprt as a multiaddr */
+		if (pos == rcu_access_pointer(clp->cl_rpcclient->cl_xprt))
+			continue;
+		seq_printf(m, ",multiaddr=%s",
+			pos->address_strings[RPC_DISPLAY_ADDR]);
+		nxprts--;
+	}
+	rcu_read_unlock();
+	xprt_switch_put(xps);
 }
 #else
 static void nfs_show_nfsv4_options(struct seq_file *m, struct nfs_server *nfss,
diff --git a/net/sunrpc/xprtmultipath.c b/net/sunrpc/xprtmultipath.c
index 360f64c..708c22a 100644
--- a/net/sunrpc/xprtmultipath.c
+++ b/net/sunrpc/xprtmultipath.c
@@ -403,6 +403,7 @@ struct rpc_xprt *xprt_iter_xprt(struct rpc_xprt_iter *xpi)
 	WARN_ON_ONCE(!rcu_read_lock_held());
 	return xprt_iter_ops(xpi)->xpi_xprt(xpi);
 }
+EXPORT_SYMBOL_GPL(xprt_iter_xprt);
 
 static
 struct rpc_xprt *xprt_iter_get_helper(struct rpc_xprt_iter *xpi,
-- 
1.8.3.1

Re: [PATCH Version 4 01/10] NFS default callback ops

[Anna Schumaker]

Hi Andy,

I think you're missing a word or two from the title of this patch.  I'm guessing it should say "rename default callback ops"?

Anna

On 05/11/2016 03:08 PM, andros@netapp.com wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> Also use with refactored exchange_id for session trunking
> 
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfs/nfs4proc.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
> index 327b8c3..da2aa2e 100644
> --- a/fs/nfs/nfs4proc.c
> +++ b/fs/nfs/nfs4proc.c
> @@ -6784,12 +6784,12 @@ nfs41_same_server_scope(struct nfs41_server_scope *a,
>  }
>  
>  static void
> -nfs4_bind_one_conn_to_session_done(struct rpc_task *task, void *calldata)
> +nfs4_proc_default_done(struct rpc_task *task, void *calldata)
>  {
>  }
>  
> -static const struct rpc_call_ops nfs4_bind_one_conn_to_session_ops = {
> -	.rpc_call_done =  &nfs4_bind_one_conn_to_session_done,
> +static const struct rpc_call_ops nfs4_proc_default_ops = {
> +	.rpc_call_done =  &nfs4_proc_default_done,
>  };
>  
>  /*
> @@ -6820,7 +6820,7 @@ int nfs4_proc_bind_one_conn_to_session(struct rpc_clnt *clnt,
>  	struct rpc_task_setup task_setup_data = {
>  		.rpc_client = clnt,
>  		.rpc_xprt = xprt,
> -		.callback_ops = &nfs4_bind_one_conn_to_session_ops,
> +		.callback_ops = &nfs4_proc_default_ops,
>  		.rpc_message = &msg,
>  		.flags = RPC_TASK_TIMEOUT,
>  	};
> 

Re: [PATCH Version 4 01/10] NFS default callback ops

[Adamson, Andy]

DQo+IE9uIE1heSAxNywgMjAxNiwgYXQgNDoxNyBQTSwgU2NodW1ha2VyLCBBbm5hIDxBbm5hLlNj
aHVtYWtlckBuZXRhcHAuY29tPiB3cm90ZToNCj4gDQo+IEhpIEFuZHksDQo+IA0KPiBJIHRoaW5r
IHlvdSdyZSBtaXNzaW5nIGEgd29yZCBvciB0d28gZnJvbSB0aGUgdGl0bGUgb2YgdGhpcyBwYXRj
aC4gIEknbSBndWVzc2luZyBpdCBzaG91bGQgc2F5ICJyZW5hbWUgZGVmYXVsdCBjYWxsYmFjayBv
cHPigJ0/DQoNClllcy4NCg0K4oCUPkFuZHkNCj4gDQo+IEFubmENCj4gDQo+IE9uIDA1LzExLzIw
MTYgMDM6MDggUE0sIGFuZHJvc0BuZXRhcHAuY29tIHdyb3RlOg0KPj4gRnJvbTogQW5keSBBZGFt
c29uIDxhbmRyb3NAbmV0YXBwLmNvbT4NCj4+IA0KPj4gQWxzbyB1c2Ugd2l0aCByZWZhY3RvcmVk
IGV4Y2hhbmdlX2lkIGZvciBzZXNzaW9uIHRydW5raW5nDQo+PiANCj4+IFNpZ25lZC1vZmYtYnk6
IEFuZHkgQWRhbXNvbiA8YW5kcm9zQG5ldGFwcC5jb20+DQo+PiAtLS0NCj4+IGZzL25mcy9uZnM0
cHJvYy5jIHwgOCArKysrLS0tLQ0KPj4gMSBmaWxlIGNoYW5nZWQsIDQgaW5zZXJ0aW9ucygrKSwg
NCBkZWxldGlvbnMoLSkNCj4+IA0KPj4gZGlmZiAtLWdpdCBhL2ZzL25mcy9uZnM0cHJvYy5jIGIv
ZnMvbmZzL25mczRwcm9jLmMNCj4+IGluZGV4IDMyN2I4YzMuLmRhMmFhMmUgMTAwNjQ0DQo+PiAt
LS0gYS9mcy9uZnMvbmZzNHByb2MuYw0KPj4gKysrIGIvZnMvbmZzL25mczRwcm9jLmMNCj4+IEBA
IC02Nzg0LDEyICs2Nzg0LDEyIEBAIG5mczQxX3NhbWVfc2VydmVyX3Njb3BlKHN0cnVjdCBuZnM0
MV9zZXJ2ZXJfc2NvcGUgKmEsDQo+PiB9DQo+PiANCj4+IHN0YXRpYyB2b2lkDQo+PiAtbmZzNF9i
aW5kX29uZV9jb25uX3RvX3Nlc3Npb25fZG9uZShzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2ssIHZvaWQg
KmNhbGxkYXRhKQ0KPj4gK25mczRfcHJvY19kZWZhdWx0X2RvbmUoc3RydWN0IHJwY190YXNrICp0
YXNrLCB2b2lkICpjYWxsZGF0YSkNCj4+IHsNCj4+IH0NCj4+IA0KPj4gLXN0YXRpYyBjb25zdCBz
dHJ1Y3QgcnBjX2NhbGxfb3BzIG5mczRfYmluZF9vbmVfY29ubl90b19zZXNzaW9uX29wcyA9IHsN
Cj4+IC0JLnJwY19jYWxsX2RvbmUgPSAgJm5mczRfYmluZF9vbmVfY29ubl90b19zZXNzaW9uX2Rv
bmUsDQo+PiArc3RhdGljIGNvbnN0IHN0cnVjdCBycGNfY2FsbF9vcHMgbmZzNF9wcm9jX2RlZmF1
bHRfb3BzID0gew0KPj4gKwkucnBjX2NhbGxfZG9uZSA9ICAmbmZzNF9wcm9jX2RlZmF1bHRfZG9u
ZSwNCj4+IH07DQo+PiANCj4+IC8qDQo+PiBAQCAtNjgyMCw3ICs2ODIwLDcgQEAgaW50IG5mczRf
cHJvY19iaW5kX29uZV9jb25uX3RvX3Nlc3Npb24oc3RydWN0IHJwY19jbG50ICpjbG50LA0KPj4g
CXN0cnVjdCBycGNfdGFza19zZXR1cCB0YXNrX3NldHVwX2RhdGEgPSB7DQo+PiAJCS5ycGNfY2xp
ZW50ID0gY2xudCwNCj4+IAkJLnJwY194cHJ0ID0geHBydCwNCj4+IC0JCS5jYWxsYmFja19vcHMg
PSAmbmZzNF9iaW5kX29uZV9jb25uX3RvX3Nlc3Npb25fb3BzLA0KPj4gKwkJLmNhbGxiYWNrX29w
cyA9ICZuZnM0X3Byb2NfZGVmYXVsdF9vcHMsDQo+PiAJCS5ycGNfbWVzc2FnZSA9ICZtc2csDQo+
PiAJCS5mbGFncyA9IFJQQ19UQVNLX1RJTUVPVVQsDQo+PiAJfTsNCj4+IA0KPiANCg0K

Re: [PATCH Version 4 03/10] NFS refactor nfs4_check_serverowner_major_id

[Anna Schumaker]

Hi Andy,

On 05/11/2016 03:08 PM, andros@netapp.com wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> For session trunking, to compare nfs41_exchange_id_res with
> existing nfs_client
> 
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfs/nfs4client.c | 9 ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)
> 
> diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
> index c9177a4..5a14fda 100644
> --- a/fs/nfs/nfs4client.c
> +++ b/fs/nfs/nfs4client.c
> @@ -578,11 +578,9 @@ static bool nfs4_match_clientids(u64 a, u64 b)
>   * Returns true if the server major ids match
>   */
>  static bool
> -nfs4_check_clientid_trunking(struct nfs_client *a, struct nfs_client *b)
> +nfs4_check_serverowner_major_id(struct nfs41_server_owner *o1,
> +				struct nfs41_server_owner *o2)
>  {

It looks like you update the dprintk() in this function with the next patch.  Can you please update it in this one, instead?

Thanks,
Anna

> -	struct nfs41_server_owner *o1 = a->cl_serverowner;
> -	struct nfs41_server_owner *o2 = b->cl_serverowner;
> -
>  	if (o1->major_id_sz != o2->major_id_sz)
>  		goto out_major_mismatch;
>  	if (memcmp(o1->major_id, o2->major_id, o1->major_id_sz) != 0)
> @@ -658,7 +656,8 @@ int nfs41_walk_client_list(struct nfs_client *new,
>  		 * client id trunking. In either case, we want to fall back
>  		 * to using the existing nfs_client.
>  		 */
> -		if (!nfs4_check_clientid_trunking(pos, new))
> +		if (!nfs4_check_serverowner_major_id(pos->cl_serverowner,
> +						     new->cl_serverowner))
>  			continue;
>  
>  		/* Unlike NFSv4.0, we know that NFSv4.1 always uses the
> 

Re: [PATCH Version 4 07/10] NFS probe pseudo-fs for replicas

[Anna Schumaker]

Hi Andy,

On 05/11/2016 03:08 PM, andros@netapp.com wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> Probe an NFSv4 Pseudo-fs for fs_locations replicas. Replicas on a Pseudo-fs
> returns multipath addresses for the server which will be tested
> for session trunking.
> 
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfs/internal.h       |  2 ++
>  fs/nfs/nfs4getroot.c    |  3 ++
>  fs/nfs/nfs4proc.c       | 84 +++++++++++++++++++++++++++++++++++++++++++++++++
>  fs/nfs/nfs4xdr.c        | 14 ++++++++-
>  include/linux/nfs_xdr.h |  4 +--
>  5 files changed, 104 insertions(+), 3 deletions(-)
> 
> diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
> index f1d1d2c..4a65288 100644
> --- a/fs/nfs/internal.h
> +++ b/fs/nfs/internal.h
> @@ -528,6 +528,8 @@ extern int nfs40_walk_client_list(struct nfs_client *clp,
>  extern int nfs41_walk_client_list(struct nfs_client *clp,
>  				struct nfs_client **result,
>  				struct rpc_cred *cred);
> +extern void nfs4_get_pseudofs_replicas(struct nfs_server *server,
> +				struct nfs_fh *mntfh);
>  
>  static inline struct inode *nfs_igrab_and_active(struct inode *inode)
>  {
> diff --git a/fs/nfs/nfs4getroot.c b/fs/nfs/nfs4getroot.c
> index 039b3eb..4c1f5e4 100644
> --- a/fs/nfs/nfs4getroot.c
> +++ b/fs/nfs/nfs4getroot.c
> @@ -36,6 +36,9 @@ int nfs4_get_rootfh(struct nfs_server *server, struct nfs_fh *mntfh, bool auth_p
>  	}
>  
>  	memcpy(&server->fsid, &fsinfo.fattr->fsid, sizeof(server->fsid));
> +
> +	nfs4_get_pseudofs_replicas(server, mntfh);
> +
>  out:
>  	nfs_free_fattr(fsinfo.fattr);
>  	dprintk("<-- nfs4_get_rootfh() = %d\n", ret);
> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
> index 70537f0..8263ee8 100644
> --- a/fs/nfs/nfs4proc.c
> +++ b/fs/nfs/nfs4proc.c
> @@ -3253,6 +3253,90 @@ static int nfs4_do_find_root_sec(struct nfs_server *server,
>  	return nfs_v4_minor_ops[mv]->find_root_sec(server, fhandle, info);
>  }
>  
> +static int _nfs4_proc_fs_locations_probe(struct nfs_server *server,
> +					 struct nfs_fh *fhandle,
> +					 struct nfs4_fs_locations *fs_locations,
> +					 struct page *page)
> +{
> +	u32 bitmask[3] = {
> +		[0] = FATTR4_WORD0_FS_LOCATIONS,
> +	};
> +	struct nfs4_fs_locations_arg args = {
> +		.fh = fhandle,
> +		.page = page,
> +		.bitmask = bitmask,
> +		.replicas = 1,
> +	};
> +	struct nfs4_fs_locations_res res = {
> +		.fs_locations = fs_locations,
> +		.replicas = 1,
> +	};
> +	struct rpc_message msg = {
> +		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_FS_LOCATIONS],
> +		.rpc_argp = &args,
> +		.rpc_resp = &res,
> +	};
> +	struct rpc_clnt *clnt = server->nfs_client->cl_rpcclient;
> +	int status = 0;
> +
> +	nfs_fattr_init(&fs_locations->fattr);
> +	fs_locations->nlocations = 0;
> +	fs_locations->server = server;
> +	status = nfs4_call_sync(clnt, server, &msg, &args.seq_args,
> +				&res.seq_res, 0);
> +	return status;
> +}

I don't think we need the status variable here.  Let's just directly return the result of nfs4_call_sync().


> +
> +static int nfs4_proc_fs_locations_probe(struct nfs_server *server,
> +					struct nfs_fh *fhandle,
> +					struct nfs4_fs_locations *fs_locations,
> +					struct page *page)
> +{
> +	struct nfs4_exception exception = { };
> +	int err;
> +
> +	do {
> +		err = _nfs4_proc_fs_locations_probe(server, fhandle,
> +						fs_locations, page);
> +		err = nfs4_handle_exception(server, err, &exception);
> +	} while (exception.retry);
> +	return err;
> +}
> +
> +
> +/**
> + * Probe the pseudo filesystem for an fs_locations replicas list.
> + * Note: The replicas list on a pseudofs is a list of multipath
> + * addresses
> + */
> +void nfs4_get_pseudofs_replicas(struct nfs_server *server,
> +				       struct nfs_fh *mntfh)
> +{
> +	struct page *page = NULL;
> +	struct nfs4_fs_locations *locations = NULL;
> +	int status;
> +
> +	if (server->nfs_client->cl_minorversion == 0)
> +		return;
> +
> +	page = alloc_page(GFP_KERNEL);
> +	if (page == NULL)
> +		goto out;
> +	locations = kmalloc(sizeof(struct nfs4_fs_locations), GFP_KERNEL);
> +	if (locations == NULL)
> +		goto out;
> +
> +	status = nfs4_proc_fs_locations_probe(server, mntfh, locations, page);
> +	if (status != 0)
> +		goto out;
> +
> +	/* test replicas for session trunking here */
> +out:
> +	if (page)
> +		__free_page(page);
> +	kfree(locations);
> +}
> +
>  /**
>   * nfs4_proc_get_rootfh - get file handle for server's pseudoroot
>   * @server: initialized nfs_server handle
> diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
> index 88474a4..4dfd136 100644
> --- a/fs/nfs/nfs4xdr.c
> +++ b/fs/nfs/nfs4xdr.c
> @@ -2685,7 +2685,11 @@ static void nfs4_xdr_enc_fs_locations(struct rpc_rqst *req,
>  
>  	encode_compound_hdr(xdr, req, &hdr);
>  	encode_sequence(xdr, &args->seq_args, &hdr);
> -	if (args->migration) {
> +	if (args->replicas) {
> +		encode_putfh(xdr, args->fh, &hdr);
> +		replen = hdr.replen;
> +		encode_fs_locations(xdr, args->bitmask, &hdr);
> +	} else if (args->migration) {

I really don't like the amount of duplicated code going into this if/else block.  Maybe it can be cleaned up like this? (untested and uncompiled)

@@ -2685,22 +2685,15 @@ static void nfs4_xdr_enc_fs_locations(struct rpc_rqst *req,
 
        encode_compound_hdr(xdr, req, &hdr);
        encode_sequence(xdr, &args->seq_args, &hdr);
-       if (args->replicas) {
+       if (args->replicas || args->migration)
                encode_putfh(xdr, args->fh, &hdr);
-               replen = hdr.replen;
-               encode_fs_locations(xdr, args->bitmask, &hdr);
-       } else if (args->migration) {
-               encode_putfh(xdr, args->fh, &hdr);
-               replen = hdr.replen;
-               encode_fs_locations(xdr, args->bitmask, &hdr);
-               if (args->renew)
-                       encode_renew(xdr, args->clientid, &hdr);
-       } else {
+       else
                encode_putfh(xdr, args->dir_fh, &hdr);
-               encode_lookup(xdr, args->name, &hdr);
-               replen = hdr.replen;
-               encode_fs_locations(xdr, args->bitmask, &hdr);
-       }
+       replen = hdr.replen;
+       encode_fs_locations(xdr, args->bitmask, &hdr);
+
+       if (args->migration && args->renew)
+               encode_renew(xdr, args->clientid, &hdr);
 
        /* Set up reply kvec to capture returned fs_locations array. */


Thanks,
Anna

>  		encode_putfh(xdr, args->fh, &hdr);
>  		replen = hdr.replen;
>  		encode_fs_locations(xdr, args->bitmask, &hdr);
> @@ -6916,6 +6920,14 @@ static int nfs4_xdr_dec_fs_locations(struct rpc_rqst *req,
>  	status = decode_putfh(xdr);
>  	if (status)
>  		goto out;
> +	if (res->replicas) {
> +		xdr_enter_page(xdr, PAGE_SIZE);
> +		status = decode_getfattr_generic(xdr,
> +					&res->fs_locations->fattr,
> +					 NULL, res->fs_locations,
> +					 NULL, res->fs_locations->server);
> +		goto out;
> +	}
>  	if (res->migration) {
>  		xdr_enter_page(xdr, PAGE_SIZE);
>  		status = decode_getfattr_generic(xdr,
> diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
> index d320906..ab9a08a2 100644
> --- a/include/linux/nfs_xdr.h
> +++ b/include/linux/nfs_xdr.h
> @@ -1120,13 +1120,13 @@ struct nfs4_fs_locations_arg {
>  	struct page *page;
>  	const u32 *bitmask;
>  	clientid4 clientid;
> -	unsigned char migration:1, renew:1;
> +	unsigned char migration:1, renew:1, replicas:1;
>  };
>  
>  struct nfs4_fs_locations_res {
>  	struct nfs4_sequence_res	seq_res;
>  	struct nfs4_fs_locations       *fs_locations;
> -	unsigned char			migration:1, renew:1;
> +	unsigned char			migration:1, renew:1, replicas:1;
>  };
>  
>  struct nfs4_secinfo4 {
> 

Re: [PATCH Version 4 08/10] NFS test and add multipaths for session trunking

[Anna Schumaker]

Hi Andy,

On 05/11/2016 03:08 PM, andros@netapp.com wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> Test the multipath addresses returned by nfs4_get_pseudofs_replicas
> for session trunking.
> 
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfs/nfs4_fs.h           |   7 ++++
>  fs/nfs/nfs4proc.c          | 100 ++++++++++++++++++++++++++++++++++++++++++++-
>  net/sunrpc/xprtmultipath.c |   3 ++
>  3 files changed, 109 insertions(+), 1 deletion(-)
> 

<snip>

> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
> index 8263ee8..827ae51 100644
> --- a/fs/nfs/nfs4proc.c
> +++ b/fs/nfs/nfs4proc.c
> @@ -3303,6 +3304,59 @@ static int nfs4_proc_fs_locations_probe(struct nfs_server *server,
>  	return err;
>  }
>  
> +/**
> + * Test the multipath addresses returned by nfs4_get_pseudofs_replicas
> + * for session trunking.
> + *
> + * Add session trunking aliases to the cl_rpcclient
> + */
> +void nfs4_test_multipath(struct nfs4_fs_locations *locations,
> +			struct nfs_client *clp)
> +{
> +	struct nfs4_add_xprt_data xprtdata = {
> +		.clp = clp,
> +	};
> +	int i;
> +
> +	if (!clp->cl_mvops->session_trunk || locations == NULL ||
> +	    locations->nlocations <= 0)
> +		return;
> +
> +	xprtdata.cred = nfs4_get_clid_cred(clp);
> +
> +	for (i = 0; i < locations->nlocations; i++) {
> +		struct nfs4_fs_location *loc = &locations->locations[i];
> +		int i;
> +
> +		for (i = 0; i < loc->nservers; i++) {
> +			struct nfs4_string *server = &loc->servers[i];
> +			struct sockaddr_storage addr;
> +			size_t addrlen;
> +			struct xprt_create xprt_args = {
> +				.ident = XPRT_TRANSPORT_TCP,
> +				.net = clp->cl_net,
> +			};
> +
> +			addrlen = rpc_pton(clp->cl_net, server->data,
> +					server->len, (struct sockaddr *)&addr,
> +					sizeof(addr));
> +
> +			xprt_args.dstaddr = (struct sockaddr *)&addr;
> +			xprt_args.addrlen = addrlen;
> +			xprt_args.servername = server->data;
> +
> +			/**
> +			 * Check for session trunking. Add this address as
> +			 * an alias if session trunking is permitted.
> +			 */
> +			rpc_clnt_add_xprt(clp->cl_rpcclient, &xprt_args,
> +					clp->cl_mvops->session_trunk,
> +					&xprtdata);
> +		}
> +	}
> +	if (xprtdata.cred)
> +		put_rpccred(xprtdata.cred);
> +}
>  
>  /**
>   * Probe the pseudo filesystem for an fs_locations replicas list.
> @@ -3330,7 +3384,8 @@ void nfs4_get_pseudofs_replicas(struct nfs_server *server,
>  	if (status != 0)
>  		goto out;
>  
> -	/* test replicas for session trunking here */
> +	/* test replicas for session trunking */
> +	nfs4_test_multipath(locations, server->nfs_client);
>  out:
>  	if (page)
>  		__free_page(page);
> @@ -7291,6 +7346,47 @@ int nfs4_proc_exchange_id(struct nfs_client *clp, struct rpc_cred *cred)
>  	return _nfs4_proc_exchange_id(clp, cred, SP4_NONE, NULL);
>  }
>  
> +/**
> + * nfs4_test_session_trunk - Test for session trunking with a
> + * synchronous exchange_id call. Upon success, add a new transport
> + * to the rpc_clnt
> + *
> + * @clnt: struct rpc_clnt to get new transport
> + * @xps:  the rpc_xprt_switch to hold the new transport
> + * @xprt: the rpc_xprt to test
> + * @data: call data for _nfs4_proc_exchange_id.
> + *
> + */
> +int nfs4_test_session_trunk(struct rpc_clnt *clnt, struct rpc_xprt_switch *xps,
> +			    struct rpc_xprt *xprt, void *data)
> +{
> +	struct nfs4_add_xprt_data *xdata = (struct nfs4_add_xprt_data *)data;
> +	u32 sp4_how;
> +	int status;
> +
> +	sp4_how = (xdata->clp->cl_sp4_flags == 0 ? SP4_NONE : SP4_MACH_CRED);
> +
> +	/* Ensure these stick around for the rpc call */
> +	xps = xprt_switch_get(xps);
> +	xprt = xprt_get(xprt);

I'm curious at what point during _nfs4_proc_exchange_id() these might get released?

Thanks,
Anna

> +
> +	/* Sync call */
> +	status = _nfs4_proc_exchange_id(xdata->clp, xdata->cred, sp4_how, xprt);
> +
> +	xprt_put(xprt);
> +	xprt_switch_put(xps);
> +
> +	if (status)
> +		pr_info("NFS:   %s: Session trunking failed for %s status %d\n",
> +			xdata->clp->cl_hostname,
> +			xprt->address_strings[RPC_DISPLAY_ADDR], status);
> +	else
> +		pr_info("NFS:   %s: Session trunking succeeded for %s\n",
> +			xdata->clp->cl_hostname,
> +			xprt->address_strings[RPC_DISPLAY_ADDR]);
> +	return status;
> +}
> +
>  static int _nfs4_proc_destroy_clientid(struct nfs_client *clp,
>  		struct rpc_cred *cred)
>  {
> @@ -8882,6 +8978,7 @@ static const struct nfs4_minor_version_ops nfs_v4_1_minor_ops = {
>  	.find_root_sec = nfs41_find_root_sec,
>  	.free_lock_state = nfs41_free_lock_state,
>  	.alloc_seqid = nfs_alloc_no_seqid,
> +	.session_trunk = nfs4_test_session_trunk,
>  	.call_sync_ops = &nfs41_call_sync_ops,
>  	.reboot_recovery_ops = &nfs41_reboot_recovery_ops,
>  	.nograce_recovery_ops = &nfs41_nograce_recovery_ops,
> @@ -8910,6 +9007,7 @@ static const struct nfs4_minor_version_ops nfs_v4_2_minor_ops = {
>  	.free_lock_state = nfs41_free_lock_state,
>  	.call_sync_ops = &nfs41_call_sync_ops,
>  	.alloc_seqid = nfs_alloc_no_seqid,
> +	.session_trunk = nfs4_test_session_trunk,
>  	.reboot_recovery_ops = &nfs41_reboot_recovery_ops,
>  	.nograce_recovery_ops = &nfs41_nograce_recovery_ops,
>  	.state_renewal_ops = &nfs41_state_renewal_ops,
> diff --git a/net/sunrpc/xprtmultipath.c b/net/sunrpc/xprtmultipath.c
> index e7fd769..360f64c 100644
> --- a/net/sunrpc/xprtmultipath.c
> +++ b/net/sunrpc/xprtmultipath.c
> @@ -53,6 +53,7 @@ void rpc_xprt_switch_add_xprt(struct rpc_xprt_switch *xps,
>  		xprt_switch_add_xprt_locked(xps, xprt);
>  	spin_unlock(&xps->xps_lock);
>  }
> +EXPORT_SYMBOL_GPL(rpc_xprt_switch_add_xprt);
>  
>  static void xprt_switch_remove_xprt_locked(struct rpc_xprt_switch *xps,
>  		struct rpc_xprt *xprt)
> @@ -145,6 +146,7 @@ struct rpc_xprt_switch *xprt_switch_get(struct rpc_xprt_switch *xps)
>  		return xps;
>  	return NULL;
>  }
> +EXPORT_SYMBOL_GPL(xprt_switch_get);
>  
>  /**
>   * xprt_switch_put - Release a reference to a rpc_xprt_switch
> @@ -157,6 +159,7 @@ void xprt_switch_put(struct rpc_xprt_switch *xps)
>  	if (xps != NULL)
>  		kref_put(&xps->xps_kref, xprt_switch_free);
>  }
> +EXPORT_SYMBOL_GPL(xprt_switch_put);
>  
>  /**
>   * rpc_xprt_switch_set_roundrobin - Set a round-robin policy on rpc_xprt_switch
> 

Re: [PATCH Version 4 08/10] NFS test and add multipaths for session trunking

[Adamson, Andy]

DQo+IE9uIE1heSAxOCwgMjAxNiwgYXQgMjoyMyBQTSwgU2NodW1ha2VyLCBBbm5hIDxBbm5hLlNj
aHVtYWtlckBuZXRhcHAuY29tPiB3cm90ZToNCj4gDQo+IEhpIEFuZHksDQo+IA0KPiBPbiAwNS8x
MS8yMDE2IDAzOjA4IFBNLCBhbmRyb3NAbmV0YXBwLmNvbSB3cm90ZToNCj4+IEZyb206IEFuZHkg
QWRhbXNvbiA8YW5kcm9zQG5ldGFwcC5jb20+DQo+PiANCj4+IFRlc3QgdGhlIG11bHRpcGF0aCBh
ZGRyZXNzZXMgcmV0dXJuZWQgYnkgbmZzNF9nZXRfcHNldWRvZnNfcmVwbGljYXMNCj4+IGZvciBz
ZXNzaW9uIHRydW5raW5nLg0KPj4gDQo+PiBTaWduZWQtb2ZmLWJ5OiBBbmR5IEFkYW1zb24gPGFu
ZHJvc0BuZXRhcHAuY29tPg0KPj4gLS0tDQo+PiBmcy9uZnMvbmZzNF9mcy5oICAgICAgICAgICB8
ICAgNyArKysrDQo+PiBmcy9uZnMvbmZzNHByb2MuYyAgICAgICAgICB8IDEwMCArKysrKysrKysr
KysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKy0NCj4+IG5ldC9zdW5ycGMveHBydG11
bHRpcGF0aC5jIHwgICAzICsrDQo+PiAzIGZpbGVzIGNoYW5nZWQsIDEwOSBpbnNlcnRpb25zKCsp
LCAxIGRlbGV0aW9uKC0pDQo+PiANCj4gDQo+IDxzbmlwPg0KPiANCj4+IGRpZmYgLS1naXQgYS9m
cy9uZnMvbmZzNHByb2MuYyBiL2ZzL25mcy9uZnM0cHJvYy5jDQo+PiBpbmRleCA4MjYzZWU4Li44
MjdhZTUxIDEwMDY0NA0KPj4gLS0tIGEvZnMvbmZzL25mczRwcm9jLmMNCj4+ICsrKyBiL2ZzL25m
cy9uZnM0cHJvYy5jDQo+PiBAQCAtMzMwMyw2ICszMzA0LDU5IEBAIHN0YXRpYyBpbnQgbmZzNF9w
cm9jX2ZzX2xvY2F0aW9uc19wcm9iZShzdHJ1Y3QgbmZzX3NlcnZlciAqc2VydmVyLA0KPj4gCXJl
dHVybiBlcnI7DQo+PiB9DQo+PiANCj4+ICsvKioNCj4+ICsgKiBUZXN0IHRoZSBtdWx0aXBhdGgg
YWRkcmVzc2VzIHJldHVybmVkIGJ5IG5mczRfZ2V0X3BzZXVkb2ZzX3JlcGxpY2FzDQo+PiArICog
Zm9yIHNlc3Npb24gdHJ1bmtpbmcuDQo+PiArICoNCj4+ICsgKiBBZGQgc2Vzc2lvbiB0cnVua2lu
ZyBhbGlhc2VzIHRvIHRoZSBjbF9ycGNjbGllbnQNCj4+ICsgKi8NCj4+ICt2b2lkIG5mczRfdGVz
dF9tdWx0aXBhdGgoc3RydWN0IG5mczRfZnNfbG9jYXRpb25zICpsb2NhdGlvbnMsDQo+PiArCQkJ
c3RydWN0IG5mc19jbGllbnQgKmNscCkNCj4+ICt7DQo+PiArCXN0cnVjdCBuZnM0X2FkZF94cHJ0
X2RhdGEgeHBydGRhdGEgPSB7DQo+PiArCQkuY2xwID0gY2xwLA0KPj4gKwl9Ow0KPj4gKwlpbnQg
aTsNCj4+ICsNCj4+ICsJaWYgKCFjbHAtPmNsX212b3BzLT5zZXNzaW9uX3RydW5rIHx8IGxvY2F0
aW9ucyA9PSBOVUxMIHx8DQo+PiArCSAgICBsb2NhdGlvbnMtPm5sb2NhdGlvbnMgPD0gMCkNCj4+
ICsJCXJldHVybjsNCj4+ICsNCj4+ICsJeHBydGRhdGEuY3JlZCA9IG5mczRfZ2V0X2NsaWRfY3Jl
ZChjbHApOw0KPj4gKw0KPj4gKwlmb3IgKGkgPSAwOyBpIDwgbG9jYXRpb25zLT5ubG9jYXRpb25z
OyBpKyspIHsNCj4+ICsJCXN0cnVjdCBuZnM0X2ZzX2xvY2F0aW9uICpsb2MgPSAmbG9jYXRpb25z
LT5sb2NhdGlvbnNbaV07DQo+PiArCQlpbnQgaTsNCj4+ICsNCj4+ICsJCWZvciAoaSA9IDA7IGkg
PCBsb2MtPm5zZXJ2ZXJzOyBpKyspIHsNCj4+ICsJCQlzdHJ1Y3QgbmZzNF9zdHJpbmcgKnNlcnZl
ciA9ICZsb2MtPnNlcnZlcnNbaV07DQo+PiArCQkJc3RydWN0IHNvY2thZGRyX3N0b3JhZ2UgYWRk
cjsNCj4+ICsJCQlzaXplX3QgYWRkcmxlbjsNCj4+ICsJCQlzdHJ1Y3QgeHBydF9jcmVhdGUgeHBy
dF9hcmdzID0gew0KPj4gKwkJCQkuaWRlbnQgPSBYUFJUX1RSQU5TUE9SVF9UQ1AsDQo+PiArCQkJ
CS5uZXQgPSBjbHAtPmNsX25ldCwNCj4+ICsJCQl9Ow0KPj4gKw0KPj4gKwkJCWFkZHJsZW4gPSBy
cGNfcHRvbihjbHAtPmNsX25ldCwgc2VydmVyLT5kYXRhLA0KPj4gKwkJCQkJc2VydmVyLT5sZW4s
IChzdHJ1Y3Qgc29ja2FkZHIgKikmYWRkciwNCj4+ICsJCQkJCXNpemVvZihhZGRyKSk7DQo+PiAr
DQo+PiArCQkJeHBydF9hcmdzLmRzdGFkZHIgPSAoc3RydWN0IHNvY2thZGRyICopJmFkZHI7DQo+
PiArCQkJeHBydF9hcmdzLmFkZHJsZW4gPSBhZGRybGVuOw0KPj4gKwkJCXhwcnRfYXJncy5zZXJ2
ZXJuYW1lID0gc2VydmVyLT5kYXRhOw0KPj4gKw0KPj4gKwkJCS8qKg0KPj4gKwkJCSAqIENoZWNr
IGZvciBzZXNzaW9uIHRydW5raW5nLiBBZGQgdGhpcyBhZGRyZXNzIGFzDQo+PiArCQkJICogYW4g
YWxpYXMgaWYgc2Vzc2lvbiB0cnVua2luZyBpcyBwZXJtaXR0ZWQuDQo+PiArCQkJICovDQo+PiAr
CQkJcnBjX2NsbnRfYWRkX3hwcnQoY2xwLT5jbF9ycGNjbGllbnQsICZ4cHJ0X2FyZ3MsDQo+PiAr
CQkJCQljbHAtPmNsX212b3BzLT5zZXNzaW9uX3RydW5rLA0KPj4gKwkJCQkJJnhwcnRkYXRhKTsN
Cj4+ICsJCX0NCj4+ICsJfQ0KPj4gKwlpZiAoeHBydGRhdGEuY3JlZCkNCj4+ICsJCXB1dF9ycGNj
cmVkKHhwcnRkYXRhLmNyZWQpOw0KPj4gK30NCj4+IA0KPj4gLyoqDQo+PiAgKiBQcm9iZSB0aGUg
cHNldWRvIGZpbGVzeXN0ZW0gZm9yIGFuIGZzX2xvY2F0aW9ucyByZXBsaWNhcyBsaXN0Lg0KPj4g
QEAgLTMzMzAsNyArMzM4NCw4IEBAIHZvaWQgbmZzNF9nZXRfcHNldWRvZnNfcmVwbGljYXMoc3Ry
dWN0IG5mc19zZXJ2ZXIgKnNlcnZlciwNCj4+IAlpZiAoc3RhdHVzICE9IDApDQo+PiAJCWdvdG8g
b3V0Ow0KPj4gDQo+PiAtCS8qIHRlc3QgcmVwbGljYXMgZm9yIHNlc3Npb24gdHJ1bmtpbmcgaGVy
ZSAqLw0KPj4gKwkvKiB0ZXN0IHJlcGxpY2FzIGZvciBzZXNzaW9uIHRydW5raW5nICovDQo+PiAr
CW5mczRfdGVzdF9tdWx0aXBhdGgobG9jYXRpb25zLCBzZXJ2ZXItPm5mc19jbGllbnQpOw0KPj4g
b3V0Og0KPj4gCWlmIChwYWdlKQ0KPj4gCQlfX2ZyZWVfcGFnZShwYWdlKTsNCj4+IEBAIC03Mjkx
LDYgKzczNDYsNDcgQEAgaW50IG5mczRfcHJvY19leGNoYW5nZV9pZChzdHJ1Y3QgbmZzX2NsaWVu
dCAqY2xwLCBzdHJ1Y3QgcnBjX2NyZWQgKmNyZWQpDQo+PiAJcmV0dXJuIF9uZnM0X3Byb2NfZXhj
aGFuZ2VfaWQoY2xwLCBjcmVkLCBTUDRfTk9ORSwgTlVMTCk7DQo+PiB9DQo+PiANCj4+ICsvKioN
Cj4+ICsgKiBuZnM0X3Rlc3Rfc2Vzc2lvbl90cnVuayAtIFRlc3QgZm9yIHNlc3Npb24gdHJ1bmtp
bmcgd2l0aCBhDQo+PiArICogc3luY2hyb25vdXMgZXhjaGFuZ2VfaWQgY2FsbC4gVXBvbiBzdWNj
ZXNzLCBhZGQgYSBuZXcgdHJhbnNwb3J0DQo+PiArICogdG8gdGhlIHJwY19jbG50DQo+PiArICoN
Cj4+ICsgKiBAY2xudDogc3RydWN0IHJwY19jbG50IHRvIGdldCBuZXcgdHJhbnNwb3J0DQo+PiAr
ICogQHhwczogIHRoZSBycGNfeHBydF9zd2l0Y2ggdG8gaG9sZCB0aGUgbmV3IHRyYW5zcG9ydA0K
Pj4gKyAqIEB4cHJ0OiB0aGUgcnBjX3hwcnQgdG8gdGVzdA0KPj4gKyAqIEBkYXRhOiBjYWxsIGRh
dGEgZm9yIF9uZnM0X3Byb2NfZXhjaGFuZ2VfaWQuDQo+PiArICoNCj4+ICsgKi8NCj4+ICtpbnQg
bmZzNF90ZXN0X3Nlc3Npb25fdHJ1bmsoc3RydWN0IHJwY19jbG50ICpjbG50LCBzdHJ1Y3QgcnBj
X3hwcnRfc3dpdGNoICp4cHMsDQo+PiArCQkJICAgIHN0cnVjdCBycGNfeHBydCAqeHBydCwgdm9p
ZCAqZGF0YSkNCj4+ICt7DQo+PiArCXN0cnVjdCBuZnM0X2FkZF94cHJ0X2RhdGEgKnhkYXRhID0g
KHN0cnVjdCBuZnM0X2FkZF94cHJ0X2RhdGEgKilkYXRhOw0KPj4gKwl1MzIgc3A0X2hvdzsNCj4+
ICsJaW50IHN0YXR1czsNCj4+ICsNCj4+ICsJc3A0X2hvdyA9ICh4ZGF0YS0+Y2xwLT5jbF9zcDRf
ZmxhZ3MgPT0gMCA/IFNQNF9OT05FIDogU1A0X01BQ0hfQ1JFRCk7DQo+PiArDQo+PiArCS8qIEVu
c3VyZSB0aGVzZSBzdGljayBhcm91bmQgZm9yIHRoZSBycGMgY2FsbCAqLw0KPj4gKwl4cHMgPSB4
cHJ0X3N3aXRjaF9nZXQoeHBzKTsNCj4+ICsJeHBydCA9IHhwcnRfZ2V0KHhwcnQpOw0KPiANCj4g
SSdtIGN1cmlvdXMgYXQgd2hhdCBwb2ludCBkdXJpbmcgX25mczRfcHJvY19leGNoYW5nZV9pZCgp
IHRoZXNlIG1pZ2h0IGdldCByZWxlYXNlZD8NCg0KSSB3YXMgY29uY2VybmVkIGFib3V0IHRoZSBj
bGllbnQgdGVhcmluZyBkb3duIHRoZSBzZXJ2ZXIgbW91bnQgd2hvc2UgcnBjX2NsbnQgc3RydWN0
IGhvc3RzIHRoZSB4cHJ0X3N3aXRjaCAob3Igb3RoZXJ3aXNlIHRlYXJpbmcgZG93biB0aGUgcnBj
X2NsbnQpIHdoaWxlIHRoZSBfbmZzNF9wcm9jX2V4Y2hhbmdlX2lkIGlzIGluIGZsaWdodC4NCg0K
QUZBSUNTIHRoYXQgaXMgdGhlIHNhbWUgY29uY2VybiB0aGF0IHRoZSBvdGhlciB1c2Ugb2YgcnBj
X2NsbnRfYWRkX3hwcnQoKSBpbiBfbmZzNF9wbmZzX3YzX2RzX2Nvbm5lY3QoKSBoYXMuDQoNCnJw
Y19jbG50X3Rlc3RfYW5kX2FkZF94cHJ0ICgpIHJlZmVyZW5jZXMgdGhlIHhwcnRfc3dpdGNoIGFu
ZCB0aGUgeHBydCwgYW5kIHJwY19jYl9hZGRfeHBydF9yZWxlYXNlKCkgZGVyZWZlcmVuY2VzIHRo
ZW0uDQoNCuKAlD5BbmR5DQoNCg0KPiANCj4gVGhhbmtzLA0KPiBBbm5hDQo+IA0KPj4gKw0KPj4g
KwkvKiBTeW5jIGNhbGwgKi8NCj4+ICsJc3RhdHVzID0gX25mczRfcHJvY19leGNoYW5nZV9pZCh4
ZGF0YS0+Y2xwLCB4ZGF0YS0+Y3JlZCwgc3A0X2hvdywgeHBydCk7DQo+PiArDQo+PiArCXhwcnRf
cHV0KHhwcnQpOw0KPj4gKwl4cHJ0X3N3aXRjaF9wdXQoeHBzKTsNCj4+ICsNCj4+ICsJaWYgKHN0
YXR1cykNCj4+ICsJCXByX2luZm8oIk5GUzogICAlczogU2Vzc2lvbiB0cnVua2luZyBmYWlsZWQg
Zm9yICVzIHN0YXR1cyAlZFxuIiwNCj4+ICsJCQl4ZGF0YS0+Y2xwLT5jbF9ob3N0bmFtZSwNCj4+
ICsJCQl4cHJ0LT5hZGRyZXNzX3N0cmluZ3NbUlBDX0RJU1BMQVlfQUREUl0sIHN0YXR1cyk7DQo+
PiArCWVsc2UNCj4+ICsJCXByX2luZm8oIk5GUzogICAlczogU2Vzc2lvbiB0cnVua2luZyBzdWNj
ZWVkZWQgZm9yICVzXG4iLA0KPj4gKwkJCXhkYXRhLT5jbHAtPmNsX2hvc3RuYW1lLA0KPj4gKwkJ
CXhwcnQtPmFkZHJlc3Nfc3RyaW5nc1tSUENfRElTUExBWV9BRERSXSk7DQo+PiArCXJldHVybiBz
dGF0dXM7DQo+PiArfQ0KPj4gKw0KPj4gc3RhdGljIGludCBfbmZzNF9wcm9jX2Rlc3Ryb3lfY2xp
ZW50aWQoc3RydWN0IG5mc19jbGllbnQgKmNscCwNCj4+IAkJc3RydWN0IHJwY19jcmVkICpjcmVk
KQ0KPj4gew0KPj4gQEAgLTg4ODIsNiArODk3OCw3IEBAIHN0YXRpYyBjb25zdCBzdHJ1Y3QgbmZz
NF9taW5vcl92ZXJzaW9uX29wcyBuZnNfdjRfMV9taW5vcl9vcHMgPSB7DQo+PiAJLmZpbmRfcm9v
dF9zZWMgPSBuZnM0MV9maW5kX3Jvb3Rfc2VjLA0KPj4gCS5mcmVlX2xvY2tfc3RhdGUgPSBuZnM0
MV9mcmVlX2xvY2tfc3RhdGUsDQo+PiAJLmFsbG9jX3NlcWlkID0gbmZzX2FsbG9jX25vX3NlcWlk
LA0KPj4gKwkuc2Vzc2lvbl90cnVuayA9IG5mczRfdGVzdF9zZXNzaW9uX3RydW5rLA0KPj4gCS5j
YWxsX3N5bmNfb3BzID0gJm5mczQxX2NhbGxfc3luY19vcHMsDQo+PiAJLnJlYm9vdF9yZWNvdmVy
eV9vcHMgPSAmbmZzNDFfcmVib290X3JlY292ZXJ5X29wcywNCj4+IAkubm9ncmFjZV9yZWNvdmVy
eV9vcHMgPSAmbmZzNDFfbm9ncmFjZV9yZWNvdmVyeV9vcHMsDQo+PiBAQCAtODkxMCw2ICs5MDA3
LDcgQEAgc3RhdGljIGNvbnN0IHN0cnVjdCBuZnM0X21pbm9yX3ZlcnNpb25fb3BzIG5mc192NF8y
X21pbm9yX29wcyA9IHsNCj4+IAkuZnJlZV9sb2NrX3N0YXRlID0gbmZzNDFfZnJlZV9sb2NrX3N0
YXRlLA0KPj4gCS5jYWxsX3N5bmNfb3BzID0gJm5mczQxX2NhbGxfc3luY19vcHMsDQo+PiAJLmFs
bG9jX3NlcWlkID0gbmZzX2FsbG9jX25vX3NlcWlkLA0KPj4gKwkuc2Vzc2lvbl90cnVuayA9IG5m
czRfdGVzdF9zZXNzaW9uX3RydW5rLA0KPj4gCS5yZWJvb3RfcmVjb3Zlcnlfb3BzID0gJm5mczQx
X3JlYm9vdF9yZWNvdmVyeV9vcHMsDQo+PiAJLm5vZ3JhY2VfcmVjb3Zlcnlfb3BzID0gJm5mczQx
X25vZ3JhY2VfcmVjb3Zlcnlfb3BzLA0KPj4gCS5zdGF0ZV9yZW5ld2FsX29wcyA9ICZuZnM0MV9z
dGF0ZV9yZW5ld2FsX29wcywNCj4+IGRpZmYgLS1naXQgYS9uZXQvc3VucnBjL3hwcnRtdWx0aXBh
dGguYyBiL25ldC9zdW5ycGMveHBydG11bHRpcGF0aC5jDQo+PiBpbmRleCBlN2ZkNzY5Li4zNjBm
NjRjIDEwMDY0NA0KPj4gLS0tIGEvbmV0L3N1bnJwYy94cHJ0bXVsdGlwYXRoLmMNCj4+ICsrKyBi
L25ldC9zdW5ycGMveHBydG11bHRpcGF0aC5jDQo+PiBAQCAtNTMsNiArNTMsNyBAQCB2b2lkIHJw
Y194cHJ0X3N3aXRjaF9hZGRfeHBydChzdHJ1Y3QgcnBjX3hwcnRfc3dpdGNoICp4cHMsDQo+PiAJ
CXhwcnRfc3dpdGNoX2FkZF94cHJ0X2xvY2tlZCh4cHMsIHhwcnQpOw0KPj4gCXNwaW5fdW5sb2Nr
KCZ4cHMtPnhwc19sb2NrKTsNCj4+IH0NCj4+ICtFWFBPUlRfU1lNQk9MX0dQTChycGNfeHBydF9z
d2l0Y2hfYWRkX3hwcnQpOw0KPj4gDQo+PiBzdGF0aWMgdm9pZCB4cHJ0X3N3aXRjaF9yZW1vdmVf
eHBydF9sb2NrZWQoc3RydWN0IHJwY194cHJ0X3N3aXRjaCAqeHBzLA0KPj4gCQlzdHJ1Y3QgcnBj
X3hwcnQgKnhwcnQpDQo+PiBAQCAtMTQ1LDYgKzE0Niw3IEBAIHN0cnVjdCBycGNfeHBydF9zd2l0
Y2ggKnhwcnRfc3dpdGNoX2dldChzdHJ1Y3QgcnBjX3hwcnRfc3dpdGNoICp4cHMpDQo+PiAJCXJl
dHVybiB4cHM7DQo+PiAJcmV0dXJuIE5VTEw7DQo+PiB9DQo+PiArRVhQT1JUX1NZTUJPTF9HUEwo
eHBydF9zd2l0Y2hfZ2V0KTsNCj4+IA0KPj4gLyoqDQo+PiAgKiB4cHJ0X3N3aXRjaF9wdXQgLSBS
ZWxlYXNlIGEgcmVmZXJlbmNlIHRvIGEgcnBjX3hwcnRfc3dpdGNoDQo+PiBAQCAtMTU3LDYgKzE1
OSw3IEBAIHZvaWQgeHBydF9zd2l0Y2hfcHV0KHN0cnVjdCBycGNfeHBydF9zd2l0Y2ggKnhwcykN
Cj4+IAlpZiAoeHBzICE9IE5VTEwpDQo+PiAJCWtyZWZfcHV0KCZ4cHMtPnhwc19rcmVmLCB4cHJ0
X3N3aXRjaF9mcmVlKTsNCj4+IH0NCj4+ICtFWFBPUlRfU1lNQk9MX0dQTCh4cHJ0X3N3aXRjaF9w
dXQpOw0KPj4gDQo+PiAvKioNCj4+ICAqIHJwY194cHJ0X3N3aXRjaF9zZXRfcm91bmRyb2JpbiAt
IFNldCBhIHJvdW5kLXJvYmluIHBvbGljeSBvbiBycGNfeHBydF9zd2l0Y2gNCj4+IA0KPiANCg0K