[RFC v3 0/7] NFS Force Unmounting

[RFC v3 0/7] NFS Force Unmounting

[Joshua Watt]

After some additional testing on version 2, I realized that 1) my test
setup was bad (Doh!) and 2) using remount isn't really an ideal method
for setting the "serverfailed" mount flag. The reason being that any
userspace remount is going to touch the file system and probably invoke
some RPCs for inode validation as such. If the server is truly dead,
these RPCs will have to time out before the flag could actually be set.
This not only takes a long time, but will probably also fail, causing
the remount to fail and not set the flag, defeating the purpose.

This patch set works around this by exposing the failed server flag in
debugfs. In addition, the flag now applies at the nfs_client level
(instead of the nfs_server). Setting the flag will cancel all RPCs in
the client, as well as all nfs_servers attached to it. This ensures that
you get all the mounts, even when dealing with submounts that cross
remote device boundaries. This is also where the nosharecache option
applying to nfs_clients becomes useful... it will prevent sharing
between other explicit mount calls, but submounts will all still share a
nfs_client (and fail together).

Obviously, debugfs is not a permanent solution for this, so some
discussion needs to be had on how this will be administered "in the real
world" (I think "remount" is off the table).

Joshua Watt (7):
  SUNRPC: Add flag to kill new tasks
  SUNRPC: Expose kill_new_tasks in debugfs
  SUNRPC: Simplify client shutdown
  NFS: Add debugfs for nfs_server and nfs_client
  NFS: Propagate NFS_MOUNT_UNSHARED to clients
  NFS: Add API to fail client
  NFS: Control failed clients through debugfs

 fs/nfs/Makefile             |   2 +-
 fs/nfs/client.c             | 121 ++++++++++++++++++++++++--
 fs/nfs/debugfs.c            | 202 ++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/inode.c              |   5 ++
 fs/nfs/internal.h           |  11 +++
 fs/nfs/nfs3client.c         |   2 +
 fs/nfs/nfs4client.c         |   5 ++
 include/linux/nfs_fs_sb.h   |   8 ++
 include/linux/sunrpc/clnt.h |   1 +
 net/sunrpc/clnt.c           |  12 +--
 net/sunrpc/debugfs.c        |   4 +
 net/sunrpc/sched.c          |   3 +
 12 files changed, 360 insertions(+), 16 deletions(-)
 create mode 100644 fs/nfs/debugfs.c

-- 
2.13.6

[RFC v3 1/7] SUNRPC: Add flag to kill new tasks

[Joshua Watt]

The flag causes any new tasks that are queued to exit immediately with
-EIO instead of executing. This will allow clients (particularly NFS) to
prevents these task from delaying shutdown of the RPC session longer
than necessary.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 include/linux/sunrpc/clnt.h | 1 +
 net/sunrpc/clnt.c           | 4 +---
 net/sunrpc/sched.c          | 3 +++
 3 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/include/linux/sunrpc/clnt.h b/include/linux/sunrpc/clnt.h
index 71c237e8240e..94f4e464de1b 100644
--- a/include/linux/sunrpc/clnt.h
+++ b/include/linux/sunrpc/clnt.h
@@ -54,6 +54,7 @@ struct rpc_clnt {
 				cl_noretranstimeo: 1,/* No retransmit timeouts */
 				cl_autobind : 1,/* use getport() */
 				cl_chatty   : 1;/* be verbose */
+	bool			cl_kill_new_tasks;	/* Kill all new tasks */
 
 	struct rpc_rtt *	cl_rtt;		/* RTO estimator data */
 	const struct rpc_timeout *cl_timeout;	/* Timeout strategy */
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index df4ecb042ebe..ab6b7c59a117 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -626,6 +626,7 @@ static struct rpc_clnt *__rpc_clone_client(struct rpc_create_args *args,
 	new->cl_noretranstimeo = clnt->cl_noretranstimeo;
 	new->cl_discrtry = clnt->cl_discrtry;
 	new->cl_chatty = clnt->cl_chatty;
+	new->cl_kill_new_tasks = clnt->cl_kill_new_tasks;
 	return new;
 
 out_err:
@@ -818,9 +819,6 @@ void rpc_killall_tasks(struct rpc_clnt *clnt)
 {
 	struct rpc_task	*rovr;
 
-
-	if (list_empty(&clnt->cl_tasks))
-		return;
 	dprintk("RPC:       killing all tasks for client %p\n", clnt);
 	/*
 	 * Spin lock all_tasks to prevent changes...
diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index 0cc83839c13c..2c9fe5c0319b 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -748,6 +748,9 @@ static void __rpc_execute(struct rpc_task *task)
 	dprintk("RPC: %5u __rpc_execute flags=0x%x\n",
 			task->tk_pid, task->tk_flags);
 
+	if (task->tk_client->cl_kill_new_tasks)
+		rpc_exit(task, -EIO);
+
 	WARN_ON_ONCE(RPC_IS_QUEUED(task));
 	if (RPC_IS_QUEUED(task))
 		return;
-- 
2.13.6

[RFC v3 2/7] SUNRPC: Expose kill_new_tasks in debugfs

[Joshua Watt]

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 net/sunrpc/debugfs.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/net/sunrpc/debugfs.c b/net/sunrpc/debugfs.c
index e980d2a493de..2ca67b681c1e 100644
--- a/net/sunrpc/debugfs.c
+++ b/net/sunrpc/debugfs.c
@@ -160,6 +160,10 @@ rpc_clnt_debugfs_register(struct rpc_clnt *clnt)
 	if (!debugfs_create_symlink("xprt", clnt->cl_debugfs, name))
 		goto out_err;
 
+	if (!debugfs_create_bool("kill_new_tasks", S_IRUSR | S_IWUSR,
+				 clnt->cl_debugfs, &clnt->cl_kill_new_tasks))
+		goto out_err;
+
 	return;
 out_err:
 	debugfs_remove_recursive(clnt->cl_debugfs);
-- 
2.13.6

[RFC v3 3/7] SUNRPC: Simplify client shutdown

[Joshua Watt]

Use the flag to kill all new tasks when shutting down instead of
repeatedly killing all the pending tasks.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 net/sunrpc/clnt.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index ab6b7c59a117..239a7b5ab135 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -851,11 +851,9 @@ void rpc_shutdown_client(struct rpc_clnt *clnt)
 			clnt->cl_program->name,
 			rcu_dereference(clnt->cl_xprt)->servername);
 
-	while (!list_empty(&clnt->cl_tasks)) {
-		rpc_killall_tasks(clnt);
-		wait_event_timeout(destroy_wait,
-			list_empty(&clnt->cl_tasks), 1*HZ);
-	}
+	clnt->cl_kill_new_tasks = true;
+	rpc_killall_tasks(clnt);
+	wait_event(destroy_wait, list_empty(&clnt->cl_tasks));
 
 	rpc_release_client(clnt);
 }
-- 
2.13.6

[RFC v3 4/7] NFS: Add debugfs for nfs_server and nfs_client

[Joshua Watt]

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 fs/nfs/Makefile           |   2 +-
 fs/nfs/client.c           |  87 ++++++++++++++++++++++++++--
 fs/nfs/debugfs.c          | 143 ++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/inode.c            |   5 ++
 fs/nfs/internal.h         |  10 ++++
 fs/nfs/nfs4client.c       |   1 +
 include/linux/nfs_fs_sb.h |   5 ++
 7 files changed, 247 insertions(+), 6 deletions(-)
 create mode 100644 fs/nfs/debugfs.c

diff --git a/fs/nfs/Makefile b/fs/nfs/Makefile
index c587e3c4c6a6..9a0553888a28 100644
--- a/fs/nfs/Makefile
+++ b/fs/nfs/Makefile
@@ -6,7 +6,7 @@
 obj-$(CONFIG_NFS_FS) += nfs.o
 
 CFLAGS_nfstrace.o += -I$(src)
-nfs-y 			:= client.o dir.o file.o getroot.o inode.o super.o \
+nfs-y 			:= client.o debugfs.o dir.o file.o getroot.o inode.o super.o \
 			   io.o direct.o pagelist.o read.o symlink.o unlink.o \
 			   write.o namespace.o mount_clnt.o nfstrace.o export.o
 nfs-$(CONFIG_ROOT_NFS)	+= nfsroot.o
diff --git a/fs/nfs/client.c b/fs/nfs/client.c
index 22880ef6d8dd..7bab47d0dd07 100644
--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -142,6 +142,32 @@ void unregister_nfs_version(struct nfs_subversion *nfs)
 }
 EXPORT_SYMBOL_GPL(unregister_nfs_version);
 
+static DEFINE_IDA(nfs_client_ids);
+
+void
+nfs_cleanup_client_ids(void)
+{
+	ida_destroy(&nfs_client_ids);
+}
+
+static int
+nfs_alloc_client_id(struct nfs_client *client)
+{
+	int id;
+
+	id = ida_simple_get(&nfs_client_ids, 0, 0, GFP_KERNEL);
+	if (id < 0)
+		return id;
+	client->cl_id = id;
+	return 0;
+}
+
+static void
+nfs_free_client_id(struct nfs_client *client)
+{
+	ida_simple_remove(&nfs_client_ids, client->cl_id);
+}
+
 /*
  * Allocate a shared client record
  *
@@ -161,6 +187,8 @@ struct nfs_client *nfs_alloc_client(const struct nfs_client_initdata *cl_init)
 	if (!try_module_get(clp->cl_nfs_mod->owner))
 		goto error_dealloc;
 
+	nfs_alloc_client_id(clp);
+
 	clp->rpc_ops = clp->cl_nfs_mod->rpc_ops;
 
 	atomic_set(&clp->cl_count, 1);
@@ -249,8 +277,11 @@ void nfs_free_client(struct nfs_client *clp)
 	if (clp->cl_machine_cred != NULL)
 		put_rpccred(clp->cl_machine_cred);
 
+	nfs_client_debugfs_unregister(clp);
+
 	put_net(clp->cl_net);
 	put_nfs_version(clp->cl_nfs_mod);
+	nfs_free_client_id(clp);
 	kfree(clp->cl_hostname);
 	kfree(clp->cl_acceptor);
 	kfree(clp);
@@ -380,6 +411,17 @@ nfs_found_client(const struct nfs_client_initdata *cl_init,
 	return clp;
 }
 
+static struct nfs_client *
+init_client(struct nfs_client *new, const struct nfs_client_initdata *cl_init)
+{
+	struct nfs_client *ret =
+		cl_init->nfs_mod->rpc_ops->init_client(new, cl_init);
+
+	if (ret)
+		nfs_client_debugfs_register(new);
+	return ret;
+}
+
 /*
  * Look up a client by IP address and protocol version
  * - creates a new record if one doesn't yet exist
@@ -411,7 +453,7 @@ struct nfs_client *nfs_get_client(const struct nfs_client_initdata *cl_init)
 					&nn->nfs_client_list);
 			spin_unlock(&nn->nfs_client_lock);
 			new->cl_flags = cl_init->init_flags;
-			return rpc_ops->init_client(new, cl_init);
+			return init_client(new, cl_init);
 		}
 
 		spin_unlock(&nn->nfs_client_lock);
@@ -856,6 +898,32 @@ void nfs_server_remove_lists(struct nfs_server *server)
 }
 EXPORT_SYMBOL_GPL(nfs_server_remove_lists);
 
+static DEFINE_IDA(nfs_server_ids);
+
+void
+nfs_cleanup_server_ids(void)
+{
+	ida_destroy(&nfs_server_ids);
+}
+
+static int
+nfs_alloc_server_id(struct nfs_server *server)
+{
+	int id;
+
+	id = ida_simple_get(&nfs_server_ids, 0, 0, GFP_KERNEL);
+	if (id < 0)
+		return id;
+	server->id = id;
+	return 0;
+}
+
+static void
+nfs_free_server_id(struct nfs_server *server)
+{
+	ida_simple_remove(&nfs_server_ids, server->id);
+}
+
 /*
  * Allocate and initialise a server record
  */
@@ -867,6 +935,8 @@ struct nfs_server *nfs_alloc_server(void)
 	if (!server)
 		return NULL;
 
+	nfs_alloc_server_id(server);
+
 	server->client = server->client_acl = ERR_PTR(-EINVAL);
 
 	/* Zero out the NFS state stuff */
@@ -879,10 +949,8 @@ struct nfs_server *nfs_alloc_server(void)
 	atomic_set(&server->active, 0);
 
 	server->io_stats = nfs_alloc_iostats();
-	if (!server->io_stats) {
-		kfree(server);
-		return NULL;
-	}
+	if (!server->io_stats)
+		goto out_error;
 
 	ida_init(&server->openowner_id);
 	ida_init(&server->lockowner_id);
@@ -890,6 +958,10 @@ struct nfs_server *nfs_alloc_server(void)
 	rpc_init_wait_queue(&server->uoc_rpcwaitq, "NFS UOC");
 
 	return server;
+out_error:
+	nfs_free_server_id(server);
+	kfree(server);
+	return NULL;
 }
 EXPORT_SYMBOL_GPL(nfs_alloc_server);
 
@@ -910,9 +982,12 @@ void nfs_free_server(struct nfs_server *server)
 
 	nfs_put_client(server->nfs_client);
 
+	nfs_server_debugfs_unregister(server);
+
 	ida_destroy(&server->lockowner_id);
 	ida_destroy(&server->openowner_id);
 	nfs_free_iostats(server->io_stats);
+	nfs_free_server_id(server);
 	kfree(server);
 	nfs_release_automount_timer();
 }
@@ -973,6 +1048,7 @@ struct nfs_server *nfs_create_server(struct nfs_mount_info *mount_info,
 	nfs_server_insert_lists(server);
 	server->mount_time = jiffies;
 	nfs_free_fattr(fattr);
+	nfs_server_debugfs_register(server);
 	return server;
 
 error:
@@ -1033,6 +1109,7 @@ struct nfs_server *nfs_clone_server(struct nfs_server *source,
 	server->mount_time = jiffies;
 
 	nfs_free_fattr(fattr_fsinfo);
+	nfs_server_debugfs_register(server);
 	return server;
 
 out_free_server:
diff --git a/fs/nfs/debugfs.c b/fs/nfs/debugfs.c
new file mode 100644
index 000000000000..0195431427c2
--- /dev/null
+++ b/fs/nfs/debugfs.c
@@ -0,0 +1,143 @@
+// SPDX-License-Identifier: GPL-2.0
+/**
+ * debugfs interface for nfs
+ *
+ * (c) 2017 Garmin International
+ */
+
+#include <linux/debugfs.h>
+#include <linux/sunrpc/clnt.h>
+#include <linux/nfs_fs.h>
+
+#include "nfs4_fs.h"
+#include "internal.h"
+
+static struct dentry *topdir;
+static struct dentry *nfs_server_dir;
+static struct dentry *nfs_client_dir;
+
+static struct dentry*
+link_rpc_client(char const *name, struct rpc_clnt *client,
+		struct dentry *parent)
+{
+	int len;
+	char target[34]; /* "../../../sunrpc/rpc_clnt/" + 8 hex digits + '\0' */
+
+	if (IS_ERR(client) || !client->cl_debugfs)
+		return NULL;
+
+	len = snprintf(target, sizeof(target), "../../../sunrpc/rpc_clnt/%s",
+		       client->cl_debugfs->d_name.name);
+
+	if (len >= sizeof(target))
+		return NULL;
+
+	return debugfs_create_symlink(name, parent, target);
+}
+
+void
+nfs_server_debugfs_register(struct nfs_server *server)
+{
+	char name[26]; /* "../../nfs_client/" + 8 hex digits + '\0' */
+	int len;
+
+	if (server->debugfs || !nfs_server_dir)
+		return;
+
+	len = snprintf(name, sizeof(name), "%x", server->id);
+	if (len >= sizeof(name))
+		return;
+
+	server->debugfs = debugfs_create_dir(name, nfs_server_dir);
+	if (!server->debugfs)
+		return;
+
+	link_rpc_client("rpc_client", server->client, server->debugfs);
+	link_rpc_client("rpc_client_acl", server->client_acl, server->debugfs);
+
+	if (server->nfs_client->cl_debugfs) {
+		len = snprintf(name, sizeof(name), "../../nfs_client/%s",
+			       server->nfs_client->cl_debugfs->d_name.name);
+		if (len >= sizeof(name))
+			goto out_error;
+
+		if (!debugfs_create_symlink("nfs_client", server->debugfs,
+					    name))
+			goto out_error;
+	}
+
+	return;
+out_error:
+	debugfs_remove_recursive(server->debugfs);
+	server->debugfs = NULL;
+}
+EXPORT_SYMBOL_GPL(nfs_server_debugfs_register);
+
+void
+nfs_server_debugfs_unregister(struct nfs_server *server)
+{
+	debugfs_remove_recursive(server->debugfs);
+	server->debugfs = NULL;
+}
+
+void
+nfs_client_debugfs_register(struct nfs_client *client)
+{
+	char name[9]; /* 8 hex digits + '\0' */
+	int len;
+
+	if (client->cl_debugfs || !nfs_client_dir)
+		return;
+
+	len = snprintf(name, sizeof(name), "%x", client->cl_id);
+	if (len >= sizeof(name))
+		return;
+
+	client->cl_debugfs = debugfs_create_dir(name, nfs_client_dir);
+	if (!client->cl_debugfs)
+		return;
+
+	link_rpc_client("rpc_client", client->cl_rpcclient,
+			client->cl_debugfs);
+}
+
+void
+nfs_client_debugfs_unregister(struct nfs_client *client)
+{
+	debugfs_remove_recursive(client->cl_debugfs);
+	client->cl_debugfs = NULL;
+}
+
+void __exit
+nfs_debugfs_exit(void)
+{
+	debugfs_remove_recursive(topdir);
+	topdir = NULL;
+	nfs_client_dir = NULL;
+	nfs_server_dir = NULL;
+}
+
+void __init
+nfs_debugfs_init(void)
+{
+	topdir = debugfs_create_dir("nfs", NULL);
+	if (!topdir)
+		return;
+
+	nfs_server_dir = debugfs_create_dir("nfs_server", topdir);
+	if (!nfs_server_dir)
+		goto out_remove;
+
+	nfs_client_dir = debugfs_create_dir("nfs_client", topdir);
+	if (!nfs_client_dir)
+		goto out_remove;
+
+	return;
+out_remove:
+	debugfs_remove_recursive(topdir);
+	topdir = NULL;
+	nfs_server_dir = NULL;
+	nfs_client_dir = NULL;
+}
+
+
diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index 134d9f560240..28879236f949 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -2144,6 +2144,8 @@ static int __init init_nfs_fs(void)
 	if (err)
 		goto out0;
 
+	nfs_debugfs_init();
+
 	return 0;
 out0:
 	rpc_proc_unregister(&init_net, "nfs");
@@ -2181,6 +2183,9 @@ static void __exit exit_nfs_fs(void)
 	unregister_nfs_fs();
 	nfs_fs_proc_exit();
 	nfsiod_stop();
+	nfs_cleanup_server_ids();
+	nfs_cleanup_client_ids();
+	nfs_debugfs_exit();
 }
 
 /* Not quite true; I just maintain it */
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index f9a4a5524bd5..75da8185ceb8 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -156,6 +156,7 @@ extern void nfs_umount(const struct nfs_mount_request *info);
 /* client.c */
 extern const struct rpc_program nfs_program;
 extern void nfs_clients_init(struct net *net);
+void nfs_cleanup_client_ids(void);
 extern struct nfs_client *nfs_alloc_client(const struct nfs_client_initdata *);
 int nfs_create_rpc_client(struct nfs_client *, const struct nfs_client_initdata *, rpc_authflavor_t);
 struct nfs_client *nfs_get_client(const struct nfs_client_initdata *);
@@ -165,6 +166,7 @@ void nfs_server_remove_lists(struct nfs_server *);
 void nfs_init_timeout_values(struct rpc_timeout *to, int proto, int timeo, int retrans);
 int nfs_init_server_rpcclient(struct nfs_server *, const struct rpc_timeout *t,
 		rpc_authflavor_t);
+void nfs_cleanup_server_ids(void);
 struct nfs_server *nfs_alloc_server(void);
 void nfs_server_copy_userdata(struct nfs_server *, struct nfs_server *);
 
@@ -560,6 +562,14 @@ void nfs_init_cinfo_from_dreq(struct nfs_commit_info *cinfo,
 			      struct nfs_direct_req *dreq);
 extern ssize_t nfs_dreq_bytes_left(struct nfs_direct_req *dreq);
 
+/* debugfs.c */
+void nfs_server_debugfs_register(struct nfs_server *server);
+void nfs_server_debugfs_unregister(struct nfs_server *server);
+void nfs_client_debugfs_register(struct nfs_client *client);
+void nfs_client_debugfs_unregister(struct nfs_client *client);
+void __exit nfs_debugfs_exit(void);
+void __init nfs_debugfs_init(void);
+
 /* nfs4proc.c */
 extern struct nfs_client *nfs4_init_client(struct nfs_client *clp,
 			    const struct nfs_client_initdata *);
diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index e9bea90dc017..ef725106e45c 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -1086,6 +1086,7 @@ struct nfs_server *nfs4_create_server(struct nfs_mount_info *mount_info,
 	if (error < 0)
 		goto error;
 
+	nfs_server_debugfs_register(server);
 	return server;
 
 error:
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index 286b71c418b4..5dac856355f0 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -27,6 +27,7 @@ struct nfs41_impl_id;
 struct nfs_client {
 	atomic_t		cl_count;
 	atomic_t		cl_mds_count;
+	unsigned int		cl_id;		/* client id */
 	int			cl_cons_state;	/* current construction state (-ve: init error) */
 #define NFS_CS_READY		0		/* ready to be used */
 #define NFS_CS_INITING		1		/* busy initialising */
@@ -120,6 +121,7 @@ struct nfs_client {
 #endif
 
 	struct net		*cl_net;
+	struct dentry		*cl_debugfs;	/* debugfs entry */
 };
 
 /*
@@ -131,6 +133,7 @@ struct nfs_server {
 						 * that share the same client
 						 */
 	struct list_head	master_link;	/* link in master servers list */
+	unsigned int		id;		/* server id */
 	struct rpc_clnt *	client;		/* RPC client handle */
 	struct rpc_clnt *	client_acl;	/* ACL RPC client handle */
 	struct nlm_host		*nlm_host;	/* NLM client handle */
@@ -225,6 +228,8 @@ struct nfs_server {
 	unsigned short		mountd_port;
 	unsigned short		mountd_protocol;
 	struct rpc_wait_queue	uoc_rpcwaitq;
+
+	struct dentry		*debugfs;
 };
 
 /* Server capabilities */
-- 
2.13.6

[RFC v3 5/7] NFS: Propagate NFS_MOUNT_UNSHARED to clients

[Joshua Watt]

Propagating the flag allows operations to be performed on the client
without affecting other mounts to the same client, much like operations
on the superblock will not affect other superblocks.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 fs/nfs/client.c           | 9 +++++++--
 fs/nfs/nfs3client.c       | 2 ++
 fs/nfs/nfs4client.c       | 4 ++++
 include/linux/nfs_fs_sb.h | 1 +
 4 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/fs/nfs/client.c b/fs/nfs/client.c
index 7bab47d0dd07..9d88dea4bc59 100644
--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -328,6 +328,10 @@ static struct nfs_client *nfs_match_client(const struct nfs_client_initdata *dat
 		if (clp->cl_cons_state < 0)
 			continue;
 
+		/* Don't match clients that don't want to be shared */
+		if (test_bit(NFS_CS_UNSHARED, &clp->cl_flags))
+			continue;
+
 		/* Different NFS versions cannot share the same nfs_client */
 		if (clp->rpc_ops != data->nfs_mod->rpc_ops)
 			continue;
@@ -428,7 +432,7 @@ init_client(struct nfs_client *new, const struct nfs_client_initdata *cl_init)
  */
 struct nfs_client *nfs_get_client(const struct nfs_client_initdata *cl_init)
 {
-	struct nfs_client *clp, *new = NULL;
+	struct nfs_client *clp = NULL, *new = NULL;
 	struct nfs_net *nn = net_generic(cl_init->net, nfs_net_id);
 	const struct nfs_rpc_ops *rpc_ops = cl_init->nfs_mod->rpc_ops;
 
@@ -441,7 +445,8 @@ struct nfs_client *nfs_get_client(const struct nfs_client_initdata *cl_init)
 	do {
 		spin_lock(&nn->nfs_client_lock);
 
-		clp = nfs_match_client(cl_init);
+		if (!test_bit(NFS_CS_UNSHARED, &cl_init->init_flags))
+			clp = nfs_match_client(cl_init);
 		if (clp) {
 			spin_unlock(&nn->nfs_client_lock);
 			if (new)
diff --git a/fs/nfs/nfs3client.c b/fs/nfs/nfs3client.c
index 7879f2a0fcfd..b62bed41053f 100644
--- a/fs/nfs/nfs3client.c
+++ b/fs/nfs/nfs3client.c
@@ -102,6 +102,8 @@ struct nfs_client *nfs3_set_ds_client(struct nfs_server *mds_srv,
 
 	if (mds_srv->flags & NFS_MOUNT_NORESVPORT)
 		set_bit(NFS_CS_NORESVPORT, &cl_init.init_flags);
+	if (mds_srv->flags & NFS_MOUNT_UNSHARED)
+		__set_bit(NFS_CS_UNSHARED, &cl_init.init_flags);
 
 	/* Use the MDS nfs_client cl_ipaddr. */
 	nfs_init_timeout_values(&ds_timeout, ds_proto, ds_timeo, ds_retrans);
diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index ef725106e45c..d4051dda29f0 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -852,6 +852,8 @@ static int nfs4_set_client(struct nfs_server *server,
 		set_bit(NFS_CS_MIGRATION, &cl_init.init_flags);
 	if (test_bit(NFS_MIG_TSM_POSSIBLE, &server->mig_status))
 		set_bit(NFS_CS_TSM_POSSIBLE, &cl_init.init_flags);
+	if (server->flags & NFS_MOUNT_UNSHARED)
+		__set_bit(NFS_CS_UNSHARED, &cl_init.init_flags);
 
 	/* Allocate or find a client reference we can use */
 	clp = nfs_get_client(&cl_init);
@@ -910,6 +912,8 @@ struct nfs_client *nfs4_set_ds_client(struct nfs_server *mds_srv,
 
 	if (mds_srv->flags & NFS_MOUNT_NORESVPORT)
 		__set_bit(NFS_CS_NORESVPORT, &cl_init.init_flags);
+	if (mds_srv->flags & NFS_MOUNT_UNSHARED)
+		__set_bit(NFS_CS_UNSHARED, &cl_init.init_flags);
 
 	/*
 	 * Set an authflavor equual to the MDS value. Use the MDS nfs_client
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index 5dac856355f0..0eb7093faea7 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -45,6 +45,7 @@ struct nfs_client {
 #define NFS_CS_INFINITE_SLOTS	3		/* - don't limit TCP slots */
 #define NFS_CS_NO_RETRANS_TIMEOUT	4	/* - Disable retransmit timeouts */
 #define NFS_CS_TSM_POSSIBLE	5		/* - Maybe state migration */
+#define NFS_CS_UNSHARED		6		/* - Client state not shared */
 	struct sockaddr_storage	cl_addr;	/* server identifier */
 	size_t			cl_addrlen;
 	char *			cl_hostname;	/* hostname of server */
-- 
2.13.6

[RFC v3 6/7] NFS: Add API to fail client

[Joshua Watt]

Failing a client will cause all current and future RPCs sent to it to
fail with -EIO

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 fs/nfs/client.c           | 25 +++++++++++++++++++++++++
 fs/nfs/internal.h         |  1 +
 include/linux/nfs_fs_sb.h |  2 ++
 3 files changed, 28 insertions(+)

diff --git a/fs/nfs/client.c b/fs/nfs/client.c
index 9d88dea4bc59..0e41b8d5904a 100644
--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -758,6 +758,31 @@ static int nfs_init_server(struct nfs_server *server,
 	return error;
 }
 
+static void
+fail_rpc_client(struct rpc_clnt *client, bool failed)
+{
+	if (IS_ERR(client))
+		return;
+
+	client->cl_kill_new_tasks = failed;
+	if (failed)
+		rpc_killall_tasks(client);
+}
+
+void
+nfs_client_failed(struct nfs_client *client, bool failed)
+{
+	struct nfs_server *server;
+
+	client->cl_failed = failed;
+	fail_rpc_client(client->cl_rpcclient, failed);
+
+	list_for_each_entry_rcu(server, &client->cl_superblocks, client_link) {
+		fail_rpc_client(server->client, failed);
+		fail_rpc_client(server->client_acl, failed);
+	}
+}
+
 /*
  * Load up the server record from information gained in an fsinfo record
  */
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 75da8185ceb8..b713f9f38d6d 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -159,6 +159,7 @@ extern void nfs_clients_init(struct net *net);
 void nfs_cleanup_client_ids(void);
 extern struct nfs_client *nfs_alloc_client(const struct nfs_client_initdata *);
 int nfs_create_rpc_client(struct nfs_client *, const struct nfs_client_initdata *, rpc_authflavor_t);
+void nfs_client_failed(struct nfs_client *client, bool failed);
 struct nfs_client *nfs_get_client(const struct nfs_client_initdata *);
 int nfs_probe_fsinfo(struct nfs_server *server, struct nfs_fh *, struct nfs_fattr *);
 void nfs_server_insert_lists(struct nfs_server *);
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index 0eb7093faea7..6d182da286a2 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -58,6 +58,8 @@ struct nfs_client {
 	int			cl_proto;	/* Network transport protocol */
 	struct nfs_subversion *	cl_nfs_mod;	/* pointer to nfs version module */
 
+	bool			cl_failed;	/* Client has failed */
+
 	u32			cl_minorversion;/* NFSv4 minorversion */
 	struct rpc_cred		*cl_machine_cred;
 
-- 
2.13.6

[RFC v3 7/7] NFS: Control failed clients through debugfs

[Joshua Watt]

The "failed" debugfs entry can be used to get or set the failed status
of a nfs_client.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 fs/nfs/debugfs.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 59 insertions(+)

diff --git a/fs/nfs/debugfs.c b/fs/nfs/debugfs.c
index 0195431427c2..7ccf04847660 100644
--- a/fs/nfs/debugfs.c
+++ b/fs/nfs/debugfs.c
@@ -80,6 +80,56 @@ nfs_server_debugfs_unregister(struct nfs_server *server)
 	server->debugfs = NULL;
 }
 
+static int
+client_failed_show(struct seq_file *f, void *private)
+{
+	struct nfs_client *client = f->private;
+
+	seq_printf(f, "%c", client->cl_failed ? 'Y' : 'N');
+	return 0;
+}
+
+static ssize_t
+client_failed_write(struct file *file, const char __user *user_buf,
+		    size_t count, loff_t *ppos)
+{
+	struct seq_file *seq = file->private_data;
+	struct nfs_client *client = seq->private;
+	char buf[32];
+	size_t buf_size;
+	bool failed;
+	int err;
+
+	buf_size = min(count, sizeof(buf) - 1);
+	if (copy_from_user(buf, user_buf, buf_size))
+		return -EFAULT;
+
+	buf[buf_size] = '\0';
+
+	err = strtobool(buf, &failed);
+	if (err)
+		return err;
+
+	nfs_client_failed(client, failed);
+
+	return count;
+}
+
+static int
+client_failed_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, client_failed_show, inode->i_private);
+}
+
+static const struct file_operations client_failed_fops = {
+	.owner = THIS_MODULE,
+	.open = client_failed_open,
+	.read = seq_read,
+	.write = client_failed_write,
+	.llseek = seq_lseek,
+	.release = single_release
+};
+
 void
 nfs_client_debugfs_register(struct nfs_client *client)
 {
@@ -99,6 +149,15 @@ nfs_client_debugfs_register(struct nfs_client *client)
 
 	link_rpc_client("rpc_client", client->cl_rpcclient,
 			client->cl_debugfs);
+
+	if (!debugfs_create_file("failed", 0600, client->cl_debugfs, client,
+				 &client_failed_fops))
+		goto out_error;
+
+	return;
+out_error:
+	debugfs_remove_recursive(client->cl_debugfs);
+	client->cl_debugfs = NULL;
 }
 
 void
-- 
2.13.6

Re: [RFC v3 0/7] NFS Force Unmounting

[Trond Myklebust]

T24gVHVlLCAyMDE3LTExLTE0IGF0IDE3OjA2IC0wNjAwLCBKb3NodWEgV2F0dCB3cm90ZToNCj4g
QWZ0ZXIgc29tZSBhZGRpdGlvbmFsIHRlc3Rpbmcgb24gdmVyc2lvbiAyLCBJIHJlYWxpemVkIHRo
YXQgMSkgbXkNCj4gdGVzdA0KPiBzZXR1cCB3YXMgYmFkIChEb2ghKSBhbmQgMikgdXNpbmcgcmVt
b3VudCBpc24ndCByZWFsbHkgYW4gaWRlYWwNCj4gbWV0aG9kDQo+IGZvciBzZXR0aW5nIHRoZSAi
c2VydmVyZmFpbGVkIiBtb3VudCBmbGFnLiBUaGUgcmVhc29uIGJlaW5nIHRoYXQgYW55DQo+IHVz
ZXJzcGFjZSByZW1vdW50IGlzIGdvaW5nIHRvIHRvdWNoIHRoZSBmaWxlIHN5c3RlbSBhbmQgcHJv
YmFibHkNCj4gaW52b2tlDQo+IHNvbWUgUlBDcyBmb3IgaW5vZGUgdmFsaWRhdGlvbiBhcyBzdWNo
LiBJZiB0aGUgc2VydmVyIGlzIHRydWx5IGRlYWQsDQo+IHRoZXNlIFJQQ3Mgd2lsbCBoYXZlIHRv
IHRpbWUgb3V0IGJlZm9yZSB0aGUgZmxhZyBjb3VsZCBhY3R1YWxseSBiZQ0KPiBzZXQuDQo+IFRo
aXMgbm90IG9ubHkgdGFrZXMgYSBsb25nIHRpbWUsIGJ1dCB3aWxsIHByb2JhYmx5IGFsc28gZmFp
bCwgY2F1c2luZw0KPiB0aGUgcmVtb3VudCB0byBmYWlsIGFuZCBub3Qgc2V0IHRoZSBmbGFnLCBk
ZWZlYXRpbmcgdGhlIHB1cnBvc2UuDQoNCldoeT8gVGhlcmUgc2hvdWxkIGJlIG5vIHJlYXNvbiB0
byBoYXZlIHRvIHJldmFsaWRhdGUgdGhlIHBhdGggaW4gb3JkZXINCnRvIGNoYW5nZSBtb3VudCBv
cHRpb25zIG9uIGEgbW91bnQgcG9pbnQuDQoNCj4gVGhpcyBwYXRjaCBzZXQgd29ya3MgYXJvdW5k
IHRoaXMgYnkgZXhwb3NpbmcgdGhlIGZhaWxlZCBzZXJ2ZXIgZmxhZw0KPiBpbg0KPiBkZWJ1Z2Zz
LiBJbiBhZGRpdGlvbiwgdGhlIGZsYWcgbm93IGFwcGxpZXMgYXQgdGhlIG5mc19jbGllbnQgbGV2
ZWwNCj4gKGluc3RlYWQgb2YgdGhlIG5mc19zZXJ2ZXIpLiBTZXR0aW5nIHRoZSBmbGFnIHdpbGwg
Y2FuY2VsIGFsbCBSUENzIGluDQo+IHRoZSBjbGllbnQsIGFzIHdlbGwgYXMgYWxsIG5mc19zZXJ2
ZXJzIGF0dGFjaGVkIHRvIGl0LiBUaGlzIGVuc3VyZXMNCj4gdGhhdA0KPiB5b3UgZ2V0IGFsbCB0
aGUgbW91bnRzLCBldmVuIHdoZW4gZGVhbGluZyB3aXRoIHN1Ym1vdW50cyB0aGF0IGNyb3NzDQo+
IHJlbW90ZSBkZXZpY2UgYm91bmRhcmllcy4gVGhpcyBpcyBhbHNvIHdoZXJlIHRoZSBub3NoYXJl
Y2FjaGUgb3B0aW9uDQo+IGFwcGx5aW5nIHRvIG5mc19jbGllbnRzIGJlY29tZXMgdXNlZnVsLi4u
IGl0IHdpbGwgcHJldmVudCBzaGFyaW5nDQo+IGJldHdlZW4gb3RoZXIgZXhwbGljaXQgbW91bnQg
Y2FsbHMsIGJ1dCBzdWJtb3VudHMgd2lsbCBhbGwgc3RpbGwNCj4gc2hhcmUgYQ0KPiBuZnNfY2xp
ZW50IChhbmQgZmFpbCB0b2dldGhlcikuDQo+IA0KPiBPYnZpb3VzbHksIGRlYnVnZnMgaXMgbm90
IGEgcGVybWFuZW50IHNvbHV0aW9uIGZvciB0aGlzLCBzbyBzb21lDQo+IGRpc2N1c3Npb24gbmVl
ZHMgdG8gYmUgaGFkIG9uIGhvdyB0aGlzIHdpbGwgYmUgYWRtaW5pc3RlcmVkICJpbiB0aGUNCj4g
cmVhbA0KPiB3b3JsZCIgKEkgdGhpbmsgInJlbW91bnQiIGlzIG9mZiB0aGUgdGFibGUpLg0KPiAN
Cg0KV2UncmUgbm90IGFkZGluZyBhbnkgQVBJcyB0byBkZWJ1Z2ZzLiBUaGF0J3MgYSBkZWFsIGJy
ZWFrZXIuLi4NCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFpbnRh
aW5lciwgUHJpbWFyeURhdGENCnRyb25kLm15a2xlYnVzdEBwcmltYXJ5ZGF0YS5jb20NCg==

Re: [RFC v3 0/7] NFS Force Unmounting

[Joshua Watt]

On Wed, 2017-11-15 at 13:40 +0000, Trond Myklebust wrote:
> On Tue, 2017-11-14 at 17:06 -0600, Joshua Watt wrote:
> > After some additional testing on version 2, I realized that 1) my
> > test
> > setup was bad (Doh!) and 2) using remount isn't really an ideal
> > method
> > for setting the "serverfailed" mount flag. The reason being that
> > any
> > userspace remount is going to touch the file system and probably
> > invoke
> > some RPCs for inode validation as such. If the server is truly
> > dead,
> > these RPCs will have to time out before the flag could actually be
> > set.
> > This not only takes a long time, but will probably also fail,
> > causing
> > the remount to fail and not set the flag, defeating the purpose.
> 
> Why? There should be no reason to have to revalidate the path in
> order
> to change mount options on a mount point.

I figured this out... the revalidation is not required for the mount(2)
syscall when remounting. However, libmount tries to do a few file
system checks that do required validation. You can convince it to skip
these checks like so:

 mount -c 172.16.9.0:/ /tmp/nfsmount -o remount,serverfailed

The "-c" prevents canonicalizing the mount path (meaning you must
provide the canonical path on the command line), and you have to
specify the "block device" (e.g. "172.16.9.0:/") to prevent it doing
some fstatat() checks on the mount path while parsing through mtab.

It's annoying you can't do the easier:

 mount /tmp/nfsmount -o remount,serverfailed

but that's not the kernels fault.

> 
> > This patch set works around this by exposing the failed server flag
> > in
> > debugfs. In addition, the flag now applies at the nfs_client level
> > (instead of the nfs_server). Setting the flag will cancel all RPCs
> > in
> > the client, as well as all nfs_servers attached to it. This ensures
> > that
> > you get all the mounts, even when dealing with submounts that cross
> > remote device boundaries. This is also where the nosharecache
> > option
> > applying to nfs_clients becomes useful... it will prevent sharing
> > between other explicit mount calls, but submounts will all still
> > share a
> > nfs_client (and fail together).
> > 
> > Obviously, debugfs is not a permanent solution for this, so some
> > discussion needs to be had on how this will be administered "in the
> > real
> > world" (I think "remount" is off the table).
> > 
> 
> We're not adding any APIs to debugfs. That's a deal breaker...

Sure. Is the mount option still the best way forward then, or are there
other options to be explored?

> 
> -- 
> Trond Myklebust
> Linux NFS client maintainer, PrimaryData
> trond.myklebust@primarydata.com