[RFC 0/51] nfs41 server patches for review

[RFC 0/51] nfs41 server patches for review

[Benny Halevy]

Bruce,

Here's the patch series for nfs41 server side, rebased
over 2.6.28 for review.

I've cleaned up the patch as much as I could in the given
time frame but surely there's more work to be done.

I encourage others to comment on the patches as well.

Note that the following patch in the client sie patchset:
	[RFC 06/85] nfs41: common protocol definitions
contains common code and is needed for the server side
to compile and run.

Thanks,

Benny

pre-nfs41 misc. patches:
[RFC 01/51] nfsd: add etoosmall to nfserrno
[RFC 02/51] nfsd: dprint each op status in nfsd4_proc_compound
[RFC 03/51] nfsd: git rid of nfs4_cb_null_ops declaration
[RFC 04/51] nfsd: fix file comment in fs/nfsd/nfs4xdr.c

basic sessions support:
[RFC 05/51] nfsd41: Add Kconfig symbols for NFSv4.1
[RFC 06/51] nfsd41: define nfs41 error codes
[RFC 07/51] nfsd41: sessions basic data types
[RFC 08/51] nfsd41: introduce nfs4_client cl_sessions list
[RFC 09/51] nfsd41: destroy_session when client is expired
[RFC 10/51] nfsd41: sessionid hashing
[RFC 11/51] FIXME: nfsd41: reduce server lease time for nfs41

minorversion 1 support:
[RFC 12/51] nfsd41: provide support for minor version 1 at rpc level
[RFC 13/51] FIXME: nfsd41: introduce current_session
[RFC 14/51] nfsd41: introduce nfs41_{get,set}_slot_state
[RFC 15/51] FIXME: nfsd41: free up slot unless operation is dropped
[RFC 16/51] nfsd41: stateid handling
[RFC 17/51] nfsd41: clientid handling
[RFC 18/51] nfsd41: access_valid
[RFC 19/51] nfsd41: add OPEN4_SHARE_ACCESS_WANT nfs4_stateid bmap
[RFC 20/51] nfsd: last_byte_offset
[Hmm, this one actually belongs in the pre-nfs41 section]

nfs41 ops:
[RFC 21/51] nfsd41: xdr stubs
[RFC 22/51] nfsd41: proc stubs
[RFC 23/51] nfsd41: exchange_id operation
[RFC 24/51] nfsd41: print exchange flags when purging client
[RFC 25/51] nfsd41: create_session operation
[RFC 26/51] nfsd41: destroy_session operation
[RFC 27/51] nfsd41: sequence operation

backchannel support:
[RFC 28/51] FIXME: nfsd41: sunrpc: Added rpc server-side backchannel handling
[RFC 29/51] nfsd: BUG_ON_UNLOCKED_STATE
[RFC 30/51] nfsd: lock state around nfs4_put_delegation in nfsd_break_deleg_cb err path
[RFC 31/51] FIXME: nfsd: kref_get cb_client while doing the callback

callback support:
[RFC 32/51] nfsd41: callback infrastructure
[RFC 33/51] nfsd41: introduce cl_cb_mutex
[RFC 34/51] nfsd41: cb_sequence callback
[RFC 35/51] nfsd41: introduce nfs4_cb_call_sync for nfs4 and nfs41
[RFC 36/51] nfsd41: cb_recall callback

CREATE_EXCLUSIVE4_1:
[RFC 37/51] nfsd41: pass writable attrs mask to nfsd4_decode_fattr
[RFC 38/51] nfsd41: support for 3-word long attribute bitmask
[RFC 39/51] nfsd41: SUPPATTR_EXCLCREAT attribute
[RFC 40/51] nfsd41: CREATE_EXCLUSIVE4_1

replay cache:
[RFC 41/51] sunrpc: Add deferral save and restore state callback
[RFC 42/51] nfsd: save and restore defer result pages
[RFC 43/51] nfsd: deferral processing
[RFC 44/51] nfsd41: slab cache for current session
[RFC 45/51] nfsd41: DRC save, restore, and clear functions
[RFC 46/51] nfsd41: nfsd nfsd4_sequence DRC logic
[RFC 47/51] nfsd41: enforce NFS4ERR_SEQUENCE_POS operation order rules
[RFC 48/51] nfsd41: nfsd DRC logic
[RFC 49/51] nfsd41: clear DRC cache on free_session
[RFC 50/51] nfsd41: Add a create session replay cache
[RFC 51/51] nfsd41: print DRC statistics

[RFC 01/51] nfsd: add etoosmall to nfserrno

[Benny Halevy]

From: Dean Hildebrand <dhildeb@us.ibm.com>

Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfsproc.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfsproc.c b/fs/nfsd/nfsproc.c
index 5cffeca..6f7f263 100644
--- a/fs/nfsd/nfsproc.c
+++ b/fs/nfsd/nfsproc.c
@@ -622,6 +622,7 @@ nfserrno (int errno)
 		{ nfserr_badname, -ESRCH },
 		{ nfserr_io, -ETXTBSY },
 		{ nfserr_notsupp, -EOPNOTSUPP },
+		{ nfserr_toosmall, -ETOOSMALL },
 	};
 	int	i;
 
-- 
1.6.0.2

Re: [pnfs] [RFC 0/51] nfs41 server patches for review

[Benny Halevy]

The patchset is also available on
git://linux-nfs.org/~bhalevy/linux-pnfs.git nfs41-2.6.28-review

Cumulated diff for the server side changes:

http://git.linux-nfs.org/?p=bhalevy/linux-pnfs.git;a=commitdiff_plain;h=nfs41-2.6.28-rc4-server-review-2008-11-10;hp=nfs41-2.6.28-rc4-client-review-2008-11-10

Benny

On Nov. 10, 2008, 22:12 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> Bruce,
> 
> Here's the patch series for nfs41 server side, rebased
> over 2.6.28 for review.
> 
> I've cleaned up the patch as much as I could in the given
> time frame but surely there's more work to be done.
> 
> I encourage others to comment on the patches as well.
> 
> Note that the following patch in the client sie patchset:
> 	[RFC 06/85] nfs41: common protocol definitions
> contains common code and is needed for the server side
> to compile and run.
> 
> Thanks,
> 
> Benny
> 
> pre-nfs41 misc. patches:
> [RFC 01/51] nfsd: add etoosmall to nfserrno
> [RFC 02/51] nfsd: dprint each op status in nfsd4_proc_compound
> [RFC 03/51] nfsd: git rid of nfs4_cb_null_ops declaration
> [RFC 04/51] nfsd: fix file comment in fs/nfsd/nfs4xdr.c
> 
> basic sessions support:
> [RFC 05/51] nfsd41: Add Kconfig symbols for NFSv4.1
> [RFC 06/51] nfsd41: define nfs41 error codes
> [RFC 07/51] nfsd41: sessions basic data types
> [RFC 08/51] nfsd41: introduce nfs4_client cl_sessions list
> [RFC 09/51] nfsd41: destroy_session when client is expired
> [RFC 10/51] nfsd41: sessionid hashing
> [RFC 11/51] FIXME: nfsd41: reduce server lease time for nfs41
> 
> minorversion 1 support:
> [RFC 12/51] nfsd41: provide support for minor version 1 at rpc level
> [RFC 13/51] FIXME: nfsd41: introduce current_session
> [RFC 14/51] nfsd41: introduce nfs41_{get,set}_slot_state
> [RFC 15/51] FIXME: nfsd41: free up slot unless operation is dropped
> [RFC 16/51] nfsd41: stateid handling
> [RFC 17/51] nfsd41: clientid handling
> [RFC 18/51] nfsd41: access_valid
> [RFC 19/51] nfsd41: add OPEN4_SHARE_ACCESS_WANT nfs4_stateid bmap
> [RFC 20/51] nfsd: last_byte_offset
> [Hmm, this one actually belongs in the pre-nfs41 section]
> 
> nfs41 ops:
> [RFC 21/51] nfsd41: xdr stubs
> [RFC 22/51] nfsd41: proc stubs
> [RFC 23/51] nfsd41: exchange_id operation
> [RFC 24/51] nfsd41: print exchange flags when purging client
> [RFC 25/51] nfsd41: create_session operation
> [RFC 26/51] nfsd41: destroy_session operation
> [RFC 27/51] nfsd41: sequence operation
> 
> backchannel support:
> [RFC 28/51] FIXME: nfsd41: sunrpc: Added rpc server-side backchannel handling
> [RFC 29/51] nfsd: BUG_ON_UNLOCKED_STATE
> [RFC 30/51] nfsd: lock state around nfs4_put_delegation in nfsd_break_deleg_cb err path
> [RFC 31/51] FIXME: nfsd: kref_get cb_client while doing the callback
> 
> callback support:
> [RFC 32/51] nfsd41: callback infrastructure
> [RFC 33/51] nfsd41: introduce cl_cb_mutex
> [RFC 34/51] nfsd41: cb_sequence callback
> [RFC 35/51] nfsd41: introduce nfs4_cb_call_sync for nfs4 and nfs41
> [RFC 36/51] nfsd41: cb_recall callback
> 
> CREATE_EXCLUSIVE4_1:
> [RFC 37/51] nfsd41: pass writable attrs mask to nfsd4_decode_fattr
> [RFC 38/51] nfsd41: support for 3-word long attribute bitmask
> [RFC 39/51] nfsd41: SUPPATTR_EXCLCREAT attribute
> [RFC 40/51] nfsd41: CREATE_EXCLUSIVE4_1
> 
> replay cache:
> [RFC 41/51] sunrpc: Add deferral save and restore state callback
> [RFC 42/51] nfsd: save and restore defer result pages
> [RFC 43/51] nfsd: deferral processing
> [RFC 44/51] nfsd41: slab cache for current session
> [RFC 45/51] nfsd41: DRC save, restore, and clear functions
> [RFC 46/51] nfsd41: nfsd nfsd4_sequence DRC logic
> [RFC 47/51] nfsd41: enforce NFS4ERR_SEQUENCE_POS operation order rules
> [RFC 48/51] nfsd41: nfsd DRC logic
> [RFC 49/51] nfsd41: clear DRC cache on free_session
> [RFC 50/51] nfsd41: Add a create session replay cache
> [RFC 51/51] nfsd41: print DRC statistics
> _______________________________________________
> pNFS mailing list
> pNFS@linux-nfs.org
> http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs

[RFC 01/51] nfsd: add etoosmall to nfserrno

[Benny Halevy]

From: Dean Hildebrand <dhildeb@us.ibm.com>

Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfsproc.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfsproc.c b/fs/nfsd/nfsproc.c
index 5cffeca..6f7f263 100644
--- a/fs/nfsd/nfsproc.c
+++ b/fs/nfsd/nfsproc.c
@@ -622,6 +622,7 @@ nfserrno (int errno)
 		{ nfserr_badname, -ESRCH },
 		{ nfserr_io, -ETXTBSY },
 		{ nfserr_notsupp, -EOPNOTSUPP },
+		{ nfserr_toosmall, -ETOOSMALL },
 	};
 	int	i;
 
-- 
1.6.0.2

[RFC 02/51] nfsd: dprint each op status in nfsd4_proc_compound

[Benny Halevy]

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 669461e..9fa60a3 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -946,6 +946,11 @@ encode_op:
 			nfsd4_encode_operation(resp, op);
 			status = op->status;
 		}
+
+		dprintk("nfsv4 compound op %p opcnt %d #%d: %d: status %d\n",
+			args->ops, args->opcnt, resp->opcnt, op->opnum,
+			be32_to_cpu(status));
+
 		if (cstate->replay_owner) {
 			nfs4_put_stateowner(cstate->replay_owner);
 			cstate->replay_owner = NULL;
-- 
1.6.0.2

Re: [pnfs] [RFC 02/51] nfsd: dprint each op status in nfsd4_proc_compound

[Benny Halevy]

On Nov. 10, 2008, 22:41 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 2008-11-12:
keep both dprintk's or just the bottom one (with the status)
in general: explain the kind of problem we need this for.

> ---
>  fs/nfsd/nfs4proc.c |    5 +++++
>  1 files changed, 5 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index 669461e..9fa60a3 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -946,6 +946,11 @@ encode_op:
>  			nfsd4_encode_operation(resp, op);
>  			status = op->status;
>  		}
> +
> +		dprintk("nfsv4 compound op %p opcnt %d #%d: %d: status %d\n",
> +			args->ops, args->opcnt, resp->opcnt, op->opnum,
> +			be32_to_cpu(status));
> +
>  		if (cstate->replay_owner) {
>  			nfs4_put_stateowner(cstate->replay_owner);
>  			cstate->replay_owner = NULL;

[RFC 03/51] nfsd: git rid of nfs4_cb_null_ops declaration

[Benny Halevy]

There's no use for nfs4_cb_null_ops's declaration in fs/nfsd/nfs4callback.c

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4callback.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)

diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
index 094747a..e198ead 100644
--- a/fs/nfsd/nfs4callback.c
+++ b/fs/nfsd/nfs4callback.c
@@ -53,9 +53,6 @@
 #define NFSPROC4_CB_NULL 0
 #define NFSPROC4_CB_COMPOUND 1
 
-/* declarations */
-static const struct rpc_call_ops nfs4_cb_null_ops;
-
 /* Index of predefined Linux callback client operations */
 
 enum {
-- 
1.6.0.2

[RFC 04/51] nfsd: fix file comment in fs/nfsd/nfs4xdr.c

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4xdr.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index afcdf4b..f7dec05 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -1,5 +1,5 @@
 /*
- *  fs/nfs/nfs4xdr.c
+ *  fs/nfsd/nfs4xdr.c
  *
  *  Server-side XDR for NFSv4
  *
-- 
1.6.0.2

Re: [pnfs] [RFC 04/51] nfsd: fix file comment in fs/nfsd/nfs4xdr.c

[Benny Halevy]

On Nov. 10, 2008, 22:41 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 2008-11-12: delete comment instead.
also, for new files the policy today is not to have these
comments at all.

> ---
>  fs/nfsd/nfs4xdr.c |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index afcdf4b..f7dec05 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -1,5 +1,5 @@
>  /*
> - *  fs/nfs/nfs4xdr.c
> + *  fs/nfsd/nfs4xdr.c
>   *
>   *  Server-side XDR for NFSv4
>   *

[RFC 05/51] nfsd41: Add Kconfig symbols for NFSv4.1

[Benny Halevy]

From: Ricardo Labiaga <ricardo.labiaga@netapp.com>

Added CONFIG_NFSD_V4_1 and made it depend upon NFSD_V4 and EXPERIMENTAL

Signed-off-by: Ricardo Labiaga <ricardo.labiaga@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/Kconfig |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/fs/Kconfig b/fs/Kconfig
index 9363634..aee944e 100644
--- a/fs/Kconfig
+++ b/fs/Kconfig
@@ -1317,6 +1317,15 @@ config NFSD_V4
 
 	  If unsure, say N.
 
+config NFSD_V4_1
+	bool "NFS server support for NFSv4.1 (EXPERIMENTAL)"
+	depends on NFSD_V4 && EXPERIMENTAL
+	help
+	  This option enables support for minor version 1 of the NFSv4 protocol
+	  (draft-ietf-nfsv4-minorversion1) in the kernel's NFS server.
+
+	  If unsure, say N.
+
 config LOCKD
 	tristate
 
-- 
1.6.0.2

[RFC 06/51] nfsd41: define nfs41 error codes

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 include/linux/nfsd/nfsd.h |   16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/include/linux/nfsd/nfsd.h b/include/linux/nfsd/nfsd.h
index 2126940..9703540 100644
--- a/include/linux/nfsd/nfsd.h
+++ b/include/linux/nfsd/nfsd.h
@@ -251,6 +251,22 @@ void		nfsd_lockd_shutdown(void);
 #define	nfserr_locked		__constant_htonl(NFSERR_LOCKED)
 #define	nfserr_wrongsec		__constant_htonl(NFSERR_WRONGSEC)
 #define	nfserr_replay_me	__constant_htonl(NFSERR_REPLAY_ME)
+#define	nfserr_badiomode	__constant_htonl(NFSERR_BADIOMODE)
+#define	nfserr_badlayout	__constant_htonl(NFSERR_BADLAYOUT)
+#define nfserr_bad_session_digest	__constant_htonl(NFSERR_BAD_SESSION_DIGEST)
+#define nfserr_badsession	__constant_htonl(NFSERR_BADSESSION)
+#define nfserr_badslot	__constant_htonl(NFSERR_BADSLOT)
+#define nfserr_complete_already	__constant_htonl(NFSERR_COMPLETE_ALREADY)
+#define nfserr_conn_not_bound_to_session	__constant_htonl(NFSERR_CONN_NOT_BOUND_TO_SESSION)
+#define nfserr_deleg_already_wanted	__constant_htonl(NFSERR_DELEG_ALREADY_WANTED)
+#define nfserr_dirdeleg_unavail	__constant_htonl(NFSERR_DIRDELEG_UNAVAIL)
+#define	nfserr_layouttrylater		__constant_htonl(NFSERR_LAYOUTTRYLATER)
+#define	nfserr_layoutunavailable	__constant_htonl(NFSERR_LAYOUTUNAVAILABLE)
+#define	nfserr_nomatching_layout	__constant_htonl(NFSERR_NOMATCHING_LAYOUT)
+#define nfserr_recallconflict	__constant_htonl(NFSERR_RECALLCONFLICT)
+#define	nfserr_unknown_layouttype	__constant_htonl(NFSERR_UNKNOWN_LAYOUTTYPE)
+#define	nfserr_seq_misordered		__constant_htonl(NFSERR_SEQ_MISORDERED)
+#define nfserr_sequence_pos	__constant_htonl(NFSERR_SEQUENCE_POS)
 
 /* error codes for internal use */
 /* if a request fails due to kmalloc failure, it gets dropped.
-- 
1.6.0.2

[RFC 07/51] nfsd41: sessions basic data types

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c        |   20 ++++++++++++
 include/linux/nfsd/state.h |   70 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index b91a2f1..8c1b5d9 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -310,6 +310,26 @@ static struct list_head	unconf_id_hashtbl[CLIENT_HASH_SIZE];
 static struct list_head client_lru;
 static struct list_head close_lru;
 
+#if defined(CONFIG_NFSD_V4_1)
+static void
+destroy_session(struct nfs41_session *ses)
+{
+	list_del(&ses->se_hash);
+	list_del(&ses->se_perclnt);
+	nfs41_put_session(ses);
+}
+
+void
+free_session(struct kref *kref)
+{
+	struct nfs41_session *ses;
+
+	ses = container_of(kref, struct nfs41_session, se_ref);
+	kfree(ses->se_slots);
+	kfree(ses);
+}
+#endif /* CONFIG_NFSD_V4_1 */
+
 static inline void
 renew_client(struct nfs4_client *clp)
 {
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index 8f492e3..2f28ad4 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -100,6 +100,76 @@ struct nfs4_callback {
 	struct rpc_clnt *       cb_client;
 };
 
+#if defined(CONFIG_NFSD_V4_1)
+/*
+ * nfs41_channel
+ *
+ * for both forward and back channels
+ */
+struct nfs41_channel {
+	u32	ch_headerpad_sz;
+	u32	ch_maxreq_sz;
+	u32	ch_maxresp_sz;
+	u32	ch_maxresp_cached;
+	u32	ch_maxops;
+	u32	ch_maxreqs;	/* number of slots */
+};
+
+/* Maximum number of slots per session - XXX arbitrary */
+#define NFS41_MAX_SLOTS 64
+
+/* slot states */
+enum {
+	NFS4_SLOT_AVAILABLE,
+	NFS4_SLOT_INPROGRESS
+};
+
+/*
+ * nfs41_slot
+ *
+ * for now, just slot sequence number - will hold DRC for this slot.
+ */
+struct nfs41_slot {
+	atomic_t		sl_state;
+	struct nfs41_session	*sl_session;
+	u32			sl_seqid;
+};
+
+/*
+ * nfs41_session
+ */
+struct nfs41_session {
+	struct kref		se_ref;
+	struct list_head	se_hash;	/* hash by sessionid_t */
+	struct list_head	se_perclnt;
+	u32			se_flags;
+	struct nfs4_client	*se_client;	/* for expire_client */
+	nfs41_sessionid		se_sessionid;
+	struct nfs41_channel	se_forward;
+	struct nfs41_slot	*se_slots;	/* forward channel slots */
+};
+
+#define se_fheaderpad_sz	se_forward.ch_headerpad_sz
+#define se_fmaxreq_sz		se_forward.ch_maxreq_sz
+#define se_fmaxresp_sz		se_forward.ch_maxresp_sz
+#define se_fmaxresp_cached	se_forward.ch_maxresp_cached
+#define se_fmaxops		se_forward.ch_maxops
+#define se_fnumslots		se_forward.ch_maxreqs
+
+static inline void
+nfs41_put_session(struct nfs41_session *ses)
+{
+	extern void free_session(struct kref *kref);
+	kref_put(&ses->se_ref, free_session);
+}
+
+static inline void
+nfs41_get_session(struct nfs41_session *ses)
+{
+	kref_get(&ses->se_ref);
+}
+#endif /* CONFIG_NFSD_V4_1 */
+
 #define HEXDIR_LEN     33 /* hex version of 16 byte md5 of cl_name plus '\0' */
 
 /*
-- 
1.6.0.2

Re: [pnfs] [RFC 07/51] nfsd41: sessions basic data types

[Benny Halevy]

On Nov. 10, 2008, 22:43 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4state.c        |   20 ++++++++++++
>  include/linux/nfsd/state.h |   70 ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 90 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index b91a2f1..8c1b5d9 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -310,6 +310,26 @@ static struct list_head	unconf_id_hashtbl[CLIENT_HASH_SIZE];
>  static struct list_head client_lru;
>  static struct list_head close_lru;
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +static void
> +destroy_session(struct nfs41_session *ses)
> +{
> +	list_del(&ses->se_hash);
> +	list_del(&ses->se_perclnt);
> +	nfs41_put_session(ses);
> +}

rename to release_session.

> +
> +void
> +free_session(struct kref *kref)
> +{
> +	struct nfs41_session *ses;
> +
> +	ses = container_of(kref, struct nfs41_session, se_ref);
> +	kfree(ses->se_slots);
> +	kfree(ses);
> +}
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  static inline void
>  renew_client(struct nfs4_client *clp)
>  {
> diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
> index 8f492e3..2f28ad4 100644
> --- a/include/linux/nfsd/state.h
> +++ b/include/linux/nfsd/state.h
> @@ -100,6 +100,76 @@ struct nfs4_callback {
>  	struct rpc_clnt *       cb_client;
>  };
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +/*
> + * nfs41_channel
> + *
> + * for both forward and back channels

review 11-12: 
channel attributes (either forward or backchannel)

> + */
> +struct nfs41_channel {

review 11-12: naming?
nfsd4_* for server names and
nfs4_ for client names

> +	u32	ch_headerpad_sz;

not used, always return zero for now.

> +	u32	ch_maxreq_sz;

we don't have to save that, just reply our max
based on svc_max_payload. ok to keep it.

> +	u32	ch_maxresp_sz;
> +	u32	ch_maxresp_cached;

we'll need at least one per-session max. variable
maybe a global max.

> +	u32	ch_maxops;

todo: nfsd_compoundargs has a max size for the preallocated ops
array. need to define that in a header file
use for both nfs4 and nfs41. (preallocated per nfsd thread)

> +	u32	ch_maxreqs;	/* number of slots */
> +};
> +
> +/* Maximum number of slots per session - XXX arbitrary */
> +#define NFS41_MAX_SLOTS 64
> +
> +/* slot states */
> +enum {
> +	NFS4_SLOT_AVAILABLE,
> +	NFS4_SLOT_INPROGRESS
> +};
> +
> +/*
> + * nfs41_slot
> + *
> + * for now, just slot sequence number - will hold DRC for this slot.
> + */
> +struct nfs41_slot {
> +	atomic_t		sl_state;

11-12: use a bitmap for the AVAIL/INPROG state.

> +	struct nfs41_session	*sl_session;

see if we can keep the pointer in the cur_sess struct

> +	u32			sl_seqid;
> +};
> +
> +/*
> + * nfs41_session
> + */
> +struct nfs41_session {
> +	struct kref		se_ref;
> +	struct list_head	se_hash;	/* hash by sessionid_t */

s/sessionid_t/sessionid/

> +	struct list_head	se_perclnt;
> +	u32			se_flags;

unused at the moment.

> +	struct nfs4_client	*se_client;	/* for expire_client */
> +	nfs41_sessionid		se_sessionid;
> +	struct nfs41_channel	se_forward;

look into whether we should embed the channel fields here
if what we use for the fore channel is different enough
from what we use for the back channel.

> +	struct nfs41_slot	*se_slots;	/* forward channel slots */
> +};
> +
> +#define se_fheaderpad_sz	se_forward.ch_headerpad_sz
> +#define se_fmaxreq_sz		se_forward.ch_maxreq_sz
> +#define se_fmaxresp_sz		se_forward.ch_maxresp_sz
> +#define se_fmaxresp_cached	se_forward.ch_maxresp_cached
> +#define se_fmaxops		se_forward.ch_maxops
> +#define se_fnumslots		se_forward.ch_maxreqs
> +
> +static inline void
> +nfs41_put_session(struct nfs41_session *ses)
> +{
> +	extern void free_session(struct kref *kref);
> +	kref_put(&ses->se_ref, free_session);
> +}
> +
> +static inline void
> +nfs41_get_session(struct nfs41_session *ses)
> +{
> +	kref_get(&ses->se_ref);
> +}
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  #define HEXDIR_LEN     33 /* hex version of 16 byte md5 of cl_name plus '\0' */
>  
>  /*

[RFC 08/51] nfsd41: introduce nfs4_client cl_sessions list

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c        |    3 +++
 include/linux/nfsd/state.h |    3 +++
 2 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 8c1b5d9..7406826 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -457,6 +457,9 @@ static struct nfs4_client *create_client(struct xdr_netobj name, char *recdir)
 	INIT_LIST_HEAD(&clp->cl_strhash);
 	INIT_LIST_HEAD(&clp->cl_openowners);
 	INIT_LIST_HEAD(&clp->cl_delegations);
+#if defined(CONFIG_NFSD_V4_1)
+	INIT_LIST_HEAD(&clp->cl_sessions);
+#endif /* CONFIG_NFSD_V4_1 */
 	INIT_LIST_HEAD(&clp->cl_lru);
 	return clp;
 }
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index 2f28ad4..28c42ea 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -187,6 +187,9 @@ struct nfs4_client {
 	struct list_head	cl_strhash; 	/* hash by cl_name */
 	struct list_head	cl_openowners;
 	struct list_head	cl_delegations;
+#if defined(CONFIG_NFSD_V4_1)
+	struct list_head	cl_sessions;
+#endif /* CONFIG_NFSD_V4_1 */
 	struct list_head        cl_lru;         /* tail queue */
 	struct xdr_netobj	cl_name; 	/* id generated by client */
 	char                    cl_recdir[HEXDIR_LEN]; /* recovery dir */
-- 
1.6.0.2

Re: [pnfs] [RFC 08/51] nfsd41: introduce nfs4_client cl_sessions list

[Benny Halevy]

On Nov. 10, 2008, 22:43 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4state.c        |    3 +++
>  include/linux/nfsd/state.h |    3 +++
>  2 files changed, 6 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 8c1b5d9..7406826 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -457,6 +457,9 @@ static struct nfs4_client *create_client(struct xdr_netobj name, char *recdir)
>  	INIT_LIST_HEAD(&clp->cl_strhash);
>  	INIT_LIST_HEAD(&clp->cl_openowners);
>  	INIT_LIST_HEAD(&clp->cl_delegations);
> +#if defined(CONFIG_NFSD_V4_1)
> +	INIT_LIST_HEAD(&clp->cl_sessions);
> +#endif /* CONFIG_NFSD_V4_1 */
>  	INIT_LIST_HEAD(&clp->cl_lru);
>  	return clp;
>  }
> diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
> index 2f28ad4..28c42ea 100644
> --- a/include/linux/nfsd/state.h
> +++ b/include/linux/nfsd/state.h
> @@ -187,6 +187,9 @@ struct nfs4_client {
>  	struct list_head	cl_strhash; 	/* hash by cl_name */
>  	struct list_head	cl_openowners;
>  	struct list_head	cl_delegations;
> +#if defined(CONFIG_NFSD_V4_1)
> +	struct list_head	cl_sessions;
> +#endif /* CONFIG_NFSD_V4_1 */

review 11-12: move this ifdef down and unify with
cl_exchange_flags et-al.

>  	struct list_head        cl_lru;         /* tail queue */
>  	struct xdr_netobj	cl_name; 	/* id generated by client */
>  	char                    cl_recdir[HEXDIR_LEN]; /* recovery dir */

[RFC 09/51] nfsd41: destroy_session when client is expired

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 7406826..bf4d2c2 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -414,6 +414,9 @@ expire_client(struct nfs4_client *clp)
 	struct nfs4_stateowner *sop;
 	struct nfs4_delegation *dp;
 	struct list_head reaplist;
+#if defined(CONFIG_NFSD_V4_1)
+	struct nfs41_session  *ses;
+#endif /* CONFIG_NFSD_V4_1 */
 
 	dprintk("NFSD: expire_client cl_count %d\n",
 	                    atomic_read(&clp->cl_count));
@@ -440,6 +443,13 @@ expire_client(struct nfs4_client *clp)
 		sop = list_entry(clp->cl_openowners.next, struct nfs4_stateowner, so_perclient);
 		release_stateowner(sop);
 	}
+#if defined(CONFIG_NFSD_V4_1)
+	while (!list_empty(&clp->cl_sessions)) {
+		ses = list_entry(clp->cl_sessions.next, struct nfs41_session,
+				 se_perclnt);
+		destroy_session(ses);
+	}
+#endif /* CONFIG_NFSD_V4_1 */
 	put_nfs4_client(clp);
 }
 
-- 
1.6.0.2

[RFC 10/51] nfsd41: sessionid hashing

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

Use crc32 of sessionid to look it up on the server.
Select CRC32 when NFSD_V4_1 is selected in fs/Kconfig.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/Kconfig          |    1 +
 fs/nfsd/nfs4state.c |   51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+), 0 deletions(-)

diff --git a/fs/Kconfig b/fs/Kconfig
index aee944e..cb0e3de 100644
--- a/fs/Kconfig
+++ b/fs/Kconfig
@@ -1320,6 +1320,7 @@ config NFSD_V4
 config NFSD_V4_1
 	bool "NFS server support for NFSv4.1 (EXPERIMENTAL)"
 	depends on NFSD_V4 && EXPERIMENTAL
+	select CRC32
 	help
 	  This option enables support for minor version 1 of the NFSv4 protocol
 	  (draft-ietf-nfsv4-minorversion1) in the kernel's NFS server.
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index bf4d2c2..0e1bc72 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -52,6 +52,7 @@
 #include <linux/namei.h>
 #include <linux/swap.h>
 #include <linux/mutex.h>
+#include <linux/crc32.h>
 #include <linux/lockd/bind.h>
 #include <linux/module.h>
 
@@ -311,6 +312,52 @@ static struct list_head client_lru;
 static struct list_head close_lru;
 
 #if defined(CONFIG_NFSD_V4_1)
+/* Use a prime for hash table size */
+#define SESSION_HASH_SIZE	1031
+static struct list_head sessionid_hashtbl[SESSION_HASH_SIZE];
+
+static int
+hash_sessionid(nfs41_sessionid *sessionid)
+{
+	u32 csum = 0;
+	int idx;
+
+	csum = crc32(0, sessionid, sizeof(*sessionid));
+	idx = csum % SESSION_HASH_SIZE;
+	dprintk("%s IDX: %u csum %u\n", __func__, idx, csum);
+	return idx;
+}
+
+static inline void
+dump_sessionid(const char *fn, nfs41_sessionid *sessionid)
+{
+	u32 *ptr = (u32 *)(*sessionid);
+	dprintk("%s: %u:%u:%u:%u\n", fn, ptr[0], ptr[1], ptr[2], ptr[3]);
+}
+
+struct nfs41_session *
+find_in_sessionid_hashtbl(nfs41_sessionid *sessionid)
+{
+	struct nfs41_session *elem;
+	int idx;
+
+	dump_sessionid(__func__, sessionid);
+	idx = hash_sessionid(sessionid);
+	dprintk("%s: idx is %d\n", __func__, idx);
+	/* Search in the appropriate list */
+	list_for_each_entry(elem, &sessionid_hashtbl[idx], se_hash) {
+		dump_sessionid("list traversal", &elem->se_sessionid);
+		if (!memcmp(elem->se_sessionid, sessionid,
+			    sizeof(nfs41_sessionid))) {
+			dprintk("%s: found session %p\n", __func__, elem);
+			return elem;
+		}
+	}
+
+	dprintk("%s: session not found\n", __func__);
+	return NULL;
+}
+
 static void
 destroy_session(struct nfs41_session *ses)
 {
@@ -3154,6 +3201,10 @@ nfs4_state_init(void)
 		INIT_LIST_HEAD(&unconf_str_hashtbl[i]);
 		INIT_LIST_HEAD(&unconf_id_hashtbl[i]);
 	}
+#if defined(CONFIG_NFSD_V4_1)
+	for (i = 0; i < SESSION_HASH_SIZE; i++)
+		INIT_LIST_HEAD(&sessionid_hashtbl[i]);
+#endif /* CONFIG_NFSD_V4_1 */
 	for (i = 0; i < FILE_HASH_SIZE; i++) {
 		INIT_LIST_HEAD(&file_hashtbl[i]);
 	}
-- 
1.6.0.2

Re: [pnfs] [RFC 10/51] nfsd41: sessionid hashing

[Benny Halevy]

On Nov. 10, 2008, 22:44 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> Use crc32 of sessionid to look it up on the server.
> Select CRC32 when NFSD_V4_1 is selected in fs/Kconfig.
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/Kconfig          |    1 +
>  fs/nfsd/nfs4state.c |   51 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 52 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/Kconfig b/fs/Kconfig
> index aee944e..cb0e3de 100644
> --- a/fs/Kconfig
> +++ b/fs/Kconfig
> @@ -1320,6 +1320,7 @@ config NFSD_V4
>  config NFSD_V4_1
>  	bool "NFS server support for NFSv4.1 (EXPERIMENTAL)"
>  	depends on NFSD_V4 && EXPERIMENTAL
> +	select CRC32
>  	help
>  	  This option enables support for minor version 1 of the NFSv4 protocol
>  	  (draft-ietf-nfsv4-minorversion1) in the kernel's NFS server.
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index bf4d2c2..0e1bc72 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -52,6 +52,7 @@
>  #include <linux/namei.h>
>  #include <linux/swap.h>
>  #include <linux/mutex.h>
> +#include <linux/crc32.h>
>  #include <linux/lockd/bind.h>
>  #include <linux/module.h>
>  
> @@ -311,6 +312,52 @@ static struct list_head client_lru;
>  static struct list_head close_lru;
>  
>  #if defined(CONFIG_NFSD_V4_1)
> +/* Use a prime for hash table size */
> +#define SESSION_HASH_SIZE	1031

review 11-12: reduce to 512

> +static struct list_head sessionid_hashtbl[SESSION_HASH_SIZE];
> +
> +static int
> +hash_sessionid(nfs41_sessionid *sessionid)
> +{
> +	u32 csum = 0;
> +	int idx;
> +
> +	csum = crc32(0, sessionid, sizeof(*sessionid));

why use crc32?
re-use one of the available in linux/hash.h

> +	idx = csum % SESSION_HASH_SIZE;
> +	dprintk("%s IDX: %u csum %u\n", __func__, idx, csum);

ditch that

> +	return idx;
> +}
> +
> +static inline void
> +dump_sessionid(const char *fn, nfs41_sessionid *sessionid)
> +{
> +	u32 *ptr = (u32 *)(*sessionid);
> +	dprintk("%s: %u:%u:%u:%u\n", fn, ptr[0], ptr[1], ptr[2], ptr[3]);
> +}
> +
> +struct nfs41_session *
> +find_in_sessionid_hashtbl(nfs41_sessionid *sessionid)
> +{
> +	struct nfs41_session *elem;
> +	int idx;
> +
> +	dump_sessionid(__func__, sessionid);
> +	idx = hash_sessionid(sessionid);
> +	dprintk("%s: idx is %d\n", __func__, idx);
> +	/* Search in the appropriate list */
> +	list_for_each_entry(elem, &sessionid_hashtbl[idx], se_hash) {
> +		dump_sessionid("list traversal", &elem->se_sessionid);
> +		if (!memcmp(elem->se_sessionid, sessionid,
> +			    sizeof(nfs41_sessionid))) {
> +			dprintk("%s: found session %p\n", __func__, elem);

can probably get rid of the dprintk

> +			return elem;
> +		}
> +	}
> +
> +	dprintk("%s: session not found\n", __func__);
> +	return NULL;
> +}
> +
>  static void
>  destroy_session(struct nfs41_session *ses)
>  {
> @@ -3154,6 +3201,10 @@ nfs4_state_init(void)
>  		INIT_LIST_HEAD(&unconf_str_hashtbl[i]);
>  		INIT_LIST_HEAD(&unconf_id_hashtbl[i]);
>  	}
> +#if defined(CONFIG_NFSD_V4_1)
> +	for (i = 0; i < SESSION_HASH_SIZE; i++)
> +		INIT_LIST_HEAD(&sessionid_hashtbl[i]);
> +#endif /* CONFIG_NFSD_V4_1 */
>  	for (i = 0; i < FILE_HASH_SIZE; i++) {
>  		INIT_LIST_HEAD(&file_hashtbl[i]);
>  	}

[RFC 11/51] FIXME: nfsd41: reduce server lease time for nfs41

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

FIXME: bhalevy: find out why this is needed.  Is this only for debugging?

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 0e1bc72..dcb075d 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -59,8 +59,13 @@
 #define NFSDDBG_FACILITY                NFSDDBG_PROC
 
 /* Globals */
+#if defined(CONFIG_NFSD_V4_1)
+static time_t lease_time = 20;     /* default lease time */
+static time_t user_lease_time = 20;
+#else /* CONFIG_NFSD_V4_1 */
 static time_t lease_time = 90;     /* default lease time */
 static time_t user_lease_time = 90;
+#endif /* CONFIG_NFSD_V4_1 */
 static time_t boot_time;
 static u32 current_ownerid = 1;
 static u32 current_fileid = 1;
-- 
1.6.0.2

Re: [pnfs] [RFC 11/51] FIXME: nfsd41: reduce server lease time for nfs41

[Benny Halevy]

review 11-12: get rid of this patch (at least for submit),
can be set in run time.

On Nov. 10, 2008, 22:44 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> FIXME: bhalevy: find out why this is needed.  Is this only for debugging?
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4state.c |    5 +++++
>  1 files changed, 5 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 0e1bc72..dcb075d 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -59,8 +59,13 @@
>  #define NFSDDBG_FACILITY                NFSDDBG_PROC
>  
>  /* Globals */
> +#if defined(CONFIG_NFSD_V4_1)
> +static time_t lease_time = 20;     /* default lease time */
> +static time_t user_lease_time = 20;
> +#else /* CONFIG_NFSD_V4_1 */
>  static time_t lease_time = 90;     /* default lease time */
>  static time_t user_lease_time = 90;
> +#endif /* CONFIG_NFSD_V4_1 */
>  static time_t boot_time;
>  static u32 current_ownerid = 1;
>  static u32 current_fileid = 1;

[RFC 12/51] nfsd41: provide support for minor version 1 at rpc level

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 include/linux/nfsd/nfsd.h |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/include/linux/nfsd/nfsd.h b/include/linux/nfsd/nfsd.h
index 9703540..ffb7897 100644
--- a/include/linux/nfsd/nfsd.h
+++ b/include/linux/nfsd/nfsd.h
@@ -24,7 +24,11 @@
  * nfsd version
  */
 #define NFSD_VERSION		"0.5"
+#if defined(CONFIG_NFSD_V4_1)
+#define NFSD_SUPPORTED_MINOR_VERSION	1
+#else /* CONFIG_NFSD_V4_1 */
 #define NFSD_SUPPORTED_MINOR_VERSION	0
+#endif /* CONFIG_NFSD_V4_1 */
 
 /*
  * Flags for nfsd_permission
-- 
1.6.0.2

Re: [pnfs] [RFC 12/51] nfsd41: provide support for minor version 1 at rpc level

[Benny Halevy]

On Nov. 10, 2008, 22:44 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 11-12: move down the patchset for bisectability

> ---
>  include/linux/nfsd/nfsd.h |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
> 
> diff --git a/include/linux/nfsd/nfsd.h b/include/linux/nfsd/nfsd.h
> index 9703540..ffb7897 100644
> --- a/include/linux/nfsd/nfsd.h
> +++ b/include/linux/nfsd/nfsd.h
> @@ -24,7 +24,11 @@
>   * nfsd version
>   */
>  #define NFSD_VERSION		"0.5"

review 11-12: send a patch to remove NFSD_VERSION

> +#if defined(CONFIG_NFSD_V4_1)
> +#define NFSD_SUPPORTED_MINOR_VERSION	1
> +#else /* CONFIG_NFSD_V4_1 */
>  #define NFSD_SUPPORTED_MINOR_VERSION	0
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  /*
>   * Flags for nfsd_permission

[RFC 13/51] FIXME: nfsd41: introduce current_session

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

FIXME: use a mem pool for current session allocation.

The current session is allocated and freed for every compound rpc.
It holds the session id and slot used by this compound.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c         |   26 ++++++++++++++++++++++++--
 include/linux/nfsd/state.h |   12 ++++++++++++
 include/linux/nfsd/xdr4.h  |    3 +++
 3 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 9fa60a3..bfebeaa 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -861,11 +861,14 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
 		    struct nfsd4_compoundargs *args,
 		    struct nfsd4_compoundres *resp)
 {
-	struct nfsd4_op	*op;
+	struct nfsd4_op	*op = NULL;
 	struct nfsd4_operation *opdesc;
 	struct nfsd4_compound_state *cstate = NULL;
 	int		slack_bytes;
 	__be32		status;
+#if defined(CONFIG_NFSD_V4_1)
+	struct current_session *current_ses = NULL;
+#endif /* CONFIG_NFSD_V4_1 */
 
 	resp->xbuf = &rqstp->rq_res;
 	resp->p = rqstp->rq_res.head[0].iov_base + rqstp->rq_res.head[0].iov_len;
@@ -890,6 +893,17 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
 	if (cstate == NULL)
 		goto out;
 
+#if defined(CONFIG_NFSD_V4_1)
+	if (args->minorversion == 1) {
+		/* FIXME: use kmem_cache */
+		current_ses = kzalloc(sizeof(*current_ses), GFP_KERNEL);
+		if (current_ses == NULL)
+			goto out_free;
+	}
+	/* DM: current_ses must be NULL for minorversion 0 */
+	cstate->current_ses = current_ses;
+#endif /* CONFIG_NFSD_V4_1 */
+
 	status = nfs_ok;
 	while (!status && resp->opcnt < args->opcnt) {
 		op = &args->ops[resp->opcnt++];
@@ -961,7 +975,15 @@ encode_op:
 
 		nfsd4_increment_op_stats(op->opnum);
 	}
-
+out_free:
+#if defined(CONFIG_NFSD_V4_1)
+	if (current_ses) {
+		struct nfs41_slot *cs_slot = current_ses->cs_slot;
+		if (cs_slot)
+			nfs41_put_session(cs_slot->sl_session);
+		kfree(current_ses);
+	}
+#endif /* CONFIG_NFSD_V4_1 */
 	cstate_free(cstate);
 out:
 	nfsd4_release_compoundargs(args);
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index 28c42ea..7684188 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -168,6 +168,18 @@ nfs41_get_session(struct nfs41_session *ses)
 {
 	kref_get(&ses->se_ref);
 }
+
+/* formatted contents of nfs41_sessionid */
+struct nfsd_sessionid {
+	clientid_t	clientid;
+	u32		boot_time;
+	u32		sequence;
+};
+
+struct current_session {
+	struct nfsd_sessionid	cs_sid;
+	struct nfs41_slot	*cs_slot;
+};
 #endif /* CONFIG_NFSD_V4_1 */
 
 #define HEXDIR_LEN     33 /* hex version of 16 byte md5 of cl_name plus '\0' */
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index 27bd3e3..88f7cd6 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -48,6 +48,9 @@ struct nfsd4_compound_state {
 	struct svc_fh current_fh;
 	struct svc_fh save_fh;
 	struct nfs4_stateowner *replay_owner;
+#if defined(CONFIG_NFSD_V4_1)
+	struct current_session *current_ses;
+#endif /* CONFIG_NFSD_V4_1 */
 };
 
 struct nfsd4_change_info {
-- 
1.6.0.2

Re: [pnfs] [RFC 13/51] FIXME: nfsd41: introduce current_session

[Benny Halevy]

On Nov. 10, 2008, 22:45 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> FIXME: use a mem pool for current session allocation.
> 

review 11-12: this code is replaced by the DRC patches.
move the latter forward in the patchset.

> The current session is allocated and freed for every compound rpc.
> It holds the session id and slot used by this compound.
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4proc.c         |   26 ++++++++++++++++++++++++--
>  include/linux/nfsd/state.h |   12 ++++++++++++
>  include/linux/nfsd/xdr4.h  |    3 +++
>  3 files changed, 39 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index 9fa60a3..bfebeaa 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -861,11 +861,14 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
>  		    struct nfsd4_compoundargs *args,
>  		    struct nfsd4_compoundres *resp)
>  {
> -	struct nfsd4_op	*op;
> +	struct nfsd4_op	*op = NULL;
>  	struct nfsd4_operation *opdesc;
>  	struct nfsd4_compound_state *cstate = NULL;
>  	int		slack_bytes;
>  	__be32		status;
> +#if defined(CONFIG_NFSD_V4_1)
> +	struct current_session *current_ses = NULL;
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  	resp->xbuf = &rqstp->rq_res;
>  	resp->p = rqstp->rq_res.head[0].iov_base + rqstp->rq_res.head[0].iov_len;
> @@ -890,6 +893,17 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
>  	if (cstate == NULL)
>  		goto out;
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +	if (args->minorversion == 1) {
> +		/* FIXME: use kmem_cache */
> +		current_ses = kzalloc(sizeof(*current_ses), GFP_KERNEL);
> +		if (current_ses == NULL)
> +			goto out_free;
> +	}
> +	/* DM: current_ses must be NULL for minorversion 0 */
> +	cstate->current_ses = current_ses;
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  	status = nfs_ok;
>  	while (!status && resp->opcnt < args->opcnt) {
>  		op = &args->ops[resp->opcnt++];
> @@ -961,7 +975,15 @@ encode_op:
>  
>  		nfsd4_increment_op_stats(op->opnum);
>  	}
> -
> +out_free:
> +#if defined(CONFIG_NFSD_V4_1)
> +	if (current_ses) {
> +		struct nfs41_slot *cs_slot = current_ses->cs_slot;
> +		if (cs_slot)
> +			nfs41_put_session(cs_slot->sl_session);
> +		kfree(current_ses);
> +	}
> +#endif /* CONFIG_NFSD_V4_1 */
>  	cstate_free(cstate);
>  out:
>  	nfsd4_release_compoundargs(args);
> diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
> index 28c42ea..7684188 100644
> --- a/include/linux/nfsd/state.h
> +++ b/include/linux/nfsd/state.h
> @@ -168,6 +168,18 @@ nfs41_get_session(struct nfs41_session *ses)
>  {
>  	kref_get(&ses->se_ref);
>  }
> +
> +/* formatted contents of nfs41_sessionid */
> +struct nfsd_sessionid {
> +	clientid_t	clientid;
> +	u32		boot_time;
> +	u32		sequence;
> +};
> +
> +struct current_session {
> +	struct nfsd_sessionid	cs_sid;
> +	struct nfs41_slot	*cs_slot;
> +};
>  #endif /* CONFIG_NFSD_V4_1 */
>  
>  #define HEXDIR_LEN     33 /* hex version of 16 byte md5 of cl_name plus '\0' */
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index 27bd3e3..88f7cd6 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -48,6 +48,9 @@ struct nfsd4_compound_state {
>  	struct svc_fh current_fh;
>  	struct svc_fh save_fh;
>  	struct nfs4_stateowner *replay_owner;
> +#if defined(CONFIG_NFSD_V4_1)
> +	struct current_session *current_ses;
> +#endif /* CONFIG_NFSD_V4_1 */
>  };
>  
>  struct nfsd4_change_info {

[RFC 14/51] nfsd41: introduce nfs41_{get,set}_slot_state

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c        |   12 ++++++++++++
 include/linux/nfsd/state.h |    3 +++
 2 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index dcb075d..23e83e2 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -321,6 +321,18 @@ static struct list_head close_lru;
 #define SESSION_HASH_SIZE	1031
 static struct list_head sessionid_hashtbl[SESSION_HASH_SIZE];
 
+int
+nfs41_get_slot_state(struct nfs41_slot *slot)
+{
+	return atomic_read(&slot->sl_state);
+}
+
+void
+nfs41_set_slot_state(struct nfs41_slot *slot, int state)
+{
+	atomic_set(&slot->sl_state, state);
+}
+
 static int
 hash_sessionid(nfs41_sessionid *sessionid)
 {
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index 7684188..85cdaf1 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -356,6 +356,9 @@ struct nfs4_stateid {
 	((err) != nfserr_stale_stateid) &&      \
 	((err) != nfserr_bad_stateid))
 
+#if defined(CONFIG_NFSD_V4_1)
+extern void nfs41_set_slot_state(struct nfs41_slot *, int);
+#endif /* CONFIG_NFSD_V4_1 */
 extern __be32 nfs4_preprocess_stateid_op(struct svc_fh *current_fh,
 		stateid_t *stateid, int flags, struct file **filp);
 extern void nfs4_lock_state(void);
-- 
1.6.0.2

Re: [pnfs] [RFC 14/51] nfsd41: introduce nfs41_{get,set}_slot_state

[Benny Halevy]

On Nov. 10, 2008, 22:45 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 11-12: change slot state into a bitmap
do not need atomic ops since done under the state lock.

> ---
>  fs/nfsd/nfs4state.c        |   12 ++++++++++++
>  include/linux/nfsd/state.h |    3 +++
>  2 files changed, 15 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index dcb075d..23e83e2 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -321,6 +321,18 @@ static struct list_head close_lru;
>  #define SESSION_HASH_SIZE	1031
>  static struct list_head sessionid_hashtbl[SESSION_HASH_SIZE];
>  
> +int
> +nfs41_get_slot_state(struct nfs41_slot *slot)
> +{
> +	return atomic_read(&slot->sl_state);
> +}
> +
> +void
> +nfs41_set_slot_state(struct nfs41_slot *slot, int state)
> +{
> +	atomic_set(&slot->sl_state, state);
> +}
> +
>  static int
>  hash_sessionid(nfs41_sessionid *sessionid)
>  {
> diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
> index 7684188..85cdaf1 100644
> --- a/include/linux/nfsd/state.h
> +++ b/include/linux/nfsd/state.h
> @@ -356,6 +356,9 @@ struct nfs4_stateid {
>  	((err) != nfserr_stale_stateid) &&      \
>  	((err) != nfserr_bad_stateid))
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +extern void nfs41_set_slot_state(struct nfs41_slot *, int);
> +#endif /* CONFIG_NFSD_V4_1 */
>  extern __be32 nfs4_preprocess_stateid_op(struct svc_fh *current_fh,
>  		stateid_t *stateid, int flags, struct file **filp);
>  extern void nfs4_lock_state(void);

[RFC 15/51] FIXME: nfsd41: free up slot unless operation is dropped

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

Mark slot is NFS4_SLOT_AVAILABLE before exiting nfsd4_proc_compound
unless op->status == nfserr_dropit

FIXME: bhalevy: verify slot gets freed eventually in the dropit case.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c |    9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index bfebeaa..876f285 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -979,8 +979,15 @@ out_free:
 #if defined(CONFIG_NFSD_V4_1)
 	if (current_ses) {
 		struct nfs41_slot *cs_slot = current_ses->cs_slot;
-		if (cs_slot)
+		if (cs_slot) {
+			if (op && op->status != nfserr_dropit) {
+				dprintk("%s: SET SLOT STATE TO AVAILABLE\n",
+					__func__);
+				nfs41_set_slot_state(cs_slot,
+						     NFS4_SLOT_AVAILABLE);
+			}
 			nfs41_put_session(cs_slot->sl_session);
+		}
 		kfree(current_ses);
 	}
 #endif /* CONFIG_NFSD_V4_1 */
-- 
1.6.0.2

Re: [pnfs] [RFC 15/51] FIXME: nfsd41: free up slot unless operation is dropped

[Benny Halevy]

On Nov. 10, 2008, 22:45 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> Mark slot is NFS4_SLOT_AVAILABLE before exiting nfsd4_proc_compound
> unless op->status == nfserr_dropit
> 
> FIXME: bhalevy: verify slot gets freed eventually in the dropit case.
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 11-12: code replaced by DRC

> ---
>  fs/nfsd/nfs4proc.c |    9 ++++++++-
>  1 files changed, 8 insertions(+), 1 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index bfebeaa..876f285 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -979,8 +979,15 @@ out_free:
>  #if defined(CONFIG_NFSD_V4_1)
>  	if (current_ses) {
>  		struct nfs41_slot *cs_slot = current_ses->cs_slot;
> -		if (cs_slot)
> +		if (cs_slot) {
> +			if (op && op->status != nfserr_dropit) {
> +				dprintk("%s: SET SLOT STATE TO AVAILABLE\n",
> +					__func__);
> +				nfs41_set_slot_state(cs_slot,
> +						     NFS4_SLOT_AVAILABLE);
> +			}
>  			nfs41_put_session(cs_slot->sl_session);
> +		}
>  		kfree(current_ses);
>  	}
>  #endif /* CONFIG_NFSD_V4_1 */

[RFC 16/51] nfsd41: stateid handling

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

Propagate the minorversion from open to the stateowner.

On first open set seqid to 1 and mark state confirmed

Skip seqid processing for NFSv4.1

4.1 is allowed to ignore the generation number when it is zero
whereas 4.0 returns bad_stateid or stale stateid.
Propagate minorversion to all stateful ops and down to check_stateid_generation.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c         |   20 +++++++++++++++-----
 fs/nfsd/nfs4state.c        |   37 +++++++++++++++++++++++++++++++------
 fs/nfsd/nfs4xdr.c          |    8 ++++++++
 include/linux/nfsd/state.h |    2 ++
 include/linux/nfsd/xdr4.h  |    5 +++++
 5 files changed, 61 insertions(+), 11 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 876f285..b17948b 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -511,6 +511,7 @@ nfsd4_read(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	   struct nfsd4_read *read)
 {
 	__be32 status;
+	int flags = 0;
 
 	/* no need to check permission - this will be done in nfsd_read() */
 
@@ -518,11 +519,14 @@ nfsd4_read(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	if (read->rd_offset >= OFFSET_MAX)
 		return nfserr_inval;
 
+	flags = CHECK_FH | RD_STATE;
+	if (read->rd_minorversion == 1)
+		flags |= NFS_4_1;
 	nfs4_lock_state();
 	/* check stateid */
 	if ((status = nfs4_preprocess_stateid_op(&cstate->current_fh,
 				&read->rd_stateid,
-				CHECK_FH | RD_STATE, &read->rd_filp))) {
+				flags, &read->rd_filp))) {
 		dprintk("NFSD: nfsd4_read: couldn't process stateid!\n");
 		goto out;
 	}
@@ -649,12 +653,15 @@ static __be32
 nfsd4_setattr(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	      struct nfsd4_setattr *setattr)
 {
-	__be32 status = nfs_ok;
+	__be32 status = nfs_ok, flags = 0;
 
 	if (setattr->sa_iattr.ia_valid & ATTR_SIZE) {
+		flags = CHECK_FH | WR_STATE;
+		if (setattr->sa_minorversion == 1)
+			flags |= NFS_4_1;
 		nfs4_lock_state();
 		status = nfs4_preprocess_stateid_op(&cstate->current_fh,
-			&setattr->sa_stateid, CHECK_FH | WR_STATE, NULL);
+			&setattr->sa_stateid, flags, NULL);
 		nfs4_unlock_state();
 		if (status) {
 			dprintk("NFSD: nfsd4_setattr: couldn't process stateid!\n");
@@ -684,16 +691,19 @@ nfsd4_write(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	stateid_t *stateid = &write->wr_stateid;
 	struct file *filp = NULL;
 	u32 *p;
-	__be32 status = nfs_ok;
+	__be32 status = nfs_ok, flags = 0;
 
 	/* no need to check permission - this will be done in nfsd_write() */
 
 	if (write->wr_offset >= OFFSET_MAX)
 		return nfserr_inval;
 
+	flags = CHECK_FH | WR_STATE;
+	if (write->wr_minorversion == 1)
+		flags |= NFS_4_1;
 	nfs4_lock_state();
 	status = nfs4_preprocess_stateid_op(&cstate->current_fh, stateid,
-					CHECK_FH | WR_STATE, &filp);
+					flags, &filp);
 	if (filp)
 		get_file(filp);
 	nfs4_unlock_state();
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 23e83e2..2e6e9d5 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -1115,6 +1115,7 @@ alloc_init_open_stateowner(unsigned int strhashval, struct nfs4_client *clp, str
 	sop->so_client = clp;
 	sop->so_seqid = open->op_seqid;
 	sop->so_confirmed = 0;
+	sop->so_minorversion = open->op_minorversion;
 	rp = &sop->so_replay;
 	rp->rp_status = nfserr_serverfault;
 	rp->rp_buflen = 0;
@@ -1524,6 +1525,9 @@ nfsd4_process_open1(struct nfsd4_open *open)
 		open->op_stateowner = NULL;
 		goto renew;
 	}
+	/* Skip seqid processing for NFSv4.1 */
+	if (open->op_minorversion == 1)
+		goto renew;
 	if (open->op_seqid == sop->so_seqid - 1) {
 		if (sop->so_replay.rp_buflen)
 			return nfserr_replay_me;
@@ -1852,9 +1856,14 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
 			release_stateid(stp, OPEN_STATE);
 			goto out;
 		}
+		if (open->op_minorversion == 1)
+			update_stateid(&stp->st_stateid);
 	}
 	memcpy(&open->op_stateid, &stp->st_stateid, sizeof(stateid_t));
 
+	if (open->op_minorversion == 1)
+		open->op_stateowner->so_confirmed = 1;
+
 	/*
 	* Attempt to hand out a delegation. No error return, because the
 	* OPEN succeeds even if we fail.
@@ -1875,7 +1884,7 @@ out:
 	* To finish the open response, we just need to set the rflags.
 	*/
 	open->op_rflags = NFS4_OPEN_RESULT_LOCKTYPE_POSIX;
-	if (!open->op_stateowner->so_confirmed)
+	if (!open->op_stateowner->so_confirmed && !open->op_minorversion)
 		open->op_rflags |= NFS4_OPEN_RESULT_CONFIRM;
 
 	return status;
@@ -2096,8 +2105,16 @@ io_during_grace_disallowed(struct inode *inode, int flags)
 		&& mandatory_lock(inode);
 }
 
-static int check_stateid_generation(stateid_t *in, stateid_t *ref)
+static int check_stateid_generation(stateid_t *in, stateid_t *ref,
+				    u32 minorversion)
 {
+	/*
+	 * 4.1 is allowed to ignore the generation number when it is zero
+	 * whereas 4.0 returns bad_stateid or stale stateid.
+	 */
+	if (minorversion && in->si_generation == 0)
+		goto out;
+
 	/* If the client sends us a stateid from the future, it's buggy: */
 	if (in->si_generation > ref->si_generation)
 		return nfserr_bad_stateid;
@@ -2113,6 +2130,7 @@ static int check_stateid_generation(stateid_t *in, stateid_t *ref)
 	 */
 	if (in->si_generation < ref->si_generation)
 		return nfserr_old_stateid;
+out:
 	return nfs_ok;
 }
 
@@ -2164,7 +2182,8 @@ nfs4_preprocess_stateid_op(struct svc_fh *current_fh, stateid_t *stateid, int fl
 			goto out;
 		stidp = &stp->st_stateid;
 	}
-	status = check_stateid_generation(stateid, stidp);
+	status = check_stateid_generation(stateid, stidp,
+					  (flags & NFS_4_1) != 0);
 	if (status)
 		goto out;
 	if (stp) {
@@ -2275,7 +2294,7 @@ nfs4_preprocess_seqid_op(struct svc_fh *current_fh, u32 seqid, stateid_t *statei
 	*  For the moment, we ignore the possibility of 
 	*  generation number wraparound.
 	*/
-	if (seqid != sop->so_seqid)
+	if (sop->so_minorversion == 0 && seqid != sop->so_seqid)
 		goto check_replay;
 
 	if (sop->so_confirmed && flags & CONFIRM) {
@@ -2288,7 +2307,8 @@ nfs4_preprocess_seqid_op(struct svc_fh *current_fh, u32 seqid, stateid_t *statei
 				" confirmed yet!\n");
 		return nfserr_bad_stateid;
 	}
-	status = check_stateid_generation(stateid, &stp->st_stateid);
+	status = check_stateid_generation(stateid, &stp->st_stateid,
+					  sop->so_minorversion);
 	if (status)
 		return status;
 	renew_client(sop->so_client);
@@ -2480,13 +2500,17 @@ nfsd4_delegreturn(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 		  struct nfsd4_delegreturn *dr)
 {
 	__be32 status;
+	int flags = 0;
 
 	if ((status = fh_verify(rqstp, &cstate->current_fh, S_IFREG, 0)))
 		goto out;
 
 	nfs4_lock_state();
+	flags |= DELEG_RET;
+	if (dr->dr_minorversion == 1)
+		flags |= NFS_4_1;
 	status = nfs4_preprocess_stateid_op(&cstate->current_fh,
-					    &dr->dr_stateid, DELEG_RET, NULL);
+					    &dr->dr_stateid, flags, NULL);
 	nfs4_unlock_state();
 out:
 	return status;
@@ -2659,6 +2683,7 @@ alloc_init_lock_stateowner(unsigned int strhashval, struct nfs4_client *clp, str
 	 * case of new lockowners; so increment the lock seqid manually: */
 	sop->so_seqid = lock->lk_new_lock_seqid + 1;
 	sop->so_confirmed = 1;
+	sop->so_minorversion = open_stp->st_stateowner->so_minorversion;
 	rp = &sop->so_replay;
 	rp->rp_status = nfserr_serverfault;
 	rp->rp_buflen = 0;
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index f7dec05..9b26ba9 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -633,6 +633,8 @@ nfsd4_decode_open(struct nfsd4_compoundargs *argp, struct nfsd4_open *open)
 	open->op_iattr.ia_valid = 0;
 	open->op_stateowner = NULL;
 
+	open->op_minorversion = argp->minorversion;
+
 	/* seqid, share_access, share_deny, clientid, ownerlen */
 	READ_BUF(16 + sizeof(clientid_t));
 	READ32(open->op_seqid);
@@ -756,6 +758,8 @@ nfsd4_decode_read(struct nfsd4_compoundargs *argp, struct nfsd4_read *read)
 {
 	DECODE_HEAD;
 
+	read->rd_minorversion = argp->minorversion;
+
 	status = nfsd4_decode_stateid(argp, &read->rd_stateid);
 	if (status)
 		return status;
@@ -850,6 +854,8 @@ nfsd4_decode_setattr(struct nfsd4_compoundargs *argp, struct nfsd4_setattr *seta
 {
 	__be32 status;
 
+	setattr->sa_minorversion = argp->minorversion;
+
 	status = nfsd4_decode_stateid(argp, &setattr->sa_stateid);
 	if (status)
 		return status;
@@ -939,6 +945,8 @@ nfsd4_decode_write(struct nfsd4_compoundargs *argp, struct nfsd4_write *write)
 	int len;
 	DECODE_HEAD;
 
+	write->wr_minorversion = argp->minorversion;
+
 	status = nfsd4_decode_stateid(argp, &write->wr_stateid);
 	if (status)
 		return status;
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index 85cdaf1..5cb4142 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -290,6 +290,7 @@ struct nfs4_stateowner {
 	u32                     so_seqid;
 	struct xdr_netobj       so_owner;     /* open owner name */
 	int                     so_confirmed; /* successful OPEN_CONFIRM? */
+	u32			so_minorversion;
 	struct nfs4_replay	so_replay;
 };
 
@@ -349,6 +350,7 @@ struct nfs4_stateid {
 #define WR_STATE	        0x00000020
 #define CLOSE_STATE             0x00000040
 #define DELEG_RET               0x00000080
+#define NFS_4_1			0x00000100
 
 #define seqid_mutating_err(err)                       \
 	(((err) != nfserr_stale_clientid) &&    \
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index 88f7cd6..81c249c 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -105,6 +105,7 @@ struct nfsd4_create {
 
 struct nfsd4_delegreturn {
 	stateid_t	dr_stateid;
+	u32		dr_minorversion;
 };
 
 struct nfsd4_getattr {
@@ -223,6 +224,7 @@ struct nfsd4_open {
 	u32		op_recall;          /* recall */
 	struct nfsd4_change_info  op_cinfo; /* response */
 	u32		op_rflags;          /* response */
+	u32		op_minorversion;    /* used during processing */
 	int		op_truncate;        /* used during processing */
 	struct nfs4_stateowner *op_stateowner; /* used during processing */
 	struct nfs4_acl *op_acl;
@@ -255,6 +257,7 @@ struct nfsd4_read {
 	
 	struct svc_rqst *rd_rqstp;          /* response */
 	struct svc_fh * rd_fhp;             /* response */
+	u32		rd_minorversion;    /* processing */
 };
 
 struct nfsd4_readdir {
@@ -304,6 +307,7 @@ struct nfsd4_secinfo {
 
 struct nfsd4_setattr {
 	stateid_t	sa_stateid;         /* request */
+	u32		sa_minorversion;    /* processing */
 	u32		sa_bmval[2];        /* request */
 	struct iattr	sa_iattr;           /* request */
 	struct nfs4_acl *sa_acl;
@@ -345,6 +349,7 @@ struct nfsd4_write {
 	u32		wr_bytes_written;   /* response */
 	u32		wr_how_written;     /* response */
 	nfs4_verifier	wr_verifier;        /* response */
+	u32		wr_minorversion;    /* processing */
 };
 
 struct nfsd4_op {
-- 
1.6.0.2

Re: [pnfs] [RFC 16/51] nfsd41: stateid handling

[Benny Halevy]

On Nov. 10, 2008, 22:46 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> Propagate the minorversion from open to the stateowner.
> 
> On first open set seqid to 1 and mark state confirmed
> 
> Skip seqid processing for NFSv4.1
> 
> 4.1 is allowed to ignore the generation number when it is zero
> whereas 4.0 returns bad_stateid or stale stateid.
> Propagate minorversion to all stateful ops and down to check_stateid_generation.
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 11-12: look into using a minorversion in the coumpound cstate/args/res
rather than adding a minorversion member to each op's xdr struct.
* update_stateid should skip over seqid 0 when wrapping around;

> ---
>  fs/nfsd/nfs4proc.c         |   20 +++++++++++++++-----
>  fs/nfsd/nfs4state.c        |   37 +++++++++++++++++++++++++++++++------
>  fs/nfsd/nfs4xdr.c          |    8 ++++++++
>  include/linux/nfsd/state.h |    2 ++
>  include/linux/nfsd/xdr4.h  |    5 +++++
>  5 files changed, 61 insertions(+), 11 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index 876f285..b17948b 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -511,6 +511,7 @@ nfsd4_read(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	   struct nfsd4_read *read)
>  {
>  	__be32 status;
> +	int flags = 0;
>  
>  	/* no need to check permission - this will be done in nfsd_read() */
>  
> @@ -518,11 +519,14 @@ nfsd4_read(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	if (read->rd_offset >= OFFSET_MAX)
>  		return nfserr_inval;
>  
> +	flags = CHECK_FH | RD_STATE;
> +	if (read->rd_minorversion == 1)
> +		flags |= NFS_4_1;
>  	nfs4_lock_state();
>  	/* check stateid */
>  	if ((status = nfs4_preprocess_stateid_op(&cstate->current_fh,
>  				&read->rd_stateid,
> -				CHECK_FH | RD_STATE, &read->rd_filp))) {
> +				flags, &read->rd_filp))) {
>  		dprintk("NFSD: nfsd4_read: couldn't process stateid!\n");
>  		goto out;
>  	}
> @@ -649,12 +653,15 @@ static __be32
>  nfsd4_setattr(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	      struct nfsd4_setattr *setattr)
>  {
> -	__be32 status = nfs_ok;
> +	__be32 status = nfs_ok, flags = 0;
>  
>  	if (setattr->sa_iattr.ia_valid & ATTR_SIZE) {
> +		flags = CHECK_FH | WR_STATE;
> +		if (setattr->sa_minorversion == 1)
> +			flags |= NFS_4_1;
>  		nfs4_lock_state();
>  		status = nfs4_preprocess_stateid_op(&cstate->current_fh,
> -			&setattr->sa_stateid, CHECK_FH | WR_STATE, NULL);
> +			&setattr->sa_stateid, flags, NULL);
>  		nfs4_unlock_state();
>  		if (status) {
>  			dprintk("NFSD: nfsd4_setattr: couldn't process stateid!\n");
> @@ -684,16 +691,19 @@ nfsd4_write(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	stateid_t *stateid = &write->wr_stateid;
>  	struct file *filp = NULL;
>  	u32 *p;
> -	__be32 status = nfs_ok;
> +	__be32 status = nfs_ok, flags = 0;
>  
>  	/* no need to check permission - this will be done in nfsd_write() */
>  
>  	if (write->wr_offset >= OFFSET_MAX)
>  		return nfserr_inval;
>  
> +	flags = CHECK_FH | WR_STATE;
> +	if (write->wr_minorversion == 1)
> +		flags |= NFS_4_1;
>  	nfs4_lock_state();
>  	status = nfs4_preprocess_stateid_op(&cstate->current_fh, stateid,
> -					CHECK_FH | WR_STATE, &filp);
> +					flags, &filp);
>  	if (filp)
>  		get_file(filp);
>  	nfs4_unlock_state();
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 23e83e2..2e6e9d5 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -1115,6 +1115,7 @@ alloc_init_open_stateowner(unsigned int strhashval, struct nfs4_client *clp, str
>  	sop->so_client = clp;
>  	sop->so_seqid = open->op_seqid;
>  	sop->so_confirmed = 0;
> +	sop->so_minorversion = open->op_minorversion;
>  	rp = &sop->so_replay;
>  	rp->rp_status = nfserr_serverfault;
>  	rp->rp_buflen = 0;
> @@ -1524,6 +1525,9 @@ nfsd4_process_open1(struct nfsd4_open *open)
>  		open->op_stateowner = NULL;
>  		goto renew;
>  	}
> +	/* Skip seqid processing for NFSv4.1 */
> +	if (open->op_minorversion == 1)
> +		goto renew;
>  	if (open->op_seqid == sop->so_seqid - 1) {
>  		if (sop->so_replay.rp_buflen)
>  			return nfserr_replay_me;
> @@ -1852,9 +1856,14 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
>  			release_stateid(stp, OPEN_STATE);
>  			goto out;
>  		}
> +		if (open->op_minorversion == 1)
> +			update_stateid(&stp->st_stateid);
>  	}
>  	memcpy(&open->op_stateid, &stp->st_stateid, sizeof(stateid_t));
>  
> +	if (open->op_minorversion == 1)
> +		open->op_stateowner->so_confirmed = 1;
> +
>  	/*
>  	* Attempt to hand out a delegation. No error return, because the
>  	* OPEN succeeds even if we fail.
> @@ -1875,7 +1884,7 @@ out:
>  	* To finish the open response, we just need to set the rflags.
>  	*/
>  	open->op_rflags = NFS4_OPEN_RESULT_LOCKTYPE_POSIX;
> -	if (!open->op_stateowner->so_confirmed)
> +	if (!open->op_stateowner->so_confirmed && !open->op_minorversion)
>  		open->op_rflags |= NFS4_OPEN_RESULT_CONFIRM;
>  
>  	return status;
> @@ -2096,8 +2105,16 @@ io_during_grace_disallowed(struct inode *inode, int flags)
>  		&& mandatory_lock(inode);
>  }
>  
> -static int check_stateid_generation(stateid_t *in, stateid_t *ref)
> +static int check_stateid_generation(stateid_t *in, stateid_t *ref,
> +				    u32 minorversion)

review 11-12: rename to bool check_zero_seq

>  {
> +	/*
> +	 * 4.1 is allowed to ignore the generation number when it is zero
> +	 * whereas 4.0 returns bad_stateid or stale stateid.
> +	 */
> +	if (minorversion && in->si_generation == 0)
> +		goto out;
> +
>  	/* If the client sends us a stateid from the future, it's buggy: */
>  	if (in->si_generation > ref->si_generation)
>  		return nfserr_bad_stateid;
> @@ -2113,6 +2130,7 @@ static int check_stateid_generation(stateid_t *in, stateid_t *ref)
>  	 */
>  	if (in->si_generation < ref->si_generation)
>  		return nfserr_old_stateid;
> +out:
>  	return nfs_ok;
>  }
>  
> @@ -2164,7 +2182,8 @@ nfs4_preprocess_stateid_op(struct svc_fh *current_fh, stateid_t *stateid, int fl
>  			goto out;
>  		stidp = &stp->st_stateid;
>  	}
> -	status = check_stateid_generation(stateid, stidp);
> +	status = check_stateid_generation(stateid, stidp,
> +					  (flags & NFS_4_1) != 0);
>  	if (status)
>  		goto out;
>  	if (stp) {
> @@ -2275,7 +2294,7 @@ nfs4_preprocess_seqid_op(struct svc_fh *current_fh, u32 seqid, stateid_t *statei
>  	*  For the moment, we ignore the possibility of 
>  	*  generation number wraparound.
>  	*/
> -	if (seqid != sop->so_seqid)
> +	if (sop->so_minorversion == 0 && seqid != sop->so_seqid)
>  		goto check_replay;
>  
>  	if (sop->so_confirmed && flags & CONFIRM) {
> @@ -2288,7 +2307,8 @@ nfs4_preprocess_seqid_op(struct svc_fh *current_fh, u32 seqid, stateid_t *statei
>  				" confirmed yet!\n");
>  		return nfserr_bad_stateid;
>  	}
> -	status = check_stateid_generation(stateid, &stp->st_stateid);
> +	status = check_stateid_generation(stateid, &stp->st_stateid,
> +					  sop->so_minorversion);
>  	if (status)
>  		return status;
>  	renew_client(sop->so_client);
> @@ -2480,13 +2500,17 @@ nfsd4_delegreturn(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  		  struct nfsd4_delegreturn *dr)
>  {
>  	__be32 status;
> +	int flags = 0;
>  
>  	if ((status = fh_verify(rqstp, &cstate->current_fh, S_IFREG, 0)))
>  		goto out;
>  
>  	nfs4_lock_state();
> +	flags |= DELEG_RET;
> +	if (dr->dr_minorversion == 1)
> +		flags |= NFS_4_1;
>  	status = nfs4_preprocess_stateid_op(&cstate->current_fh,
> -					    &dr->dr_stateid, DELEG_RET, NULL);
> +					    &dr->dr_stateid, flags, NULL);
>  	nfs4_unlock_state();
>  out:
>  	return status;
> @@ -2659,6 +2683,7 @@ alloc_init_lock_stateowner(unsigned int strhashval, struct nfs4_client *clp, str
>  	 * case of new lockowners; so increment the lock seqid manually: */
>  	sop->so_seqid = lock->lk_new_lock_seqid + 1;
>  	sop->so_confirmed = 1;
> +	sop->so_minorversion = open_stp->st_stateowner->so_minorversion;
>  	rp = &sop->so_replay;
>  	rp->rp_status = nfserr_serverfault;
>  	rp->rp_buflen = 0;
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index f7dec05..9b26ba9 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -633,6 +633,8 @@ nfsd4_decode_open(struct nfsd4_compoundargs *argp, struct nfsd4_open *open)
>  	open->op_iattr.ia_valid = 0;
>  	open->op_stateowner = NULL;
>  
> +	open->op_minorversion = argp->minorversion;
> +
>  	/* seqid, share_access, share_deny, clientid, ownerlen */
>  	READ_BUF(16 + sizeof(clientid_t));
>  	READ32(open->op_seqid);
> @@ -756,6 +758,8 @@ nfsd4_decode_read(struct nfsd4_compoundargs *argp, struct nfsd4_read *read)
>  {
>  	DECODE_HEAD;
>  
> +	read->rd_minorversion = argp->minorversion;
> +
>  	status = nfsd4_decode_stateid(argp, &read->rd_stateid);
>  	if (status)
>  		return status;
> @@ -850,6 +854,8 @@ nfsd4_decode_setattr(struct nfsd4_compoundargs *argp, struct nfsd4_setattr *seta
>  {
>  	__be32 status;
>  
> +	setattr->sa_minorversion = argp->minorversion;
> +
>  	status = nfsd4_decode_stateid(argp, &setattr->sa_stateid);
>  	if (status)
>  		return status;
> @@ -939,6 +945,8 @@ nfsd4_decode_write(struct nfsd4_compoundargs *argp, struct nfsd4_write *write)
>  	int len;
>  	DECODE_HEAD;
>  
> +	write->wr_minorversion = argp->minorversion;
> +
>  	status = nfsd4_decode_stateid(argp, &write->wr_stateid);
>  	if (status)
>  		return status;
> diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
> index 85cdaf1..5cb4142 100644
> --- a/include/linux/nfsd/state.h
> +++ b/include/linux/nfsd/state.h
> @@ -290,6 +290,7 @@ struct nfs4_stateowner {
>  	u32                     so_seqid;
>  	struct xdr_netobj       so_owner;     /* open owner name */
>  	int                     so_confirmed; /* successful OPEN_CONFIRM? */
> +	u32			so_minorversion;
>  	struct nfs4_replay	so_replay;
>  };
>  
> @@ -349,6 +350,7 @@ struct nfs4_stateid {
>  #define WR_STATE	        0x00000020
>  #define CLOSE_STATE             0x00000040
>  #define DELEG_RET               0x00000080
> +#define NFS_4_1			0x00000100

review 11-12: rename to CHECK_ZERO_SEQ

>  
>  #define seqid_mutating_err(err)                       \
>  	(((err) != nfserr_stale_clientid) &&    \
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index 88f7cd6..81c249c 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -105,6 +105,7 @@ struct nfsd4_create {
>  
>  struct nfsd4_delegreturn {
>  	stateid_t	dr_stateid;
> +	u32		dr_minorversion;
>  };
>  
>  struct nfsd4_getattr {
> @@ -223,6 +224,7 @@ struct nfsd4_open {
>  	u32		op_recall;          /* recall */
>  	struct nfsd4_change_info  op_cinfo; /* response */
>  	u32		op_rflags;          /* response */
> +	u32		op_minorversion;    /* used during processing */
>  	int		op_truncate;        /* used during processing */
>  	struct nfs4_stateowner *op_stateowner; /* used during processing */
>  	struct nfs4_acl *op_acl;
> @@ -255,6 +257,7 @@ struct nfsd4_read {
>  	
>  	struct svc_rqst *rd_rqstp;          /* response */
>  	struct svc_fh * rd_fhp;             /* response */
> +	u32		rd_minorversion;    /* processing */
>  };
>  
>  struct nfsd4_readdir {
> @@ -304,6 +307,7 @@ struct nfsd4_secinfo {
>  
>  struct nfsd4_setattr {
>  	stateid_t	sa_stateid;         /* request */
> +	u32		sa_minorversion;    /* processing */
>  	u32		sa_bmval[2];        /* request */
>  	struct iattr	sa_iattr;           /* request */
>  	struct nfs4_acl *sa_acl;
> @@ -345,6 +349,7 @@ struct nfsd4_write {
>  	u32		wr_bytes_written;   /* response */
>  	u32		wr_how_written;     /* response */
>  	nfs4_verifier	wr_verifier;        /* response */
> +	u32		wr_minorversion;    /* processing */
>  };
>  
>  struct nfsd4_op {

[RFC 17/51] nfsd41: clientid handling

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

Extract the clientid from sessionid to set the op_clientid on open.
Verify that the clid for other stateful ops is zero for minorversion != 0
(and do all other checks only for minorversion 0).

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c  |   13 +++++++++++++
 fs/nfsd/nfs4state.c |   21 ++++++++++++++++-----
 fs/nfsd/nfs4xdr.c   |   10 ++++++++++
 3 files changed, 39 insertions(+), 5 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index b17948b..26aae5d 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -164,6 +164,14 @@ do_open_fhandle(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nfsd4_
 	return status;
 }
 
+#if defined(CONFIG_NFSD_V4_1)
+static void
+nfsd41_set_clientid(clientid_t *clid, struct current_session *cses)
+{
+	clid->cl_boot = cses->cs_sid.clientid.cl_boot;
+	clid->cl_id = cses->cs_sid.clientid.cl_id;
+}
+#endif /* CONFIG_NFSD_V4_1 */
 
 static __be32
 nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
@@ -178,6 +186,11 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	if (open->op_create && open->op_claim_type != NFS4_OPEN_CLAIM_NULL)
 		return nfserr_inval;
 
+#if defined(CONFIG_NFSD_V4_1)
+	/* Set the NFSv4.1 client id */
+	if (open->op_minorversion)
+		nfsd41_set_clientid(&open->op_clientid, cstate->current_ses);
+#endif /* CONFIG_NFSD_V4_1 */
 	nfs4_lock_state();
 
 	/* check seqid for replay. set nfs4_owner */
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 2e6e9d5..14c1a0e 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -413,8 +413,8 @@ STALE_CLIENTID(clientid_t *clid)
 {
 	if (clid->cl_boot == boot_time)
 		return 0;
-	dprintk("NFSD stale clientid (%08x/%08x)\n", 
-			clid->cl_boot, clid->cl_id);
+	dprintk("NFSD stale clientid (%08x/%08x) boot_time %08lx\n",
+			clid->cl_boot, clid->cl_id, boot_time);
 	return 1;
 }
 
@@ -2270,7 +2270,8 @@ nfs4_preprocess_seqid_op(struct svc_fh *current_fh, u32 seqid, stateid_t *statei
 		if (lock->lk_is_new) {
 			if (!sop->so_is_open_owner)
 				return nfserr_bad_stateid;
-			if (!same_clid(&clp->cl_clientid, lockclid))
+			if (sop->so_minorversion == 0 &&
+			    !same_clid(&clp->cl_clientid, lockclid))
 			       return nfserr_bad_stateid;
 			/* stp is the open stateid */
 			status = nfs4_check_openmode(stp, lkflg);
@@ -2747,6 +2748,9 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	unsigned int strhashval;
 	unsigned int cmd;
 	int err;
+#if defined(CONFIG_NFSD_V4_1)
+	struct current_session *cses = cstate->current_ses;
+#endif /* CONFIG_NFSD_V4_1 */
 
 	dprintk("NFSD: nfsd4_lock: start=%Ld length=%Ld\n",
 		(long long) lock->lk_offset,
@@ -2773,8 +2777,10 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 		struct nfs4_file *fp;
 		
 		status = nfserr_stale_clientid;
-		if (STALE_CLIENTID(&lock->lk_new_clientid))
+#if defined(CONFIG_NFSD_V4_1)
+		if (!cses && STALE_CLIENTID(&lock->lk_new_clientid))
 			goto out;
+#endif /* CONFIG_NFSD_V4_1 */
 
 		/* validate and update open stateid and open seqid */
 		status = nfs4_preprocess_seqid_op(&cstate->current_fh,
@@ -2902,6 +2908,9 @@ nfsd4_lockt(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	struct file_lock file_lock;
 	int error;
 	__be32 status;
+#if defined(CONFIG_NFSD_V4_1)
+	struct current_session *cses = cstate->current_ses;
+#endif /* CONFIG_NFSD_V4_1 */
 
 	if (locks_in_grace())
 		return nfserr_grace;
@@ -2913,8 +2922,10 @@ nfsd4_lockt(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	nfs4_lock_state();
 
 	status = nfserr_stale_clientid;
-	if (STALE_CLIENTID(&lockt->lt_clientid))
+#if defined(CONFIG_NFSD_V4_1)
+	if (!cses && STALE_CLIENTID(&lockt->lt_clientid))
 		goto out;
+#endif /* CONFIG_NFSD_V4_1 */
 
 	if ((status = fh_verify(rqstp, &cstate->current_fh, S_IFREG, 0))) {
 		dprintk("NFSD: nfsd4_lockt: fh_verify() failed!\n");
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 9b26ba9..4675339 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -148,6 +148,11 @@ xdr_error:					\
 	}					\
 } while (0)
 
+static int zero_clientid(clientid_t *clid)
+{
+	return ((clid->cl_boot == 0) && (clid->cl_id == 0));
+}
+
 static __be32 *read_buf(struct nfsd4_compoundargs *argp, u32 nbytes)
 {
 	/* We want more bytes than seem to be available.
@@ -585,6 +590,8 @@ nfsd4_decode_lockt(struct nfsd4_compoundargs *argp, struct nfsd4_lockt *lockt)
 	READ_BUF(lockt->lt_owner.len);
 	READMEM(lockt->lt_owner.data, lockt->lt_owner.len);
 
+	if (argp->minorversion && !zero_clientid(&lockt->lt_clientid))
+		return nfserr_inval;
 	DECODE_TAIL;
 }
 
@@ -1003,6 +1010,9 @@ nfsd4_decode_release_lockowner(struct nfsd4_compoundargs *argp, struct nfsd4_rel
 	READ_BUF(rlockowner->rl_owner.len);
 	READMEM(rlockowner->rl_owner.data, rlockowner->rl_owner.len);
 
+	if (argp->minorversion && !zero_clientid(&rlockowner->rl_clientid))
+		return nfserr_inval;
+
 	DECODE_TAIL;
 }
 
-- 
1.6.0.2

Re: [pnfs] [RFC 17/51] nfsd41: clientid handling

[Benny Halevy]

On Nov. 10, 2008, 22:46 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> Extract the clientid from sessionid to set the op_clientid on open.
> Verify that the clid for other stateful ops is zero for minorversion != 0
> (and do all other checks only for minorversion 0).
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4proc.c  |   13 +++++++++++++
>  fs/nfsd/nfs4state.c |   21 ++++++++++++++++-----
>  fs/nfsd/nfs4xdr.c   |   10 ++++++++++
>  3 files changed, 39 insertions(+), 5 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index b17948b..26aae5d 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -164,6 +164,14 @@ do_open_fhandle(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nfsd4_
>  	return status;
>  }
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +static void
> +nfsd41_set_clientid(clientid_t *clid, struct current_session *cses)
> +{
> +	clid->cl_boot = cses->cs_sid.clientid.cl_boot;
> +	clid->cl_id = cses->cs_sid.clientid.cl_id;
> +}
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  static __be32
>  nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
> @@ -178,6 +186,11 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	if (open->op_create && open->op_claim_type != NFS4_OPEN_CLAIM_NULL)
>  		return nfserr_inval;
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +	/* Set the NFSv4.1 client id */
> +	if (open->op_minorversion)
> +		nfsd41_set_clientid(&open->op_clientid, cstate->current_ses);
> +#endif /* CONFIG_NFSD_V4_1 */
>  	nfs4_lock_state();
>  
>  	/* check seqid for replay. set nfs4_owner */
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 2e6e9d5..14c1a0e 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -413,8 +413,8 @@ STALE_CLIENTID(clientid_t *clid)
>  {
>  	if (clid->cl_boot == boot_time)
>  		return 0;
> -	dprintk("NFSD stale clientid (%08x/%08x)\n", 
> -			clid->cl_boot, clid->cl_id);
> +	dprintk("NFSD stale clientid (%08x/%08x) boot_time %08lx\n",
> +			clid->cl_boot, clid->cl_id, boot_time);
>  	return 1;
>  }
>  
> @@ -2270,7 +2270,8 @@ nfs4_preprocess_seqid_op(struct svc_fh *current_fh, u32 seqid, stateid_t *statei
>  		if (lock->lk_is_new) {
>  			if (!sop->so_is_open_owner)
>  				return nfserr_bad_stateid;
> -			if (!same_clid(&clp->cl_clientid, lockclid))
> +			if (sop->so_minorversion == 0 &&
> +			    !same_clid(&clp->cl_clientid, lockclid))
>  			       return nfserr_bad_stateid;
>  			/* stp is the open stateid */
>  			status = nfs4_check_openmode(stp, lkflg);
> @@ -2747,6 +2748,9 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	unsigned int strhashval;
>  	unsigned int cmd;
>  	int err;
> +#if defined(CONFIG_NFSD_V4_1)
> +	struct current_session *cses = cstate->current_ses;
> +#endif /* CONFIG_NFSD_V4_1 */

review 11-12: when cstate is going to be embedded in compoundres
this will change.

>  
>  	dprintk("NFSD: nfsd4_lock: start=%Ld length=%Ld\n",
>  		(long long) lock->lk_offset,
> @@ -2773,8 +2777,10 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  		struct nfs4_file *fp;
>  		
>  		status = nfserr_stale_clientid;
> -		if (STALE_CLIENTID(&lock->lk_new_clientid))
> +#if defined(CONFIG_NFSD_V4_1)
> +		if (!cses && STALE_CLIENTID(&lock->lk_new_clientid))
>  			goto out;
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  		/* validate and update open stateid and open seqid */
>  		status = nfs4_preprocess_seqid_op(&cstate->current_fh,
> @@ -2902,6 +2908,9 @@ nfsd4_lockt(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	struct file_lock file_lock;
>  	int error;
>  	__be32 status;
> +#if defined(CONFIG_NFSD_V4_1)
> +	struct current_session *cses = cstate->current_ses;
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  	if (locks_in_grace())
>  		return nfserr_grace;
> @@ -2913,8 +2922,10 @@ nfsd4_lockt(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	nfs4_lock_state();
>  
>  	status = nfserr_stale_clientid;
> -	if (STALE_CLIENTID(&lockt->lt_clientid))
> +#if defined(CONFIG_NFSD_V4_1)
> +	if (!cses && STALE_CLIENTID(&lockt->lt_clientid))
>  		goto out;
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  	if ((status = fh_verify(rqstp, &cstate->current_fh, S_IFREG, 0))) {
>  		dprintk("NFSD: nfsd4_lockt: fh_verify() failed!\n");
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index 9b26ba9..4675339 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -148,6 +148,11 @@ xdr_error:					\
>  	}					\
>  } while (0)
>  
> +static int zero_clientid(clientid_t *clid)
> +{
> +	return ((clid->cl_boot == 0) && (clid->cl_id == 0));
> +}
> +
>  static __be32 *read_buf(struct nfsd4_compoundargs *argp, u32 nbytes)
>  {
>  	/* We want more bytes than seem to be available.
> @@ -585,6 +590,8 @@ nfsd4_decode_lockt(struct nfsd4_compoundargs *argp, struct nfsd4_lockt *lockt)
>  	READ_BUF(lockt->lt_owner.len);
>  	READMEM(lockt->lt_owner.data, lockt->lt_owner.len);
>  
> +	if (argp->minorversion && !zero_clientid(&lockt->lt_clientid))
> +		return nfserr_inval;
>  	DECODE_TAIL;
>  }
>  
> @@ -1003,6 +1010,9 @@ nfsd4_decode_release_lockowner(struct nfsd4_compoundargs *argp, struct nfsd4_rel
>  	READ_BUF(rlockowner->rl_owner.len);
>  	READMEM(rlockowner->rl_owner.data, rlockowner->rl_owner.len);
>  
> +	if (argp->minorversion && !zero_clientid(&rlockowner->rl_clientid))
> +		return nfserr_inval;
> +
>  	DECODE_TAIL;
>  }
>  

[RFC 18/51] nfsd41: access_valid

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

For nfs41, the open share flags are used also for
delegation "wants" and "signals".  Check that they are valid.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c |   27 ++++++++++++++++++++++-----
 1 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 14c1a0e..b772469 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -1255,11 +1255,23 @@ find_file(struct inode *ino)
 	return NULL;
 }
 
-static inline int access_valid(u32 x)
+static inline int access_valid(u32 x, u32 minorversion)
 {
-	if (x < NFS4_SHARE_ACCESS_READ)
+	if ((x & NFS4_SHARE_DENY_MASK) < NFS4_SHARE_ACCESS_READ)
 		return 0;
-	if (x > NFS4_SHARE_ACCESS_BOTH)
+	if ((x & NFS4_SHARE_DENY_MASK) > NFS4_SHARE_ACCESS_BOTH)
+		return 0;
+	x &= ~NFS4_SHARE_DENY_MASK;
+#if defined(CONFIG_NFSD_V4_1)
+	if (minorversion && x) {
+		if ((x & NFS4_SHARE_WANT_MASK) > NFS4_SHARE_WANT_CANCEL)
+			return 0;
+		if ((x & NFS4_SHARE_WHEN_MASK) > NFS4_SHARE_PUSH_DELEG_WHEN_UNCONTENDED)
+			return 0;
+		x &= ~(NFS4_SHARE_WANT_MASK | NFS4_SHARE_WHEN_MASK);
+	}
+#endif /* CONFIG_NFSD_V4_1 */
+	if (x)
 		return 0;
 	return 1;
 }
@@ -1805,7 +1817,7 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
 	__be32 status;
 
 	status = nfserr_inval;
-	if (!access_valid(open->op_share_access)
+	if (!access_valid(open->op_share_access, open->op_minorversion)
 			|| !deny_valid(open->op_share_deny))
 		goto out;
 	/*
@@ -2405,12 +2417,17 @@ nfsd4_open_downgrade(struct svc_rqst *rqstp,
 	__be32 status;
 	struct nfs4_stateid *stp;
 	unsigned int share_access;
+	u32 minorversion = 0;
 
 	dprintk("NFSD: nfsd4_open_downgrade on file %.*s\n", 
 			(int)cstate->current_fh.fh_dentry->d_name.len,
 			cstate->current_fh.fh_dentry->d_name.name);
 
-	if (!access_valid(od->od_share_access)
+#if defined(CONFIG_NFSD_V4_1)
+	if (cstate->current_ses)
+		minorversion = 1;
+#endif /* CONFIG_NFSD_V4_1 */
+	if (!access_valid(od->od_share_access, minorversion)
 			|| !deny_valid(od->od_share_deny))
 		return nfserr_inval;
 
-- 
1.6.0.2

Re: [pnfs] [RFC 18/51] nfsd41: access_valid

[Benny Halevy]

On Nov. 10, 2008, 22:46 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Marc Eshel <eshel@almaden.ibm.com>
> 
> For nfs41, the open share flags are used also for
> delegation "wants" and "signals".  Check that they are valid.
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4state.c |   27 ++++++++++++++++++++++-----
>  1 files changed, 22 insertions(+), 5 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 14c1a0e..b772469 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -1255,11 +1255,23 @@ find_file(struct inode *ino)
>  	return NULL;
>  }
>  
> -static inline int access_valid(u32 x)
> +static inline int access_valid(u32 x, u32 minorversion)
>  {
> -	if (x < NFS4_SHARE_ACCESS_READ)
> +	if ((x & NFS4_SHARE_DENY_MASK) < NFS4_SHARE_ACCESS_READ)
>  		return 0;
> -	if (x > NFS4_SHARE_ACCESS_BOTH)
> +	if ((x & NFS4_SHARE_DENY_MASK) > NFS4_SHARE_ACCESS_BOTH)
> +		return 0;
> +	x &= ~NFS4_SHARE_DENY_MASK;
> +#if defined(CONFIG_NFSD_V4_1)
> +	if (minorversion && x) {
> +		if ((x & NFS4_SHARE_WANT_MASK) > NFS4_SHARE_WANT_CANCEL)
> +			return 0;
> +		if ((x & NFS4_SHARE_WHEN_MASK) > NFS4_SHARE_PUSH_DELEG_WHEN_UNCONTENDED)
> +			return 0;
> +		x &= ~(NFS4_SHARE_WANT_MASK | NFS4_SHARE_WHEN_MASK);
> +	}
> +#endif /* CONFIG_NFSD_V4_1 */
> +	if (x)
>  		return 0;
>  	return 1;
>  }
> @@ -1805,7 +1817,7 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
>  	__be32 status;
>  
>  	status = nfserr_inval;
> -	if (!access_valid(open->op_share_access)
> +	if (!access_valid(open->op_share_access, open->op_minorversion)

review 11-13: use minorversion via struct svc_rqst.rq_resp

>  			|| !deny_valid(open->op_share_deny))
>  		goto out;
>  	/*
> @@ -2405,12 +2417,17 @@ nfsd4_open_downgrade(struct svc_rqst *rqstp,
>  	__be32 status;
>  	struct nfs4_stateid *stp;
>  	unsigned int share_access;
> +	u32 minorversion = 0;

review 11-13: no need for this hack

>  
>  	dprintk("NFSD: nfsd4_open_downgrade on file %.*s\n", 
>  			(int)cstate->current_fh.fh_dentry->d_name.len,
>  			cstate->current_fh.fh_dentry->d_name.name);
>  
> -	if (!access_valid(od->od_share_access)
> +#if defined(CONFIG_NFSD_V4_1)
> +	if (cstate->current_ses)
> +		minorversion = 1;
> +#endif /* CONFIG_NFSD_V4_1 */
> +	if (!access_valid(od->od_share_access, minorversion)
>  			|| !deny_valid(od->od_share_deny))
>  		return nfserr_inval;
>  

[RFC 19/51] nfsd41: add OPEN4_SHARE_ACCESS_WANT nfs4_stateid bmap

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

Separate the access bits from the want bits and enable __set_bit to
work correctly with st_access_bmap.

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index b772469..db79c0c 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -1186,7 +1186,8 @@ init_stateid(struct nfs4_stateid *stp, struct nfs4_file *fp, struct nfsd4_open *
 	stp->st_stateid.si_generation = 0;
 	stp->st_access_bmap = 0;
 	stp->st_deny_bmap = 0;
-	__set_bit(open->op_share_access, &stp->st_access_bmap);
+	__set_bit(open->op_share_access & ~NFS4_SHARE_WANT_MASK,
+		  &stp->st_access_bmap);
 	__set_bit(open->op_share_deny, &stp->st_deny_bmap);
 	stp->st_openstp = NULL;
 }
-- 
1.6.0.2

Re: [pnfs] [RFC 19/51] nfsd41: add OPEN4_SHARE_ACCESS_WANT nfs4_stateid bmap

[Benny Halevy]

On Nov. 10, 2008, 22:47 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> Separate the access bits from the want bits and enable __set_bit to
> work correctly with st_access_bmap.
> 
> Signed-off-by: Andy Adamson<andros@netapp.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4state.c |    3 ++-
>  1 files changed, 2 insertions(+), 1 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index b772469..db79c0c 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -1186,7 +1186,8 @@ init_stateid(struct nfs4_stateid *stp, struct nfs4_file *fp, struct nfsd4_open *
>  	stp->st_stateid.si_generation = 0;
>  	stp->st_access_bmap = 0;
>  	stp->st_deny_bmap = 0;
> -	__set_bit(open->op_share_access, &stp->st_access_bmap);
> +	__set_bit(open->op_share_access & ~NFS4_SHARE_WANT_MASK,

review 11-13: should be (open->op_share_access & NFS4_SHARE_DENY_MASK)
also, define NFS4_SHARE_ACCESS_MASK (== NFS4_SHARE_DENY_MASK)
and use it in access_valid() as well.

> +		  &stp->st_access_bmap);
>  	__set_bit(open->op_share_deny, &stp->st_deny_bmap);
>  	stp->st_openstp = NULL;
>  }

[RFC 20/51] nfsd: last_byte_offset

[Benny Halevy]

refactor the nfs4 server lock code to use last_byte_offset
to compute the last byte covered by the lock.  Check for overflow
so that the last byte is set to NFS4_LENGTH_EOF if offset + len
wraps around.

Also, use NFS4_LENGTH_EOF for ~(u64)0 where appropriate.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c |   42 ++++++++++++++++++++++++++----------------
 1 files changed, 26 insertions(+), 16 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index db79c0c..ca2e3e0 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -2544,6 +2544,26 @@ out:
 #define LOCK_HASH_SIZE             (1 << LOCK_HASH_BITS)
 #define LOCK_HASH_MASK             (LOCK_HASH_SIZE - 1)
 
+static inline u64
+end_offset(u64 start, u64 len)
+{
+	u64 end;
+
+	end = start + len;
+	return end >= start ? end: NFS4_LENGTH_EOF;
+}
+
+/* last octet in a range */
+static inline u64
+last_byte_offset(u64 start, u64 len)
+{
+	u64 end;
+
+	BUG_ON(!len);
+	end = start + len;
+	return end > start ? end - 1: NFS4_LENGTH_EOF;
+}
+
 #define lockownerid_hashval(id) \
         ((id) & LOCK_HASH_MASK)
 
@@ -2646,8 +2666,8 @@ nfs4_set_lock_denied(struct file_lock *fl, struct nfsd4_lock_denied *deny)
 		deny->ld_clientid.cl_id = 0;
 	}
 	deny->ld_start = fl->fl_start;
-	deny->ld_length = ~(u64)0;
-	if (fl->fl_end != ~(u64)0)
+	deny->ld_length = NFS4_LENGTH_EOF;
+	if (fl->fl_end != NFS4_LENGTH_EOF)
 		deny->ld_length = fl->fl_end - fl->fl_start + 1;        
 	deny->ld_type = NFS4_READ_LT;
 	if (fl->fl_type != F_RDLCK)
@@ -2745,7 +2765,7 @@ out:
 static int
 check_lock_length(u64 offset, u64 length)
 {
-	return ((length == 0)  || ((length != ~(u64)0) &&
+	return ((length == 0)  || ((length != NFS4_LENGTH_EOF) &&
 	     LOFF_OVERFLOW(offset, length)));
 }
 
@@ -2870,11 +2890,7 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	file_lock.fl_lmops = &nfsd_posix_mng_ops;
 
 	file_lock.fl_start = lock->lk_offset;
-	if ((lock->lk_length == ~(u64)0) || 
-			LOFF_OVERFLOW(lock->lk_offset, lock->lk_length))
-		file_lock.fl_end = ~(u64)0;
-	else
-		file_lock.fl_end = lock->lk_offset + lock->lk_length - 1;
+	file_lock.fl_end = last_byte_offset(lock->lk_offset, lock->lk_length);
 	nfs4_transform_lock_offset(&file_lock);
 
 	/*
@@ -2978,10 +2994,7 @@ nfsd4_lockt(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	file_lock.fl_lmops = &nfsd_posix_mng_ops;
 
 	file_lock.fl_start = lockt->lt_offset;
-	if ((lockt->lt_length == ~(u64)0) || LOFF_OVERFLOW(lockt->lt_offset, lockt->lt_length))
-		file_lock.fl_end = ~(u64)0;
-	else
-		file_lock.fl_end = lockt->lt_offset + lockt->lt_length - 1;
+	file_lock.fl_end = last_byte_offset(lockt->lt_offset, lockt->lt_length);
 
 	nfs4_transform_lock_offset(&file_lock);
 
@@ -3045,10 +3058,7 @@ nfsd4_locku(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	file_lock.fl_lmops = &nfsd_posix_mng_ops;
 	file_lock.fl_start = locku->lu_offset;
 
-	if ((locku->lu_length == ~(u64)0) || LOFF_OVERFLOW(locku->lu_offset, locku->lu_length))
-		file_lock.fl_end = ~(u64)0;
-	else
-		file_lock.fl_end = locku->lu_offset + locku->lu_length - 1;
+	file_lock.fl_end = last_byte_offset(locku->lu_offset, locku->lu_length);
 	nfs4_transform_lock_offset(&file_lock);
 
 	/*
-- 
1.6.0.2

Re: [pnfs] [RFC 20/51] nfsd: last_byte_offset

[Benny Halevy]

On Nov. 10, 2008, 22:47 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> refactor the nfs4 server lock code to use last_byte_offset
> to compute the last byte covered by the lock.  Check for overflow
> so that the last byte is set to NFS4_LENGTH_EOF if offset + len
> wraps around.
> 
> Also, use NFS4_LENGTH_EOF for ~(u64)0 where appropriate.
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 11-12: check what cases are covered by pynfs testing,
add more cases as Andy captured.

Move this patch forward into the misc. section.

> ---
>  fs/nfsd/nfs4state.c |   42 ++++++++++++++++++++++++++----------------
>  1 files changed, 26 insertions(+), 16 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index db79c0c..ca2e3e0 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -2544,6 +2544,26 @@ out:
>  #define LOCK_HASH_SIZE             (1 << LOCK_HASH_BITS)
>  #define LOCK_HASH_MASK             (LOCK_HASH_SIZE - 1)
>  
> +static inline u64
> +end_offset(u64 start, u64 len)
> +{
> +	u64 end;
> +
> +	end = start + len;
> +	return end >= start ? end: NFS4_LENGTH_EOF;
> +}
> +
> +/* last octet in a range */
> +static inline u64
> +last_byte_offset(u64 start, u64 len)
> +{
> +	u64 end;
> +
> +	BUG_ON(!len);
> +	end = start + len;
> +	return end > start ? end - 1: NFS4_LENGTH_EOF;
> +}
> +
>  #define lockownerid_hashval(id) \
>          ((id) & LOCK_HASH_MASK)
>  
> @@ -2646,8 +2666,8 @@ nfs4_set_lock_denied(struct file_lock *fl, struct nfsd4_lock_denied *deny)
>  		deny->ld_clientid.cl_id = 0;
>  	}
>  	deny->ld_start = fl->fl_start;
> -	deny->ld_length = ~(u64)0;
> -	if (fl->fl_end != ~(u64)0)
> +	deny->ld_length = NFS4_LENGTH_EOF;
> +	if (fl->fl_end != NFS4_LENGTH_EOF)
>  		deny->ld_length = fl->fl_end - fl->fl_start + 1;        
>  	deny->ld_type = NFS4_READ_LT;
>  	if (fl->fl_type != F_RDLCK)
> @@ -2745,7 +2765,7 @@ out:
>  static int
>  check_lock_length(u64 offset, u64 length)
>  {
> -	return ((length == 0)  || ((length != ~(u64)0) &&
> +	return ((length == 0)  || ((length != NFS4_LENGTH_EOF) &&
>  	     LOFF_OVERFLOW(offset, length)));
>  }
>  
> @@ -2870,11 +2890,7 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	file_lock.fl_lmops = &nfsd_posix_mng_ops;
>  
>  	file_lock.fl_start = lock->lk_offset;
> -	if ((lock->lk_length == ~(u64)0) || 
> -			LOFF_OVERFLOW(lock->lk_offset, lock->lk_length))
> -		file_lock.fl_end = ~(u64)0;
> -	else
> -		file_lock.fl_end = lock->lk_offset + lock->lk_length - 1;
> +	file_lock.fl_end = last_byte_offset(lock->lk_offset, lock->lk_length);
>  	nfs4_transform_lock_offset(&file_lock);
>  
>  	/*
> @@ -2978,10 +2994,7 @@ nfsd4_lockt(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	file_lock.fl_lmops = &nfsd_posix_mng_ops;
>  
>  	file_lock.fl_start = lockt->lt_offset;
> -	if ((lockt->lt_length == ~(u64)0) || LOFF_OVERFLOW(lockt->lt_offset, lockt->lt_length))
> -		file_lock.fl_end = ~(u64)0;
> -	else
> -		file_lock.fl_end = lockt->lt_offset + lockt->lt_length - 1;
> +	file_lock.fl_end = last_byte_offset(lockt->lt_offset, lockt->lt_length);
>  
>  	nfs4_transform_lock_offset(&file_lock);
>  
> @@ -3045,10 +3058,7 @@ nfsd4_locku(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	file_lock.fl_lmops = &nfsd_posix_mng_ops;
>  	file_lock.fl_start = locku->lu_offset;
>  
> -	if ((locku->lu_length == ~(u64)0) || LOFF_OVERFLOW(locku->lu_offset, locku->lu_length))
> -		file_lock.fl_end = ~(u64)0;
> -	else
> -		file_lock.fl_end = locku->lu_offset + locku->lu_length - 1;
> +	file_lock.fl_end = last_byte_offset(locku->lu_offset, locku->lu_length);
>  	nfs4_transform_lock_offset(&file_lock);
>  
>  	/*

[RFC 21/51] nfsd41: xdr stubs

[Benny Halevy]

define stubs for exchange_id, create_session, destroy_session, and
sequence.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4xdr.c         |  151 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/nfsd/xdr4.h |   24 +++++++
 2 files changed, 175 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 4675339..fc613cd 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -1016,6 +1016,36 @@ nfsd4_decode_release_lockowner(struct nfsd4_compoundargs *argp, struct nfsd4_rel
 	DECODE_TAIL;
 }
 
+#if defined(CONFIG_NFSD_V4_1)
+static __be32
+nfsd4_decode_exchange_id(struct nfsd4_compoundargs *argp,
+			 struct nfsd4_exchange_id *clid)
+{
+	return nfserr_opnotsupp;	/* stub */
+}
+
+static __be32
+nfsd4_decode_create_session(struct nfsd4_compoundargs *argp,
+			    struct nfsd4_create_session *sess)
+{
+	return nfserr_opnotsupp;	/* stub */
+}
+
+static __be32
+nfsd4_decode_destroy_session(struct nfsd4_compoundargs *argp,
+			     struct nfsd4_destroy_session *destroy_session)
+{
+	return nfserr_opnotsupp;	/* stub */
+}
+
+static __be32
+nfsd4_decode_sequence(struct nfsd4_compoundargs *argp,
+		      struct nfsd4_sequence *seq)
+{
+	return nfserr_opnotsupp;	/* stub */
+}
+#endif /* CONFIG_NFSD_V4_1 */
+
 static __be32
 nfsd4_decode_noop(struct nfsd4_compoundargs *argp, void *p)
 {
@@ -1070,6 +1100,69 @@ static nfsd4_dec nfsd4_dec_ops[] = {
 	[OP_RELEASE_LOCKOWNER]	= (nfsd4_dec)nfsd4_decode_release_lockowner,
 };
 
+#if defined(CONFIG_NFSD_V4_1)
+static nfsd4_dec nfsd41_dec_ops[] = {
+	[OP_ACCESS]		(nfsd4_dec)nfsd4_decode_access,
+	[OP_CLOSE]		(nfsd4_dec)nfsd4_decode_close,
+	[OP_COMMIT]		(nfsd4_dec)nfsd4_decode_commit,
+	[OP_CREATE]		(nfsd4_dec)nfsd4_decode_create,
+	[OP_DELEGPURGE]		(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_DELEGRETURN]	(nfsd4_dec)nfsd4_decode_delegreturn,
+	[OP_GETATTR]		(nfsd4_dec)nfsd4_decode_getattr,
+	[OP_GETFH]		(nfsd4_dec)nfsd4_decode_noop,
+	[OP_LINK]		(nfsd4_dec)nfsd4_decode_link,
+	[OP_LOCK]		(nfsd4_dec)nfsd4_decode_lock,
+	[OP_LOCKT]		(nfsd4_dec)nfsd4_decode_lockt,
+	[OP_LOCKU]		(nfsd4_dec)nfsd4_decode_locku,
+	[OP_LOOKUP]		(nfsd4_dec)nfsd4_decode_lookup,
+	[OP_LOOKUPP]		(nfsd4_dec)nfsd4_decode_noop,
+	[OP_NVERIFY]		(nfsd4_dec)nfsd4_decode_verify,
+	[OP_OPEN]		(nfsd4_dec)nfsd4_decode_open,
+	[OP_OPENATTR]		(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_OPEN_CONFIRM]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_OPEN_DOWNGRADE]	(nfsd4_dec)nfsd4_decode_open_downgrade,
+	[OP_PUTFH]		(nfsd4_dec)nfsd4_decode_putfh,
+	[OP_PUTPUBFH]		(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_PUTROOTFH]		(nfsd4_dec)nfsd4_decode_noop,
+	[OP_READ]		(nfsd4_dec)nfsd4_decode_read,
+	[OP_READDIR]		(nfsd4_dec)nfsd4_decode_readdir,
+	[OP_READLINK]		(nfsd4_dec)nfsd4_decode_noop,
+	[OP_REMOVE]		(nfsd4_dec)nfsd4_decode_remove,
+	[OP_RENAME]		(nfsd4_dec)nfsd4_decode_rename,
+	[OP_RENEW]		(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_RESTOREFH]		(nfsd4_dec)nfsd4_decode_noop,
+	[OP_SAVEFH]		(nfsd4_dec)nfsd4_decode_noop,
+	[OP_SECINFO]		(nfsd4_dec)nfsd4_decode_secinfo,
+	[OP_SETATTR]		(nfsd4_dec)nfsd4_decode_setattr,
+	[OP_SETCLIENTID]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_SETCLIENTID_CONFIRM](nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_VERIFY]		(nfsd4_dec)nfsd4_decode_verify,
+	[OP_WRITE]		(nfsd4_dec)nfsd4_decode_write,
+	[OP_RELEASE_LOCKOWNER]	(nfsd4_dec)nfsd4_decode_notsupp,
+
+	/* new operations for NFSv4.1 */
+	[OP_BACKCHANNEL_CTL]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_BIND_CONN_TO_SESSION](nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_EXCHANGE_ID]	(nfsd4_dec)nfsd4_decode_exchange_id,
+	[OP_CREATE_SESSION]	(nfsd4_dec)nfsd4_decode_create_session,
+	[OP_DESTROY_SESSION]	(nfsd4_dec)nfsd4_decode_destroy_session,
+	[OP_FREE_STATEID]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_GET_DIR_DELEGATION]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_GETDEVICEINFO]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_GETDEVICELIST]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_LAYOUTCOMMIT]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_LAYOUTGET]		(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_LAYOUTRETURN]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_SECINFO_NO_NAME]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_SEQUENCE]		(nfsd4_dec)nfsd4_decode_sequence,
+	[OP_SET_SSV]		(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_TEST_STATEID]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_WANT_DELEGATION]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_DESTROY_CLIENTID]	(nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_RECLAIM_COMPLETE]	(nfsd4_dec)nfsd4_decode_notsupp,
+};
+#endif /* CONFIG_NFSD_V4_1 */
+
 struct nfsd4_minorversion_ops {
 	nfsd4_dec *decoders;
 	int nops;
@@ -1077,6 +1170,9 @@ struct nfsd4_minorversion_ops {
 
 static struct nfsd4_minorversion_ops nfsd4_minorversion[] = {
 	[0] = { nfsd4_dec_ops, ARRAY_SIZE(nfsd4_dec_ops) },
+#if defined(CONFIG_NFSD_V4_1)
+	[1] = { nfsd41_dec_ops, ARRAY_SIZE(nfsd41_dec_ops) },
+#endif /* CONFIG_NFSD_V4_1 */
 };
 
 static __be32
@@ -2591,6 +2687,40 @@ nfsd4_encode_write(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_w
 	return nfserr;
 }
 
+#if defined(CONFIG_NFSD_V4_1)
+static __be32
+nfsd4_encode_exchange_id(struct nfsd4_compoundres *resp, int nfserr,
+			 struct nfsd4_exchange_id *exid)
+{
+	/* stub */
+	return nfserr;
+}
+
+static __be32
+nfsd4_encode_create_session(struct nfsd4_compoundres *resp, int nfserr,
+			    struct nfsd4_create_session *sess)
+{
+	/* stub */
+	return nfserr;
+}
+
+static __be32
+nfsd4_encode_destroy_session(struct nfsd4_compoundres *resp, int nfserr,
+			     struct nfsd4_destroy_session *destroy_session)
+{
+	/* stub */
+	return nfserr;
+}
+
+static __be32
+nfsd4_encode_sequence(struct nfsd4_compoundres *resp, int nfserr,
+		      struct nfsd4_sequence *seq)
+{
+	/* stub */
+	return nfserr;
+}
+#endif /* CONFIG_NFSD_V4_1 */
+
 static __be32
 nfsd4_encode_noop(struct nfsd4_compoundres *resp, __be32 nfserr, void *p)
 {
@@ -2636,6 +2766,27 @@ static nfsd4_enc nfsd4_enc_ops[] = {
 	[OP_VERIFY]		= (nfsd4_enc)nfsd4_encode_noop,
 	[OP_WRITE]		= (nfsd4_enc)nfsd4_encode_write,
 	[OP_RELEASE_LOCKOWNER]	= (nfsd4_enc)nfsd4_encode_noop,
+#if defined(CONFIG_NFSD_V4_1)
+	[OP_BACKCHANNEL_CTL]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_BIND_CONN_TO_SESSION] = (nfsd4_enc)nfsd4_encode_noop,
+	[OP_EXCHANGE_ID]	= (nfsd4_enc)nfsd4_encode_exchange_id,
+	[OP_CREATE_SESSION]	= (nfsd4_enc)nfsd4_encode_create_session,
+	[OP_DESTROY_SESSION]	= (nfsd4_enc)nfsd4_encode_destroy_session,
+	[OP_FREE_STATEID]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_GET_DIR_DELEGATION]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_GETDEVICEINFO]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_GETDEVICELIST]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_LAYOUTCOMMIT]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_LAYOUTGET]		= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_LAYOUTRETURN]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_SECINFO_NO_NAME]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_SEQUENCE]		= (nfsd4_enc)nfsd4_encode_sequence,
+	[OP_SET_SSV]		= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_TEST_STATEID]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_WANT_DELEGATION]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_DESTROY_CLIENTID]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_RECLAIM_COMPLETE]	= (nfsd4_enc)nfsd4_encode_noop,
+#endif /* CONFIG_NFSD_V4_1 */
 };
 
 void
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index 81c249c..b864007 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -352,6 +352,24 @@ struct nfsd4_write {
 	u32		wr_minorversion;    /* processing */
 };
 
+#if defined(CONFIG_NFSD_V4_1)
+struct nfsd4_exchange_id {
+	int	foo;	/* stub */
+};
+
+struct nfsd4_create_session {
+	int	foo;	/* stub */
+};
+
+struct nfsd4_sequence {
+	int	foo;	/* stub */
+};
+
+struct nfsd4_destroy_session {
+	int	foo;	/* stub */
+};
+#endif /* CONFIG_NFSD_V4_1 */
+
 struct nfsd4_op {
 	int					opnum;
 	__be32					status;
@@ -386,6 +404,12 @@ struct nfsd4_op {
 		struct nfsd4_verify		verify;
 		struct nfsd4_write		write;
 		struct nfsd4_release_lockowner	release_lockowner;
+#if defined(CONFIG_NFSD_V4_1)
+		struct nfsd4_exchange_id	exchange_id;
+		struct nfsd4_create_session	create_session;
+		struct nfsd4_destroy_session	destroy_session;
+		struct nfsd4_sequence		sequence;
+#endif /* CONFIG_NFSD_V4_1 */
 	} u;
 	struct nfs4_replay *			replay;
 };
-- 
1.6.0.2

Re: [pnfs] [RFC 21/51] nfsd41: xdr stubs

[Benny Halevy]

On Nov. 10, 2008, 22:47 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> define stubs for exchange_id, create_session, destroy_session, and
> sequence.
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4xdr.c         |  151 +++++++++++++++++++++++++++++++++++++++++++++
>  include/linux/nfsd/xdr4.h |   24 +++++++
>  2 files changed, 175 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index 4675339..fc613cd 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -1016,6 +1016,36 @@ nfsd4_decode_release_lockowner(struct nfsd4_compoundargs *argp, struct nfsd4_rel
>  	DECODE_TAIL;
>  }
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +static __be32
> +nfsd4_decode_exchange_id(struct nfsd4_compoundargs *argp,
> +			 struct nfsd4_exchange_id *clid)
> +{
> +	return nfserr_opnotsupp;	/* stub */
> +}
> +
> +static __be32
> +nfsd4_decode_create_session(struct nfsd4_compoundargs *argp,
> +			    struct nfsd4_create_session *sess)
> +{
> +	return nfserr_opnotsupp;	/* stub */
> +}
> +
> +static __be32
> +nfsd4_decode_destroy_session(struct nfsd4_compoundargs *argp,
> +			     struct nfsd4_destroy_session *destroy_session)
> +{
> +	return nfserr_opnotsupp;	/* stub */
> +}
> +
> +static __be32
> +nfsd4_decode_sequence(struct nfsd4_compoundargs *argp,
> +		      struct nfsd4_sequence *seq)
> +{
> +	return nfserr_opnotsupp;	/* stub */
> +}
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  static __be32
>  nfsd4_decode_noop(struct nfsd4_compoundargs *argp, void *p)
>  {
> @@ -1070,6 +1100,69 @@ static nfsd4_dec nfsd4_dec_ops[] = {
>  	[OP_RELEASE_LOCKOWNER]	= (nfsd4_dec)nfsd4_decode_release_lockowner,
>  };
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +static nfsd4_dec nfsd41_dec_ops[] = {
> +	[OP_ACCESS]		(nfsd4_dec)nfsd4_decode_access,
> +	[OP_CLOSE]		(nfsd4_dec)nfsd4_decode_close,
> +	[OP_COMMIT]		(nfsd4_dec)nfsd4_decode_commit,
> +	[OP_CREATE]		(nfsd4_dec)nfsd4_decode_create,
> +	[OP_DELEGPURGE]		(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_DELEGRETURN]	(nfsd4_dec)nfsd4_decode_delegreturn,
> +	[OP_GETATTR]		(nfsd4_dec)nfsd4_decode_getattr,
> +	[OP_GETFH]		(nfsd4_dec)nfsd4_decode_noop,
> +	[OP_LINK]		(nfsd4_dec)nfsd4_decode_link,
> +	[OP_LOCK]		(nfsd4_dec)nfsd4_decode_lock,
> +	[OP_LOCKT]		(nfsd4_dec)nfsd4_decode_lockt,
> +	[OP_LOCKU]		(nfsd4_dec)nfsd4_decode_locku,
> +	[OP_LOOKUP]		(nfsd4_dec)nfsd4_decode_lookup,
> +	[OP_LOOKUPP]		(nfsd4_dec)nfsd4_decode_noop,
> +	[OP_NVERIFY]		(nfsd4_dec)nfsd4_decode_verify,
> +	[OP_OPEN]		(nfsd4_dec)nfsd4_decode_open,
> +	[OP_OPENATTR]		(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_OPEN_CONFIRM]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_OPEN_DOWNGRADE]	(nfsd4_dec)nfsd4_decode_open_downgrade,
> +	[OP_PUTFH]		(nfsd4_dec)nfsd4_decode_putfh,
> +	[OP_PUTPUBFH]		(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_PUTROOTFH]		(nfsd4_dec)nfsd4_decode_noop,
> +	[OP_READ]		(nfsd4_dec)nfsd4_decode_read,
> +	[OP_READDIR]		(nfsd4_dec)nfsd4_decode_readdir,
> +	[OP_READLINK]		(nfsd4_dec)nfsd4_decode_noop,
> +	[OP_REMOVE]		(nfsd4_dec)nfsd4_decode_remove,
> +	[OP_RENAME]		(nfsd4_dec)nfsd4_decode_rename,
> +	[OP_RENEW]		(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_RESTOREFH]		(nfsd4_dec)nfsd4_decode_noop,
> +	[OP_SAVEFH]		(nfsd4_dec)nfsd4_decode_noop,
> +	[OP_SECINFO]		(nfsd4_dec)nfsd4_decode_secinfo,
> +	[OP_SETATTR]		(nfsd4_dec)nfsd4_decode_setattr,
> +	[OP_SETCLIENTID]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_SETCLIENTID_CONFIRM](nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_VERIFY]		(nfsd4_dec)nfsd4_decode_verify,
> +	[OP_WRITE]		(nfsd4_dec)nfsd4_decode_write,
> +	[OP_RELEASE_LOCKOWNER]	(nfsd4_dec)nfsd4_decode_notsupp,
> +
> +	/* new operations for NFSv4.1 */
> +	[OP_BACKCHANNEL_CTL]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_BIND_CONN_TO_SESSION](nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_EXCHANGE_ID]	(nfsd4_dec)nfsd4_decode_exchange_id,
> +	[OP_CREATE_SESSION]	(nfsd4_dec)nfsd4_decode_create_session,
> +	[OP_DESTROY_SESSION]	(nfsd4_dec)nfsd4_decode_destroy_session,
> +	[OP_FREE_STATEID]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_GET_DIR_DELEGATION]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_GETDEVICEINFO]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_GETDEVICELIST]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_LAYOUTCOMMIT]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_LAYOUTGET]		(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_LAYOUTRETURN]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_SECINFO_NO_NAME]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_SEQUENCE]		(nfsd4_dec)nfsd4_decode_sequence,
> +	[OP_SET_SSV]		(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_TEST_STATEID]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_WANT_DELEGATION]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_DESTROY_CLIENTID]	(nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_RECLAIM_COMPLETE]	(nfsd4_dec)nfsd4_decode_notsupp,
> +};
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  struct nfsd4_minorversion_ops {
>  	nfsd4_dec *decoders;
>  	int nops;
> @@ -1077,6 +1170,9 @@ struct nfsd4_minorversion_ops {
>  
>  static struct nfsd4_minorversion_ops nfsd4_minorversion[] = {
>  	[0] = { nfsd4_dec_ops, ARRAY_SIZE(nfsd4_dec_ops) },
> +#if defined(CONFIG_NFSD_V4_1)
> +	[1] = { nfsd41_dec_ops, ARRAY_SIZE(nfsd41_dec_ops) },
> +#endif /* CONFIG_NFSD_V4_1 */
>  };
>  
>  static __be32
> @@ -2591,6 +2687,40 @@ nfsd4_encode_write(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_w
>  	return nfserr;
>  }
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +static __be32
> +nfsd4_encode_exchange_id(struct nfsd4_compoundres *resp, int nfserr,
> +			 struct nfsd4_exchange_id *exid)
> +{
> +	/* stub */
> +	return nfserr;
> +}
> +
> +static __be32
> +nfsd4_encode_create_session(struct nfsd4_compoundres *resp, int nfserr,
> +			    struct nfsd4_create_session *sess)
> +{
> +	/* stub */
> +	return nfserr;
> +}
> +
> +static __be32
> +nfsd4_encode_destroy_session(struct nfsd4_compoundres *resp, int nfserr,
> +			     struct nfsd4_destroy_session *destroy_session)
> +{
> +	/* stub */
> +	return nfserr;
> +}
> +
> +static __be32
> +nfsd4_encode_sequence(struct nfsd4_compoundres *resp, int nfserr,
> +		      struct nfsd4_sequence *seq)
> +{
> +	/* stub */
> +	return nfserr;
> +}
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  static __be32
>  nfsd4_encode_noop(struct nfsd4_compoundres *resp, __be32 nfserr, void *p)
>  {

review 11-13: add comment about why we don't need nfsd4_enc_ops
per minorversion like we do for the decoding vector.

> @@ -2636,6 +2766,27 @@ static nfsd4_enc nfsd4_enc_ops[] = {
>  	[OP_VERIFY]		= (nfsd4_enc)nfsd4_encode_noop,
>  	[OP_WRITE]		= (nfsd4_enc)nfsd4_encode_write,
>  	[OP_RELEASE_LOCKOWNER]	= (nfsd4_enc)nfsd4_encode_noop,
> +#if defined(CONFIG_NFSD_V4_1)
> +	[OP_BACKCHANNEL_CTL]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_BIND_CONN_TO_SESSION] = (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_EXCHANGE_ID]	= (nfsd4_enc)nfsd4_encode_exchange_id,
> +	[OP_CREATE_SESSION]	= (nfsd4_enc)nfsd4_encode_create_session,
> +	[OP_DESTROY_SESSION]	= (nfsd4_enc)nfsd4_encode_destroy_session,
> +	[OP_FREE_STATEID]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_GET_DIR_DELEGATION]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_GETDEVICEINFO]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_GETDEVICELIST]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_LAYOUTCOMMIT]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_LAYOUTGET]		= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_LAYOUTRETURN]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_SECINFO_NO_NAME]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_SEQUENCE]		= (nfsd4_enc)nfsd4_encode_sequence,
> +	[OP_SET_SSV]		= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_TEST_STATEID]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_WANT_DELEGATION]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_DESTROY_CLIENTID]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_RECLAIM_COMPLETE]	= (nfsd4_enc)nfsd4_encode_noop,
> +#endif /* CONFIG_NFSD_V4_1 */
>  };
>  
>  void
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index 81c249c..b864007 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -352,6 +352,24 @@ struct nfsd4_write {
>  	u32		wr_minorversion;    /* processing */
>  };
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +struct nfsd4_exchange_id {
> +	int	foo;	/* stub */
> +};
> +
> +struct nfsd4_create_session {
> +	int	foo;	/* stub */
> +};
> +
> +struct nfsd4_sequence {
> +	int	foo;	/* stub */
> +};
> +
> +struct nfsd4_destroy_session {
> +	int	foo;	/* stub */
> +};
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  struct nfsd4_op {
>  	int					opnum;
>  	__be32					status;
> @@ -386,6 +404,12 @@ struct nfsd4_op {
>  		struct nfsd4_verify		verify;
>  		struct nfsd4_write		write;
>  		struct nfsd4_release_lockowner	release_lockowner;
> +#if defined(CONFIG_NFSD_V4_1)
> +		struct nfsd4_exchange_id	exchange_id;
> +		struct nfsd4_create_session	create_session;
> +		struct nfsd4_destroy_session	destroy_session;
> +		struct nfsd4_sequence		sequence;
> +#endif /* CONFIG_NFSD_V4_1 */
>  	} u;
>  	struct nfs4_replay *			replay;
>  };

[RFC 22/51] nfsd41: proc stubs

[Benny Halevy]

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c        |   30 +++++++++++++++++++++++++++++-
 fs/nfsd/nfs4state.c       |   34 ++++++++++++++++++++++++++++++++++
 include/linux/nfsd/xdr4.h |   14 ++++++++++++++
 3 files changed, 77 insertions(+), 1 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 26aae5d..6f6d221 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1021,7 +1021,13 @@ out:
 	return status;
 }
 
-static struct nfsd4_operation nfsd4_ops[OP_RELEASE_LOCKOWNER+1] = {
+#if defined(CONFIG_NFSD_V4_1)
+#define NFSD4_LAST_OP OP_RECLAIM_COMPLETE
+#else /* CONFIG_NFSD_V4_1 */
+#define NFSD4_LAST_OP OP_RELEASE_LOCKOWNER
+#endif /* CONFIG_NFSD_V4_1 */
+
+static struct nfsd4_operation nfsd4_ops[NFSD4_LAST_OP+1] = {
 	[OP_ACCESS] = {
 		.op_func = (nfsd4op_func)nfsd4_access,
 		.op_name = "OP_ACCESS",
@@ -1171,6 +1177,28 @@ static struct nfsd4_operation nfsd4_ops[OP_RELEASE_LOCKOWNER+1] = {
 		.op_flags = ALLOWED_WITHOUT_FH | ALLOWED_ON_ABSENT_FS,
 		.op_name = "OP_RELEASE_LOCKOWNER",
 	},
+#if defined(CONFIG_NFSD_V4_1)
+	[OP_EXCHANGE_ID] = {
+		.op_func = (nfsd4op_func)nfsd4_exchange_id,
+		.op_flags = ALLOWED_WITHOUT_FH,
+		.op_name = "OP_EXCHANGE_ID",
+	},
+	[OP_CREATE_SESSION] = {
+		.op_func = (nfsd4op_func)nfsd4_create_session,
+		.op_flags = ALLOWED_WITHOUT_FH,
+		.op_name = "OP_CREATE_SESSION",
+	},
+	[OP_DESTROY_SESSION] = {
+		.op_func = (nfsd4op_func)nfsd4_destroy_session,
+		.op_flags = ALLOWED_WITHOUT_FH,
+		.op_name = "OP_DESTROY_SESSION",
+	},
+	[OP_SEQUENCE] = {
+		.op_func = (nfsd4op_func)nfsd4_sequence,
+		.op_flags = ALLOWED_WITHOUT_FH,
+		.op_name = "OP_SEQUENCE",
+	},
+#endif /* CONFIG_NFSD_V4_1 */
 };
 
 static const char *nfsd4_op_name(unsigned opnum)
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index ca2e3e0..5c96c39 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -780,6 +780,40 @@ out_err:
 	return;
 }
 
+#if defined(CONFIG_NFSD_V4_1)
+__be32
+nfsd4_exchange_id(struct svc_rqst *rqstp,
+		  struct nfsd4_compound_state *cstate,
+		  struct nfsd4_exchange_id *clid)
+{
+	return -1;	/* stub */
+}
+
+__be32
+nfsd4_create_session(struct svc_rqst *rqstp,
+		     struct nfsd4_compound_state *cstate,
+		     struct nfsd4_create_session *cr_ses)
+{
+	return -1;	/* stub */
+}
+
+__be32
+nfsd4_destroy_session(struct svc_rqst *r,
+		      struct nfsd4_compound_state *cstate,
+		      struct nfsd4_destroy_session *sessionid)
+{
+	return -1;	/* stub */
+}
+
+__be32
+nfsd4_sequence(struct svc_rqst *r,
+	       struct nfsd4_compound_state *cstate,
+	       struct nfsd4_sequence *seq)
+{
+	return -1;	/* stub */
+}
+#endif /* CONFIG_NFSD_V4_1 */
+
 __be32
 nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 		  struct nfsd4_setclientid *setclid)
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index b864007..d7dbdab 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -480,6 +480,20 @@ extern __be32 nfsd4_setclientid(struct svc_rqst *rqstp,
 extern __be32 nfsd4_setclientid_confirm(struct svc_rqst *rqstp,
 		struct nfsd4_compound_state *,
 		struct nfsd4_setclientid_confirm *setclientid_confirm);
+#if defined(CONFIG_NFSD_V4_1)
+extern __be32 nfsd4_exchange_id(struct svc_rqst *rqstp,
+		struct nfsd4_compound_state *,
+		struct nfsd4_exchange_id *);
+extern __be32 nfsd4_create_session(struct svc_rqst *,
+		struct nfsd4_compound_state *,
+		struct nfsd4_create_session *);
+extern __be32 nfsd4_sequence(struct svc_rqst *,
+		struct nfsd4_compound_state *,
+		struct nfsd4_sequence *);
+extern __be32 nfsd4_destroy_session(struct svc_rqst *,
+		struct nfsd4_compound_state *,
+		struct nfsd4_destroy_session *);
+#endif /* CONFIG_NFSD_V4_1 */
 extern __be32 nfsd4_process_open1(struct nfsd4_open *open);
 extern __be32 nfsd4_process_open2(struct svc_rqst *rqstp,
 		struct svc_fh *current_fh, struct nfsd4_open *open);
-- 
1.6.0.2

Re: [pnfs] [RFC 22/51] nfsd41: proc stubs

[Benny Halevy]

On Nov. 10, 2008, 22:48 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4proc.c        |   30 +++++++++++++++++++++++++++++-
>  fs/nfsd/nfs4state.c       |   34 ++++++++++++++++++++++++++++++++++
>  include/linux/nfsd/xdr4.h |   14 ++++++++++++++
>  3 files changed, 77 insertions(+), 1 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index 26aae5d..6f6d221 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -1021,7 +1021,13 @@ out:
>  	return status;
>  }
>  
> -static struct nfsd4_operation nfsd4_ops[OP_RELEASE_LOCKOWNER+1] = {
> +#if defined(CONFIG_NFSD_V4_1)
> +#define NFSD4_LAST_OP OP_RECLAIM_COMPLETE
> +#else /* CONFIG_NFSD_V4_1 */
> +#define NFSD4_LAST_OP OP_RELEASE_LOCKOWNER
> +#endif /* CONFIG_NFSD_V4_1 */
> +
> +static struct nfsd4_operation nfsd4_ops[NFSD4_LAST_OP+1] = {

review 11-13: get rid of NFSD4_LAST_OP

>  	[OP_ACCESS] = {
>  		.op_func = (nfsd4op_func)nfsd4_access,
>  		.op_name = "OP_ACCESS",
> @@ -1171,6 +1177,28 @@ static struct nfsd4_operation nfsd4_ops[OP_RELEASE_LOCKOWNER+1] = {
>  		.op_flags = ALLOWED_WITHOUT_FH | ALLOWED_ON_ABSENT_FS,
>  		.op_name = "OP_RELEASE_LOCKOWNER",
>  	},
> +#if defined(CONFIG_NFSD_V4_1)
> +	[OP_EXCHANGE_ID] = {
> +		.op_func = (nfsd4op_func)nfsd4_exchange_id,
> +		.op_flags = ALLOWED_WITHOUT_FH,
> +		.op_name = "OP_EXCHANGE_ID",
> +	},
> +	[OP_CREATE_SESSION] = {
> +		.op_func = (nfsd4op_func)nfsd4_create_session,
> +		.op_flags = ALLOWED_WITHOUT_FH,
> +		.op_name = "OP_CREATE_SESSION",
> +	},
> +	[OP_DESTROY_SESSION] = {
> +		.op_func = (nfsd4op_func)nfsd4_destroy_session,
> +		.op_flags = ALLOWED_WITHOUT_FH,
> +		.op_name = "OP_DESTROY_SESSION",
> +	},
> +	[OP_SEQUENCE] = {
> +		.op_func = (nfsd4op_func)nfsd4_sequence,
> +		.op_flags = ALLOWED_WITHOUT_FH,
> +		.op_name = "OP_SEQUENCE",
> +	},
> +#endif /* CONFIG_NFSD_V4_1 */
>  };
>  
>  static const char *nfsd4_op_name(unsigned opnum)
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index ca2e3e0..5c96c39 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -780,6 +780,40 @@ out_err:
>  	return;
>  }
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +__be32
> +nfsd4_exchange_id(struct svc_rqst *rqstp,
> +		  struct nfsd4_compound_state *cstate,
> +		  struct nfsd4_exchange_id *clid)
> +{
> +	return -1;	/* stub */
> +}
> +
> +__be32
> +nfsd4_create_session(struct svc_rqst *rqstp,
> +		     struct nfsd4_compound_state *cstate,
> +		     struct nfsd4_create_session *cr_ses)
> +{
> +	return -1;	/* stub */
> +}
> +
> +__be32
> +nfsd4_destroy_session(struct svc_rqst *r,
> +		      struct nfsd4_compound_state *cstate,
> +		      struct nfsd4_destroy_session *sessionid)
> +{
> +	return -1;	/* stub */
> +}
> +
> +__be32
> +nfsd4_sequence(struct svc_rqst *r,
> +	       struct nfsd4_compound_state *cstate,
> +	       struct nfsd4_sequence *seq)
> +{
> +	return -1;	/* stub */
> +}
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  __be32
>  nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  		  struct nfsd4_setclientid *setclid)
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index b864007..d7dbdab 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -480,6 +480,20 @@ extern __be32 nfsd4_setclientid(struct svc_rqst *rqstp,
>  extern __be32 nfsd4_setclientid_confirm(struct svc_rqst *rqstp,
>  		struct nfsd4_compound_state *,
>  		struct nfsd4_setclientid_confirm *setclientid_confirm);
> +#if defined(CONFIG_NFSD_V4_1)
> +extern __be32 nfsd4_exchange_id(struct svc_rqst *rqstp,
> +		struct nfsd4_compound_state *,
> +		struct nfsd4_exchange_id *);
> +extern __be32 nfsd4_create_session(struct svc_rqst *,
> +		struct nfsd4_compound_state *,
> +		struct nfsd4_create_session *);
> +extern __be32 nfsd4_sequence(struct svc_rqst *,
> +		struct nfsd4_compound_state *,
> +		struct nfsd4_sequence *);
> +extern __be32 nfsd4_destroy_session(struct svc_rqst *,
> +		struct nfsd4_compound_state *,
> +		struct nfsd4_destroy_session *);
> +#endif /* CONFIG_NFSD_V4_1 */
>  extern __be32 nfsd4_process_open1(struct nfsd4_open *open);
>  extern __be32 nfsd4_process_open2(struct svc_rqst *rqstp,
>  		struct svc_fh *current_fh, struct nfsd4_open *open);

[RFC 23/51] nfsd41: exchange_id operation

[Benny Halevy]

Implement the exchange_id operation confoming to
http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26

Based on the client provided name, hash a client id.
If a confirmed one is found, compare the op's creds and
verifier.  If the creds match and the verifier is different
then expire the old client (client re-incarnated), otherwise,
if both match, assume it's a replay and ignore it.

If an unconfirmed client is found, then copy the new creds
and verifer if need update, otherwise assume replay.

The client is moved to a confirmed state on create_session.

In the nfs41 branch set the exchange_id flags to
EXCHGID4_FLAG_USE_NON_PNFS | EXCHGID4_FLAG_SUPP_MOVED_REFER
(pNFS is not supported, Referrals are supported,
Migration is not.).

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c        |  113 ++++++++++++++++++++++++++++++++++-
 fs/nfsd/nfs4xdr.c          |  143 +++++++++++++++++++++++++++++++++++++++++++-
 include/linux/nfsd/state.h |    4 +
 include/linux/nfsd/xdr4.h  |    7 ++-
 4 files changed, 263 insertions(+), 4 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 5c96c39..76b2870 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -781,12 +781,123 @@ out_err:
 }
 
 #if defined(CONFIG_NFSD_V4_1)
+/*
+ * Set the exchange_id flags returned by the server.
+ */
+static void
+nfsd4_set_ex_flags(struct nfs4_client *new, struct nfsd4_exchange_id *clid)
+{
+	/* pNFS is not supported */
+	new->cl_exchange_flags |= EXCHGID4_FLAG_USE_NON_PNFS;
+
+	/* Referrals are supported, Migration is not. */
+	new->cl_exchange_flags |= EXCHGID4_FLAG_SUPP_MOVED_REFER;
+
+	/* set the wire flags to return to client. */
+	clid->flags = new->cl_exchange_flags;
+}
+
 __be32
 nfsd4_exchange_id(struct svc_rqst *rqstp,
 		  struct nfsd4_compound_state *cstate,
 		  struct nfsd4_exchange_id *clid)
 {
-	return -1;	/* stub */
+	struct nfs4_client *unconf, *conf, *new;
+	int status;
+	unsigned int		strhashval;
+	char			dname[HEXDIR_LEN];
+	nfs4_verifier		verf = clid->verifier;
+	u32			ip_addr = svc_addr_in(rqstp)->sin_addr.s_addr;
+	struct xdr_netobj clname = {
+		.len = clid->id_len,
+		.data = clid->id,
+	};
+
+	dprintk("%s rqstp=%p clid=%p clname.len=%u clname.data=%p "
+		" ip_addr=%u flags %x\n",
+		__func__, rqstp, clid, clname.len, clname.data,
+		ip_addr, clid->flags);
+
+	if (!check_name(clname) || (clid->flags & EXCHGID4_INVAL_FLAG_MASK))
+		return nfserr_inval;
+
+	status = nfs4_make_rec_clidname(dname, &clname);
+
+	if (status)
+		goto error;
+
+	strhashval = clientstr_hashval(dname);
+
+	nfs4_lock_state();
+	status = nfserr_clid_inuse;
+
+	conf = find_confirmed_client_by_str(dname, strhashval);
+	if (conf) {
+		if (!same_creds(&conf->cl_cred, &rqstp->rq_cred) ||
+		    (ip_addr != conf->cl_addr)) {
+			/* Client collision: send nfserr_clid_inuse */
+			goto out;
+		}
+
+		if (!same_verf(&verf, &conf->cl_verifier)) {
+			/* Client reboot: destroy old state */
+			expire_client(conf);
+			goto out_new;
+		}
+		/* router replay */
+		goto out;
+	}
+
+	unconf  = find_unconfirmed_client_by_str(dname, strhashval);
+	if (unconf) {
+		status = nfs_ok;
+		/* Found an unconfirmed record */
+		if (!same_creds(&unconf->cl_cred, &rqstp->rq_cred)) {
+			/* Principal changed: update to the new principal
+			 * and send nfs_ok */
+			copy_cred(&unconf->cl_cred, &rqstp->rq_cred);
+		}
+
+		if (!same_verf(&unconf->cl_verifier, &verf)) {
+			/* Reboot before confirmation: update the verifier and
+			 * send nfs_ok */
+			copy_verf(unconf, &verf);
+			new = unconf;
+			goto out_copy;
+		}
+		goto out;
+	}
+
+out_new:
+	/* Normal case */
+	status = nfserr_resource;
+	new = create_client(clname, dname);
+
+	if (new == NULL)
+		goto out;
+
+	copy_verf(new, &verf);
+	copy_cred(&new->cl_cred, &rqstp->rq_cred);
+	new->cl_addr = ip_addr;
+	gen_clid(new);
+	gen_confirm(new);
+	add_to_unconfirmed(new, strhashval);
+out_copy:
+	clid->clientid.cl_boot = new->cl_clientid.cl_boot;
+	clid->clientid.cl_id = new->cl_clientid.cl_id;
+
+	new->cl_seqid = clid->seqid = 1;
+	nfsd4_set_ex_flags(new, clid);
+
+	dprintk("nfsd4_exchange_id seqid %d flags %x\n",
+		new->cl_seqid, new->cl_exchange_flags);
+	status = nfs_ok;
+
+out:
+	nfs4_unlock_state();
+error:
+	dprintk("nfsd4_exchange_id returns %d\n", ntohl(status));
+	return status;
 }
 
 __be32
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index fc613cd..055f1ad 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -1021,7 +1021,98 @@ static __be32
 nfsd4_decode_exchange_id(struct nfsd4_compoundargs *argp,
 			 struct nfsd4_exchange_id *clid)
 {
-	return nfserr_opnotsupp;	/* stub */
+	int dummy;
+	DECODE_HEAD;
+
+	READ_BUF(NFS4_VERIFIER_SIZE);
+	COPYMEM(clid->verifier.data, NFS4_VERIFIER_SIZE);
+
+	READ_BUF(4);
+	READ32(clid->id_len);
+
+	READ_BUF(clid->id_len);
+	SAVEMEM(clid->id, clid->id_len);
+
+	READ_BUF(4);
+	READ32(clid->flags);
+
+	/* Ignore state_protect4_a */
+	READ_BUF(4);
+	READ32(dummy);
+	switch (dummy) {
+	case SP4_NONE:
+		break;
+	case SP4_MACH_CRED:
+		/* spo_must_enforce */
+		READ_BUF(4);
+		READ32(dummy);
+		READ_BUF(dummy * 4);
+		p += dummy;
+
+		/* spo_must_allow */
+		READ_BUF(4);
+		READ32(dummy);
+		READ_BUF(dummy * 4);
+		p += dummy;
+		break;
+	case SP4_SSV:
+		/* ssp_ops */
+		READ_BUF(4);
+		READ32(dummy);
+		READ_BUF(dummy * 4);
+		p += dummy;
+
+		READ_BUF(4);
+		READ32(dummy);
+		READ_BUF(dummy * 4);
+		p += dummy;
+
+		/* ssp_hash_algs<> */
+		READ_BUF(4);
+		READ32(dummy);
+		READ_BUF(dummy);
+		p += XDR_QUADLEN(dummy);
+
+		/* ssp_encr_algs<> */
+		READ_BUF(4);
+		READ32(dummy);
+		READ_BUF(dummy);
+		p += XDR_QUADLEN(dummy);
+
+		/* ssp_window and ssp_num_gss_handles */
+		READ_BUF(8);
+		READ32(dummy);
+		READ32(dummy);
+		break;
+	default:
+		goto xdr_error;
+	}
+
+	/* Ignore Implementation ID */
+	READ_BUF(4);    /* nfs_impl_id4 array length */
+	READ32(dummy);
+
+	if (dummy > 1)
+		goto xdr_error;
+
+	if (dummy == 1) {
+		/* nii_domain */
+		READ_BUF(4);
+		READ32(dummy);
+		READ_BUF(dummy);
+		p += XDR_QUADLEN(dummy);
+
+		/* nii_name */
+		READ_BUF(4);
+		READ32(dummy);
+		READ_BUF(dummy);
+		p += XDR_QUADLEN(dummy);
+
+		/* nii_date */
+		READ_BUF(12);
+		p += 3;
+	}
+	DECODE_TAIL;
 }
 
 static __be32
@@ -2692,7 +2783,55 @@ static __be32
 nfsd4_encode_exchange_id(struct nfsd4_compoundres *resp, int nfserr,
 			 struct nfsd4_exchange_id *exid)
 {
-	/* stub */
+	ENCODE_HEAD;
+	char *major_id = "fixme_please";
+	char *server_scope = "fixme_please";
+	int major_id_sz;
+	int server_scope_sz;
+	uint64_t minor_id = 0;
+
+	if (nfserr)
+		goto out;
+
+	/* XXX FIXME We currently use ia dummy  as the major id. Need to change
+	 * this to something more meaningful...
+	 */
+	major_id_sz = strlen(major_id);
+	server_scope_sz = strlen(server_scope);
+
+	RESERVE_SPACE(
+		8 /* eir_clientid */ +
+		4 /* eir_sequenceid */ +
+		4 /* eir_flags */ +
+		4 /* spr_how (SP4_NONE) */ +
+		8 /* so_minor_id */ +
+		4 /* so_major_id.len */ +
+		(XDR_QUADLEN(major_id_sz) * 4) +
+		4 /* eir_server_scope.len */ +
+		(XDR_QUADLEN(server_scope_sz) * 4) +
+		4 /* eir_server_impl_id.count (0) */);
+
+	WRITEMEM(&exid->clientid, 8);
+	WRITE32(exid->seqid);
+	WRITE32(exid->flags);
+
+	/* state_protect4_r */
+	WRITE32(SP4_NONE);
+
+	/* The server_owner struct */
+	WRITE64(minor_id);      /* Minor id */
+	/* major id */
+	WRITE32(major_id_sz);
+	WRITEMEM(major_id, major_id_sz);
+
+	/* Server scope */
+	WRITE32(server_scope_sz);
+	WRITEMEM(server_scope, server_scope_sz);
+
+	/* Implementation id */
+	WRITE32(0);	/* zero length nfs_impl_id4 array */
+	ADJUST_ARGS();
+out:
 	return nfserr;
 }
 
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index 5cb4142..9bbfd88 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -214,6 +214,10 @@ struct nfs4_client {
 	struct nfs4_callback	cl_callback;    /* callback info */
 	atomic_t		cl_count;	/* ref count */
 	u32			cl_firststate;	/* recovery dir creation */
+#if defined(CONFIG_NFSD_V4_1)
+	u32			cl_seqid;	/* seqid for create_session */
+	u32			cl_exchange_flags;
+#endif /* CONFIG_NFSD_V4_1 */
 };
 
 /* struct nfs4_client_reset
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index d7dbdab..aee8996 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -354,7 +354,12 @@ struct nfsd4_write {
 
 #if defined(CONFIG_NFSD_V4_1)
 struct nfsd4_exchange_id {
-	int	foo;	/* stub */
+	nfs4_verifier	verifier;
+	u32		id_len;
+	char		*id;
+	u32		flags;
+	clientid_t	clientid;
+	u32		seqid;
 };
 
 struct nfsd4_create_session {
-- 
1.6.0.2

Re: [pnfs] [RFC 23/51] nfsd41: exchange_id operation

[Benny Halevy]

On Nov. 10, 2008, 22:48 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> Implement the exchange_id operation confoming to
> http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26
> 
> Based on the client provided name, hash a client id.
> If a confirmed one is found, compare the op's creds and
> verifier.  If the creds match and the verifier is different
> then expire the old client (client re-incarnated), otherwise,
> if both match, assume it's a replay and ignore it.
> 
> If an unconfirmed client is found, then copy the new creds
> and verifer if need update, otherwise assume replay.
> 
> The client is moved to a confirmed state on create_session.
> 
> In the nfs41 branch set the exchange_id flags to
> EXCHGID4_FLAG_USE_NON_PNFS | EXCHGID4_FLAG_SUPP_MOVED_REFER
> (pNFS is not supported, Referrals are supported,
> Migration is not.).
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4state.c        |  113 ++++++++++++++++++++++++++++++++++-
>  fs/nfsd/nfs4xdr.c          |  143 +++++++++++++++++++++++++++++++++++++++++++-
>  include/linux/nfsd/state.h |    4 +
>  include/linux/nfsd/xdr4.h  |    7 ++-
>  4 files changed, 263 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 5c96c39..76b2870 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -781,12 +781,123 @@ out_err:
>  }
>  
>  #if defined(CONFIG_NFSD_V4_1)
> +/*
> + * Set the exchange_id flags returned by the server.
> + */
> +static void
> +nfsd4_set_ex_flags(struct nfs4_client *new, struct nfsd4_exchange_id *clid)
> +{
> +	/* pNFS is not supported */
> +	new->cl_exchange_flags |= EXCHGID4_FLAG_USE_NON_PNFS;
> +
> +	/* Referrals are supported, Migration is not. */
> +	new->cl_exchange_flags |= EXCHGID4_FLAG_SUPP_MOVED_REFER;
> +
> +	/* set the wire flags to return to client. */
> +	clid->flags = new->cl_exchange_flags;
> +}
> +

review 11-13: Andy to add a "TODO" comment.
Document the ramfications of the present implementation
shortcuts we made.
Document server owner implementation.

>  __be32
>  nfsd4_exchange_id(struct svc_rqst *rqstp,
>  		  struct nfsd4_compound_state *cstate,
>  		  struct nfsd4_exchange_id *clid)
>  {
> -	return -1;	/* stub */
> +	struct nfs4_client *unconf, *conf, *new;
> +	int status;
> +	unsigned int		strhashval;
> +	char			dname[HEXDIR_LEN];
> +	nfs4_verifier		verf = clid->verifier;
> +	u32			ip_addr = svc_addr_in(rqstp)->sin_addr.s_addr;
> +	struct xdr_netobj clname = {
> +		.len = clid->id_len,
> +		.data = clid->id,
> +	};
> +
> +	dprintk("%s rqstp=%p clid=%p clname.len=%u clname.data=%p "
> +		" ip_addr=%u flags %x\n",
> +		__func__, rqstp, clid, clname.len, clname.data,
> +		ip_addr, clid->flags);
> +
> +	if (!check_name(clname) || (clid->flags & EXCHGID4_INVAL_FLAG_MASK))
> +		return nfserr_inval;
> +
> +	status = nfs4_make_rec_clidname(dname, &clname);
> +
> +	if (status)
> +		goto error;
> +
> +	strhashval = clientstr_hashval(dname);
> +
> +	nfs4_lock_state();
> +	status = nfserr_clid_inuse;
> +
> +	conf = find_confirmed_client_by_str(dname, strhashval);

review 11-13: pass a minorversion flag down to find_{,un}confirmed_client*()
check cl_exchange_id != 0 for minorversion 1
to logically separate nfs4.0 clients from nfs4.1

> +	if (conf) {
> +		if (!same_creds(&conf->cl_cred, &rqstp->rq_cred) ||
> +		    (ip_addr != conf->cl_addr)) {
> +			/* Client collision: send nfserr_clid_inuse */
> +			goto out;
> +		}
> +
> +		if (!same_verf(&verf, &conf->cl_verifier)) {
> +			/* Client reboot: destroy old state */
> +			expire_client(conf);
> +			goto out_new;
> +		}
> +		/* router replay */

review 11-13: for now just return success.

> +		goto out;
> +	}
> +
> +	unconf  = find_unconfirmed_client_by_str(dname, strhashval);
> +	if (unconf) {
> +		status = nfs_ok;
> +		/* Found an unconfirmed record */
> +		if (!same_creds(&unconf->cl_cred, &rqstp->rq_cred)) {
> +			/* Principal changed: update to the new principal
> +			 * and send nfs_ok */
> +			copy_cred(&unconf->cl_cred, &rqstp->rq_cred);
> +		}
> +
> +		if (!same_verf(&unconf->cl_verifier, &verf)) {
> +			/* Reboot before confirmation: update the verifier and
> +			 * send nfs_ok */
> +			copy_verf(unconf, &verf);
> +			new = unconf;
> +			goto out_copy;
> +		}
> +		goto out;
> +	}
> +
> +out_new:
> +	/* Normal case */
> +	status = nfserr_resource;
> +	new = create_client(clname, dname);
> +
> +	if (new == NULL)
> +		goto out;
> +
> +	copy_verf(new, &verf);
> +	copy_cred(&new->cl_cred, &rqstp->rq_cred);
> +	new->cl_addr = ip_addr;
> +	gen_clid(new);
> +	gen_confirm(new);
> +	add_to_unconfirmed(new, strhashval);
> +out_copy:
> +	clid->clientid.cl_boot = new->cl_clientid.cl_boot;
> +	clid->clientid.cl_id = new->cl_clientid.cl_id;
> +
> +	new->cl_seqid = clid->seqid = 1;
> +	nfsd4_set_ex_flags(new, clid);
> +
> +	dprintk("nfsd4_exchange_id seqid %d flags %x\n",
> +		new->cl_seqid, new->cl_exchange_flags);
> +	status = nfs_ok;
> +
> +out:
> +	nfs4_unlock_state();
> +error:
> +	dprintk("nfsd4_exchange_id returns %d\n", ntohl(status));
> +	return status;
>  }
>  
>  __be32
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index fc613cd..055f1ad 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -1021,7 +1021,98 @@ static __be32
>  nfsd4_decode_exchange_id(struct nfsd4_compoundargs *argp,
>  			 struct nfsd4_exchange_id *clid)
>  {
> -	return nfserr_opnotsupp;	/* stub */
> +	int dummy;
> +	DECODE_HEAD;
> +
> +	READ_BUF(NFS4_VERIFIER_SIZE);
> +	COPYMEM(clid->verifier.data, NFS4_VERIFIER_SIZE);
> +
> +	READ_BUF(4);
> +	READ32(clid->id_len);
> +
> +	READ_BUF(clid->id_len);
> +	SAVEMEM(clid->id, clid->id_len);
> +
> +	READ_BUF(4);
> +	READ32(clid->flags);
> +
> +	/* Ignore state_protect4_a */
> +	READ_BUF(4);
> +	READ32(dummy);

review 11-13: store sp_how, return NFS4ERR_ENCR_ALG_UNSUPP
error from proc later on.

> +	switch (dummy) {
> +	case SP4_NONE:
> +		break;
> +	case SP4_MACH_CRED:
> +		/* spo_must_enforce */
> +		READ_BUF(4);
> +		READ32(dummy);
> +		READ_BUF(dummy * 4);
> +		p += dummy;
> +
> +		/* spo_must_allow */
> +		READ_BUF(4);
> +		READ32(dummy);
> +		READ_BUF(dummy * 4);
> +		p += dummy;
> +		break;
> +	case SP4_SSV:
> +		/* ssp_ops */
> +		READ_BUF(4);
> +		READ32(dummy);
> +		READ_BUF(dummy * 4);
> +		p += dummy;
> +
> +		READ_BUF(4);
> +		READ32(dummy);
> +		READ_BUF(dummy * 4);
> +		p += dummy;
> +
> +		/* ssp_hash_algs<> */
> +		READ_BUF(4);
> +		READ32(dummy);
> +		READ_BUF(dummy);
> +		p += XDR_QUADLEN(dummy);
> +
> +		/* ssp_encr_algs<> */
> +		READ_BUF(4);
> +		READ32(dummy);
> +		READ_BUF(dummy);
> +		p += XDR_QUADLEN(dummy);
> +
> +		/* ssp_window and ssp_num_gss_handles */
> +		READ_BUF(8);
> +		READ32(dummy);
> +		READ32(dummy);
> +		break;
> +	default:
> +		goto xdr_error;
> +	}
> +
> +	/* Ignore Implementation ID */
> +	READ_BUF(4);    /* nfs_impl_id4 array length */
> +	READ32(dummy);
> +
> +	if (dummy > 1)
> +		goto xdr_error;
> +
> +	if (dummy == 1) {
> +		/* nii_domain */
> +		READ_BUF(4);
> +		READ32(dummy);
> +		READ_BUF(dummy);
> +		p += XDR_QUADLEN(dummy);
> +
> +		/* nii_name */
> +		READ_BUF(4);
> +		READ32(dummy);
> +		READ_BUF(dummy);
> +		p += XDR_QUADLEN(dummy);
> +
> +		/* nii_date */
> +		READ_BUF(12);
> +		p += 3;
> +	}
> +	DECODE_TAIL;
>  }
>  
>  static __be32
> @@ -2692,7 +2783,55 @@ static __be32
>  nfsd4_encode_exchange_id(struct nfsd4_compoundres *resp, int nfserr,
>  			 struct nfsd4_exchange_id *exid)
>  {
> -	/* stub */
> +	ENCODE_HEAD;
> +	char *major_id = "fixme_please";
> +	char *server_scope = "fixme_please";
> +	int major_id_sz;
> +	int server_scope_sz;
> +	uint64_t minor_id = 0;

review 11-13: use utsname for major_id (and copy to server_scope)
MUST DO before submitting.

> +
> +	if (nfserr)
> +		goto out;
> +
> +	/* XXX FIXME We currently use ia dummy  as the major id. Need to change
> +	 * this to something more meaningful...
> +	 */
> +	major_id_sz = strlen(major_id);
> +	server_scope_sz = strlen(server_scope);
> +
> +	RESERVE_SPACE(
> +		8 /* eir_clientid */ +
> +		4 /* eir_sequenceid */ +
> +		4 /* eir_flags */ +
> +		4 /* spr_how (SP4_NONE) */ +
> +		8 /* so_minor_id */ +
> +		4 /* so_major_id.len */ +
> +		(XDR_QUADLEN(major_id_sz) * 4) +
> +		4 /* eir_server_scope.len */ +
> +		(XDR_QUADLEN(server_scope_sz) * 4) +
> +		4 /* eir_server_impl_id.count (0) */);
> +
> +	WRITEMEM(&exid->clientid, 8);
> +	WRITE32(exid->seqid);
> +	WRITE32(exid->flags);
> +
> +	/* state_protect4_r */
> +	WRITE32(SP4_NONE);

review 11-13: BUG_ON(sp_how (that we saved previously) != SP4_NONE)

> +
> +	/* The server_owner struct */
> +	WRITE64(minor_id);      /* Minor id */
> +	/* major id */
> +	WRITE32(major_id_sz);
> +	WRITEMEM(major_id, major_id_sz);
> +
> +	/* Server scope */
> +	WRITE32(server_scope_sz);
> +	WRITEMEM(server_scope, server_scope_sz);
> +
> +	/* Implementation id */
> +	WRITE32(0);	/* zero length nfs_impl_id4 array */
> +	ADJUST_ARGS();
> +out:
>  	return nfserr;
>  }
>  
> diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
> index 5cb4142..9bbfd88 100644
> --- a/include/linux/nfsd/state.h
> +++ b/include/linux/nfsd/state.h
> @@ -214,6 +214,10 @@ struct nfs4_client {
>  	struct nfs4_callback	cl_callback;    /* callback info */
>  	atomic_t		cl_count;	/* ref count */
>  	u32			cl_firststate;	/* recovery dir creation */
> +#if defined(CONFIG_NFSD_V4_1)
> +	u32			cl_seqid;	/* seqid for create_session */
> +	u32			cl_exchange_flags;
> +#endif /* CONFIG_NFSD_V4_1 */
>  };
>  
>  /* struct nfs4_client_reset
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index d7dbdab..aee8996 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -354,7 +354,12 @@ struct nfsd4_write {
>  
>  #if defined(CONFIG_NFSD_V4_1)
>  struct nfsd4_exchange_id {
> -	int	foo;	/* stub */
> +	nfs4_verifier	verifier;
> +	u32		id_len;
> +	char		*id;
> +	u32		flags;
> +	clientid_t	clientid;
> +	u32		seqid;
>  };
>  
>  struct nfsd4_create_session {

[RFC 24/51] nfsd41: print exchange flags when purging client

[Benny Halevy]

From: Dean Hildebrand <seattleplus@gmail.com>

Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 76b2870..cbbb733 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -2114,8 +2114,14 @@ nfs4_laundromat(void)
 				clientid_val = t;
 			break;
 		}
-		dprintk("NFSD: purging unused client (clientid %08x)\n",
+#if defined(CONFIG_NFSD_V4_1)
+		dprintk("NFSD: purging unused client"
+			"(clientid %08x cl_exchange_flags %x)\n",
+			clp->cl_clientid.cl_id, clp->cl_exchange_flags);
+#else
+		dprintk("NFSD: purging unused client(clientid %08x)\n",
 			clp->cl_clientid.cl_id);
+#endif /* CONFIG_NFSD_V4_1 */
 		nfsd4_remove_clid_dir(clp);
 		expire_client(clp);
 	}
-- 
1.6.0.2

Re: [pnfs] [RFC 24/51] nfsd41: print exchange flags when purging client

[Benny Halevy]

On Nov. 10, 2008, 22:48 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Dean Hildebrand <seattleplus@gmail.com>
> 
> Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 11-13: get rid of this patch.

> ---
>  fs/nfsd/nfs4state.c |    8 +++++++-
>  1 files changed, 7 insertions(+), 1 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 76b2870..cbbb733 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -2114,8 +2114,14 @@ nfs4_laundromat(void)
>  				clientid_val = t;
>  			break;
>  		}
> -		dprintk("NFSD: purging unused client (clientid %08x)\n",
> +#if defined(CONFIG_NFSD_V4_1)
> +		dprintk("NFSD: purging unused client"
> +			"(clientid %08x cl_exchange_flags %x)\n",
> +			clp->cl_clientid.cl_id, clp->cl_exchange_flags);
> +#else
> +		dprintk("NFSD: purging unused client(clientid %08x)\n",
>  			clp->cl_clientid.cl_id);
> +#endif /* CONFIG_NFSD_V4_1 */
>  		nfsd4_remove_clid_dir(clp);
>  		expire_client(clp);
>  	}

[RFC 25/51] nfsd41: create_session operation

[Benny Halevy]

From: Ricardo Labiaga <ricardo.labiaga@netapp.com>

Implement the create_session operation confoming to
http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26

Look up the client id (generated by the server on exchange_id,
given by the client on create_session).
If neither a confirmed or unconfirmed client is found
then the client id is stale
If a confirmed cilent is found (i.e. we already received
create_session for it) then compare the sequence id
to determine if it's a replay or possibly a mis-ordered rpc.
If the seqid is in order, update the confirmed client seqid
and procedd with updating the session parameters.

If an unconfirmed client_id is found then verify the creds
and seqid.  If both match move the client id to confirmed state
and proceed with processing the create_session.

Currently, we do not support persistent sessions, and RDMA.

alloc_init_session generates a new sessionid and creates
a session structure.

TODO: improve resource pre-allocation and negotiate session
parameters accordingly.

Signed-off-by: Marc Eshel <eshel@almaden.ibm.com>
Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c        |  139 +++++++++++++++++++++++++++++++++++++++++-
 fs/nfsd/nfs4xdr.c          |  147 +++++++++++++++++++++++++++++++++++++++++++-
 include/linux/nfsd/state.h |    3 +
 include/linux/nfsd/xdr4.h  |   21 ++++++-
 4 files changed, 306 insertions(+), 4 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index cbbb733..73bc8de 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -73,6 +73,9 @@ static u32 current_delegid = 1;
 static u32 nfs4_init;
 static stateid_t zerostateid;             /* bits all 0 */
 static stateid_t onestateid;              /* bits all 1 */
+#if defined(CONFIG_NFSD_V4_1)
+static u64 current_sessionid = 1;
+#endif
 
 #define ZERO_STATEID(stateid) (!memcmp((stateid), &zerostateid, sizeof(stateid_t)))
 #define ONE_STATEID(stateid)  (!memcmp((stateid), &onestateid, sizeof(stateid_t)))
@@ -352,6 +355,72 @@ dump_sessionid(const char *fn, nfs41_sessionid *sessionid)
 	dprintk("%s: %u:%u:%u:%u\n", fn, ptr[0], ptr[1], ptr[2], ptr[3]);
 }
 
+static void
+gen_sessionid(struct nfs41_session *ses)
+{
+	struct nfs4_client *clp = ses->se_client;
+	u32 *p = (u32 *)ses->se_sessionid;
+
+	*p++ = clp->cl_clientid.cl_boot;
+	*p++ = clp->cl_clientid.cl_id;
+	*p++ = (u32)boot_time;
+	*p++ = current_sessionid++;
+	BUG_ON((char *)p - (char *)ses->se_sessionid !=
+	       sizeof(ses->se_sessionid));
+}
+
+static int
+alloc_init_session(struct nfs4_client *clp, struct nfsd4_create_session *cses)
+{
+	struct nfs41_session *new;
+	int idx, status = nfserr_resource, slotsize, i;
+
+	new = kzalloc(sizeof(*new), GFP_KERNEL);
+	if (!new)
+		goto out;
+
+	if (cses->fore_channel.maxreqs >= NFS41_MAX_SLOTS)
+		cses->fore_channel.maxreqs = NFS41_MAX_SLOTS;
+	new->se_fnumslots = cses->fore_channel.maxreqs;
+	slotsize = new->se_fnumslots * sizeof(struct nfs41_slot);
+
+	new->se_slots = kzalloc(slotsize, GFP_KERNEL);
+	if (!new->se_slots)
+		goto out_free;
+
+	for (i = 0; i < new->se_fnumslots; i++) {
+		new->se_slots[i].sl_session = new;
+		nfs41_set_slot_state(&new->se_slots[i], NFS4_SLOT_AVAILABLE);
+	}
+
+	new->se_client = clp;
+	gen_sessionid(new);
+	idx = hash_sessionid(&new->se_sessionid);
+	memcpy(&clp->cl_sessionid, &new->se_sessionid, sizeof(nfs41_sessionid));
+
+	new->se_flags = cses->flags;
+
+	/* for now, accept the client values */
+	new->se_fheaderpad_sz = cses->fore_channel.headerpadsz;
+	new->se_fmaxreq_sz = cses->fore_channel.maxreq_sz;
+	new->se_fmaxresp_sz = cses->fore_channel.maxresp_sz;
+	new->se_fmaxresp_cached = cses->fore_channel.maxresp_cached;
+	new->se_fmaxops = cses->fore_channel.maxops;
+
+	kref_init(&new->se_ref);
+	INIT_LIST_HEAD(&new->se_hash);
+	INIT_LIST_HEAD(&new->se_perclnt);
+	list_add(&new->se_hash, &sessionid_hashtbl[idx]);
+	list_add(&new->se_perclnt, &clp->cl_sessions);
+
+	status = nfs_ok;
+out:
+	return status;
+out_free:
+	kfree(new);
+	goto out;
+}
+
 struct nfs41_session *
 find_in_sessionid_hashtbl(nfs41_sessionid *sessionid)
 {
@@ -905,7 +974,75 @@ nfsd4_create_session(struct svc_rqst *rqstp,
 		     struct nfsd4_compound_state *cstate,
 		     struct nfsd4_create_session *cr_ses)
 {
-	return -1;	/* stub */
+	u32 ip_addr = svc_addr_in(rqstp)->sin_addr.s_addr;
+	struct nfs4_client *conf, *unconf;
+	__u32   max_blocksize = svc_max_payload(rqstp);
+	int status = 0;
+
+	if (STALE_CLIENTID(&cr_ses->clientid))
+		return nfserr_stale_clientid;
+
+	nfs4_lock_state();
+	unconf = find_unconfirmed_client(&cr_ses->clientid);
+	conf = find_confirmed_client(&cr_ses->clientid);
+
+	if (!conf && !unconf) {
+		status = nfserr_stale_clientid;
+		goto out;
+	}
+	if (conf) {
+		status = nfs_ok;
+		if (conf->cl_seqid == cr_ses->seqid) {
+			dprintk("Got a create_session replay! seqid= %d\n",
+				conf->cl_seqid);
+			goto out_replay;
+		} else if (cr_ses->seqid != conf->cl_seqid + 1) {
+			status = nfserr_seq_misordered;
+			dprintk("Sequence misordered!\n");
+			dprintk("Expected seqid= %d but got seqid= %d\n",
+				conf->cl_seqid, cr_ses->seqid);
+			goto out;
+		}
+		conf->cl_seqid++;
+	} else if (unconf) {
+		if (!same_creds(&unconf->cl_cred, &rqstp->rq_cred) ||
+		    (ip_addr != unconf->cl_addr)) {
+			status = nfserr_clid_inuse;
+			goto out;
+		}
+
+		if (unconf->cl_seqid != cr_ses->seqid) {
+			status = nfserr_seq_misordered;
+			goto out;
+		}
+
+		move_to_confirmed(unconf);
+
+		/*
+		 * We do not support RDMA or persistent sessions
+		 */
+		cr_ses->flags &= ~SESSION4_PERSIST;
+		cr_ses->flags &= ~SESSION4_RDMA;
+
+		conf = unconf;
+	}
+
+	status = alloc_init_session(conf, cr_ses);
+
+out_replay:
+	memcpy(cr_ses->sessionid, conf->cl_sessionid, 16);
+	cr_ses->seqid = conf->cl_seqid;
+	cr_ses->fore_channel.maxreq_sz = max_blocksize;
+	cr_ses->fore_channel.maxresp_sz = max_blocksize;
+	cr_ses->fore_channel.maxresp_cached = max_blocksize;
+	cr_ses->back_channel.maxreq_sz = max_blocksize;
+	cr_ses->back_channel.maxresp_sz = max_blocksize;
+	cr_ses->back_channel.maxresp_cached = max_blocksize;
+
+out:
+	nfs4_unlock_state();
+	dprintk("%s returns %d\n", __func__, ntohl(status));
+	return status;
 }
 
 __be32
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 055f1ad..02c82d4 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -1119,7 +1119,108 @@ static __be32
 nfsd4_decode_create_session(struct nfsd4_compoundargs *argp,
 			    struct nfsd4_create_session *sess)
 {
-	return nfserr_opnotsupp;	/* stub */
+	DECODE_HEAD;
+
+	u32 dummy;
+	char *machine_name;
+	int i;
+	int nr_secflavs;
+
+	READ_BUF(16);
+	COPYMEM(&sess->clientid, 8);
+	READ32(sess->seqid);
+	READ32(sess->flags);
+
+	/* Fore channel attrs */
+	READ_BUF(28);
+	READ32(sess->fore_channel.headerpadsz);
+	READ32(sess->fore_channel.maxreq_sz);
+	READ32(sess->fore_channel.maxresp_sz);
+	READ32(sess->fore_channel.maxresp_cached);
+	READ32(sess->fore_channel.maxops);
+	READ32(sess->fore_channel.maxreqs);
+	READ32(sess->fore_channel.nr_rdma_attrs);
+	if (sess->fore_channel.nr_rdma_attrs == 1) {
+		READ_BUF(4);
+		READ32(sess->fore_channel.rdma_attrs);
+	} else if (sess->fore_channel.nr_rdma_attrs > 1) {
+		dprintk("Too many fore channel attr bitmaps!\n");
+		goto xdr_error;
+	}
+
+	/* Back channel attrs */
+	READ_BUF(28);
+	READ32(sess->back_channel.headerpadsz);
+	READ32(sess->back_channel.maxreq_sz);
+	READ32(sess->back_channel.maxresp_sz);
+	READ32(sess->back_channel.maxresp_cached);
+	READ32(sess->back_channel.maxops);
+	READ32(sess->back_channel.maxreqs);
+	READ32(sess->back_channel.nr_rdma_attrs);
+	if (sess->back_channel.nr_rdma_attrs == 1) {
+		READ_BUF(4);
+		READ32(sess->back_channel.rdma_attrs);
+	} else if (sess->back_channel.nr_rdma_attrs > 1) {
+		dprintk("Too many back channel attr bitmaps!\n");
+		goto xdr_error;
+	}
+
+	READ_BUF(8);
+	READ32(sess->callback_prog);
+
+	/* callback_sec_params4 */
+	READ32(nr_secflavs);
+	for (i = 0; i < nr_secflavs; ++i) {
+		READ_BUF(4);
+		READ32(dummy);
+		switch (dummy) {
+		case RPC_AUTH_NULL:
+			/* Nothing to read */
+			break;
+		case RPC_AUTH_UNIX:
+			READ_BUF(8);
+			/* stamp */
+			READ32(dummy);
+
+			/* machine name */
+			READ32(dummy);
+			READ_BUF(dummy);
+			SAVEMEM(machine_name, dummy);
+
+			/* uid, gid */
+			READ_BUF(8);
+			READ32(sess->uid);
+			READ32(sess->gid);
+
+			/* more gids */
+			READ_BUF(4);
+			READ32(dummy);
+			READ_BUF(dummy * 4);
+			for (i = 0; i < dummy; ++i)
+				READ32(dummy);
+			break;
+		case RPC_AUTH_GSS:
+			dprintk("RPC_AUTH_GSS callback secflavor "
+				"not supported!\n");
+			READ_BUF(8);
+			/* gcbp_service */
+			READ32(dummy);
+			/* gcbp_handle_from_server */
+			READ32(dummy);
+			READ_BUF(dummy);
+			p += XDR_QUADLEN(dummy);
+			/* gcbp_handle_from_client */
+			READ_BUF(4);
+			READ32(dummy);
+			READ_BUF(dummy);
+			p += XDR_QUADLEN(dummy);
+			break;
+		default:
+			dprintk("Illegal callback secflavor\n");
+			return nfserr_inval;
+		}
+	}
+	DECODE_TAIL;
 }
 
 static __be32
@@ -2839,7 +2940,49 @@ static __be32
 nfsd4_encode_create_session(struct nfsd4_compoundres *resp, int nfserr,
 			    struct nfsd4_create_session *sess)
 {
-	/* stub */
+	ENCODE_HEAD;
+
+	if (nfserr)
+		goto out;
+
+	RESERVE_SPACE(24);
+	WRITEMEM(sess->sessionid, sizeof(sess->sessionid));
+	WRITE32(sess->seqid);
+	WRITE32(sess->flags);
+	ADJUST_ARGS();
+
+	RESERVE_SPACE(28);
+	WRITE32(sess->fore_channel.headerpadsz);
+	WRITE32(sess->fore_channel.maxreq_sz);
+	WRITE32(sess->fore_channel.maxresp_sz);
+	WRITE32(sess->fore_channel.maxresp_cached);
+	WRITE32(sess->fore_channel.maxops);
+	WRITE32(sess->fore_channel.maxreqs);
+	WRITE32(sess->fore_channel.nr_rdma_attrs);
+	ADJUST_ARGS();
+
+	if (sess->fore_channel.nr_rdma_attrs) {
+		RESERVE_SPACE(4);
+		WRITE32(sess->fore_channel.rdma_attrs);
+		ADJUST_ARGS();
+	}
+
+	RESERVE_SPACE(28);
+	WRITE32(sess->back_channel.headerpadsz);
+	WRITE32(sess->back_channel.maxreq_sz);
+	WRITE32(sess->back_channel.maxresp_sz);
+	WRITE32(sess->back_channel.maxresp_cached);
+	WRITE32(sess->back_channel.maxops);
+	WRITE32(sess->back_channel.maxreqs);
+	WRITE32(sess->back_channel.nr_rdma_attrs);
+	ADJUST_ARGS();
+
+	if (sess->back_channel.nr_rdma_attrs) {
+		RESERVE_SPACE(4);
+		WRITE32(sess->back_channel.rdma_attrs);
+		ADJUST_ARGS();
+	}
+out:
 	return nfserr;
 }
 
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index 9bbfd88..94b5366 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -217,6 +217,9 @@ struct nfs4_client {
 #if defined(CONFIG_NFSD_V4_1)
 	u32			cl_seqid;	/* seqid for create_session */
 	u32			cl_exchange_flags;
+	nfs41_sessionid		cl_sessionid;
+
+	struct svc_xprt		*cl_cb_xprt;	/* 4.1 callback transport */
 #endif /* CONFIG_NFSD_V4_1 */
 };
 
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index aee8996..75d93ab 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -362,8 +362,27 @@ struct nfsd4_exchange_id {
 	u32		seqid;
 };
 
+struct nfsd4_channel {
+	u32		headerpadsz;
+	u32		maxreq_sz;
+	u32		maxresp_sz;
+	u32		maxresp_cached;
+	u32		maxops;
+	u32		maxreqs;
+	u32		nr_rdma_attrs;
+	u32		rdma_attrs;
+};
+
 struct nfsd4_create_session {
-	int	foo;	/* stub */
+	clientid_t		clientid;
+	nfs41_sessionid		sessionid;
+	u32			seqid;
+	u32			flags;
+	struct nfsd4_channel	fore_channel;
+	struct nfsd4_channel	back_channel;
+	u32			callback_prog;
+	u32			uid;
+	u32			gid;
 };
 
 struct nfsd4_sequence {
-- 
1.6.0.2

Re: [pnfs] [RFC 25/51] nfsd41: create_session operation

[Benny Halevy]

On Nov. 10, 2008, 22:49 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Ricardo Labiaga <ricardo.labiaga@netapp.com>
> 
> Implement the create_session operation confoming to
> http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26
> 
> Look up the client id (generated by the server on exchange_id,
> given by the client on create_session).
> If neither a confirmed or unconfirmed client is found
> then the client id is stale
> If a confirmed cilent is found (i.e. we already received
> create_session for it) then compare the sequence id
> to determine if it's a replay or possibly a mis-ordered rpc.
> If the seqid is in order, update the confirmed client seqid
> and procedd with updating the session parameters.
> 
> If an unconfirmed client_id is found then verify the creds
> and seqid.  If both match move the client id to confirmed state
> and proceed with processing the create_session.
> 
> Currently, we do not support persistent sessions, and RDMA.
> 
> alloc_init_session generates a new sessionid and creates
> a session structure.
> 
> TODO: improve resource pre-allocation and negotiate session
> parameters accordingly.
> 
> Signed-off-by: Marc Eshel <eshel@almaden.ibm.com>
> Signed-off-by: Dean Hildebrand <dhildeb@us.ibm.com>
> Signed-off-by: Andy Adamson<andros@netapp.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4state.c        |  139 +++++++++++++++++++++++++++++++++++++++++-
>  fs/nfsd/nfs4xdr.c          |  147 +++++++++++++++++++++++++++++++++++++++++++-
>  include/linux/nfsd/state.h |    3 +
>  include/linux/nfsd/xdr4.h  |   21 ++++++-
>  4 files changed, 306 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index cbbb733..73bc8de 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -73,6 +73,9 @@ static u32 current_delegid = 1;
>  static u32 nfs4_init;
>  static stateid_t zerostateid;             /* bits all 0 */
>  static stateid_t onestateid;              /* bits all 1 */
> +#if defined(CONFIG_NFSD_V4_1)
> +static u64 current_sessionid = 1;
> +#endif
>  
>  #define ZERO_STATEID(stateid) (!memcmp((stateid), &zerostateid, sizeof(stateid_t)))
>  #define ONE_STATEID(stateid)  (!memcmp((stateid), &onestateid, sizeof(stateid_t)))
> @@ -352,6 +355,72 @@ dump_sessionid(const char *fn, nfs41_sessionid *sessionid)
>  	dprintk("%s: %u:%u:%u:%u\n", fn, ptr[0], ptr[1], ptr[2], ptr[3]);
>  }
>  
> +static void
> +gen_sessionid(struct nfs41_session *ses)
> +{
> +	struct nfs4_client *clp = ses->se_client;
> +	u32 *p = (u32 *)ses->se_sessionid;
> +
> +	*p++ = clp->cl_clientid.cl_boot;
> +	*p++ = clp->cl_clientid.cl_id;
> +	*p++ = (u32)boot_time;

revewiw 11-13: find something useful to put here
boot_time is already part of the clientid part.
(or reserve as 0)

> +	*p++ = current_sessionid++;
> +	BUG_ON((char *)p - (char *)ses->se_sessionid !=
> +	       sizeof(ses->se_sessionid));

remove BUG_ON

> +}
> +
> +static int
> +alloc_init_session(struct nfs4_client *clp, struct nfsd4_create_session *cses)
> +{
> +	struct nfs41_session *new;
> +	int idx, status = nfserr_resource, slotsize, i;
> +
> +	new = kzalloc(sizeof(*new), GFP_KERNEL);
> +	if (!new)
> +		goto out;
> +
> +	if (cses->fore_channel.maxreqs >= NFS41_MAX_SLOTS)
> +		cses->fore_channel.maxreqs = NFS41_MAX_SLOTS;
> +	new->se_fnumslots = cses->fore_channel.maxreqs;
> +	slotsize = new->se_fnumslots * sizeof(struct nfs41_slot);
> +
> +	new->se_slots = kzalloc(slotsize, GFP_KERNEL);
> +	if (!new->se_slots)
> +		goto out_free;
> +
> +	for (i = 0; i < new->se_fnumslots; i++) {
> +		new->se_slots[i].sl_session = new;
> +		nfs41_set_slot_state(&new->se_slots[i], NFS4_SLOT_AVAILABLE);
> +	}
> +
> +	new->se_client = clp;
> +	gen_sessionid(new);
> +	idx = hash_sessionid(&new->se_sessionid);
> +	memcpy(&clp->cl_sessionid, &new->se_sessionid, sizeof(nfs41_sessionid));
> +
> +	new->se_flags = cses->flags;
> +
> +	/* for now, accept the client values */
> +	new->se_fheaderpad_sz = cses->fore_channel.headerpadsz;
> +	new->se_fmaxreq_sz = cses->fore_channel.maxreq_sz;
> +	new->se_fmaxresp_sz = cses->fore_channel.maxresp_sz;
> +	new->se_fmaxresp_cached = cses->fore_channel.maxresp_cached;
> +	new->se_fmaxops = cses->fore_channel.maxops;

review 11-13: this is being rewritten by Andy.

> +
> +	kref_init(&new->se_ref);
> +	INIT_LIST_HEAD(&new->se_hash);
> +	INIT_LIST_HEAD(&new->se_perclnt);
> +	list_add(&new->se_hash, &sessionid_hashtbl[idx]);
> +	list_add(&new->se_perclnt, &clp->cl_sessions);
> +
> +	status = nfs_ok;
> +out:
> +	return status;
> +out_free:
> +	kfree(new);
> +	goto out;
> +}
> +
>  struct nfs41_session *
>  find_in_sessionid_hashtbl(nfs41_sessionid *sessionid)
>  {
> @@ -905,7 +974,75 @@ nfsd4_create_session(struct svc_rqst *rqstp,
>  		     struct nfsd4_compound_state *cstate,
>  		     struct nfsd4_create_session *cr_ses)
>  {
> -	return -1;	/* stub */
> +	u32 ip_addr = svc_addr_in(rqstp)->sin_addr.s_addr;
> +	struct nfs4_client *conf, *unconf;
> +	__u32   max_blocksize = svc_max_payload(rqstp);
> +	int status = 0;
> +
> +	if (STALE_CLIENTID(&cr_ses->clientid))
> +		return nfserr_stale_clientid;

review 11-13: no need to check that here since we're returning the
same error if we can't find it. Do the same for setclientid_confirm.

> +
> +	nfs4_lock_state();
> +	unconf = find_unconfirmed_client(&cr_ses->clientid);
> +	conf = find_confirmed_client(&cr_ses->clientid);
> +
> +	if (!conf && !unconf) {
> +		status = nfserr_stale_clientid;
> +		goto out;
> +	}

review 11-13: move downward as the final else lag.

> +	if (conf) {
> +		status = nfs_ok;
> +		if (conf->cl_seqid == cr_ses->seqid) {
> +			dprintk("Got a create_session replay! seqid= %d\n",
> +				conf->cl_seqid);
> +			goto out_replay;
> +		} else if (cr_ses->seqid != conf->cl_seqid + 1) {
> +			status = nfserr_seq_misordered;
> +			dprintk("Sequence misordered!\n");
> +			dprintk("Expected seqid= %d but got seqid= %d\n",
> +				conf->cl_seqid, cr_ses->seqid);
> +			goto out;
> +		}
> +		conf->cl_seqid++;
> +	} else if (unconf) {
> +		if (!same_creds(&unconf->cl_cred, &rqstp->rq_cred) ||
> +		    (ip_addr != unconf->cl_addr)) {
> +			status = nfserr_clid_inuse;
> +			goto out;
> +		}
> +
> +		if (unconf->cl_seqid != cr_ses->seqid) {
> +			status = nfserr_seq_misordered;
> +			goto out;
> +		}
> +
> +		move_to_confirmed(unconf);
> +
> +		/*
> +		 * We do not support RDMA or persistent sessions
> +		 */
> +		cr_ses->flags &= ~SESSION4_PERSIST;
> +		cr_ses->flags &= ~SESSION4_RDMA;
> +
> +		conf = unconf;
> +	}
> +
> +	status = alloc_init_session(conf, cr_ses);
> +
> +out_replay:
> +	memcpy(cr_ses->sessionid, conf->cl_sessionid, 16);
> +	cr_ses->seqid = conf->cl_seqid;
> +	cr_ses->fore_channel.maxreq_sz = max_blocksize;
> +	cr_ses->fore_channel.maxresp_sz = max_blocksize;
> +	cr_ses->fore_channel.maxresp_cached = max_blocksize;
> +	cr_ses->back_channel.maxreq_sz = max_blocksize;
> +	cr_ses->back_channel.maxresp_sz = max_blocksize;
> +	cr_ses->back_channel.maxresp_cached = max_blocksize;
> +
> +out:
> +	nfs4_unlock_state();
> +	dprintk("%s returns %d\n", __func__, ntohl(status));
> +	return status;
>  }
>  
>  __be32
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index 055f1ad..02c82d4 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -1119,7 +1119,108 @@ static __be32
>  nfsd4_decode_create_session(struct nfsd4_compoundargs *argp,
>  			    struct nfsd4_create_session *sess)
>  {
> -	return nfserr_opnotsupp;	/* stub */
> +	DECODE_HEAD;
> +
> +	u32 dummy;
> +	char *machine_name;
> +	int i;
> +	int nr_secflavs;
> +
> +	READ_BUF(16);
> +	COPYMEM(&sess->clientid, 8);
> +	READ32(sess->seqid);
> +	READ32(sess->flags);
> +
> +	/* Fore channel attrs */
> +	READ_BUF(28);
> +	READ32(sess->fore_channel.headerpadsz);
> +	READ32(sess->fore_channel.maxreq_sz);
> +	READ32(sess->fore_channel.maxresp_sz);
> +	READ32(sess->fore_channel.maxresp_cached);
> +	READ32(sess->fore_channel.maxops);
> +	READ32(sess->fore_channel.maxreqs);
> +	READ32(sess->fore_channel.nr_rdma_attrs);
> +	if (sess->fore_channel.nr_rdma_attrs == 1) {
> +		READ_BUF(4);
> +		READ32(sess->fore_channel.rdma_attrs);
> +	} else if (sess->fore_channel.nr_rdma_attrs > 1) {
> +		dprintk("Too many fore channel attr bitmaps!\n");
> +		goto xdr_error;
> +	}
> +
> +	/* Back channel attrs */
> +	READ_BUF(28);
> +	READ32(sess->back_channel.headerpadsz);
> +	READ32(sess->back_channel.maxreq_sz);
> +	READ32(sess->back_channel.maxresp_sz);
> +	READ32(sess->back_channel.maxresp_cached);
> +	READ32(sess->back_channel.maxops);
> +	READ32(sess->back_channel.maxreqs);
> +	READ32(sess->back_channel.nr_rdma_attrs);
> +	if (sess->back_channel.nr_rdma_attrs == 1) {
> +		READ_BUF(4);
> +		READ32(sess->back_channel.rdma_attrs);
> +	} else if (sess->back_channel.nr_rdma_attrs > 1) {
> +		dprintk("Too many back channel attr bitmaps!\n");
> +		goto xdr_error;
> +	}
> +
> +	READ_BUF(8);
> +	READ32(sess->callback_prog);
> +
> +	/* callback_sec_params4 */
> +	READ32(nr_secflavs);
> +	for (i = 0; i < nr_secflavs; ++i) {
> +		READ_BUF(4);
> +		READ32(dummy);
> +		switch (dummy) {
> +		case RPC_AUTH_NULL:
> +			/* Nothing to read */
> +			break;
> +		case RPC_AUTH_UNIX:
> +			READ_BUF(8);
> +			/* stamp */
> +			READ32(dummy);
> +
> +			/* machine name */
> +			READ32(dummy);
> +			READ_BUF(dummy);
> +			SAVEMEM(machine_name, dummy);
> +
> +			/* uid, gid */
> +			READ_BUF(8);
> +			READ32(sess->uid);
> +			READ32(sess->gid);
> +
> +			/* more gids */
> +			READ_BUF(4);
> +			READ32(dummy);
> +			READ_BUF(dummy * 4);
> +			for (i = 0; i < dummy; ++i)
> +				READ32(dummy);
> +			break;
> +		case RPC_AUTH_GSS:
> +			dprintk("RPC_AUTH_GSS callback secflavor "
> +				"not supported!\n");
> +			READ_BUF(8);
> +			/* gcbp_service */
> +			READ32(dummy);
> +			/* gcbp_handle_from_server */
> +			READ32(dummy);
> +			READ_BUF(dummy);
> +			p += XDR_QUADLEN(dummy);
> +			/* gcbp_handle_from_client */
> +			READ_BUF(4);
> +			READ32(dummy);
> +			READ_BUF(dummy);
> +			p += XDR_QUADLEN(dummy);
> +			break;
> +		default:
> +			dprintk("Illegal callback secflavor\n");
> +			return nfserr_inval;
> +		}
> +	}
> +	DECODE_TAIL;
>  }
>  
>  static __be32
> @@ -2839,7 +2940,49 @@ static __be32
>  nfsd4_encode_create_session(struct nfsd4_compoundres *resp, int nfserr,
>  			    struct nfsd4_create_session *sess)
>  {
> -	/* stub */
> +	ENCODE_HEAD;
> +
> +	if (nfserr)
> +		goto out;
> +
> +	RESERVE_SPACE(24);
> +	WRITEMEM(sess->sessionid, sizeof(sess->sessionid));
> +	WRITE32(sess->seqid);
> +	WRITE32(sess->flags);
> +	ADJUST_ARGS();
> +
> +	RESERVE_SPACE(28);
> +	WRITE32(sess->fore_channel.headerpadsz);
> +	WRITE32(sess->fore_channel.maxreq_sz);
> +	WRITE32(sess->fore_channel.maxresp_sz);
> +	WRITE32(sess->fore_channel.maxresp_cached);
> +	WRITE32(sess->fore_channel.maxops);
> +	WRITE32(sess->fore_channel.maxreqs);
> +	WRITE32(sess->fore_channel.nr_rdma_attrs);
> +	ADJUST_ARGS();
> +
> +	if (sess->fore_channel.nr_rdma_attrs) {
> +		RESERVE_SPACE(4);
> +		WRITE32(sess->fore_channel.rdma_attrs);
> +		ADJUST_ARGS();
> +	}
> +
> +	RESERVE_SPACE(28);
> +	WRITE32(sess->back_channel.headerpadsz);
> +	WRITE32(sess->back_channel.maxreq_sz);
> +	WRITE32(sess->back_channel.maxresp_sz);
> +	WRITE32(sess->back_channel.maxresp_cached);
> +	WRITE32(sess->back_channel.maxops);
> +	WRITE32(sess->back_channel.maxreqs);
> +	WRITE32(sess->back_channel.nr_rdma_attrs);
> +	ADJUST_ARGS();
> +
> +	if (sess->back_channel.nr_rdma_attrs) {
> +		RESERVE_SPACE(4);
> +		WRITE32(sess->back_channel.rdma_attrs);
> +		ADJUST_ARGS();
> +	}
> +out:
>  	return nfserr;
>  }
>  
> diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
> index 9bbfd88..94b5366 100644
> --- a/include/linux/nfsd/state.h
> +++ b/include/linux/nfsd/state.h
> @@ -217,6 +217,9 @@ struct nfs4_client {
>  #if defined(CONFIG_NFSD_V4_1)
>  	u32			cl_seqid;	/* seqid for create_session */
>  	u32			cl_exchange_flags;
> +	nfs41_sessionid		cl_sessionid;
> +
> +	struct svc_xprt		*cl_cb_xprt;	/* 4.1 callback transport */
>  #endif /* CONFIG_NFSD_V4_1 */
>  };
>  
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index aee8996..75d93ab 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -362,8 +362,27 @@ struct nfsd4_exchange_id {
>  	u32		seqid;
>  };
>  
> +struct nfsd4_channel {
> +	u32		headerpadsz;
> +	u32		maxreq_sz;
> +	u32		maxresp_sz;
> +	u32		maxresp_cached;
> +	u32		maxops;
> +	u32		maxreqs;
> +	u32		nr_rdma_attrs;
> +	u32		rdma_attrs;
> +};
> +
>  struct nfsd4_create_session {
> -	int	foo;	/* stub */
> +	clientid_t		clientid;
> +	nfs41_sessionid		sessionid;
> +	u32			seqid;
> +	u32			flags;
> +	struct nfsd4_channel	fore_channel;
> +	struct nfsd4_channel	back_channel;
> +	u32			callback_prog;
> +	u32			uid;
> +	u32			gid;
>  };
>  
>  struct nfsd4_sequence {

[RFC 26/51] nfsd41: destroy_session operation

[Benny Halevy]

Implement the destory_session operation confoming to
http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c       |   26 +++++++++++++++++++++++++-
 fs/nfsd/nfs4xdr.c         |    7 +++++--
 include/linux/nfsd/xdr4.h |    2 +-
 3 files changed, 31 insertions(+), 4 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 73bc8de..77b89b6 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -1050,7 +1050,31 @@ nfsd4_destroy_session(struct svc_rqst *r,
 		      struct nfsd4_compound_state *cstate,
 		      struct nfsd4_destroy_session *sessionid)
 {
-	return -1;	/* stub */
+	struct nfs41_session *ses;
+	u32 status = nfserr_badsession;
+
+	/* Notes:
+	 * - The confirmed nfs4_client->cl_sessionid holds destroyed sessinid
+	 * - Should we return nfserr_back_chan_busy if waiting for
+	 *   callbacks on to-be-destroyed session?
+	 * - Do we need to clear any callback info from previous session?
+	 */
+
+	dump_sessionid(__func__, &sessionid->sessionid);
+	nfs4_lock_state();
+	ses = find_in_sessionid_hashtbl(&sessionid->sessionid);
+	if (!ses)
+		goto out;
+
+	/* wait for callbacks */
+	shutdown_callback_client(ses->se_client);
+
+	destroy_session(ses);
+	status = nfs_ok;
+out:
+	nfs4_unlock_state();
+	dprintk("%s returns %d\n", __func__, ntohl(status));
+	return status;
 }
 
 __be32
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 02c82d4..f49ddde 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -1227,7 +1227,11 @@ static __be32
 nfsd4_decode_destroy_session(struct nfsd4_compoundargs *argp,
 			     struct nfsd4_destroy_session *destroy_session)
 {
-	return nfserr_opnotsupp;	/* stub */
+	DECODE_HEAD;
+	READ_BUF(NFS4_MAX_SESSIONID_LEN);
+	COPYMEM(destroy_session->sessionid, NFS4_MAX_SESSIONID_LEN);
+
+	DECODE_TAIL;
 }
 
 static __be32
@@ -2990,7 +2994,6 @@ static __be32
 nfsd4_encode_destroy_session(struct nfsd4_compoundres *resp, int nfserr,
 			     struct nfsd4_destroy_session *destroy_session)
 {
-	/* stub */
 	return nfserr;
 }
 
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index 75d93ab..8b166d8 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -390,7 +390,7 @@ struct nfsd4_sequence {
 };
 
 struct nfsd4_destroy_session {
-	int	foo;	/* stub */
+	nfs41_sessionid		sessionid;
 };
 #endif /* CONFIG_NFSD_V4_1 */
 
-- 
1.6.0.2

Re: [pnfs] [RFC 26/51] nfsd41: destroy_session operation

[Benny Halevy]

On Nov. 10, 2008, 22:49 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> Implement the destory_session operation confoming to
> http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4state.c       |   26 +++++++++++++++++++++++++-
>  fs/nfsd/nfs4xdr.c         |    7 +++++--
>  include/linux/nfsd/xdr4.h |    2 +-
>  3 files changed, 31 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 73bc8de..77b89b6 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -1050,7 +1050,31 @@ nfsd4_destroy_session(struct svc_rqst *r,
>  		      struct nfsd4_compound_state *cstate,
>  		      struct nfsd4_destroy_session *sessionid)
>  {
> -	return -1;	/* stub */
> +	struct nfs41_session *ses;
> +	u32 status = nfserr_badsession;
> +
> +	/* Notes:
> +	 * - The confirmed nfs4_client->cl_sessionid holds destroyed sessinid
> +	 * - Should we return nfserr_back_chan_busy if waiting for
> +	 *   callbacks on to-be-destroyed session?
> +	 * - Do we need to clear any callback info from previous session?
> +	 */
> +
> +	dump_sessionid(__func__, &sessionid->sessionid);
> +	nfs4_lock_state();
> +	ses = find_in_sessionid_hashtbl(&sessionid->sessionid);
> +	if (!ses)
> +		goto out;
> +
> +	/* wait for callbacks */
> +	shutdown_callback_client(ses->se_client);

review 11-12: we need ref. counting for that backchannel
to support trunking.
clp->cl_callback.cb_client is shared?

> +
> +	destroy_session(ses);
> +	status = nfs_ok;
> +out:
> +	nfs4_unlock_state();
> +	dprintk("%s returns %d\n", __func__, ntohl(status));
> +	return status;
>  }
>  
>  __be32
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index 02c82d4..f49ddde 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -1227,7 +1227,11 @@ static __be32
>  nfsd4_decode_destroy_session(struct nfsd4_compoundargs *argp,
>  			     struct nfsd4_destroy_session *destroy_session)
>  {
> -	return nfserr_opnotsupp;	/* stub */
> +	DECODE_HEAD;
> +	READ_BUF(NFS4_MAX_SESSIONID_LEN);
> +	COPYMEM(destroy_session->sessionid, NFS4_MAX_SESSIONID_LEN);
> +
> +	DECODE_TAIL;
>  }
>  
>  static __be32
> @@ -2990,7 +2994,6 @@ static __be32
>  nfsd4_encode_destroy_session(struct nfsd4_compoundres *resp, int nfserr,
>  			     struct nfsd4_destroy_session *destroy_session)
>  {
> -	/* stub */
>  	return nfserr;
>  }
>  
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index 75d93ab..8b166d8 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -390,7 +390,7 @@ struct nfsd4_sequence {
>  };
>  
>  struct nfsd4_destroy_session {
> -	int	foo;	/* stub */
> +	nfs41_sessionid		sessionid;
>  };
>  #endif /* CONFIG_NFSD_V4_1 */
>  

[RFC 27/51] nfsd41: sequence operation

[Benny Halevy]

Implement the sequece operation conforming to
http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26

Check for stale clientid (as derived from the sessionid).
Enforce slotid range and exactly-once semantics using
the slotid and seqid.

If everything went well renew the client lease and
mark the slot INPROGRESS.

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c       |   88 ++++++++++++++++++++++++++++++++++++++++++++-
 fs/nfsd/nfs4xdr.c         |   27 +++++++++++++-
 include/linux/nfsd/xdr4.h |    8 ++++-
 3 files changed, 119 insertions(+), 4 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 77b89b6..1a9d121 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -1077,12 +1077,98 @@ out:
 	return status;
 }
 
+static int
+check_slot_seqid(u32 seqid, struct nfs41_slot *slot)
+{
+	dprintk("%s enter. seqid %d slot->sl_seqid %d\n", __func__, seqid,
+		slot->sl_seqid);
+	/* Normal */
+	if (likely(seqid == slot->sl_seqid + 1))
+		return nfs_ok;
+	/* Replay */
+	if (seqid == slot->sl_seqid)
+		return NFSERR_REPLAY_ME;
+	/* Wraparound */
+	if (seqid == 1 && (slot->sl_seqid + 1) == 0)
+		return nfs_ok;
+	/* Misordered replay or misordered new request */
+	return nfserr_seq_misordered;
+}
+
 __be32
 nfsd4_sequence(struct svc_rqst *r,
 	       struct nfsd4_compound_state *cstate,
 	       struct nfsd4_sequence *seq)
 {
-	return -1;	/* stub */
+	struct nfs41_session *elem;
+	struct nfs41_slot *slot;
+	struct current_session *c_ses = cstate->current_ses;
+	int status;
+
+	if (STALE_CLIENTID((clientid_t *)seq->sessionid))
+		return nfserr_badsession;
+
+	nfs4_lock_state();
+	status = nfserr_badsession;
+	elem = find_in_sessionid_hashtbl(&seq->sessionid);
+	if (!elem)
+		goto out;
+
+	status = nfserr_badslot;
+	if (seq->slotid >= elem->se_fnumslots)
+		goto out;
+
+	slot = &elem->se_slots[seq->slotid];
+	dprintk("%s: slotid %d\n", __func__, seq->slotid);
+
+	/* Server post op_sequence compound processing had an upcall which
+	 * resulted in replaying the compound processing including the
+	 * already processed op_sequence. Set current_session
+	 * but don't bump slot->sl_seqid which was incremented in successful
+	 * op_sequence processing prior to upcall.
+	 */
+	if (nfs41_get_slot_state(slot) == NFS4_SLOT_INPROGRESS) {
+		dprintk("%s: NFS4_SLOT_INPROGRESS. set current_session\n",
+			__func__);
+		goto set_curr_ses;
+	}
+
+	status = check_slot_seqid(seq->seqid, slot);
+	if (status == NFSERR_REPLAY_ME)
+		goto replay;
+	else if (status)
+		goto out;
+
+	/* Success! bump slot seqid and renew clientid */
+	slot->sl_seqid = seq->seqid;
+	renew_client(elem->se_client);
+	dprintk("%s: set NFS4_SLOT_INPROGRESS\n", __func__);
+	nfs41_set_slot_state(slot, NFS4_SLOT_INPROGRESS);
+
+set_curr_ses:
+	/* Set current_session. hold reference until done processing compound.
+	 * nfs41_put_session called only if cs_slot is set
+	 */
+	memcpy(&c_ses->cs_sid, &seq->sessionid, sizeof(c_ses->cs_sid));
+	BUG_ON(sizeof(c_ses->cs_sid) != sizeof(seq->sessionid));
+	c_ses->cs_slot = slot;
+	nfs41_get_session(slot->sl_session);
+
+	/* FIXME: for now just initialize target_highest_slotid and flags
+	 * response fields */
+	seq->target_maxslots = seq->maxslots;
+	seq->status_flags = 0;
+
+	status = nfs_ok;
+out:
+	dprintk("%s: return %d\n", __func__, ntohl(status));
+	nfs4_unlock_state();
+	return status;
+replay:
+	dprintk("%s: REPLAY - AKKKK! no code yet! return BAD SESSION\n",
+		__func__);
+	status = nfserr_badsession;
+	goto out;
 }
 #endif /* CONFIG_NFSD_V4_1 */
 
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index f49ddde..6db74f7 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -1238,7 +1238,16 @@ static __be32
 nfsd4_decode_sequence(struct nfsd4_compoundargs *argp,
 		      struct nfsd4_sequence *seq)
 {
-	return nfserr_opnotsupp;	/* stub */
+	DECODE_HEAD;
+
+	READ_BUF(NFS4_MAX_SESSIONID_LEN + 16);
+	COPYMEM(seq->sessionid, NFS4_MAX_SESSIONID_LEN);
+	READ32(seq->seqid);
+	READ32(seq->slotid);
+	READ32(seq->maxslots);
+	READ32(seq->catchthis);
+
+	DECODE_TAIL;
 }
 #endif /* CONFIG_NFSD_V4_1 */
 
@@ -3001,7 +3010,21 @@ static __be32
 nfsd4_encode_sequence(struct nfsd4_compoundres *resp, int nfserr,
 		      struct nfsd4_sequence *seq)
 {
-	/* stub */
+	ENCODE_HEAD;
+
+	if (nfserr)
+		goto out;
+
+	RESERVE_SPACE(NFS4_MAX_SESSIONID_LEN + 20);
+	WRITEMEM(seq->sessionid, NFS4_MAX_SESSIONID_LEN);
+	WRITE32(seq->seqid);
+	WRITE32(seq->slotid);
+	WRITE32(seq->maxslots);
+	WRITE32(seq->target_maxslots);
+	WRITE32(seq->status_flags);
+
+	ADJUST_ARGS();
+out:
 	return nfserr;
 }
 #endif /* CONFIG_NFSD_V4_1 */
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index 8b166d8..d78ba3c 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -386,7 +386,13 @@ struct nfsd4_create_session {
 };
 
 struct nfsd4_sequence {
-	int	foo;	/* stub */
+	nfs41_sessionid		sessionid;		/* request/response */
+	u32			seqid;			/* request/response */
+	u32			slotid;			/* request/response */
+	u32			maxslots;		/* request/response */
+	u32			catchthis;		/* request */
+	u32			target_maxslots;	/* response */
+	u32			status_flags;		/* response */
 };
 
 struct nfsd4_destroy_session {
-- 
1.6.0.2

Re: [pnfs] [RFC 27/51] nfsd41: sequence operation

[Benny Halevy]

On Nov. 10, 2008, 22:49 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> Implement the sequece operation conforming to
> http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26
> 
> Check for stale clientid (as derived from the sessionid).
> Enforce slotid range and exactly-once semantics using
> the slotid and seqid.
> 
> If everything went well renew the client lease and
> mark the slot INPROGRESS.
> 
> Signed-off-by: Andy Adamson<andros@netapp.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4state.c       |   88 ++++++++++++++++++++++++++++++++++++++++++++-
>  fs/nfsd/nfs4xdr.c         |   27 +++++++++++++-
>  include/linux/nfsd/xdr4.h |    8 ++++-
>  3 files changed, 119 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 77b89b6..1a9d121 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -1077,12 +1077,98 @@ out:
>  	return status;
>  }
>  
> +static int
> +check_slot_seqid(u32 seqid, struct nfs41_slot *slot)
> +{
> +	dprintk("%s enter. seqid %d slot->sl_seqid %d\n", __func__, seqid,
> +		slot->sl_seqid);
> +	/* Normal */
> +	if (likely(seqid == slot->sl_seqid + 1))
> +		return nfs_ok;
> +	/* Replay */
> +	if (seqid == slot->sl_seqid)
> +		return NFSERR_REPLAY_ME;
> +	/* Wraparound */
> +	if (seqid == 1 && (slot->sl_seqid + 1) == 0)
> +		return nfs_ok;
> +	/* Misordered replay or misordered new request */
> +	return nfserr_seq_misordered;
> +}
> +
>  __be32
>  nfsd4_sequence(struct svc_rqst *r,
>  	       struct nfsd4_compound_state *cstate,
>  	       struct nfsd4_sequence *seq)
>  {
> -	return -1;	/* stub */
> +	struct nfs41_session *elem;
> +	struct nfs41_slot *slot;
> +	struct current_session *c_ses = cstate->current_ses;
> +	int status;
> +
> +	if (STALE_CLIENTID((clientid_t *)seq->sessionid))
> +		return nfserr_badsession;

review 11-13: get rid of that.
the sessionid mustn't be in the hash table.

> +
> +	nfs4_lock_state();
> +	status = nfserr_badsession;
> +	elem = find_in_sessionid_hashtbl(&seq->sessionid);

review 11-13: s/elem/session/

> +	if (!elem)
> +		goto out;
> +
> +	status = nfserr_badslot;
> +	if (seq->slotid >= elem->se_fnumslots)
> +		goto out;
> +
> +	slot = &elem->se_slots[seq->slotid];
> +	dprintk("%s: slotid %d\n", __func__, seq->slotid);
> +
> +	/* Server post op_sequence compound processing had an upcall which
> +	 * resulted in replaying the compound processing including the
> +	 * already processed op_sequence. Set current_session
> +	 * but don't bump slot->sl_seqid which was incremented in successful
> +	 * op_sequence processing prior to upcall.
> +	 */
> +	if (nfs41_get_slot_state(slot) == NFS4_SLOT_INPROGRESS) {
> +		dprintk("%s: NFS4_SLOT_INPROGRESS. set current_session\n",
> +			__func__);
> +		goto set_curr_ses;
> +	}

squash DRC changes in here.

> +
> +	status = check_slot_seqid(seq->seqid, slot);
> +	if (status == NFSERR_REPLAY_ME)
> +		goto replay;
> +	else if (status)
> +		goto out;
> +
> +	/* Success! bump slot seqid and renew clientid */
> +	slot->sl_seqid = seq->seqid;
> +	renew_client(elem->se_client);
> +	dprintk("%s: set NFS4_SLOT_INPROGRESS\n", __func__);
> +	nfs41_set_slot_state(slot, NFS4_SLOT_INPROGRESS);
> +
> +set_curr_ses:
> +	/* Set current_session. hold reference until done processing compound.
> +	 * nfs41_put_session called only if cs_slot is set
> +	 */
> +	memcpy(&c_ses->cs_sid, &seq->sessionid, sizeof(c_ses->cs_sid));
> +	BUG_ON(sizeof(c_ses->cs_sid) != sizeof(seq->sessionid));

get rid of this BUG_ON.

> +	c_ses->cs_slot = slot;
> +	nfs41_get_session(slot->sl_session);
> +
> +	/* FIXME: for now just initialize target_highest_slotid and flags
> +	 * response fields */
> +	seq->target_maxslots = seq->maxslots;
> +	seq->status_flags = 0;
> +
> +	status = nfs_ok;
> +out:
> +	dprintk("%s: return %d\n", __func__, ntohl(status));
> +	nfs4_unlock_state();
> +	return status;
> +replay:
> +	dprintk("%s: REPLAY - AKKKK! no code yet! return BAD SESSION\n",
> +		__func__);
> +	status = nfserr_badsession;
> +	goto out;
>  }
>  #endif /* CONFIG_NFSD_V4_1 */
>  
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index f49ddde..6db74f7 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -1238,7 +1238,16 @@ static __be32
>  nfsd4_decode_sequence(struct nfsd4_compoundargs *argp,
>  		      struct nfsd4_sequence *seq)
>  {
> -	return nfserr_opnotsupp;	/* stub */
> +	DECODE_HEAD;
> +
> +	READ_BUF(NFS4_MAX_SESSIONID_LEN + 16);
> +	COPYMEM(seq->sessionid, NFS4_MAX_SESSIONID_LEN);
> +	READ32(seq->seqid);
> +	READ32(seq->slotid);
> +	READ32(seq->maxslots);
> +	READ32(seq->catchthis);
> +
> +	DECODE_TAIL;
>  }
>  #endif /* CONFIG_NFSD_V4_1 */
>  
> @@ -3001,7 +3010,21 @@ static __be32
>  nfsd4_encode_sequence(struct nfsd4_compoundres *resp, int nfserr,
>  		      struct nfsd4_sequence *seq)
>  {
> -	/* stub */
> +	ENCODE_HEAD;
> +
> +	if (nfserr)
> +		goto out;
> +
> +	RESERVE_SPACE(NFS4_MAX_SESSIONID_LEN + 20);
> +	WRITEMEM(seq->sessionid, NFS4_MAX_SESSIONID_LEN);
> +	WRITE32(seq->seqid);
> +	WRITE32(seq->slotid);
> +	WRITE32(seq->maxslots);
> +	WRITE32(seq->target_maxslots);
> +	WRITE32(seq->status_flags);
> +
> +	ADJUST_ARGS();
> +out:
>  	return nfserr;
>  }
>  #endif /* CONFIG_NFSD_V4_1 */
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index 8b166d8..d78ba3c 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -386,7 +386,13 @@ struct nfsd4_create_session {
>  };
>  
>  struct nfsd4_sequence {
> -	int	foo;	/* stub */
> +	nfs41_sessionid		sessionid;		/* request/response */
> +	u32			seqid;			/* request/response */
> +	u32			slotid;			/* request/response */
> +	u32			maxslots;		/* request/response */
> +	u32			catchthis;		/* request */
> +	u32			target_maxslots;	/* response */
> +	u32			status_flags;		/* response */

review 11-13: get rid of target_maxslots and status_flags for now.

>  };
>  
>  struct nfsd4_destroy_session {

[RFC 28/51] FIXME: nfsd41: sunrpc: Added rpc server-side backchannel handling

[Benny Halevy]

FIXME: bhalevy: write up commit message

Signed-off-by: Mike Sager <sager@netapp.com>
Signed-off-by: Marc Eshel <eshel@almaden.ibm.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 include/linux/sunrpc/clnt.h    |    1 +
 include/linux/sunrpc/svcsock.h |    1 +
 include/linux/sunrpc/xprt.h    |    4 +
 net/sunrpc/clnt.c              |    1 +
 net/sunrpc/svcsock.c           |   76 +++++++++++
 net/sunrpc/xprt.c              |   41 ++++++-
 net/sunrpc/xprtsock.c          |  284 +++++++++++++++++++++++++++++++++++++++-
 7 files changed, 398 insertions(+), 10 deletions(-)

diff --git a/include/linux/sunrpc/clnt.h b/include/linux/sunrpc/clnt.h
index c6a1179..fe6a699 100644
--- a/include/linux/sunrpc/clnt.h
+++ b/include/linux/sunrpc/clnt.h
@@ -118,6 +118,7 @@ struct rpc_create_args {
 	unsigned char		minorversion;
 	rpc_authflavor_t	authflavor;
 	unsigned long		flags;
+	struct svc_sock		*bc_sock;	/* NFSv4.1 backchannel */
 };
 
 /* Values for "flags" field */
diff --git a/include/linux/sunrpc/svcsock.h b/include/linux/sunrpc/svcsock.h
index 6bb1ec4..e01a1c6 100644
--- a/include/linux/sunrpc/svcsock.h
+++ b/include/linux/sunrpc/svcsock.h
@@ -28,6 +28,7 @@ struct svc_sock {
 	/* private TCP part */
 	u32			sk_reclen;	/* length of record */
 	u32			sk_tcplen;	/* current read length */
+	struct rpc_xprt	       *sk_bc_xprt;	/* NFSv4.1 backchannel xprt */
 };
 
 /*
diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 45a92e2..982dbbc 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -180,6 +180,9 @@ struct rpc_xprt {
 	spinlock_t		reserve_lock;	/* lock slot table */
 	u32			xid;		/* Next XID value to use */
 	struct rpc_task *	snd_task;	/* Task blocked in send */
+#if defined(CONFIG_NFSD_V4_1)
+	struct svc_sock		*bc_sock;	/* NFSv4.1 backchannel */
+#endif /* CONFIG_NFSD_V4_1 */
 #if defined(CONFIG_NFS_V4_1)
 	struct svc_serv		*bc_serv;	/* The RPC service which will */
 						/* process the callback */
@@ -232,6 +235,7 @@ struct xprt_create {
 	struct sockaddr *	srcaddr;	/* optional local address */
 	struct sockaddr *	dstaddr;	/* remote peer address */
 	size_t			addrlen;
+	struct svc_sock		*bc_sock;	/* NFSv4.1 backchannel */
 };
 
 struct xprt_class {
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index fcadd8e..5e8fba0 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -262,6 +262,7 @@ struct rpc_clnt *rpc_create(struct rpc_create_args *args)
 		.srcaddr = args->saddress,
 		.dstaddr = args->address,
 		.addrlen = args->addrsize,
+		.bc_sock = args->bc_sock,
 	};
 	char servername[48];
 
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index c2b6fa4..0cc826d 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -49,6 +49,7 @@
 #include <linux/sunrpc/msg_prot.h>
 #include <linux/sunrpc/svcsock.h>
 #include <linux/sunrpc/stats.h>
+#include <linux/sunrpc/xprt.h>
 
 #define RPCDBG_FACILITY	RPCDBG_SVCXPRT
 
@@ -795,6 +796,9 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp)
 	int		len;
 	struct kvec *vec;
 	int pnum, vlen;
+#if defined(CONFIG_NFSD_V4_1)
+	struct rpc_rqst *req = NULL;
+#endif
 
 	dprintk("svc: tcp_recv %p data %d conn %d close %d\n",
 		svsk, test_bit(XPT_DATA, &svsk->sk_xprt.xpt_flags),
@@ -878,12 +882,71 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp)
 	len = svsk->sk_reclen;
 	set_bit(XPT_DATA, &svsk->sk_xprt.xpt_flags);
 
+	/*
+	 * We have enough data for the whole tcp record. Let's try and read the
+	 * first 8 bytes to get the xid and the call direction. We can use this
+	 * to figure out if this is a call or a reply to a callback. If
+	 * sk_reclen is < 8 (xid and calldir), then this is a malformed packet.
+	 * In that case, don't bother with the calldir and just read the data.
+	 * It will be rejected in svc_process.
+	 */
+
 	vec = rqstp->rq_vec;
 	vec[0] = rqstp->rq_arg.head[0];
 	vlen = PAGE_SIZE;
+
+	if (len >= 8) {
+		u32 *p;
+		u32 xid;
+		u32 calldir;
+
+		len = svc_recvfrom(rqstp, vec, 1, 8);
+		if (len < 0)
+			goto error;
+
+		p = (u32 *)vec[0].iov_base;
+		p = (u32 *)rqstp->rq_arg.head[0].iov_base;
+		xid = *p++;
+		calldir = *p;
+
+#if defined(CONFIG_NFSD_V4_1)
+		if (calldir) {
+			/* REPLY */
+			if (svsk->sk_bc_xprt)
+				req = xprt_lookup_rqst(svsk->sk_bc_xprt, xid);
+			if (req) {
+				memcpy(&req->rq_private_buf, &req->rq_rcv_buf,
+					sizeof(struct xdr_buf));
+				vec[0] = req->rq_private_buf.head[0];
+			} else
+				printk(KERN_NOTICE
+					"%s: Got unrecognized reply: "
+					"calldir 0x%x sk_bc_xprt %p xid %08x\n",
+					__func__, ntohl(calldir),
+					svsk->sk_bc_xprt, xid);
+		}
+
+		if (!calldir || !req)
+			vec[0] = rqstp->rq_arg.head[0];
+
+#else /* CONFIG_NFSD_V4_1 */
+		vec[0] = rqstp->rq_arg.head[0];
+#endif /* CONFIG_NFSD_V4_1 */
+		vec[0].iov_base += 8;
+		vec[0].iov_len -= 8;
+		len = svsk->sk_reclen - 8;
+		vlen -= 8;
+	}
+
 	pnum = 1;
 	while (vlen < len) {
+#if defined(CONFIG_NFSD_V4_1)
+		vec[pnum].iov_base = (req) ?
+			page_address(req->rq_private_buf.pages[pnum - 1]):
+			page_address(rqstp->rq_pages[pnum]);
+#else /* CONFIG_NFSD_V4_1 */
 		vec[pnum].iov_base = page_address(rqstp->rq_pages[pnum]);
+#endif /* CONFIG_NFSD_V4_1 */
 		vec[pnum].iov_len = PAGE_SIZE;
 		pnum++;
 		vlen += PAGE_SIZE;
@@ -895,6 +958,18 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp)
 	if (len < 0)
 		goto error;
 
+	/*
+	 * Account for the 8 bytes we read earlier
+	 */
+	len += 8;
+
+#if defined(CONFIG_NFSD_V4_1)
+	if (req) {
+		xprt_complete_rqst(req->rq_task, len);
+		len = 0;
+		goto out;
+	}
+#endif /* CONFIG_NFSD_V4_1 */
 	dprintk("svc: TCP complete record (%d bytes)\n", len);
 	rqstp->rq_arg.len = len;
 	rqstp->rq_arg.page_base = 0;
@@ -908,6 +983,7 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp)
 	rqstp->rq_xprt_ctxt   = NULL;
 	rqstp->rq_prot	      = IPPROTO_TCP;
 
+out:
 	/* Reset TCP read info */
 	svsk->sk_reclen = 0;
 	svsk->sk_tcplen = 0;
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index e71f1be..5eeb660 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1022,6 +1022,27 @@ void xprt_release(struct rpc_task *task)
 	spin_unlock(&xprt->reserve_lock);
 }
 
+/*
+ * The autoclose function for the back channel
+ *
+ * The callback channel should never close the channel,
+ * let the forechannel do that.
+ */
+static void bc_autoclose(struct work_struct *work)
+{
+	return;
+}
+
+
+/*
+ * The autodisconnect routine for the back channel. We never disconnect
+ */
+static void
+bc_init_autodisconnect(unsigned long data)
+{
+	return;
+}
+
 /**
  * xprt_create_transport - create an RPC transport
  * @args: rpc transport creation arguments
@@ -1063,9 +1084,16 @@ found:
 	INIT_LIST_HEAD(&xprt->bc_pa_list);
 #endif /* CONFIG_NFS_V4_1 */
 
-	INIT_WORK(&xprt->task_cleanup, xprt_autoclose);
-	setup_timer(&xprt->timer, xprt_init_autodisconnect,
-			(unsigned long)xprt);
+	if (args->bc_sock) {
+		INIT_WORK(&xprt->task_cleanup, bc_autoclose);
+		setup_timer(&xprt->timer, bc_init_autodisconnect,
+			    (unsigned long)xprt);
+	} else {
+		INIT_WORK(&xprt->task_cleanup, xprt_autoclose);
+		setup_timer(&xprt->timer, xprt_init_autodisconnect,
+			    (unsigned long)xprt);
+	}
+
 	xprt->last_used = jiffies;
 	xprt->cwnd = RPC_INITCWND;
 	xprt->bind_index = 0;
@@ -1085,6 +1113,13 @@ found:
 	dprintk("RPC:       created transport %p with %u slots\n", xprt,
 			xprt->max_reqs);
 
+	/*
+	 * Since we don't want connections for the backchannel, we set
+	 * the xprt status to connected
+	 */
+	if (args->bc_sock)
+		xprt_set_connected(xprt);
+
 	return xprt;
 }
 
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 8513d50..5531e50 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -32,6 +32,7 @@
 #include <linux/tcp.h>
 #include <linux/sunrpc/clnt.h>
 #include <linux/sunrpc/sched.h>
+#include <linux/sunrpc/svcsock.h>
 #include <linux/sunrpc/xprtsock.h>
 #include <linux/file.h>
 #ifdef CONFIG_NFS_V4_1
@@ -2018,6 +2019,221 @@ static void xs_tcp_print_stats(struct rpc_xprt *xprt, struct seq_file *seq)
 			xprt->stat.bklog_u);
 }
 
+#if defined(CONFIG_NFSD_V4_1)
+/*
+ * The connect worker for the backchannel
+ * This should never be called as we should never need to connect
+ */
+static void bc_connect_worker(struct work_struct *work)
+{
+	BUG();
+}
+
+/*
+ * The set_port routine of the rpc_xprt_ops. This is related to the portmapper
+ * and should never be called
+ */
+
+static void bc_set_port(struct rpc_xprt *xprt, unsigned short port)
+{
+	BUG();
+}
+
+/*
+ * The connect routine for the backchannel rpc_xprt ops
+ * Again, should never be called!
+ */
+
+static void bc_connect(struct rpc_task *task)
+{
+	BUG();
+}
+
+struct rpc_buffer {
+	size_t	len;
+	char	data[];
+};
+/*
+ * Allocate a bunch of pages for a scratch buffer for the rpc code. The reason
+ * we allocate pages instead doing a kmalloc like rpc_malloc is because we want
+ * to use the server side send routines.
+ */
+void *bc_malloc(struct rpc_task *task, size_t size)
+{
+	struct page *page;
+	struct rpc_buffer *buf;
+
+	BUG_ON(size > PAGE_SIZE - sizeof(struct rpc_buffer));
+	page = alloc_page(GFP_KERNEL);
+
+	if (!page)
+		return NULL;
+
+	buf = page_address(page);
+	buf->len = PAGE_SIZE;
+
+	return buf->data;
+}
+
+/*
+ * Free the space allocated in the bc_alloc routine
+ */
+void bc_free(void *buffer)
+{
+	struct rpc_buffer *buf;
+
+	if (!buffer)
+		return;
+
+	buf = container_of(buffer, struct rpc_buffer, data);
+	free_pages((unsigned long)buf, get_order(buf->len));
+}
+
+/*
+ * Use the svc_sock to send the callback. Must be called with svsk->sk_mutex
+ * held. Borrows heavily from svc_tcp_sendto and xs_tcp_semd_request.
+ */
+static int bc_sendto(struct rpc_rqst *req)
+{
+	int total_len;
+	int len;
+	int size;
+	int result;
+	struct xdr_buf *xbufp = &req->rq_snd_buf;
+	struct page **pages = xbufp->pages;
+	unsigned int flags = MSG_MORE;
+	unsigned int pglen = xbufp->page_len;
+	size_t base = xbufp->page_base;
+	struct rpc_xprt *xprt = req->rq_xprt;
+	struct sock_xprt *transport =
+				container_of(xprt, struct sock_xprt, xprt);
+	struct socket *sock = transport->sock;
+
+	total_len = xbufp->len;
+
+	/*
+	 * Set up the rpc header and record marker stuff
+	 */
+	xs_encode_tcp_record_marker(xbufp);
+
+	/*
+	 * The RPC message is divided into 3 pieces:
+	 * - The header: This is what most of the smaller RPC messages consist
+	 *   of. Often the whole message is in this.
+	 *
+	 *   - xdr->pages: This is a list of pages that contain data, for
+	 *   example in a write request or while using rpcsec gss
+	 *
+	 *   - The tail: This is the rest of the rpc message
+	 *
+	 *  First we send the header, then the pages and then finally the tail.
+	 *  The code borrows heavily from svc_sendto.
+	 */
+
+	/*
+	 * Send the head
+	 */
+	if (total_len == xbufp->head[0].iov_len)
+		flags = 0;
+
+	len = sock->ops->sendpage(sock, virt_to_page(xbufp->head[0].iov_base),
+			(unsigned long)xbufp->head[0].iov_base & ~PAGE_MASK,
+			xbufp->head[0].iov_len, flags);
+
+	if (len != xbufp->head[0].iov_len)
+		goto out;
+
+	/*
+	 * send page data
+	 *
+	 * Check the amount of data to be sent. If it is less than the
+	 * remaining page, then send it else send the current page
+	 */
+
+	size = PAGE_SIZE - base < pglen ? PAGE_SIZE - base : pglen;
+	while (pglen > 0) {
+		if (total_len == size)
+			flags = 0;
+		result = sock->ops->sendpage(sock, *pages, base, size, flags);
+		if (result > 0)
+			len += result;
+		if (result != size)
+			goto out;
+		total_len -= size;
+		pglen -= size;
+		size = PAGE_SIZE < pglen ? PAGE_SIZE : pglen;
+		base = 0;
+		pages++;
+	}
+	/*
+	 * send tail
+	 */
+	if (xbufp->tail[0].iov_len) {
+		result = sock->ops->sendpage(sock,
+			xbufp->tail[0].iov_base,
+			(unsigned long)xbufp->tail[0].iov_base & ~PAGE_MASK,
+			xbufp->tail[0].iov_len,
+			0);
+
+		if (result > 0)
+			len += result;
+	}
+out:
+	if (len != xbufp->len)
+		printk(KERN_NOTICE "Error sending entire callback!\n");
+
+	return len;
+}
+
+/*
+ * The send routine. Borrows from svc_send
+ */
+static int bc_send_request(struct rpc_task *task)
+{
+	struct rpc_rqst *req = task->tk_rqstp;
+	struct rpc_xprt *bc_xprt = req->rq_xprt;
+	struct svc_xprt	*xprt;
+	struct svc_sock         *svsk;
+	u32                     len;
+
+	dprintk("sending request with xid: %08x\n", ntohl(req->rq_xid));
+	/*
+	 * Get the server socket associated with this callback xprt
+	 */
+	svsk = bc_xprt->bc_sock;
+	xprt = &svsk->sk_xprt;
+
+	mutex_lock(&xprt->xpt_mutex);
+	if (test_bit(XPT_DEAD, &xprt->xpt_flags))
+		len = -ENOTCONN;
+	else
+		len = bc_sendto(req);
+	mutex_unlock(&xprt->xpt_mutex);
+
+	return 0;
+
+}
+
+/*
+ * The close routine. Since this is client initiated, we do nothing
+ */
+
+static void bc_close(struct rpc_xprt *xprt)
+{
+	return;
+}
+
+/*
+ * The xprt destroy routine. Again, because this connection is client
+ * initiated, we do nothing
+ */
+
+static void bc_destroy(struct rpc_xprt *xprt)
+{
+	return;
+}
+#endif /* CONFIG_NFSD_V4_1 */
+
 static struct rpc_xprt_ops xs_udp_ops = {
 	.set_buffer_size	= xs_udp_set_buffer_size,
 	.reserve_xprt		= xprt_reserve_xprt_cong,
@@ -2054,6 +2270,26 @@ static struct rpc_xprt_ops xs_tcp_ops = {
 	.print_stats		= xs_tcp_print_stats,
 };
 
+#if defined(CONFIG_NFSD_V4_1)
+/*
+ * The rpc_xprt_ops for the server backchannel
+ */
+
+static struct rpc_xprt_ops bc_tcp_ops = {
+	.reserve_xprt		= xprt_reserve_xprt,
+	.release_xprt		= xprt_release_xprt,
+	.set_port		= bc_set_port,
+	.connect		= bc_connect,
+	.buf_alloc		= bc_malloc,
+	.buf_free		= bc_free,
+	.send_request		= bc_send_request,
+	.set_retrans_timeout	= xprt_set_retrans_timeout_def,
+	.close			= bc_close,
+	.destroy		= bc_destroy,
+	.print_stats		= xs_tcp_print_stats,
+};
+#endif /* CONFIG_NFSD_V4_1 */
+
 static struct rpc_xprt *xs_setup_xprt(struct xprt_create *args,
 				      unsigned int slot_table_size)
 {
@@ -2186,13 +2422,31 @@ static struct rpc_xprt *xs_setup_tcp(struct xprt_create *args)
 	xprt->tsh_size = sizeof(rpc_fraghdr) / sizeof(u32);
 	xprt->max_payload = RPC_MAX_FRAGMENT_SIZE;
 
-	xprt->bind_timeout = XS_BIND_TO;
-	xprt->connect_timeout = XS_TCP_CONN_TO;
-	xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
-	xprt->idle_timeout = XS_IDLE_DISC_TO;
+#if defined(CONFIG_NFSD_V4_1)
+	if (args->bc_sock) {
+		/* backchannel */
+		xprt_set_bound(xprt);
+		INIT_DELAYED_WORK(&transport->connect_worker,
+				  bc_connect_worker);
+		xprt->bind_timeout = 0;
+		xprt->connect_timeout = 0;
+		xprt->reestablish_timeout = 0;
+		xprt->idle_timeout = (~0);
 
-	xprt->ops = &xs_tcp_ops;
-	xprt->timeout = &xs_tcp_default_timeout;
+		/*
+		 * The backchannel uses the same socket connection as the
+		 * forechannel
+		 */
+		xprt->bc_sock = args->bc_sock;
+		xprt->bc_sock->sk_bc_xprt = xprt;
+		transport->sock = xprt->bc_sock->sk_sock;
+		transport->inet = xprt->bc_sock->sk_sk;
+
+		xprt->ops = &bc_tcp_ops;
+
+		goto next;
+	}
+#endif /* CONFIG_NFSD_V4_1 */
 
 	switch (addr->sa_family) {
 	case AF_INET:
@@ -2200,13 +2454,29 @@ static struct rpc_xprt *xs_setup_tcp(struct xprt_create *args)
 			xprt_set_bound(xprt);
 
 		INIT_DELAYED_WORK(&transport->connect_worker, xs_tcp_connect_worker4);
-		xs_format_ipv4_peer_addresses(xprt, "tcp", RPCBIND_NETID_TCP);
 		break;
 	case AF_INET6:
 		if (((struct sockaddr_in6 *)addr)->sin6_port != htons(0))
 			xprt_set_bound(xprt);
 
 		INIT_DELAYED_WORK(&transport->connect_worker, xs_tcp_connect_worker6);
+		break;
+	}
+	xprt->bind_timeout = XS_BIND_TO;
+	xprt->connect_timeout = XS_TCP_CONN_TO;
+	xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
+	xprt->idle_timeout = XS_IDLE_DISC_TO;
+
+	xprt->ops = &xs_tcp_ops;
+
+next:
+	xprt->timeout = &xs_tcp_default_timeout;
+
+	switch (addr->sa_family) {
+	case AF_INET:
+		xs_format_ipv4_peer_addresses(xprt, "tcp", RPCBIND_NETID_TCP);
+		break;
+	case AF_INET6:
 		xs_format_ipv6_peer_addresses(xprt, "tcp", RPCBIND_NETID_TCP6);
 		break;
 	default:
-- 
1.6.0.2

Re: [pnfs] [RFC 28/51] FIXME: nfsd41: sunrpc: Added rpc server-side backchannel handling

[Benny Halevy]

On Nov. 10, 2008, 22:50 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> FIXME: bhalevy: write up commit message
> 
> Signed-off-by: Mike Sager <sager@netapp.com>
> Signed-off-by: Marc Eshel <eshel@almaden.ibm.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 11-13: look into using a similar design to the rdma xprt.
we're essentially adding a new transport type which will eventually
be multi-xprt and will work over multiple connections of different
xprts.

> ---
>  include/linux/sunrpc/clnt.h    |    1 +
>  include/linux/sunrpc/svcsock.h |    1 +
>  include/linux/sunrpc/xprt.h    |    4 +
>  net/sunrpc/clnt.c              |    1 +
>  net/sunrpc/svcsock.c           |   76 +++++++++++
>  net/sunrpc/xprt.c              |   41 ++++++-
>  net/sunrpc/xprtsock.c          |  284 +++++++++++++++++++++++++++++++++++++++-
>  7 files changed, 398 insertions(+), 10 deletions(-)
> 
> diff --git a/include/linux/sunrpc/clnt.h b/include/linux/sunrpc/clnt.h
> index c6a1179..fe6a699 100644
> --- a/include/linux/sunrpc/clnt.h
> +++ b/include/linux/sunrpc/clnt.h
> @@ -118,6 +118,7 @@ struct rpc_create_args {
>  	unsigned char		minorversion;
>  	rpc_authflavor_t	authflavor;
>  	unsigned long		flags;
> +	struct svc_sock		*bc_sock;	/* NFSv4.1 backchannel */
>  };
>  
>  /* Values for "flags" field */
> diff --git a/include/linux/sunrpc/svcsock.h b/include/linux/sunrpc/svcsock.h
> index 6bb1ec4..e01a1c6 100644
> --- a/include/linux/sunrpc/svcsock.h
> +++ b/include/linux/sunrpc/svcsock.h
> @@ -28,6 +28,7 @@ struct svc_sock {
>  	/* private TCP part */
>  	u32			sk_reclen;	/* length of record */
>  	u32			sk_tcplen;	/* current read length */
> +	struct rpc_xprt	       *sk_bc_xprt;	/* NFSv4.1 backchannel xprt */
>  };
>  
>  /*
> diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
> index 45a92e2..982dbbc 100644
> --- a/include/linux/sunrpc/xprt.h
> +++ b/include/linux/sunrpc/xprt.h
> @@ -180,6 +180,9 @@ struct rpc_xprt {
>  	spinlock_t		reserve_lock;	/* lock slot table */
>  	u32			xid;		/* Next XID value to use */
>  	struct rpc_task *	snd_task;	/* Task blocked in send */
> +#if defined(CONFIG_NFSD_V4_1)

review 11-13: get rid of #ifdef?
straighten out the naming convention!!!!!!!
(bc_svc_* on the client/rpc receiver and bc_* on the server, rpc sender)

> +	struct svc_sock		*bc_sock;	/* NFSv4.1 backchannel */
> +#endif /* CONFIG_NFSD_V4_1 */
>  #if defined(CONFIG_NFS_V4_1)
>  	struct svc_serv		*bc_serv;	/* The RPC service which will */
>  						/* process the callback */
> @@ -232,6 +235,7 @@ struct xprt_create {
>  	struct sockaddr *	srcaddr;	/* optional local address */
>  	struct sockaddr *	dstaddr;	/* remote peer address */
>  	size_t			addrlen;
> +	struct svc_sock		*bc_sock;	/* NFSv4.1 backchannel */
>  };
>  
>  struct xprt_class {
> diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
> index fcadd8e..5e8fba0 100644
> --- a/net/sunrpc/clnt.c
> +++ b/net/sunrpc/clnt.c
> @@ -262,6 +262,7 @@ struct rpc_clnt *rpc_create(struct rpc_create_args *args)
>  		.srcaddr = args->saddress,
>  		.dstaddr = args->address,
>  		.addrlen = args->addrsize,
> +		.bc_sock = args->bc_sock,
>  	};
>  	char servername[48];
>  
> diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
> index c2b6fa4..0cc826d 100644
> --- a/net/sunrpc/svcsock.c
> +++ b/net/sunrpc/svcsock.c
> @@ -49,6 +49,7 @@
>  #include <linux/sunrpc/msg_prot.h>
>  #include <linux/sunrpc/svcsock.h>
>  #include <linux/sunrpc/stats.h>
> +#include <linux/sunrpc/xprt.h>
>  
>  #define RPCDBG_FACILITY	RPCDBG_SVCXPRT
>  
> @@ -795,6 +796,9 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp)
>  	int		len;
>  	struct kvec *vec;
>  	int pnum, vlen;
> +#if defined(CONFIG_NFSD_V4_1)
> +	struct rpc_rqst *req = NULL;
> +#endif
>  
>  	dprintk("svc: tcp_recv %p data %d conn %d close %d\n",
>  		svsk, test_bit(XPT_DATA, &svsk->sk_xprt.xpt_flags),
> @@ -878,12 +882,71 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp)
>  	len = svsk->sk_reclen;
>  	set_bit(XPT_DATA, &svsk->sk_xprt.xpt_flags);
>  
> +	/*
> +	 * We have enough data for the whole tcp record. Let's try and read the
> +	 * first 8 bytes to get the xid and the call direction. We can use this
> +	 * to figure out if this is a call or a reply to a callback. If
> +	 * sk_reclen is < 8 (xid and calldir), then this is a malformed packet.
> +	 * In that case, don't bother with the calldir and just read the data.
> +	 * It will be rejected in svc_process.
> +	 */
> +
>  	vec = rqstp->rq_vec;
>  	vec[0] = rqstp->rq_arg.head[0];
>  	vlen = PAGE_SIZE;
> +
> +	if (len >= 8) {
> +		u32 *p;
> +		u32 xid;
> +		u32 calldir;
> +
> +		len = svc_recvfrom(rqstp, vec, 1, 8);
> +		if (len < 0)
> +			goto error;
> +
> +		p = (u32 *)vec[0].iov_base;
> +		p = (u32 *)rqstp->rq_arg.head[0].iov_base;
> +		xid = *p++;
> +		calldir = *p;
> +
> +#if defined(CONFIG_NFSD_V4_1)
> +		if (calldir) {
> +			/* REPLY */
> +			if (svsk->sk_bc_xprt)
> +				req = xprt_lookup_rqst(svsk->sk_bc_xprt, xid);
> +			if (req) {
> +				memcpy(&req->rq_private_buf, &req->rq_rcv_buf,
> +					sizeof(struct xdr_buf));
> +				vec[0] = req->rq_private_buf.head[0];
> +			} else
> +				printk(KERN_NOTICE
> +					"%s: Got unrecognized reply: "
> +					"calldir 0x%x sk_bc_xprt %p xid %08x\n",
> +					__func__, ntohl(calldir),
> +					svsk->sk_bc_xprt, xid);
> +		}
> +
> +		if (!calldir || !req)
> +			vec[0] = rqstp->rq_arg.head[0];
> +
> +#else /* CONFIG_NFSD_V4_1 */
> +		vec[0] = rqstp->rq_arg.head[0];
> +#endif /* CONFIG_NFSD_V4_1 */
> +		vec[0].iov_base += 8;
> +		vec[0].iov_len -= 8;
> +		len = svsk->sk_reclen - 8;
> +		vlen -= 8;
> +	}
> +
>  	pnum = 1;
>  	while (vlen < len) {
> +#if defined(CONFIG_NFSD_V4_1)
> +		vec[pnum].iov_base = (req) ?
> +			page_address(req->rq_private_buf.pages[pnum - 1]):
> +			page_address(rqstp->rq_pages[pnum]);
> +#else /* CONFIG_NFSD_V4_1 */
>  		vec[pnum].iov_base = page_address(rqstp->rq_pages[pnum]);
> +#endif /* CONFIG_NFSD_V4_1 */
>  		vec[pnum].iov_len = PAGE_SIZE;
>  		pnum++;
>  		vlen += PAGE_SIZE;
> @@ -895,6 +958,18 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp)
>  	if (len < 0)
>  		goto error;
>  
> +	/*
> +	 * Account for the 8 bytes we read earlier
> +	 */
> +	len += 8;
> +
> +#if defined(CONFIG_NFSD_V4_1)
> +	if (req) {
> +		xprt_complete_rqst(req->rq_task, len);
> +		len = 0;
> +		goto out;
> +	}
> +#endif /* CONFIG_NFSD_V4_1 */
>  	dprintk("svc: TCP complete record (%d bytes)\n", len);
>  	rqstp->rq_arg.len = len;
>  	rqstp->rq_arg.page_base = 0;
> @@ -908,6 +983,7 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp)
>  	rqstp->rq_xprt_ctxt   = NULL;
>  	rqstp->rq_prot	      = IPPROTO_TCP;
>  
> +out:
>  	/* Reset TCP read info */
>  	svsk->sk_reclen = 0;
>  	svsk->sk_tcplen = 0;
> diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
> index e71f1be..5eeb660 100644
> --- a/net/sunrpc/xprt.c
> +++ b/net/sunrpc/xprt.c
> @@ -1022,6 +1022,27 @@ void xprt_release(struct rpc_task *task)
>  	spin_unlock(&xprt->reserve_lock);
>  }
>  
> +/*
> + * The autoclose function for the back channel
> + *
> + * The callback channel should never close the channel,
> + * let the forechannel do that.
> + */
> +static void bc_autoclose(struct work_struct *work)
> +{
> +	return;
> +}
> +
> +
> +/*
> + * The autodisconnect routine for the back channel. We never disconnect
> + */
> +static void
> +bc_init_autodisconnect(unsigned long data)

review 11-13: rename to noop_init_autodisconnect and noop_autoclose.
> +{
> +	return;
> +}
> +
>  /**
>   * xprt_create_transport - create an RPC transport
>   * @args: rpc transport creation arguments
> @@ -1063,9 +1084,16 @@ found:
>  	INIT_LIST_HEAD(&xprt->bc_pa_list);
>  #endif /* CONFIG_NFS_V4_1 */
>  
> -	INIT_WORK(&xprt->task_cleanup, xprt_autoclose);
> -	setup_timer(&xprt->timer, xprt_init_autodisconnect,
> -			(unsigned long)xprt);
> +	if (args->bc_sock) {
> +		INIT_WORK(&xprt->task_cleanup, bc_autoclose);
> +		setup_timer(&xprt->timer, bc_init_autodisconnect,
> +			    (unsigned long)xprt);
> +	} else {
> +		INIT_WORK(&xprt->task_cleanup, xprt_autoclose);
> +		setup_timer(&xprt->timer, xprt_init_autodisconnect,
> +			    (unsigned long)xprt);
> +	}
> +
>  	xprt->last_used = jiffies;
>  	xprt->cwnd = RPC_INITCWND;
>  	xprt->bind_index = 0;
> @@ -1085,6 +1113,13 @@ found:
>  	dprintk("RPC:       created transport %p with %u slots\n", xprt,
>  			xprt->max_reqs);
>  
> +	/*
> +	 * Since we don't want connections for the backchannel, we set
> +	 * the xprt status to connected
> +	 */
> +	if (args->bc_sock)
> +		xprt_set_connected(xprt);
> +
>  	return xprt;
>  }
>  
> diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
> index 8513d50..5531e50 100644
> --- a/net/sunrpc/xprtsock.c
> +++ b/net/sunrpc/xprtsock.c
> @@ -32,6 +32,7 @@
>  #include <linux/tcp.h>
>  #include <linux/sunrpc/clnt.h>
>  #include <linux/sunrpc/sched.h>
> +#include <linux/sunrpc/svcsock.h>
>  #include <linux/sunrpc/xprtsock.h>
>  #include <linux/file.h>
>  #ifdef CONFIG_NFS_V4_1
> @@ -2018,6 +2019,221 @@ static void xs_tcp_print_stats(struct rpc_xprt *xprt, struct seq_file *seq)
>  			xprt->stat.bklog_u);
>  }
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +/*
> + * The connect worker for the backchannel
> + * This should never be called as we should never need to connect
> + */
> +static void bc_connect_worker(struct work_struct *work)
> +{
> +	BUG();
> +}
> +
> +/*
> + * The set_port routine of the rpc_xprt_ops. This is related to the portmapper
> + * and should never be called
> + */
> +
> +static void bc_set_port(struct rpc_xprt *xprt, unsigned short port)
> +{
> +	BUG();
> +}
> +
> +/*
> + * The connect routine for the backchannel rpc_xprt ops
> + * Again, should never be called!
> + */
> +
> +static void bc_connect(struct rpc_task *task)
> +{
> +	BUG();
> +}
> +
> +struct rpc_buffer {
> +	size_t	len;
> +	char	data[];
> +};
> +/*
> + * Allocate a bunch of pages for a scratch buffer for the rpc code. The reason
> + * we allocate pages instead doing a kmalloc like rpc_malloc is because we want
> + * to use the server side send routines.
> + */
> +void *bc_malloc(struct rpc_task *task, size_t size)
> +{
> +	struct page *page;
> +	struct rpc_buffer *buf;
> +
> +	BUG_ON(size > PAGE_SIZE - sizeof(struct rpc_buffer));
> +	page = alloc_page(GFP_KERNEL);
> +
> +	if (!page)
> +		return NULL;
> +
> +	buf = page_address(page);
> +	buf->len = PAGE_SIZE;
> +
> +	return buf->data;
> +}
> +
> +/*
> + * Free the space allocated in the bc_alloc routine
> + */
> +void bc_free(void *buffer)
> +{
> +	struct rpc_buffer *buf;
> +
> +	if (!buffer)
> +		return;
> +
> +	buf = container_of(buffer, struct rpc_buffer, data);
> +	free_pages((unsigned long)buf, get_order(buf->len));
> +}
> +
> +/*
> + * Use the svc_sock to send the callback. Must be called with svsk->sk_mutex

review 11-13: revise comment. s/svsk->sk_mutex/svc_xprt.xpt_mutex/

> + * held. Borrows heavily from svc_tcp_sendto and xs_tcp_semd_request.

ask Ricardo / Mike Sager if svc_send can be refactored so we'd reuse
the common code.

> + */
> +static int bc_sendto(struct rpc_rqst *req)
> +{
> +	int total_len;
> +	int len;
> +	int size;
> +	int result;
> +	struct xdr_buf *xbufp = &req->rq_snd_buf;
> +	struct page **pages = xbufp->pages;
> +	unsigned int flags = MSG_MORE;
> +	unsigned int pglen = xbufp->page_len;
> +	size_t base = xbufp->page_base;
> +	struct rpc_xprt *xprt = req->rq_xprt;
> +	struct sock_xprt *transport =
> +				container_of(xprt, struct sock_xprt, xprt);
> +	struct socket *sock = transport->sock;
> +
> +	total_len = xbufp->len;
> +
> +	/*
> +	 * Set up the rpc header and record marker stuff
> +	 */
> +	xs_encode_tcp_record_marker(xbufp);
> +
> +	/*
> +	 * The RPC message is divided into 3 pieces:
> +	 * - The header: This is what most of the smaller RPC messages consist
> +	 *   of. Often the whole message is in this.
> +	 *
> +	 *   - xdr->pages: This is a list of pages that contain data, for
> +	 *   example in a write request or while using rpcsec gss
> +	 *
> +	 *   - The tail: This is the rest of the rpc message
> +	 *
> +	 *  First we send the header, then the pages and then finally the tail.
> +	 *  The code borrows heavily from svc_sendto.
> +	 */
> +
> +	/*
> +	 * Send the head
> +	 */
> +	if (total_len == xbufp->head[0].iov_len)
> +		flags = 0;
> +
> +	len = sock->ops->sendpage(sock, virt_to_page(xbufp->head[0].iov_base),
> +			(unsigned long)xbufp->head[0].iov_base & ~PAGE_MASK,
> +			xbufp->head[0].iov_len, flags);
> +
> +	if (len != xbufp->head[0].iov_len)
> +		goto out;
> +
> +	/*
> +	 * send page data
> +	 *
> +	 * Check the amount of data to be sent. If it is less than the
> +	 * remaining page, then send it else send the current page
> +	 */
> +
> +	size = PAGE_SIZE - base < pglen ? PAGE_SIZE - base : pglen;
> +	while (pglen > 0) {
> +		if (total_len == size)
> +			flags = 0;
> +		result = sock->ops->sendpage(sock, *pages, base, size, flags);
> +		if (result > 0)
> +			len += result;
> +		if (result != size)
> +			goto out;
> +		total_len -= size;
> +		pglen -= size;
> +		size = PAGE_SIZE < pglen ? PAGE_SIZE : pglen;
> +		base = 0;
> +		pages++;
> +	}
> +	/*
> +	 * send tail
> +	 */
> +	if (xbufp->tail[0].iov_len) {
> +		result = sock->ops->sendpage(sock,
> +			xbufp->tail[0].iov_base,
> +			(unsigned long)xbufp->tail[0].iov_base & ~PAGE_MASK,
> +			xbufp->tail[0].iov_len,
> +			0);
> +
> +		if (result > 0)
> +			len += result;
> +	}
> +out:
> +	if (len != xbufp->len)
> +		printk(KERN_NOTICE "Error sending entire callback!\n");
> +
> +	return len;
> +}
> +
> +/*
> + * The send routine. Borrows from svc_send
> + */
> +static int bc_send_request(struct rpc_task *task)
> +{
> +	struct rpc_rqst *req = task->tk_rqstp;
> +	struct rpc_xprt *bc_xprt = req->rq_xprt;
> +	struct svc_xprt	*xprt;
> +	struct svc_sock         *svsk;
> +	u32                     len;
> +
> +	dprintk("sending request with xid: %08x\n", ntohl(req->rq_xid));
> +	/*
> +	 * Get the server socket associated with this callback xprt
> +	 */
> +	svsk = bc_xprt->bc_sock;
> +	xprt = &svsk->sk_xprt;
> +
> +	mutex_lock(&xprt->xpt_mutex);
> +	if (test_bit(XPT_DEAD, &xprt->xpt_flags))
> +		len = -ENOTCONN;
> +	else
> +		len = bc_sendto(req);
> +	mutex_unlock(&xprt->xpt_mutex);
> +
> +	return 0;
> +
> +}
> +
> +/*
> + * The close routine. Since this is client initiated, we do nothing
> + */
> +
> +static void bc_close(struct rpc_xprt *xprt)
> +{
> +	return;
> +}
> +
> +/*
> + * The xprt destroy routine. Again, because this connection is client
> + * initiated, we do nothing
> + */
> +
> +static void bc_destroy(struct rpc_xprt *xprt)
> +{
> +	return;
> +}
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  static struct rpc_xprt_ops xs_udp_ops = {
>  	.set_buffer_size	= xs_udp_set_buffer_size,
>  	.reserve_xprt		= xprt_reserve_xprt_cong,
> @@ -2054,6 +2270,26 @@ static struct rpc_xprt_ops xs_tcp_ops = {
>  	.print_stats		= xs_tcp_print_stats,
>  };
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +/*
> + * The rpc_xprt_ops for the server backchannel
> + */
> +
> +static struct rpc_xprt_ops bc_tcp_ops = {
> +	.reserve_xprt		= xprt_reserve_xprt,
> +	.release_xprt		= xprt_release_xprt,
> +	.set_port		= bc_set_port,
> +	.connect		= bc_connect,
> +	.buf_alloc		= bc_malloc,
> +	.buf_free		= bc_free,
> +	.send_request		= bc_send_request,
> +	.set_retrans_timeout	= xprt_set_retrans_timeout_def,
> +	.close			= bc_close,
> +	.destroy		= bc_destroy,
> +	.print_stats		= xs_tcp_print_stats,
> +};
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  static struct rpc_xprt *xs_setup_xprt(struct xprt_create *args,
>  				      unsigned int slot_table_size)
>  {
> @@ -2186,13 +2422,31 @@ static struct rpc_xprt *xs_setup_tcp(struct xprt_create *args)
>  	xprt->tsh_size = sizeof(rpc_fraghdr) / sizeof(u32);
>  	xprt->max_payload = RPC_MAX_FRAGMENT_SIZE;
>  
> -	xprt->bind_timeout = XS_BIND_TO;
> -	xprt->connect_timeout = XS_TCP_CONN_TO;
> -	xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
> -	xprt->idle_timeout = XS_IDLE_DISC_TO;
> +#if defined(CONFIG_NFSD_V4_1)
> +	if (args->bc_sock) {

review 11-13: get rid if the #ifdef and next: label
(which is just the "else" lag.

> +		/* backchannel */
> +		xprt_set_bound(xprt);
> +		INIT_DELAYED_WORK(&transport->connect_worker,
> +				  bc_connect_worker);
> +		xprt->bind_timeout = 0;
> +		xprt->connect_timeout = 0;
> +		xprt->reestablish_timeout = 0;
> +		xprt->idle_timeout = (~0);
>  
> -	xprt->ops = &xs_tcp_ops;
> -	xprt->timeout = &xs_tcp_default_timeout;
> +		/*
> +		 * The backchannel uses the same socket connection as the
> +		 * forechannel
> +		 */
> +		xprt->bc_sock = args->bc_sock;
> +		xprt->bc_sock->sk_bc_xprt = xprt;
> +		transport->sock = xprt->bc_sock->sk_sock;
> +		transport->inet = xprt->bc_sock->sk_sk;
> +
> +		xprt->ops = &bc_tcp_ops;
> +
> +		goto next;
> +	}
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  	switch (addr->sa_family) {
>  	case AF_INET:
> @@ -2200,13 +2454,29 @@ static struct rpc_xprt *xs_setup_tcp(struct xprt_create *args)
			xprt_set_bound(xprt);
 
 		INIT_DELAYED_WORK(&transport->connect_worker, xs_tcp_connect_worker4);
-		xs_format_ipv4_peer_addresses(xprt, "tcp", RPCBIND_NETID_TCP);
 		break;
 	case AF_INET6:
 		if (((struct sockaddr_in6 *)addr)->sin6_port != htons(0))
>  			xprt_set_bound(xprt);
>  
 		INIT_DELAYED_WORK(&transport->connect_worker, xs_tcp_connect_worker6);
+		break;
+	}
+	xprt->bind_timeout = XS_BIND_TO;
+	xprt->connect_timeout = XS_TCP_CONN_TO;
+	xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
+	xprt->idle_timeout = XS_IDLE_DISC_TO;
+
+	xprt->ops = &xs_tcp_ops;
+
+next:
+	xprt->timeout = &xs_tcp_default_timeout;
+
+	switch (addr->sa_family) {
+	case AF_INET:
+		xs_format_ipv4_peer_addresses(xprt, "tcp", RPCBIND_NETID_TCP);
+		break;
+	case AF_INET6:
 		xs_format_ipv6_peer_addresses(xprt, "tcp", RPCBIND_NETID_TCP6);
 		break;
 	default:
-- 
1.6.0.1

>  		INIT_DELAYED_WORK(&transport->connect_worker, xs_tcp_connect_worker4);
> -		xs_format_ipv4_peer_addresses(xprt, "tcp", RPCBIND_NETID_TCP);
>  		break;
>  	case AF_INET6:
>  		if (((struct sockaddr_in6 *)addr)->sin6_port != htons(0))
>  			xprt_set_bound(xprt);
>  
>  		INIT_DELAYED_WORK(&transport->connect_worker, xs_tcp_connect_worker6);
> +		break;
> +	}
> +	xprt->bind_timeout = XS_BIND_TO;
> +	xprt->connect_timeout = XS_TCP_CONN_TO;
> +	xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
> +	xprt->idle_timeout = XS_IDLE_DISC_TO;
> +
> +	xprt->ops = &xs_tcp_ops;
> +
> +next:
> +	xprt->timeout = &xs_tcp_default_timeout;
> +
> +	switch (addr->sa_family) {
> +	case AF_INET:
> +		xs_format_ipv4_peer_addresses(xprt, "tcp", RPCBIND_NETID_TCP);
> +		break;
> +	case AF_INET6:
>  		xs_format_ipv6_peer_addresses(xprt, "tcp", RPCBIND_NETID_TCP6);
>  		break;
>  	default:

[RFC 29/51] nfsd: BUG_ON_UNLOCKED_STATE

[Benny Halevy]

Make sure fi_hash is always traversed and manipulated under the state lock.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 1a9d121..7fe074a 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -100,6 +100,8 @@ static struct kmem_cache *file_slab = NULL;
 static struct kmem_cache *stateid_slab = NULL;
 static struct kmem_cache *deleg_slab = NULL;
 
+#define BUG_ON_UNLOCKED_STATE() BUG_ON(mutex_trylock(&client_mutex))
+
 void
 nfs4_lock_state(void)
 {
@@ -149,6 +151,7 @@ free_nfs4_file(struct kref *kref)
 static inline void
 put_nfs4_file(struct nfs4_file *fi)
 {
+	BUG_ON_UNLOCKED_STATE();
 	kref_put(&fi->fi_ref, free_nfs4_file);
 }
 
@@ -554,6 +557,8 @@ expire_client(struct nfs4_client *clp)
 	dprintk("NFSD: expire_client cl_count %d\n",
 	                    atomic_read(&clp->cl_count));
 
+	BUG_ON_UNLOCKED_STATE();
+
 	INIT_LIST_HEAD(&reaplist);
 	spin_lock(&recall_lock);
 	while (!list_empty(&clp->cl_delegations)) {
@@ -1625,6 +1630,7 @@ find_openstateowner_str(unsigned int hashval, struct nfsd4_open *open)
 {
 	struct nfs4_stateowner *so = NULL;
 
+	BUG_ON_UNLOCKED_STATE();
 	list_for_each_entry(so, &ownerstr_hashtbl[hashval], so_strhash) {
 		if (same_owner_str(so, &open->op_owner, &open->op_clientid))
 			return so;
@@ -1639,6 +1645,7 @@ find_file(struct inode *ino)
 	unsigned int hashval = file_hashval(ino);
 	struct nfs4_file *fp;
 
+	BUG_ON_UNLOCKED_STATE();
 	list_for_each_entry(fp, &file_hashtbl[hashval], fi_hash) {
 		if (fp->fi_inode == ino) {
 			get_nfs4_file(fp);
-- 
1.6.0.2

[RFC 30/51] nfsd: lock state around nfs4_put_delegation in nfsd_break_deleg_cb err path

[Benny Halevy]

If kthread_run fails we put the client and delegation.  This requires
the state lock since the fi_hash is manipulated in the process.
Use nfs4_lock_state_nested as nfsd_break_deleg_cb may be called
back by the filesystem while under the state lock (e.g. in the close path)

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 7fe074a..0b45680 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -94,6 +94,7 @@ static void nfs4_set_recdir(char *recdir);
  * 	unconfstr_hashtbl[], uncofid_hashtbl[].
  */
 static DEFINE_MUTEX(client_mutex);
+static struct thread_info *client_mutex_owner;
 
 static struct kmem_cache *stateowner_slab = NULL;
 static struct kmem_cache *file_slab = NULL;
@@ -106,14 +107,26 @@ void
 nfs4_lock_state(void)
 {
 	mutex_lock(&client_mutex);
+	client_mutex_owner = current_thread_info();
 }
 
 void
 nfs4_unlock_state(void)
 {
+	BUG_ON(client_mutex_owner != current_thread_info());
+	client_mutex_owner = NULL;
 	mutex_unlock(&client_mutex);
 }
 
+static int
+nfs4_lock_state_nested(void)
+{
+	if (client_mutex_owner == current_thread_info())
+		return 0;
+	nfs4_lock_state();
+	return 1;
+}
+
 static inline u32
 opaque_hashval(const void *ptr, int nbytes)
 {
@@ -1798,6 +1811,7 @@ void nfsd_break_deleg_cb(struct file_lock *fl)
 {
 	struct nfs4_delegation *dp=  (struct nfs4_delegation *)fl->fl_owner;
 	struct task_struct *t;
+	int did_lock;
 
 	dprintk("NFSD nfsd_break_deleg_cb: dp %p fl %p\n",dp,fl);
 	if (!dp)
@@ -1832,8 +1846,11 @@ void nfsd_break_deleg_cb(struct file_lock *fl)
 		printk(KERN_INFO "NFSD: Callback thread failed for "
 			"for client (clientid %08x/%08x)\n",
 			clp->cl_clientid.cl_boot, clp->cl_clientid.cl_id);
+		did_lock = nfs4_lock_state_nested();
 		put_nfs4_client(dp->dl_client);
 		nfs4_put_delegation(dp);
+		if (did_lock)
+			nfs4_unlock_state();
 	}
 }
 
-- 
1.6.0.2

[RFC 31/51] FIXME: nfsd: kref_get cb_client while doing the callback

[Benny Halevy]

From: Marc Eshel <eshel@almaden.ibm.com>

FIXME: bhalevy: I think this patch should be dropped.
First, dp->dl_client is correctly refcounted while in do_recall and
should protect dp->dl_client->cl_callback.cb_client from going away.
Second, the rpc_clnt isn't dereferenced on the error-free path so
apparently this patch introduces a leak.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c |   13 +++++++++++++
 net/sunrpc/clnt.c   |    1 +
 2 files changed, 14 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 0b45680..55fd003 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -1810,6 +1810,7 @@ static
 void nfsd_break_deleg_cb(struct file_lock *fl)
 {
 	struct nfs4_delegation *dp=  (struct nfs4_delegation *)fl->fl_owner;
+	struct rpc_clnt *clnt;
 	struct task_struct *t;
 	int did_lock;
 
@@ -1817,6 +1818,17 @@ void nfsd_break_deleg_cb(struct file_lock *fl)
 	if (!dp)
 		return;
 
+	did_lock = nfs4_lock_state_nested();
+	clnt = dp->dl_client->cl_callback.cb_client;
+	if (!atomic_read(&dp->dl_client->cl_callback.cb_set) || !clnt) {
+		if (did_lock)
+			nfs4_unlock_state();
+		return;
+	}
+	kref_get(&clnt->cl_kref);
+	if (did_lock)
+		nfs4_unlock_state();
+
 	/* We're assuming the state code never drops its reference
 	 * without first removing the lease.  Since we're in this lease
 	 * callback (and since the lease code is serialized by the kernel
@@ -1846,6 +1858,7 @@ void nfsd_break_deleg_cb(struct file_lock *fl)
 		printk(KERN_INFO "NFSD: Callback thread failed for "
 			"for client (clientid %08x/%08x)\n",
 			clp->cl_clientid.cl_boot, clp->cl_clientid.cl_id);
+		rpc_release_client(clnt);
 		did_lock = nfs4_lock_state_nested();
 		put_nfs4_client(dp->dl_client);
 		nfs4_put_delegation(dp);
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 5e8fba0..144f3bd 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -464,6 +464,7 @@ rpc_release_client(struct rpc_clnt *clnt)
 		wake_up(&destroy_wait);
 	kref_put(&clnt->cl_kref, rpc_free_auth);
 }
+EXPORT_SYMBOL_GPL(rpc_release_client);
 
 /**
  * rpc_bind_new_program - bind a new RPC program to an existing client
-- 
1.6.0.2

[RFC 32/51] nfsd41: callback infrastructure

[Benny Halevy]

Keep the xprt used for create_session in cl_cb_xprt.
Mark cl_callback.cb_minorversion = 1 and remember
the client provided cl_callback.cb_prog rpc program number.
Use it to probe the callback path.

Define xdr sizes and code nfs41 cb_compound header to be able
to send a null callback rpc.

Define stubs for other nfs41 callbacks.

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4callback.c     |  189 +++++++++++++++++++++++++++++++++++++++-----
 fs/nfsd/nfs4state.c        |   14 +++
 include/linux/nfsd/state.h |    3 +-
 3 files changed, 186 insertions(+), 20 deletions(-)

diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
index e198ead..b1d2299 100644
--- a/fs/nfsd/nfs4callback.c
+++ b/fs/nfsd/nfs4callback.c
@@ -43,6 +43,7 @@
 #include <linux/sunrpc/xdr.h>
 #include <linux/sunrpc/svc.h>
 #include <linux/sunrpc/clnt.h>
+#include <linux/sunrpc/svcsock.h>
 #include <linux/nfsd/nfsd.h>
 #include <linux/nfsd/state.h>
 #include <linux/sunrpc/sched.h>
@@ -52,16 +53,24 @@
 
 #define NFSPROC4_CB_NULL 0
 #define NFSPROC4_CB_COMPOUND 1
+#define NFS4_STATEID_SIZE 16
+
+#if defined(CONFIG_NFSD_V4_1)
+#define NFS4_CB_PROGRAM 0x40000000
+#endif
 
 /* Index of predefined Linux callback client operations */
 
 enum {
-        NFSPROC4_CLNT_CB_NULL = 0,
+	NFSPROC4_CLNT_CB_NULL = 0,
 	NFSPROC4_CLNT_CB_RECALL,
+	NFSPROC4_CLNT_CB_SEQUENCE,
 };
 
 enum nfs_cb_opnum4 {
 	OP_CB_RECALL            = 4,
+	OP_CB_LAYOUT            = 5,
+	OP_CB_SEQUENCE          = 11,
 };
 
 #define NFS4_MAXTAGLEN		20
@@ -77,10 +86,40 @@ enum nfs_cb_opnum4 {
 #define NFS4_enc_cb_recall_sz		(cb_compound_enc_hdr_sz +       \
 					1 + enc_stateid_sz +            \
 					enc_nfs4_fh_sz)
-
 #define NFS4_dec_cb_recall_sz		(cb_compound_dec_hdr_sz  +      \
 					op_dec_sz)
 
+#if defined(CONFIG_NFSD_V4_1)
+#define NFS41_enc_cb_null_sz		0
+#define NFS41_dec_cb_null_sz		0
+#define cb_compound41_enc_hdr_sz	4
+#define cb_compound41_dec_hdr_sz	(3 + (NFS4_MAXTAGLEN >> 2))
+#define sessionid_sz			(NFS4_MAX_SESSIONID_LEN >> 2)
+#define cb_sequence41_enc_sz		(sessionid_sz + 4 +             \
+					1 /* no referring calls list yet */)
+#define cb_sequence41_dec_sz		(op_dec_sz + sessionid_sz + 4)
+#define NFS41_enc_cb_recall_sz		(cb_compound41_enc_hdr_sz +     \
+					cb_sequence41_enc_sz +          \
+					1 + enc_stateid_sz +            \
+					enc_nfs4_fh_sz)
+#define NFS41_dec_cb_recall_sz		(cb_compound_dec_hdr_sz  +      \
+					cb_sequence41_dec_sz +          \
+					op_dec_sz)
+#else /* defined(CONFIG_NFSD_V4_1) */
+
+struct nfs41_cb_sequence;
+
+#endif /* defined(CONFIG_NFSD_V4_1) */
+
+struct nfs4_rpc_args {
+	void                     *args_op;
+	struct nfs41_cb_sequence *args_seq;
+};
+
+struct nfs4_rpc_res {
+	struct nfs41_cb_sequence *res_seq;
+};
+
 /*
 * Generic encode routines from fs/nfs/nfs4xdr.c
 */
@@ -135,11 +174,18 @@ xdr_error:                                      \
 		return -EIO; \
 	} \
 } while (0)
+#define COPYMEM(x,nbytes) do {			\
+	memcpy((x), p, nbytes);			\
+	p += XDR_QUADLEN(nbytes);		\
+} while (0)
 
 struct nfs4_cb_compound_hdr {
-	int		status;
-	u32		ident;
+	/* args */
+	u32		ident;		/* minorversion 0 only */
 	u32		nops;
+
+	/* res */
+	int		status;
 	u32		taglen;
 	char *		tag;
 };
@@ -208,7 +254,7 @@ encode_cb_compound_hdr(struct xdr_stream *xdr, struct nfs4_cb_compound_hdr *hdr)
 
 	RESERVE_SPACE(16);
 	WRITE32(0);            /* tag length is always 0 */
-	WRITE32(NFS4_MINOR_VERSION);
+	WRITE32(0);		/* minorversion */
 	WRITE32(hdr->ident);
 	WRITE32(hdr->nops);
 	return 0;
@@ -230,6 +276,14 @@ encode_cb_recall(struct xdr_stream *xdr, struct nfs4_cb_recall *cb_rec)
 	return 0;
 }
 
+#if defined(CONFIG_NFSD_V4_1)
+static int
+encode_cb_sequence(struct xdr_stream *xdr, struct nfs41_cb_sequence *args)
+{
+	return -1;	/* stub */
+}
+#endif /* defined(CONFIG_NFSD_V4_1) */
+
 static int
 nfs4_xdr_enc_cb_null(struct rpc_rqst *req, __be32 *p)
 {
@@ -255,6 +309,29 @@ nfs4_xdr_enc_cb_recall(struct rpc_rqst *req, __be32 *p, struct nfs4_cb_recall *a
 }
 
 
+#if defined(CONFIG_NFSD_V4_1)
+static int
+encode_cb_compound41_hdr(struct xdr_stream *xdr,
+			 struct nfs4_cb_compound_hdr *hdr)
+{
+	u32 *p;
+
+	RESERVE_SPACE(16);
+	WRITE32(0);		/* tag length is always 0 */
+	WRITE32(1);		/* minorversion */
+	WRITE32(0);		/* callback_ident not used in 4.1 */
+	WRITE32(hdr->nops);
+	return 0;
+}
+
+static int
+nfs41_xdr_enc_cb_recall(struct rpc_rqst *req, u32 *p,
+			struct nfs4_rpc_args *rpc_args)
+{
+	return -1;	/* stub */
+}
+#endif /* defined(CONFIG_NFSD_V4_1) */
+
 static int
 decode_cb_compound_hdr(struct xdr_stream *xdr, struct nfs4_cb_compound_hdr *hdr){
         __be32 *p;
@@ -312,6 +389,21 @@ out:
 	return status;
 }
 
+#if defined(CONFIG_NFSD_V4_1)
+static int
+decode_cb_sequence(struct xdr_stream *xdr, struct nfs41_cb_sequence *res)
+{
+	return -1;	/* stub */
+}
+
+static int
+nfs41_xdr_dec_cb_recall(struct rpc_rqst *rqstp, u32 *p,
+			struct nfs4_rpc_res *rpc_res)
+{
+	return -1;	/* stub */
+}
+#endif /* defined(CONFIG_NFSD_V4_1) */
+
 /*
  * RPC procedure tables
  */
@@ -331,32 +423,76 @@ static struct rpc_procinfo     nfs4_cb_procedures[] = {
     PROC(CB_RECALL,    COMPOUND,   enc_cb_recall,      dec_cb_recall),
 };
 
-static struct rpc_version       nfs_cb_version4 = {
+static struct rpc_version       nfs4_cb_version1 = {
         .number                 = 1,
         .nrprocs                = ARRAY_SIZE(nfs4_cb_procedures),
         .procs                  = nfs4_cb_procedures
 };
 
-static struct rpc_version *	nfs_cb_version[] = {
+static struct rpc_version *nfs4_cb_version[] = {
 	NULL,
-	&nfs_cb_version4,
+	&nfs4_cb_version1,
 };
 
-static struct rpc_program cb_program;
+static struct rpc_program nfs4_cb_program;
 
-static struct rpc_stat cb_stats = {
-		.program	= &cb_program
+static struct rpc_stat nfs4_cb_stats = {
+	.program	= &nfs4_cb_program
 };
 
 #define NFS4_CALLBACK 0x40000000
-static struct rpc_program cb_program = {
-		.name 		= "nfs4_cb",
-		.number		= NFS4_CALLBACK,
-		.nrvers		= ARRAY_SIZE(nfs_cb_version),
-		.version	= nfs_cb_version,
-		.stats		= &cb_stats,
+static struct rpc_program nfs4_cb_program = {
+	.name 		= "nfs4_cb",
+	.number		= NFS4_CALLBACK,
+	.nrvers		= ARRAY_SIZE(nfs4_cb_version),
+	.version	= nfs4_cb_version,
+	.stats		= &nfs4_cb_stats,
+};
+
+#if defined(CONFIG_NFSD_V4_1)
+#define PROC41(proc, call, argtype, restype)                            \
+[NFSPROC4_CLNT_##proc] = {                                              \
+	.p_proc   = NFSPROC4_CB_##call,                                 \
+	.p_encode = (kxdrproc_t) nfs41_xdr_##argtype,                   \
+	.p_decode = (kxdrproc_t) nfs41_xdr_##restype,                   \
+	.p_arglen = NFS41_##argtype##_sz,                               \
+	.p_replen = NFS41_##restype##_sz,                               \
+	.p_statidx = NFSPROC4_CB_##call,                                \
+	.p_name   = #proc,                                              \
+}
+
+static struct rpc_procinfo     nfs41_cb_procedures[] = {
+	PROC(CB_NULL,        NULL,       enc_cb_null,        dec_cb_null),
+	PROC41(CB_RECALL,    COMPOUND,   enc_cb_recall,      dec_cb_recall),
 };
 
+static struct rpc_version       nfs41_cb_version1 = {
+	.number                 = 1,
+	.nrprocs                = ARRAY_SIZE(nfs41_cb_procedures),
+	.procs                  = nfs41_cb_procedures
+};
+
+static struct rpc_version *nfs41_cb_version[] = {
+	NULL,
+	&nfs41_cb_version1,
+};
+
+static struct rpc_program nfs41_cb_program;
+
+static struct rpc_stat nfs41_cb_stats = {
+	.program	= &nfs41_cb_program
+};
+
+static struct rpc_program nfs41_cb_program = {
+	.name 		= "nfs41_cb",
+	.number		= NFS4_CALLBACK,
+	.nrvers		= ARRAY_SIZE(nfs41_cb_version),
+	.version	= nfs41_cb_version,
+	.stats		= &nfs41_cb_stats,
+};
+
+#endif /* defined(CONFIG_NFSD_V4_1) */
+
 /* Reference counting, callback cleanup, etc., all look racy as heck.
  * And why is cb_set an atomic? */
 
@@ -376,9 +512,9 @@ static int do_probe_callback(void *data)
 		.address	= (struct sockaddr *)&addr,
 		.addrsize	= sizeof(addr),
 		.timeout	= &timeparms,
-		.program	= &cb_program,
+		.program	= &nfs4_cb_program,
 		.prognumber	= cb->cb_prog,
-		.version	= nfs_cb_version[1]->number,
+		.version	= nfs4_cb_version[1]->number,
 		.authflavor	= RPC_AUTH_UNIX, /* XXX: need AUTH_GSS... */
 		.flags		= (RPC_CLNT_CREATE_NOPING | RPC_CLNT_CREATE_QUIET),
 	};
@@ -395,6 +531,20 @@ static int do_probe_callback(void *data)
 	addr.sin_port = htons(cb->cb_port);
 	addr.sin_addr.s_addr = htonl(cb->cb_addr);
 
+#if defined(CONFIG_NFSD_V4_1)
+	if (cb->cb_minorversion) {
+		BUG_ON(cb->cb_minorversion != 1);
+		args.program = &nfs41_cb_program;
+		args.version = nfs41_cb_version[1]->number;
+		args.bc_sock = container_of(clp->cl_cb_xprt, struct svc_sock,
+					    sk_xprt);
+	}
+#endif /* CONFIG_NFSD_V4_1 */
+
+	dprintk("%s: program %s 0x%x nrvers %u version %u minorversion %u\n",
+		__func__, args.program->name, args.prognumber,
+		args.program->nrvers, args.version, cb->cb_minorversion);
+
 	/* Create RPC client */
 	client = rpc_create(&args);
 	if (IS_ERR(client)) {
@@ -413,6 +563,7 @@ static int do_probe_callback(void *data)
 	put_nfs4_client(clp);
 	return 0;
 out_release_client:
+	dprintk("NFSD: synchronous CB_NULL failed. status=%d\n", status);
 	rpc_shutdown_client(client);
 out_err:
 	dprintk("NFSD: warning: no callback path to client %.*s\n",
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 55fd003..edac97a 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -39,6 +39,7 @@
 #include <linux/slab.h>
 
 #include <linux/sunrpc/svc.h>
+#include <linux/sunrpc/svcsock.h>
 #include <linux/nfsd/nfsd.h>
 #include <linux/nfsd/cache.h>
 #include <linux/file.h>
@@ -544,6 +545,10 @@ static inline void
 free_client(struct nfs4_client *clp)
 {
 	shutdown_callback_client(clp);
+#if defined(CONFIG_NFSD_V4_1)
+	if (clp->cl_cb_xprt)
+		svc_xprt_put(clp->cl_cb_xprt);
+#endif /* CONFIG_NFSD_V4_1 */
 	if (clp->cl_cred.cr_group_info)
 		put_group_info(clp->cl_cred.cr_group_info);
 	kfree(clp->cl_name.data);
@@ -856,6 +861,8 @@ gen_callback(struct nfs4_client *clp, struct nfsd4_setclientid *se)
 	if ( !(parse_ipv4(se->se_callback_addr_len, se->se_callback_addr_val,
 	                 &cb->cb_addr, &cb->cb_port)))
 		goto out_err;
+
+	cb->cb_minorversion = 0;
 	cb->cb_prog = se->se_callback_prog;
 	cb->cb_ident = se->se_callback_ident;
 	return;
@@ -1042,6 +1049,13 @@ nfsd4_create_session(struct svc_rqst *rqstp,
 		cr_ses->flags &= ~SESSION4_PERSIST;
 		cr_ses->flags &= ~SESSION4_RDMA;
 
+		if (cr_ses->flags & SESSION4_BACK_CHAN) {
+			unconf->cl_cb_xprt = rqstp->rq_xprt;
+			svc_xprt_get(unconf->cl_cb_xprt);
+			unconf->cl_callback.cb_minorversion = 1;
+			unconf->cl_callback.cb_prog = cr_ses->callback_prog;
+			nfsd4_probe_callback(unconf);
+		}
 		conf = unconf;
 	}
 
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index 94b5366..918adde 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -94,7 +94,8 @@ struct nfs4_callback {
 	u32                     cb_addr;
 	unsigned short          cb_port;
 	u32                     cb_prog;
-	u32                     cb_ident;
+	u32			cb_minorversion;
+	u32			cb_ident;	/* minorversion 0 only */
 	/* RPC client info */
 	atomic_t		cb_set;     /* successful CB_NULL call */
 	struct rpc_clnt *       cb_client;
-- 
1.6.0.2

Re: [pnfs] [RFC 32/51] nfsd41: callback infrastructure

[Benny Halevy]

On Nov. 10, 2008, 22:51 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> Keep the xprt used for create_session in cl_cb_xprt.
> Mark cl_callback.cb_minorversion = 1 and remember
> the client provided cl_callback.cb_prog rpc program number.
> Use it to probe the callback path.
> 
> Define xdr sizes and code nfs41 cb_compound header to be able
> to send a null callback rpc.
> 
> Define stubs for other nfs41 callbacks.
> 
> Signed-off-by: Andy Adamson<andros@netapp.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 11-13: unify xdr function like Andy did for the nfs client

> ---
>  fs/nfsd/nfs4callback.c     |  189 +++++++++++++++++++++++++++++++++++++++-----
>  fs/nfsd/nfs4state.c        |   14 +++
>  include/linux/nfsd/state.h |    3 +-
>  3 files changed, 186 insertions(+), 20 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
> index e198ead..b1d2299 100644
> --- a/fs/nfsd/nfs4callback.c
> +++ b/fs/nfsd/nfs4callback.c
> @@ -43,6 +43,7 @@
>  #include <linux/sunrpc/xdr.h>
>  #include <linux/sunrpc/svc.h>
>  #include <linux/sunrpc/clnt.h>
> +#include <linux/sunrpc/svcsock.h>
>  #include <linux/nfsd/nfsd.h>
>  #include <linux/nfsd/state.h>
>  #include <linux/sunrpc/sched.h>
> @@ -52,16 +53,24 @@
>  
>  #define NFSPROC4_CB_NULL 0
>  #define NFSPROC4_CB_COMPOUND 1
> +#define NFS4_STATEID_SIZE 16
> +
> +#if defined(CONFIG_NFSD_V4_1)
> +#define NFS4_CB_PROGRAM 0x40000000
> +#endif
>  
>  /* Index of predefined Linux callback client operations */
>  
>  enum {
> -        NFSPROC4_CLNT_CB_NULL = 0,
> +	NFSPROC4_CLNT_CB_NULL = 0,
>  	NFSPROC4_CLNT_CB_RECALL,
> +	NFSPROC4_CLNT_CB_SEQUENCE,
>  };
>  
>  enum nfs_cb_opnum4 {
>  	OP_CB_RECALL            = 4,
> +	OP_CB_LAYOUT            = 5,
> +	OP_CB_SEQUENCE          = 11,
>  };
>  
>  #define NFS4_MAXTAGLEN		20
> @@ -77,10 +86,40 @@ enum nfs_cb_opnum4 {
>  #define NFS4_enc_cb_recall_sz		(cb_compound_enc_hdr_sz +       \
>  					1 + enc_stateid_sz +            \
>  					enc_nfs4_fh_sz)
> -
>  #define NFS4_dec_cb_recall_sz		(cb_compound_dec_hdr_sz  +      \
>  					op_dec_sz)
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +#define NFS41_enc_cb_null_sz		0
> +#define NFS41_dec_cb_null_sz		0
> +#define cb_compound41_enc_hdr_sz	4
> +#define cb_compound41_dec_hdr_sz	(3 + (NFS4_MAXTAGLEN >> 2))
> +#define sessionid_sz			(NFS4_MAX_SESSIONID_LEN >> 2)
> +#define cb_sequence41_enc_sz		(sessionid_sz + 4 +             \
> +					1 /* no referring calls list yet */)
> +#define cb_sequence41_dec_sz		(op_dec_sz + sessionid_sz + 4)
> +#define NFS41_enc_cb_recall_sz		(cb_compound41_enc_hdr_sz +     \
> +					cb_sequence41_enc_sz +          \
> +					1 + enc_stateid_sz +            \
> +					enc_nfs4_fh_sz)
> +#define NFS41_dec_cb_recall_sz		(cb_compound_dec_hdr_sz  +      \
> +					cb_sequence41_dec_sz +          \
> +					op_dec_sz)
> +#else /* defined(CONFIG_NFSD_V4_1) */
> +
> +struct nfs41_cb_sequence;
> +
> +#endif /* defined(CONFIG_NFSD_V4_1) */
> +
> +struct nfs4_rpc_args {
> +	void                     *args_op;
> +	struct nfs41_cb_sequence *args_seq;
> +};
> +
> +struct nfs4_rpc_res {
> +	struct nfs41_cb_sequence *res_seq;
> +};
> +
>  /*
>  * Generic encode routines from fs/nfs/nfs4xdr.c
>  */
> @@ -135,11 +174,18 @@ xdr_error:                                      \
>  		return -EIO; \
>  	} \
>  } while (0)
> +#define COPYMEM(x,nbytes) do {			\
> +	memcpy((x), p, nbytes);			\
> +	p += XDR_QUADLEN(nbytes);		\
> +} while (0)
>  
>  struct nfs4_cb_compound_hdr {
> -	int		status;
> -	u32		ident;
> +	/* args */
> +	u32		ident;		/* minorversion 0 only */
>  	u32		nops;
> +
> +	/* res */
> +	int		status;
>  	u32		taglen;
>  	char *		tag;
>  };
> @@ -208,7 +254,7 @@ encode_cb_compound_hdr(struct xdr_stream *xdr, struct nfs4_cb_compound_hdr *hdr)
>  
>  	RESERVE_SPACE(16);
>  	WRITE32(0);            /* tag length is always 0 */
> -	WRITE32(NFS4_MINOR_VERSION);
> +	WRITE32(0);		/* minorversion */
>  	WRITE32(hdr->ident);
>  	WRITE32(hdr->nops);
>  	return 0;
> @@ -230,6 +276,14 @@ encode_cb_recall(struct xdr_stream *xdr, struct nfs4_cb_recall *cb_rec)
>  	return 0;
>  }
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +static int
> +encode_cb_sequence(struct xdr_stream *xdr, struct nfs41_cb_sequence *args)
> +{
> +	return -1;	/* stub */
> +}
> +#endif /* defined(CONFIG_NFSD_V4_1) */
> +
>  static int
>  nfs4_xdr_enc_cb_null(struct rpc_rqst *req, __be32 *p)
>  {
> @@ -255,6 +309,29 @@ nfs4_xdr_enc_cb_recall(struct rpc_rqst *req, __be32 *p, struct nfs4_cb_recall *a
>  }
>  
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +static int
> +encode_cb_compound41_hdr(struct xdr_stream *xdr,
> +			 struct nfs4_cb_compound_hdr *hdr)
> +{
> +	u32 *p;
> +
> +	RESERVE_SPACE(16);
> +	WRITE32(0);		/* tag length is always 0 */
> +	WRITE32(1);		/* minorversion */
> +	WRITE32(0);		/* callback_ident not used in 4.1 */
> +	WRITE32(hdr->nops);
> +	return 0;
> +}
> +
> +static int
> +nfs41_xdr_enc_cb_recall(struct rpc_rqst *req, u32 *p,
> +			struct nfs4_rpc_args *rpc_args)
> +{
> +	return -1;	/* stub */
> +}
> +#endif /* defined(CONFIG_NFSD_V4_1) */
> +
>  static int
>  decode_cb_compound_hdr(struct xdr_stream *xdr, struct nfs4_cb_compound_hdr *hdr){
>          __be32 *p;
> @@ -312,6 +389,21 @@ out:
>  	return status;
>  }
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +static int
> +decode_cb_sequence(struct xdr_stream *xdr, struct nfs41_cb_sequence *res)
> +{
> +	return -1;	/* stub */
> +}
> +
> +static int
> +nfs41_xdr_dec_cb_recall(struct rpc_rqst *rqstp, u32 *p,
> +			struct nfs4_rpc_res *rpc_res)
> +{
> +	return -1;	/* stub */
> +}
> +#endif /* defined(CONFIG_NFSD_V4_1) */
> +
>  /*
>   * RPC procedure tables
>   */
> @@ -331,32 +423,76 @@ static struct rpc_procinfo     nfs4_cb_procedures[] = {
>      PROC(CB_RECALL,    COMPOUND,   enc_cb_recall,      dec_cb_recall),
>  };
>  
> -static struct rpc_version       nfs_cb_version4 = {
> +static struct rpc_version       nfs4_cb_version1 = {
>          .number                 = 1,
>          .nrprocs                = ARRAY_SIZE(nfs4_cb_procedures),
>          .procs                  = nfs4_cb_procedures
>  };
>  
> -static struct rpc_version *	nfs_cb_version[] = {
> +static struct rpc_version *nfs4_cb_version[] = {
>  	NULL,
> -	&nfs_cb_version4,
> +	&nfs4_cb_version1,
>  };
>  
> -static struct rpc_program cb_program;
> +static struct rpc_program nfs4_cb_program;
>  
> -static struct rpc_stat cb_stats = {
> -		.program	= &cb_program
> +static struct rpc_stat nfs4_cb_stats = {
> +	.program	= &nfs4_cb_program
>  };
>  
>  #define NFS4_CALLBACK 0x40000000
> -static struct rpc_program cb_program = {
> -		.name 		= "nfs4_cb",
> -		.number		= NFS4_CALLBACK,
> -		.nrvers		= ARRAY_SIZE(nfs_cb_version),
> -		.version	= nfs_cb_version,
> -		.stats		= &cb_stats,
> +static struct rpc_program nfs4_cb_program = {
> +	.name 		= "nfs4_cb",
> +	.number		= NFS4_CALLBACK,
> +	.nrvers		= ARRAY_SIZE(nfs4_cb_version),
> +	.version	= nfs4_cb_version,
> +	.stats		= &nfs4_cb_stats,
> +};
> +
> +#if defined(CONFIG_NFSD_V4_1)
> +#define PROC41(proc, call, argtype, restype)                            \
> +[NFSPROC4_CLNT_##proc] = {                                              \
> +	.p_proc   = NFSPROC4_CB_##call,                                 \
> +	.p_encode = (kxdrproc_t) nfs41_xdr_##argtype,                   \
> +	.p_decode = (kxdrproc_t) nfs41_xdr_##restype,                   \
> +	.p_arglen = NFS41_##argtype##_sz,                               \
> +	.p_replen = NFS41_##restype##_sz,                               \
> +	.p_statidx = NFSPROC4_CB_##call,                                \
> +	.p_name   = #proc,                                              \
> +}
> +
> +static struct rpc_procinfo     nfs41_cb_procedures[] = {
> +	PROC(CB_NULL,        NULL,       enc_cb_null,        dec_cb_null),
> +	PROC41(CB_RECALL,    COMPOUND,   enc_cb_recall,      dec_cb_recall),
>  };
>  
> +static struct rpc_version       nfs41_cb_version1 = {
> +	.number                 = 1,
> +	.nrprocs                = ARRAY_SIZE(nfs41_cb_procedures),
> +	.procs                  = nfs41_cb_procedures
> +};
> +
> +static struct rpc_version *nfs41_cb_version[] = {
> +	NULL,
> +	&nfs41_cb_version1,
> +};
> +
> +static struct rpc_program nfs41_cb_program;
> +
> +static struct rpc_stat nfs41_cb_stats = {
> +	.program	= &nfs41_cb_program
> +};
> +
> +static struct rpc_program nfs41_cb_program = {
> +	.name 		= "nfs41_cb",
> +	.number		= NFS4_CALLBACK,
> +	.nrvers		= ARRAY_SIZE(nfs41_cb_version),
> +	.version	= nfs41_cb_version,
> +	.stats		= &nfs41_cb_stats,
> +};
> +
> +#endif /* defined(CONFIG_NFSD_V4_1) */
> +
>  /* Reference counting, callback cleanup, etc., all look racy as heck.
>   * And why is cb_set an atomic? */
>  
> @@ -376,9 +512,9 @@ static int do_probe_callback(void *data)
>  		.address	= (struct sockaddr *)&addr,
>  		.addrsize	= sizeof(addr),
>  		.timeout	= &timeparms,
> -		.program	= &cb_program,
> +		.program	= &nfs4_cb_program,
>  		.prognumber	= cb->cb_prog,
> -		.version	= nfs_cb_version[1]->number,
> +		.version	= nfs4_cb_version[1]->number,
>  		.authflavor	= RPC_AUTH_UNIX, /* XXX: need AUTH_GSS... */
>  		.flags		= (RPC_CLNT_CREATE_NOPING | RPC_CLNT_CREATE_QUIET),
>  	};
> @@ -395,6 +531,20 @@ static int do_probe_callback(void *data)
>  	addr.sin_port = htons(cb->cb_port);
>  	addr.sin_addr.s_addr = htonl(cb->cb_addr);
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +	if (cb->cb_minorversion) {
> +		BUG_ON(cb->cb_minorversion != 1);
> +		args.program = &nfs41_cb_program;
> +		args.version = nfs41_cb_version[1]->number;
> +		args.bc_sock = container_of(clp->cl_cb_xprt, struct svc_sock,
> +					    sk_xprt);
> +	}
> +#endif /* CONFIG_NFSD_V4_1 */
> +
> +	dprintk("%s: program %s 0x%x nrvers %u version %u minorversion %u\n",
> +		__func__, args.program->name, args.prognumber,
> +		args.program->nrvers, args.version, cb->cb_minorversion);
> +
>  	/* Create RPC client */
>  	client = rpc_create(&args);
>  	if (IS_ERR(client)) {
> @@ -413,6 +563,7 @@ static int do_probe_callback(void *data)
>  	put_nfs4_client(clp);
>  	return 0;
>  out_release_client:
> +	dprintk("NFSD: synchronous CB_NULL failed. status=%d\n", status);
>  	rpc_shutdown_client(client);
>  out_err:
>  	dprintk("NFSD: warning: no callback path to client %.*s\n",
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 55fd003..edac97a 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -39,6 +39,7 @@
>  #include <linux/slab.h>
>  
>  #include <linux/sunrpc/svc.h>
> +#include <linux/sunrpc/svcsock.h>
>  #include <linux/nfsd/nfsd.h>
>  #include <linux/nfsd/cache.h>
>  #include <linux/file.h>
> @@ -544,6 +545,10 @@ static inline void
>  free_client(struct nfs4_client *clp)
>  {
>  	shutdown_callback_client(clp);
> +#if defined(CONFIG_NFSD_V4_1)
> +	if (clp->cl_cb_xprt)
> +		svc_xprt_put(clp->cl_cb_xprt);
> +#endif /* CONFIG_NFSD_V4_1 */
>  	if (clp->cl_cred.cr_group_info)
>  		put_group_info(clp->cl_cred.cr_group_info);
>  	kfree(clp->cl_name.data);
> @@ -856,6 +861,8 @@ gen_callback(struct nfs4_client *clp, struct nfsd4_setclientid *se)
>  	if ( !(parse_ipv4(se->se_callback_addr_len, se->se_callback_addr_val,
>  	                 &cb->cb_addr, &cb->cb_port)))
>  		goto out_err;
> +
> +	cb->cb_minorversion = 0;
>  	cb->cb_prog = se->se_callback_prog;
>  	cb->cb_ident = se->se_callback_ident;
>  	return;
> @@ -1042,6 +1049,13 @@ nfsd4_create_session(struct svc_rqst *rqstp,
>  		cr_ses->flags &= ~SESSION4_PERSIST;
>  		cr_ses->flags &= ~SESSION4_RDMA;
>  
> +		if (cr_ses->flags & SESSION4_BACK_CHAN) {
> +			unconf->cl_cb_xprt = rqstp->rq_xprt;
> +			svc_xprt_get(unconf->cl_cb_xprt);
> +			unconf->cl_callback.cb_minorversion = 1;
> +			unconf->cl_callback.cb_prog = cr_ses->callback_prog;
> +			nfsd4_probe_callback(unconf);
> +		}
>  		conf = unconf;
>  	}
>  
> diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
> index 94b5366..918adde 100644
> --- a/include/linux/nfsd/state.h
> +++ b/include/linux/nfsd/state.h
> @@ -94,7 +94,8 @@ struct nfs4_callback {
>  	u32                     cb_addr;
>  	unsigned short          cb_port;
>  	u32                     cb_prog;
> -	u32                     cb_ident;
> +	u32			cb_minorversion;
> +	u32			cb_ident;	/* minorversion 0 only */
>  	/* RPC client info */
>  	atomic_t		cb_set;     /* successful CB_NULL call */
>  	struct rpc_clnt *       cb_client;

[RFC 33/51] nfsd41: introduce cl_cb_mutex

[Benny Halevy]

To be used for serializing nfs41 callbacks.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c        |    1 +
 include/linux/nfsd/state.h |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index edac97a..9377ad0 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -625,6 +625,7 @@ static struct nfs4_client *create_client(struct xdr_netobj name, char *recdir)
 	INIT_LIST_HEAD(&clp->cl_delegations);
 #if defined(CONFIG_NFSD_V4_1)
 	INIT_LIST_HEAD(&clp->cl_sessions);
+	mutex_init(&clp->cl_cb_mutex);
 #endif /* CONFIG_NFSD_V4_1 */
 	INIT_LIST_HEAD(&clp->cl_lru);
 	return clp;
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index 918adde..eb8d55c 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -221,6 +221,7 @@ struct nfs4_client {
 	nfs41_sessionid		cl_sessionid;
 
 	struct svc_xprt		*cl_cb_xprt;	/* 4.1 callback transport */
+	struct mutex		cl_cb_mutex;
 #endif /* CONFIG_NFSD_V4_1 */
 };
 
-- 
1.6.0.2

[RFC 34/51] nfsd41: cb_sequence callback

[Benny Halevy]

Implement the cb_sequence callback conforming to
http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4callback.c     |   57 ++++++++++++++++++++++++++++++++++++++++++-
 include/linux/nfsd/state.h |   12 +++++++++
 2 files changed, 67 insertions(+), 2 deletions(-)

diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
index b1d2299..312fe87 100644
--- a/fs/nfsd/nfs4callback.c
+++ b/fs/nfsd/nfs4callback.c
@@ -280,7 +280,18 @@ encode_cb_recall(struct xdr_stream *xdr, struct nfs4_cb_recall *cb_rec)
 static int
 encode_cb_sequence(struct xdr_stream *xdr, struct nfs41_cb_sequence *args)
 {
-	return -1;	/* stub */
+	u32 *p;
+
+	RESERVE_SPACE(1 + NFS4_MAX_SESSIONID_LEN + 20);
+
+	WRITE32(OP_CB_SEQUENCE);
+	WRITEMEM(args->cbs_sessionid, NFS4_MAX_SESSIONID_LEN);
+	WRITE32(args->cbs_seqid);
+	WRITE32(args->cbs_slotid);
+	WRITE32(args->cbs_highest_slotid);
+	WRITE32(args->cbsa_cachethis);
+	WRITE32(0); /* FIXME: support referring_call_lists */
+	return 0;
 }
 #endif /* defined(CONFIG_NFSD_V4_1) */
 
@@ -393,7 +404,19 @@ out:
 static int
 decode_cb_sequence(struct xdr_stream *xdr, struct nfs41_cb_sequence *res)
 {
-	return -1;	/* stub */
+	int status;
+	u32 *p;
+
+	status = decode_cb_op_hdr(xdr, OP_CB_SEQUENCE);
+	if (status)
+		return status;
+	READ_BUF(NFS4_MAX_SESSIONID_LEN + 16);
+	COPYMEM(res->cbs_sessionid, NFS4_MAX_SESSIONID_LEN);
+	READ32(res->cbs_seqid);
+	READ32(res->cbs_slotid);
+	READ32(res->cbs_highest_slotid);
+	READ32(res->cbsr_target_highest_slotid);
+	return 0;
 }
 
 static int
@@ -593,6 +616,36 @@ nfsd4_probe_callback(struct nfs4_client *clp)
 	return;
 }
 
+#if defined(CONFIG_NFSD_V4_1)
+/* FIXME: cb_sequence should support referring call lists, cachethis, and multiple slots */
+static int
+nfs41_cb_sequence_setup(struct nfs4_client *clp, struct nfs41_cb_sequence *args)
+{
+	u32 *ptr = (u32 *)clp->cl_sessionid;
+	dprintk("%s: %u:%u:%u:%u\n", __func__,
+		ptr[0], ptr[1], ptr[2], ptr[3]);
+
+	mutex_lock(&clp->cl_cb_mutex);
+	memcpy(args->cbs_sessionid, clp->cl_sessionid, NFS4_MAX_SESSIONID_LEN);
+	args->cbs_seqid = ++clp->cl_cb_seq_nr;
+	args->cbs_slotid = 0;
+	args->cbs_highest_slotid = 0;
+	args->cbsa_cachethis = 0;
+	return 0;
+}
+
+static void
+nfs41_cb_sequence_done(struct nfs4_client *clp, struct nfs41_cb_sequence *res)
+{
+	u32 *ptr = (u32 *)res->cbs_sessionid;
+	dprintk("%s: %u:%u:%u:%u\n", __func__,
+		ptr[0], ptr[1], ptr[2], ptr[3]);
+
+	/* FIXME: support multiple callback slots */
+	mutex_unlock(&clp->cl_cb_mutex);
+}
+#endif /* CONFIG_NFSD_V4_1 */
+
 /*
  * called with dp->dl_count inc'ed.
  * nfs4_lock_state() may or may not have been called.
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index eb8d55c..4f37e2f 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -181,6 +181,16 @@ struct current_session {
 	struct nfsd_sessionid	cs_sid;
 	struct nfs41_slot	*cs_slot;
 };
+
+struct nfs41_cb_sequence {
+	/* args/res */
+	char			cbs_sessionid[NFS4_MAX_SESSIONID_LEN];
+	u32			cbs_seqid;
+	u32			cbs_slotid;
+	u32			cbs_highest_slotid;
+	u32			cbsa_cachethis;			/* args only */
+	u32			cbsr_target_highest_slotid;	/* res only */
+};
 #endif /* CONFIG_NFSD_V4_1 */
 
 #define HEXDIR_LEN     33 /* hex version of 16 byte md5 of cl_name plus '\0' */
@@ -222,6 +232,8 @@ struct nfs4_client {
 
 	struct svc_xprt		*cl_cb_xprt;	/* 4.1 callback transport */
 	struct mutex		cl_cb_mutex;
+	/* FIXME: support multiple callback slots */
+	u32			cl_cb_seq_nr;
 #endif /* CONFIG_NFSD_V4_1 */
 };
 
-- 
1.6.0.2

[RFC 35/51] nfsd41: introduce nfs4_cb_call_sync for nfs4 and nfs41

[Benny Halevy]

For nfs41 callbacks we need to setup the cb_sequence args and
process the result.  This will be called by code common to
nfs4 and nfs41.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4callback.c |   41 +++++++++++++++++++++++++++++++++++++++++
 1 files changed, 41 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
index 312fe87..5b17397 100644
--- a/fs/nfsd/nfs4callback.c
+++ b/fs/nfsd/nfs4callback.c
@@ -616,6 +616,12 @@ nfsd4_probe_callback(struct nfs4_client *clp)
 	return;
 }
 
+static int _nfs4_cb_sync(struct nfs4_client *clp,
+			 const struct rpc_message *msg, int flags)
+{
+	return rpc_call_sync(clp->cl_callback.cb_client, msg, RPC_TASK_SOFT);
+}
+
 #if defined(CONFIG_NFSD_V4_1)
 /* FIXME: cb_sequence should support referring call lists, cachethis, and multiple slots */
 static int
@@ -644,6 +650,41 @@ nfs41_cb_sequence_done(struct nfs4_client *clp, struct nfs41_cb_sequence *res)
 	/* FIXME: support multiple callback slots */
 	mutex_unlock(&clp->cl_cb_mutex);
 }
+
+static int _nfs41_cb_sync(struct nfs4_client *clp,
+			  struct rpc_message *msg, int flags)
+{
+	struct nfs41_cb_sequence seq;
+	struct nfs4_rpc_args *args;
+	struct nfs4_rpc_res res;
+	int status;
+
+	args = msg->rpc_argp;
+	args->args_seq = &seq;
+
+	res.res_seq = &seq;
+	msg->rpc_resp = &res;
+
+	nfs41_cb_sequence_setup(clp, &seq);
+	status = _nfs4_cb_sync(clp, msg, flags);
+	nfs41_cb_sequence_done(clp, &seq);
+
+	return status;
+}
+
+static int nfs4_cb_sync(struct nfs4_client *clp,
+			     struct rpc_message *msg, int flags)
+{
+	return clp->cl_callback.cb_minorversion ?
+		_nfs41_cb_sync(clp, msg, flags) :
+		_nfs4_cb_sync(clp, msg, flags);
+}
+#else  /* CONFIG_NFSD_V4_1 */
+static int nfs4_cb_sync(struct nfs4_client *clp,
+			     struct rpc_message *msg, int flags)
+{
+	return _nfs4_cb_sync(clp, msg, flags);
+}
 #endif /* CONFIG_NFSD_V4_1 */
 
 /*
-- 
1.6.0.2

[RFC 36/51] nfsd41: cb_recall callback

[Benny Halevy]

From: Ricardo Labiaga <ricardo.labiaga@netapp.com>

Implement the cb_recall callback conforming to
http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26

Signed-off-by: Ricardo Labiaga <ricardo.labiaga@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4callback.c |   49 +++++++++++++++++++++++++++++++++++++++++------
 1 files changed, 42 insertions(+), 7 deletions(-)

diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
index 5b17397..53ffc56 100644
--- a/fs/nfsd/nfs4callback.c
+++ b/fs/nfsd/nfs4callback.c
@@ -306,9 +306,11 @@ nfs4_xdr_enc_cb_null(struct rpc_rqst *req, __be32 *p)
 }
 
 static int
-nfs4_xdr_enc_cb_recall(struct rpc_rqst *req, __be32 *p, struct nfs4_cb_recall *args)
+nfs4_xdr_enc_cb_recall(struct rpc_rqst *req, __be32 *p,
+		       struct nfs4_rpc_args *rpc_args)
 {
 	struct xdr_stream xdr;
+	struct nfs4_cb_recall *args = rpc_args->args_op;
 	struct nfs4_cb_compound_hdr hdr = {
 		.ident = args->cbr_ident,
 		.nops   = 1,
@@ -339,7 +341,16 @@ static int
 nfs41_xdr_enc_cb_recall(struct rpc_rqst *req, u32 *p,
 			struct nfs4_rpc_args *rpc_args)
 {
-	return -1;	/* stub */
+	struct xdr_stream xdr;
+	struct nfs4_cb_recall *args = rpc_args->args_op;
+	struct nfs4_cb_compound_hdr hdr = {
+		.nops   = 2,
+	};
+
+	xdr_init_encode(&xdr, &req->rq_snd_buf, p);
+	encode_cb_compound41_hdr(&xdr, &hdr);
+	encode_cb_sequence(&xdr, rpc_args->args_seq);
+	return encode_cb_recall(&xdr, args);
 }
 #endif /* defined(CONFIG_NFSD_V4_1) */
 
@@ -423,7 +434,20 @@ static int
 nfs41_xdr_dec_cb_recall(struct rpc_rqst *rqstp, u32 *p,
 			struct nfs4_rpc_res *rpc_res)
 {
-	return -1;	/* stub */
+	struct xdr_stream xdr;
+	struct nfs4_cb_compound_hdr hdr;
+	int status;
+
+	xdr_init_decode(&xdr, &rqstp->rq_rcv_buf, p);
+	status = decode_cb_compound_hdr(&xdr, &hdr);
+	if (status)
+		goto out;
+	status = decode_cb_sequence(&xdr, rpc_res->res_seq);
+	if (status)
+		goto out;
+	status = decode_cb_op_hdr(&xdr, OP_CB_RECALL);
+out:
+	return status;
 }
 #endif /* defined(CONFIG_NFSD_V4_1) */
 
@@ -695,19 +719,28 @@ void
 nfsd4_cb_recall(struct nfs4_delegation *dp)
 {
 	struct nfs4_client *clp = dp->dl_client;
-	struct rpc_clnt *clnt = clp->cl_callback.cb_client;
 	struct nfs4_cb_recall *cbr = &dp->dl_recall;
+	struct nfs4_rpc_args args = {
+		.args_op = cbr,
+	};
 	struct rpc_message msg = {
 		.rpc_proc = &nfs4_cb_procedures[NFSPROC4_CLNT_CB_RECALL],
-		.rpc_argp = cbr,
+		.rpc_argp = &args,
 	};
 	int retries = 1;
 	int status = 0;
 
+	dprintk("%s: dp %p\n", __func__, dp);
+
 	cbr->cbr_trunc = 0; /* XXX need to implement truncate optimization */
 	cbr->cbr_dp = dp;
 
-	status = rpc_call_sync(clnt, &msg, RPC_TASK_SOFT);
+#if defined(CONFIG_NFSD_V4_1)
+	if (clp->cl_callback.cb_minorversion)
+		msg.rpc_proc = &nfs41_cb_procedures[NFSPROC4_CLNT_CB_RECALL];
+#endif /* CONFIG_NFSD_V4_1 */
+
+	status = nfs4_cb_sync(clp, &msg, RPC_TASK_SOFT);
 	while (retries--) {
 		switch (status) {
 			case -EIO:
@@ -722,13 +755,15 @@ nfsd4_cb_recall(struct nfs4_delegation *dp)
 				goto out_put_cred;
 		}
 		ssleep(2);
-		status = rpc_call_sync(clnt, &msg, RPC_TASK_SOFT);
+		status = nfs4_cb_sync(clp, &msg, RPC_TASK_SOFT);
 	}
 out_put_cred:
 	/*
 	 * Success or failure, now we're either waiting for lease expiration
 	 * or deleg_return.
 	 */
+	dprintk("%s: dp %p dl_flock %p dl_count %d\n",
+		__func__, dp, dp->dl_flock, atomic_read(&dp->dl_count));
 	put_nfs4_client(clp);
 	nfs4_put_delegation(dp);
 	return;
-- 
1.6.0.2

[RFC 37/51] nfsd41: pass writable attrs mask to nfsd4_decode_fattr

[Benny Halevy]

In preparation for EXCLUSIVE4_1

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4xdr.c |   21 +++++++++++++++------
 1 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 6db74f7..e6cad37 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -252,9 +252,14 @@ nfsd4_decode_bitmap(struct nfsd4_compoundargs *argp, u32 *bmval)
 	DECODE_TAIL;
 }
 
+static u32 nfsd_attrmask[] = {
+	NFSD_WRITEABLE_ATTRS_WORD0,
+	NFSD_WRITEABLE_ATTRS_WORD1
+};
+
 static __be32
-nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, struct iattr *iattr,
-    struct nfs4_acl **acl)
+nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, u32 *writable,
+		   struct iattr *iattr, struct nfs4_acl **acl)
 {
 	int expected_len, len = 0;
 	u32 dummy32;
@@ -272,7 +277,7 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, struct iattr *ia
 	 */
 	if ((bmval[0] & ~NFSD_SUPPORTED_ATTRS_WORD0) || (bmval[1] & ~NFSD_SUPPORTED_ATTRS_WORD1))
 		return nfserr_attrnotsupp;
-	if ((bmval[0] & ~NFSD_WRITEABLE_ATTRS_WORD0) || (bmval[1] & ~NFSD_WRITEABLE_ATTRS_WORD1))
+	if ((bmval[0] & ~writable[0]) || (bmval[1] & ~writable[1]))
 		return nfserr_inval;
 
 	READ_BUF(4);
@@ -500,7 +505,9 @@ nfsd4_decode_create(struct nfsd4_compoundargs *argp, struct nfsd4_create *create
 	if ((status = check_filename(create->cr_name, create->cr_namelen, nfserr_inval)))
 		return status;
 
-	if ((status = nfsd4_decode_fattr(argp, create->cr_bmval, &create->cr_iattr, &create->cr_acl)))
+	status = nfsd4_decode_fattr(argp, create->cr_bmval, nfsd_attrmask,
+				    &create->cr_iattr, &create->cr_acl);
+	if (status)
 		goto out;
 
 	DECODE_TAIL;
@@ -663,7 +670,9 @@ nfsd4_decode_open(struct nfsd4_compoundargs *argp, struct nfsd4_open *open)
 		switch (open->op_createmode) {
 		case NFS4_CREATE_UNCHECKED:
 		case NFS4_CREATE_GUARDED:
-			if ((status = nfsd4_decode_fattr(argp, open->op_bmval, &open->op_iattr, &open->op_acl)))
+			status = nfsd4_decode_fattr(argp, open->op_bmval,
+				nfsd_attrmask, &open->op_iattr, &open->op_acl);
+			if (status)
 				goto out;
 			break;
 		case NFS4_CREATE_EXCLUSIVE:
@@ -866,7 +875,7 @@ nfsd4_decode_setattr(struct nfsd4_compoundargs *argp, struct nfsd4_setattr *seta
 	status = nfsd4_decode_stateid(argp, &setattr->sa_stateid);
 	if (status)
 		return status;
-	return nfsd4_decode_fattr(argp, setattr->sa_bmval,
+	return nfsd4_decode_fattr(argp, setattr->sa_bmval, nfsd_attrmask,
 				  &setattr->sa_iattr, &setattr->sa_acl);
 }
 
-- 
1.6.0.2

[RFC 38/51] nfsd41: support for 3-word long attribute bitmask

[Benny Halevy]

Also, use client minorversion to generate supported attrs

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c        |   24 +++++++++----
 fs/nfsd/nfs4xdr.c         |   82 ++++++++++++++++++++++++++++++++++++--------
 include/linux/nfsd/nfsd.h |   51 +++++++++++++++++++++++++++-
 include/linux/nfsd/xdr4.h |   29 +++++++++++----
 4 files changed, 154 insertions(+), 32 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 6f6d221..5164040 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -459,6 +459,7 @@ static __be32
 nfsd4_getattr(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	      struct nfsd4_getattr *getattr)
 {
+	u32 minorversion;
 	__be32 status;
 
 	status = fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_NOP);
@@ -468,8 +469,10 @@ nfsd4_getattr(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	if (getattr->ga_bmval[1] & NFSD_WRITEONLY_ATTRS_WORD1)
 		return nfserr_inval;
 
-	getattr->ga_bmval[0] &= NFSD_SUPPORTED_ATTRS_WORD0;
-	getattr->ga_bmval[1] &= NFSD_SUPPORTED_ATTRS_WORD1;
+	minorversion = nfsd4_compound_minorversion(cstate);
+	getattr->ga_bmval[0] &= nfsd_suppattrs0(minorversion);
+	getattr->ga_bmval[1] &= nfsd_suppattrs1(minorversion);
+	getattr->ga_bmval[2] &= nfsd_suppattrs2(minorversion);
 
 	getattr->ga_fhp = &cstate->current_fh;
 	return nfs_ok;
@@ -557,6 +560,7 @@ static __be32
 nfsd4_readdir(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	      struct nfsd4_readdir *readdir)
 {
+	u32 minorversion;
 	u64 cookie = readdir->rd_cookie;
 	static const nfs4_verifier zeroverf;
 
@@ -565,8 +569,10 @@ nfsd4_readdir(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	if (readdir->rd_bmval[1] & NFSD_WRITEONLY_ATTRS_WORD1)
 		return nfserr_inval;
 
-	readdir->rd_bmval[0] &= NFSD_SUPPORTED_ATTRS_WORD0;
-	readdir->rd_bmval[1] &= NFSD_SUPPORTED_ATTRS_WORD1;
+	minorversion = nfsd4_compound_minorversion(cstate);
+	readdir->rd_bmval[0] &= nfsd_suppattrs0(minorversion);
+	readdir->rd_bmval[1] &= nfsd_suppattrs1(minorversion);
+	readdir->rd_bmval[2] &= nfsd_suppattrs2(minorversion);
 
 	if ((cookie > ~(u32)0) || (cookie == 1) || (cookie == 2) ||
 	    (cookie == 0 && memcmp(readdir->rd_verf.data, zeroverf.data, NFS4_VERIFIER_SIZE)))
@@ -754,14 +760,17 @@ _nfsd4_verify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 {
 	__be32 *buf, *p;
 	int count;
+	u32 minorversion;
 	__be32 status;
 
 	status = fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_NOP);
 	if (status)
 		return status;
 
-	if ((verify->ve_bmval[0] & ~NFSD_SUPPORTED_ATTRS_WORD0)
-	    || (verify->ve_bmval[1] & ~NFSD_SUPPORTED_ATTRS_WORD1))
+	minorversion = nfsd4_compound_minorversion(cstate);
+	if ((verify->ve_bmval[0] & ~nfsd_suppattrs0(minorversion))
+	    || (verify->ve_bmval[1] & ~nfsd_suppattrs1(minorversion))
+	    || (verify->ve_bmval[2] & ~nfsd_suppattrs2(minorversion)))
 		return nfserr_attrnotsupp;
 	if ((verify->ve_bmval[0] & FATTR4_WORD0_RDATTR_ERROR)
 	    || (verify->ve_bmval[1] & NFSD_WRITEONLY_ATTRS_WORD1))
@@ -781,7 +790,7 @@ _nfsd4_verify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 				    cstate->current_fh.fh_export,
 				    cstate->current_fh.fh_dentry, buf,
 				    &count, verify->ve_bmval,
-				    rqstp, 0);
+				    rqstp, 0, verify->ve_minorversion);
 
 	/* this means that nfsd4_encode_fattr() ran out of space */
 	if (status == nfserr_resource && count == 0)
@@ -903,6 +912,7 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
 	resp->tag = args->tag;
 	resp->opcnt = 0;
 	resp->rqstp = rqstp;
+	resp->minorversion = args->minorversion;
 
 	/*
 	 * According to RFC3010, this takes precedence over all other errors.
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index e6cad37..89e7b0d 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -237,6 +237,7 @@ nfsd4_decode_bitmap(struct nfsd4_compoundargs *argp, u32 *bmval)
 
 	bmval[0] = 0;
 	bmval[1] = 0;
+	bmval[2] = 0;
 
 	READ_BUF(4);
 	READ32(bmlen);
@@ -248,13 +249,19 @@ nfsd4_decode_bitmap(struct nfsd4_compoundargs *argp, u32 *bmval)
 		READ32(bmval[0]);
 	if (bmlen > 1)
 		READ32(bmval[1]);
+#if defined(CONFIG_NFSD_V4_1)
+	if (bmlen > 2) {
+		READ32(bmval[2]);
+	}
+#endif /* CONFIG_NFSD_V4_1 */
 
 	DECODE_TAIL;
 }
 
 static u32 nfsd_attrmask[] = {
 	NFSD_WRITEABLE_ATTRS_WORD0,
-	NFSD_WRITEABLE_ATTRS_WORD1
+	NFSD_WRITEABLE_ATTRS_WORD1,
+	NFSD_WRITEABLE_ATTRS_WORD2
 };
 
 static __be32
@@ -275,9 +282,12 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, u32 *writable,
 	 * According to spec, unsupported attributes return ERR_ATTRNOTSUPP;
 	 * read-only attributes return ERR_INVAL.
 	 */
-	if ((bmval[0] & ~NFSD_SUPPORTED_ATTRS_WORD0) || (bmval[1] & ~NFSD_SUPPORTED_ATTRS_WORD1))
+	if ((bmval[0] & ~nfsd_suppattrs0(argp->minorversion)) ||
+	    (bmval[1] & ~nfsd_suppattrs1(argp->minorversion)) ||
+	    (bmval[2] & ~nfsd_suppattrs2(argp->minorversion)))
 		return nfserr_attrnotsupp;
-	if ((bmval[0] & ~writable[0]) || (bmval[1] & ~writable[1]))
+	if ((bmval[0] & ~writable[0]) || (bmval[1] & ~writable[1]) ||
+	    (bmval[2] & ~writable[2]))
 		return nfserr_inval;
 
 	READ_BUF(4);
@@ -412,6 +422,7 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, u32 *writable,
 			goto xdr_error;
 		}
 	}
+	BUG_ON(bmval[2]);	/* no such writeable attr supported yet */
 	if (len != expected_len)
 		goto xdr_error;
 
@@ -796,6 +807,7 @@ nfsd4_decode_readdir(struct nfsd4_compoundargs *argp, struct nfsd4_readdir *read
 	COPYMEM(readdir->rd_verf.data, sizeof(readdir->rd_verf.data));
 	READ32(readdir->rd_dircount);    /* just in case you needed a useless field... */
 	READ32(readdir->rd_maxcount);
+	readdir->minorversion = argp->minorversion;
 	if ((status = nfsd4_decode_bitmap(argp, readdir->rd_bmval)))
 		goto out;
 
@@ -949,6 +961,7 @@ nfsd4_decode_verify(struct nfsd4_compoundargs *argp, struct nfsd4_verify *verify
 	READ32(verify->ve_attrlen);
 	READ_BUF(verify->ve_attrlen);
 	SAVEMEM(verify->ve_attrval, verify->ve_attrlen);
+	verify->ve_minorversion = argp->minorversion;
 
 	DECODE_TAIL;
 }
@@ -1738,10 +1751,11 @@ static __be32 fattr_handle_absent_fs(u32 *bmval0, u32 *bmval1, u32 *rdattr_err)
 __be32
 nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp,
 		struct dentry *dentry, __be32 *buffer, int *countp, u32 *bmval,
-		struct svc_rqst *rqstp, int ignore_crossmnt)
+		struct svc_rqst *rqstp, int ignore_crossmnt, u32 minorversion)
 {
 	u32 bmval0 = bmval[0];
 	u32 bmval1 = bmval[1];
+	u32 bmval2 = bmval[2];
 	struct kstat stat;
 	struct svc_fh tempfh;
 	struct kstatfs statfs;
@@ -1757,10 +1771,25 @@ nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp,
 	struct nfs4_acl *acl = NULL;
 
 	BUG_ON(bmval1 & NFSD_WRITEONLY_ATTRS_WORD1);
-	BUG_ON(bmval0 & ~NFSD_SUPPORTED_ATTRS_WORD0);
-	BUG_ON(bmval1 & ~NFSD_SUPPORTED_ATTRS_WORD1);
+	/* FIXME: change warnings back into BUG_ON */;
+	if (bmval0 & ~nfsd_suppattrs0(minorversion)) {
+		printk("%s: minorversion=%u bmval0=0x%08x suppattrs0=0x%08x\n",
+			__func__, minorversion, bmval0, nfsd_suppattrs0(minorversion));
+		WARN_ON(1);
+	}
+	if (bmval1 & ~nfsd_suppattrs1(minorversion)) {
+		printk("%s: minorversion=%u bmval1=0x%08x suppattrs1=0x%08x\n",
+			__func__, minorversion, bmval1, nfsd_suppattrs1(minorversion));
+		WARN_ON(1);
+	}
+	if (bmval2 & ~nfsd_suppattrs2(minorversion)) {
+		printk("%s: minorversion=%u bmval2=0x%08x suppattrs2=0x%08x\n",
+			__func__, minorversion, bmval2, nfsd_suppattrs2(minorversion));
+		WARN_ON(1);
+	}
 
 	if (exp->ex_fslocs.migrated) {
+		BUG_ON(bmval[2]);
 		status = fattr_handle_absent_fs(&bmval0, &bmval1, &rdattr_err);
 		if (status)
 			goto out;
@@ -1806,22 +1835,42 @@ nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp,
 	if ((buflen -= 16) < 0)
 		goto out_resource;
 
-	WRITE32(2);
-	WRITE32(bmval0);
-	WRITE32(bmval1);
+	if (unlikely(bmval2)) {
+		WRITE32(3);
+		WRITE32(bmval0);
+		WRITE32(bmval1);
+		WRITE32(bmval2);
+	} else if (likely(bmval1)) {
+		WRITE32(2);
+		WRITE32(bmval0);
+		WRITE32(bmval1);
+	} else {
+		WRITE32(1);
+		WRITE32(bmval0);
+	}
 	attrlenp = p++;                /* to be backfilled later */
 
 	if (bmval0 & FATTR4_WORD0_SUPPORTED_ATTRS) {
-		u32 word0 = NFSD_SUPPORTED_ATTRS_WORD0;
+		u32 word0 = nfsd_suppattrs0(minorversion);
+		u32 word1 = nfsd_suppattrs1(minorversion);
+		u32 word2 = nfsd_suppattrs2(minorversion);
+
 		if ((buflen -= 12) < 0)
 			goto out_resource;
 		if (!aclsupport)
 			word0 &= ~FATTR4_WORD0_ACL;
 		if (!exp->ex_fslocs.locations)
 			word0 &= ~FATTR4_WORD0_FS_LOCATIONS;
-		WRITE32(2);
-		WRITE32(word0);
-		WRITE32(NFSD_SUPPORTED_ATTRS_WORD1);
+		if (!word2) {
+			WRITE32(2);
+			WRITE32(word0);
+			WRITE32(word1);
+		} else {
+			WRITE32(3);
+			WRITE32(word0);
+			WRITE32(word1);
+			WRITE32(word2);
+		}
 	}
 	if (bmval0 & FATTR4_WORD0_TYPE) {
 		if ((buflen -= 4) < 0)
@@ -2131,6 +2180,8 @@ out_acl:
 		}
 		WRITE64(stat.ino);
 	}
+	BUG_ON(bmval2);	/* FIXME: not implemented yet */
+
 	*attrlenp = htonl((char *)p - (char *)attrlenp - 4);
 	*countp = p - buffer;
 	status = nfs_ok;
@@ -2203,7 +2254,8 @@ nfsd4_encode_dirent_fattr(struct nfsd4_readdir *cd,
 
 	}
 	nfserr = nfsd4_encode_fattr(NULL, exp, dentry, p, buflen, cd->rd_bmval,
-					cd->rd_rqstp, ignore_crossmnt);
+					cd->rd_rqstp, ignore_crossmnt,
+					cd->minorversion);
 out_put:
 	dput(dentry);
 	exp_put(exp);
@@ -2369,7 +2421,7 @@ nfsd4_encode_getattr(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4
 	buflen = resp->end - resp->p - (COMPOUND_ERR_SLACK_SPACE >> 2);
 	nfserr = nfsd4_encode_fattr(fhp, fhp->fh_export, fhp->fh_dentry,
 				    resp->p, &buflen, getattr->ga_bmval,
-				    resp->rqstp, 0);
+				    resp->rqstp, 0, resp->minorversion);
 	if (!nfserr)
 		resp->p += buflen;
 	return nfserr;
diff --git a/include/linux/nfsd/nfsd.h b/include/linux/nfsd/nfsd.h
index ffb7897..46b8110 100644
--- a/include/linux/nfsd/nfsd.h
+++ b/include/linux/nfsd/nfsd.h
@@ -321,7 +321,7 @@ extern struct timeval	nfssvc_boot;
  *    TIME_BACKUP   (unlikely to be supported any time soon)
  *    TIME_CREATE   (unlikely to be supported any time soon)
  */
-#define NFSD_SUPPORTED_ATTRS_WORD0                                                          \
+#define NFSD4_SUPPORTED_ATTRS_WORD0                                                         \
 (FATTR4_WORD0_SUPPORTED_ATTRS   | FATTR4_WORD0_TYPE         | FATTR4_WORD0_FH_EXPIRE_TYPE   \
  | FATTR4_WORD0_CHANGE          | FATTR4_WORD0_SIZE         | FATTR4_WORD0_LINK_SUPPORT     \
  | FATTR4_WORD0_SYMLINK_SUPPORT | FATTR4_WORD0_NAMED_ATTR   | FATTR4_WORD0_FSID             \
@@ -333,7 +333,7 @@ extern struct timeval	nfssvc_boot;
  | FATTR4_WORD0_MAXFILESIZE     | FATTR4_WORD0_MAXLINK      | FATTR4_WORD0_MAXNAME          \
  | FATTR4_WORD0_MAXREAD         | FATTR4_WORD0_MAXWRITE     | FATTR4_WORD0_ACL)
 
-#define NFSD_SUPPORTED_ATTRS_WORD1                                                          \
+#define NFSD4_SUPPORTED_ATTRS_WORD1                                                         \
 (FATTR4_WORD1_MODE              | FATTR4_WORD1_NO_TRUNC     | FATTR4_WORD1_NUMLINKS         \
  | FATTR4_WORD1_OWNER	        | FATTR4_WORD1_OWNER_GROUP  | FATTR4_WORD1_RAWDEV           \
  | FATTR4_WORD1_SPACE_AVAIL     | FATTR4_WORD1_SPACE_FREE   | FATTR4_WORD1_SPACE_TOTAL      \
@@ -341,6 +341,52 @@ extern struct timeval	nfssvc_boot;
  | FATTR4_WORD1_TIME_DELTA   | FATTR4_WORD1_TIME_METADATA    \
  | FATTR4_WORD1_TIME_MODIFY     | FATTR4_WORD1_TIME_MODIFY_SET | FATTR4_WORD1_MOUNTED_ON_FILEID)
 
+#define NFSD4_SUPPORTED_ATTRS_WORD2 0
+
+#if defined(CONFIG_NFSD_V4_1)
+#define NFSD4_1_SUPPORTED_ATTRS_WORD0 \
+	NFSD4_SUPPORTED_ATTRS_WORD0
+
+#define NFSD4_1_SUPPORTED_ATTRS_WORD1 \
+	NFSD4_SUPPORTED_ATTRS_WORD1
+
+#define NFSD4_1_SUPPORTED_ATTRS_WORD2 \
+	NFSD4_SUPPORTED_ATTRS_WORD2
+
+static inline u32 nfsd_suppattrs0(u32 minorversion)
+{
+	return minorversion ? NFSD4_1_SUPPORTED_ATTRS_WORD0
+			    : NFSD4_SUPPORTED_ATTRS_WORD0;
+}
+
+static inline u32 nfsd_suppattrs1(u32 minorversion)
+{
+	return minorversion ? NFSD4_1_SUPPORTED_ATTRS_WORD1
+			    : NFSD4_SUPPORTED_ATTRS_WORD1;
+}
+
+static inline u32 nfsd_suppattrs2(u32 minorversion)
+{
+	return minorversion ? NFSD4_1_SUPPORTED_ATTRS_WORD2
+			    : NFSD4_SUPPORTED_ATTRS_WORD2;
+}
+#else  /* CONFIG_NFSD_V4_1 */
+static inline u32 nfsd_suppattrs0(u32 minorversion)
+{
+	return NFSD4_SUPPORTED_ATTRS_WORD0;
+}
+
+static inline u32 nfsd_suppattrs1(u32 minorversion)
+{
+	return NFSD4_SUPPORTED_ATTRS_WORD1;
+}
+
+static inline u32 nfsd_suppattrs2(u32 minorversion)
+{
+	return NFSD4_SUPPORTED_ATTRS_WORD2;
+}
+#endif /* CONFIG_NFSD_V4_1 */
+
 /* These will return ERR_INVAL if specified in GETATTR or READDIR. */
 #define NFSD_WRITEONLY_ATTRS_WORD1							    \
 (FATTR4_WORD1_TIME_ACCESS_SET   | FATTR4_WORD1_TIME_MODIFY_SET)
@@ -351,6 +397,7 @@ extern struct timeval	nfssvc_boot;
 #define NFSD_WRITEABLE_ATTRS_WORD1                                                          \
 (FATTR4_WORD1_MODE              | FATTR4_WORD1_OWNER         | FATTR4_WORD1_OWNER_GROUP     \
  | FATTR4_WORD1_TIME_ACCESS_SET | FATTR4_WORD1_TIME_MODIFY_SET)
+#define NFSD_WRITEABLE_ATTRS_WORD2 0
 
 #endif /* CONFIG_NFSD_V4 */
 
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index d78ba3c..7812e8c 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -53,6 +53,15 @@ struct nfsd4_compound_state {
 #endif /* CONFIG_NFSD_V4_1 */
 };
 
+static inline u32 nfsd4_compound_minorversion(struct nfsd4_compound_state *cs)
+{
+#if defined(CONFIG_NFSD_V4_1)
+	return cs->current_ses ? 1 : 0;
+#else  /* CONFIG_NFSD_V4_1 */
+	return 0;
+#endif /* CONFIG_NFSD_V4_1 */
+}
+
 struct nfsd4_change_info {
 	u32		atomic;
 	u32		before_ctime_sec;
@@ -93,7 +102,7 @@ struct nfsd4_create {
 			u32 specdata2;
 		} dev;    /* NF4BLK, NF4CHR */
 	} u;
-	u32		cr_bmval[2];        /* request */
+	u32		cr_bmval[3];        /* request */
 	struct iattr	cr_iattr;           /* request */
 	struct nfsd4_change_info  cr_cinfo; /* response */
 	struct nfs4_acl *cr_acl;
@@ -109,7 +118,7 @@ struct nfsd4_delegreturn {
 };
 
 struct nfsd4_getattr {
-	u32		ga_bmval[2];        /* request */
+	u32		ga_bmval[3];        /* request */
 	struct svc_fh	*ga_fhp;            /* response */
 };
 
@@ -210,7 +219,7 @@ struct nfsd4_open {
 	stateid_t       op_delegate_stateid; /* request - response */
 	u32		op_create;     	    /* request */
 	u32		op_createmode;      /* request */
-	u32		op_bmval[2];        /* request */
+	u32		op_bmval[3];        /* request */
 	union {                             /* request */
 		struct iattr	iattr;                      /* UNCHECKED4,GUARDED4 */
 		nfs4_verifier	verf;                                /* EXCLUSIVE4 */
@@ -265,7 +274,7 @@ struct nfsd4_readdir {
 	nfs4_verifier	rd_verf;            /* request */
 	u32		rd_dircount;        /* request */
 	u32		rd_maxcount;        /* request */
-	u32		rd_bmval[2];        /* request */
+	u32		rd_bmval[3];        /* request */
 	struct svc_rqst *rd_rqstp;          /* response */
 	struct svc_fh * rd_fhp;             /* response */
 
@@ -273,6 +282,7 @@ struct nfsd4_readdir {
 	__be32 *		buffer;
 	int			buflen;
 	__be32 *		offset;
+	u32			minorversion;
 };
 
 struct nfsd4_release_lockowner {
@@ -308,7 +318,7 @@ struct nfsd4_secinfo {
 struct nfsd4_setattr {
 	stateid_t	sa_stateid;         /* request */
 	u32		sa_minorversion;    /* processing */
-	u32		sa_bmval[2];        /* request */
+	u32		sa_bmval[3];        /* request */
 	struct iattr	sa_iattr;           /* request */
 	struct nfs4_acl *sa_acl;
 };
@@ -334,9 +344,10 @@ struct nfsd4_setclientid_confirm {
 
 /* also used for NVERIFY */
 struct nfsd4_verify {
-	u32		ve_bmval[2];        /* request */
+	u32		ve_bmval[3];        /* request */
 	u32		ve_attrlen;         /* request */
 	char *		ve_attrval;         /* request */
+	u32		ve_minorversion;
 };
 
 struct nfsd4_write {
@@ -479,6 +490,7 @@ struct nfsd4_compoundres {
 	char *				tag;
 	u32				opcnt;
 	__be32 *			tagp; /* where to encode tag and  opcount */
+	u32				minorversion;
 };
 
 #define NFS4_SVC_XDRSIZE		sizeof(struct nfsd4_compoundargs)
@@ -502,8 +514,9 @@ int nfs4svc_encode_compoundres(struct svc_rqst *, __be32 *,
 void nfsd4_encode_operation(struct nfsd4_compoundres *, struct nfsd4_op *);
 void nfsd4_encode_replay(struct nfsd4_compoundres *resp, struct nfsd4_op *op);
 __be32 nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp,
-		       struct dentry *dentry, __be32 *buffer, int *countp,
-		       u32 *bmval, struct svc_rqst *, int ignore_crossmnt);
+			  struct dentry *dentry, __be32 *buffer, int *countp,
+			  u32 *bmval, struct svc_rqst *, int ignore_crossmnt,
+			  u32 minorversion);
 extern __be32 nfsd4_setclientid(struct svc_rqst *rqstp,
 		struct nfsd4_compound_state *,
 		struct nfsd4_setclientid *setclid);
-- 
1.6.0.2

Re: [pnfs] [RFC 38/51] nfsd41: support for 3-word long attribute bitmask

[Benny Halevy]

On Nov. 10, 2008, 22:53 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> Also, use client minorversion to generate supported attrs
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4proc.c        |   24 +++++++++----
>  fs/nfsd/nfs4xdr.c         |   82 ++++++++++++++++++++++++++++++++++++--------
>  include/linux/nfsd/nfsd.h |   51 +++++++++++++++++++++++++++-
>  include/linux/nfsd/xdr4.h |   29 +++++++++++----
>  4 files changed, 154 insertions(+), 32 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index 6f6d221..5164040 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -459,6 +459,7 @@ static __be32
>  nfsd4_getattr(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	      struct nfsd4_getattr *getattr)
>  {
> +	u32 minorversion;
>  	__be32 status;
>  
>  	status = fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_NOP);
> @@ -468,8 +469,10 @@ nfsd4_getattr(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	if (getattr->ga_bmval[1] & NFSD_WRITEONLY_ATTRS_WORD1)
>  		return nfserr_inval;
>  
> -	getattr->ga_bmval[0] &= NFSD_SUPPORTED_ATTRS_WORD0;
> -	getattr->ga_bmval[1] &= NFSD_SUPPORTED_ATTRS_WORD1;
> +	minorversion = nfsd4_compound_minorversion(cstate);
> +	getattr->ga_bmval[0] &= nfsd_suppattrs0(minorversion);
> +	getattr->ga_bmval[1] &= nfsd_suppattrs1(minorversion);
> +	getattr->ga_bmval[2] &= nfsd_suppattrs2(minorversion);
>  
>  	getattr->ga_fhp = &cstate->current_fh;
>  	return nfs_ok;
> @@ -557,6 +560,7 @@ static __be32
>  nfsd4_readdir(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	      struct nfsd4_readdir *readdir)
>  {
> +	u32 minorversion;
>  	u64 cookie = readdir->rd_cookie;
>  	static const nfs4_verifier zeroverf;
>  
> @@ -565,8 +569,10 @@ nfsd4_readdir(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	if (readdir->rd_bmval[1] & NFSD_WRITEONLY_ATTRS_WORD1)
>  		return nfserr_inval;
>  
> -	readdir->rd_bmval[0] &= NFSD_SUPPORTED_ATTRS_WORD0;
> -	readdir->rd_bmval[1] &= NFSD_SUPPORTED_ATTRS_WORD1;
> +	minorversion = nfsd4_compound_minorversion(cstate);
> +	readdir->rd_bmval[0] &= nfsd_suppattrs0(minorversion);
> +	readdir->rd_bmval[1] &= nfsd_suppattrs1(minorversion);
> +	readdir->rd_bmval[2] &= nfsd_suppattrs2(minorversion);
>  
>  	if ((cookie > ~(u32)0) || (cookie == 1) || (cookie == 2) ||
>  	    (cookie == 0 && memcmp(readdir->rd_verf.data, zeroverf.data, NFS4_VERIFIER_SIZE)))
> @@ -754,14 +760,17 @@ _nfsd4_verify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  {
>  	__be32 *buf, *p;
>  	int count;
> +	u32 minorversion;
>  	__be32 status;
>  
>  	status = fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_NOP);
>  	if (status)
>  		return status;
>  
> -	if ((verify->ve_bmval[0] & ~NFSD_SUPPORTED_ATTRS_WORD0)
> -	    || (verify->ve_bmval[1] & ~NFSD_SUPPORTED_ATTRS_WORD1))
> +	minorversion = nfsd4_compound_minorversion(cstate);
> +	if ((verify->ve_bmval[0] & ~nfsd_suppattrs0(minorversion))
> +	    || (verify->ve_bmval[1] & ~nfsd_suppattrs1(minorversion))
> +	    || (verify->ve_bmval[2] & ~nfsd_suppattrs2(minorversion)))
>  		return nfserr_attrnotsupp;
>  	if ((verify->ve_bmval[0] & FATTR4_WORD0_RDATTR_ERROR)
>  	    || (verify->ve_bmval[1] & NFSD_WRITEONLY_ATTRS_WORD1))
> @@ -781,7 +790,7 @@ _nfsd4_verify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  				    cstate->current_fh.fh_export,
>  				    cstate->current_fh.fh_dentry, buf,
>  				    &count, verify->ve_bmval,
> -				    rqstp, 0);
> +				    rqstp, 0, verify->ve_minorversion);
>  
>  	/* this means that nfsd4_encode_fattr() ran out of space */
>  	if (status == nfserr_resource && count == 0)
> @@ -903,6 +912,7 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
>  	resp->tag = args->tag;
>  	resp->opcnt = 0;
>  	resp->rqstp = rqstp;
> +	resp->minorversion = args->minorversion;
>  
>  	/*
>  	 * According to RFC3010, this takes precedence over all other errors.
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index e6cad37..89e7b0d 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -237,6 +237,7 @@ nfsd4_decode_bitmap(struct nfsd4_compoundargs *argp, u32 *bmval)
>  
>  	bmval[0] = 0;
>  	bmval[1] = 0;
> +	bmval[2] = 0;
>  
>  	READ_BUF(4);
>  	READ32(bmlen);
> @@ -248,13 +249,19 @@ nfsd4_decode_bitmap(struct nfsd4_compoundargs *argp, u32 *bmval)
>  		READ32(bmval[0]);
>  	if (bmlen > 1)
>  		READ32(bmval[1]);
> +#if defined(CONFIG_NFSD_V4_1)
> +	if (bmlen > 2) {
> +		READ32(bmval[2]);
> +	}
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  	DECODE_TAIL;
>  }
>  
>  static u32 nfsd_attrmask[] = {
>  	NFSD_WRITEABLE_ATTRS_WORD0,
> -	NFSD_WRITEABLE_ATTRS_WORD1
> +	NFSD_WRITEABLE_ATTRS_WORD1,
> +	NFSD_WRITEABLE_ATTRS_WORD2
>  };
>  
>  static __be32
> @@ -275,9 +282,12 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, u32 *writable,
>  	 * According to spec, unsupported attributes return ERR_ATTRNOTSUPP;
>  	 * read-only attributes return ERR_INVAL.
>  	 */
> -	if ((bmval[0] & ~NFSD_SUPPORTED_ATTRS_WORD0) || (bmval[1] & ~NFSD_SUPPORTED_ATTRS_WORD1))
> +	if ((bmval[0] & ~nfsd_suppattrs0(argp->minorversion)) ||
> +	    (bmval[1] & ~nfsd_suppattrs1(argp->minorversion)) ||
> +	    (bmval[2] & ~nfsd_suppattrs2(argp->minorversion)))
>  		return nfserr_attrnotsupp;
> -	if ((bmval[0] & ~writable[0]) || (bmval[1] & ~writable[1]))
> +	if ((bmval[0] & ~writable[0]) || (bmval[1] & ~writable[1]) ||
> +	    (bmval[2] & ~writable[2]))
>  		return nfserr_inval;
>  
>  	READ_BUF(4);
> @@ -412,6 +422,7 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, u32 *writable,
>  			goto xdr_error;
>  		}
>  	}
> +	BUG_ON(bmval[2]);	/* no such writeable attr supported yet */
>  	if (len != expected_len)
>  		goto xdr_error;
>  
> @@ -796,6 +807,7 @@ nfsd4_decode_readdir(struct nfsd4_compoundargs *argp, struct nfsd4_readdir *read
>  	COPYMEM(readdir->rd_verf.data, sizeof(readdir->rd_verf.data));
>  	READ32(readdir->rd_dircount);    /* just in case you needed a useless field... */
>  	READ32(readdir->rd_maxcount);
> +	readdir->minorversion = argp->minorversion;
>  	if ((status = nfsd4_decode_bitmap(argp, readdir->rd_bmval)))
>  		goto out;
>  
> @@ -949,6 +961,7 @@ nfsd4_decode_verify(struct nfsd4_compoundargs *argp, struct nfsd4_verify *verify
>  	READ32(verify->ve_attrlen);
>  	READ_BUF(verify->ve_attrlen);
>  	SAVEMEM(verify->ve_attrval, verify->ve_attrlen);
> +	verify->ve_minorversion = argp->minorversion;
>  
>  	DECODE_TAIL;
>  }
> @@ -1738,10 +1751,11 @@ static __be32 fattr_handle_absent_fs(u32 *bmval0, u32 *bmval1, u32 *rdattr_err)
>  __be32
>  nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp,
>  		struct dentry *dentry, __be32 *buffer, int *countp, u32 *bmval,
> -		struct svc_rqst *rqstp, int ignore_crossmnt)
> +		struct svc_rqst *rqstp, int ignore_crossmnt, u32 minorversion)

review 11-13: no need for the extra arg - we can get to the MV from
svc_rqst->rq_resp.

>  {
>  	u32 bmval0 = bmval[0];
>  	u32 bmval1 = bmval[1];
> +	u32 bmval2 = bmval[2];
>  	struct kstat stat;
>  	struct svc_fh tempfh;
>  	struct kstatfs statfs;
> @@ -1757,10 +1771,25 @@ nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp,
>  	struct nfs4_acl *acl = NULL;
>  
>  	BUG_ON(bmval1 & NFSD_WRITEONLY_ATTRS_WORD1);
> -	BUG_ON(bmval0 & ~NFSD_SUPPORTED_ATTRS_WORD0);
> -	BUG_ON(bmval1 & ~NFSD_SUPPORTED_ATTRS_WORD1);
> +	/* FIXME: change warnings back into BUG_ON */;

review 11-13: fix this.

> +	if (bmval0 & ~nfsd_suppattrs0(minorversion)) {
> +		printk("%s: minorversion=%u bmval0=0x%08x suppattrs0=0x%08x\n",
> +			__func__, minorversion, bmval0, nfsd_suppattrs0(minorversion));
> +		WARN_ON(1);
> +	}
> +	if (bmval1 & ~nfsd_suppattrs1(minorversion)) {
> +		printk("%s: minorversion=%u bmval1=0x%08x suppattrs1=0x%08x\n",
> +			__func__, minorversion, bmval1, nfsd_suppattrs1(minorversion));
> +		WARN_ON(1);
> +	}
> +	if (bmval2 & ~nfsd_suppattrs2(minorversion)) {
> +		printk("%s: minorversion=%u bmval2=0x%08x suppattrs2=0x%08x\n",
> +			__func__, minorversion, bmval2, nfsd_suppattrs2(minorversion));
> +		WARN_ON(1);
> +	}
>  
>  	if (exp->ex_fslocs.migrated) {
> +		BUG_ON(bmval[2]);
>  		status = fattr_handle_absent_fs(&bmval0, &bmval1, &rdattr_err);
>  		if (status)
>  			goto out;
> @@ -1806,22 +1835,42 @@ nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp,
>  	if ((buflen -= 16) < 0)
>  		goto out_resource;
>  
> -	WRITE32(2);
> -	WRITE32(bmval0);
> -	WRITE32(bmval1);
> +	if (unlikely(bmval2)) {
> +		WRITE32(3);
> +		WRITE32(bmval0);
> +		WRITE32(bmval1);
> +		WRITE32(bmval2);
> +	} else if (likely(bmval1)) {
> +		WRITE32(2);
> +		WRITE32(bmval0);
> +		WRITE32(bmval1);
> +	} else {
> +		WRITE32(1);
> +		WRITE32(bmval0);
> +	}
>  	attrlenp = p++;                /* to be backfilled later */
>  
>  	if (bmval0 & FATTR4_WORD0_SUPPORTED_ATTRS) {
> -		u32 word0 = NFSD_SUPPORTED_ATTRS_WORD0;
> +		u32 word0 = nfsd_suppattrs0(minorversion);
> +		u32 word1 = nfsd_suppattrs1(minorversion);
> +		u32 word2 = nfsd_suppattrs2(minorversion);
> +
>  		if ((buflen -= 12) < 0)
>  			goto out_resource;
>  		if (!aclsupport)
>  			word0 &= ~FATTR4_WORD0_ACL;
>  		if (!exp->ex_fslocs.locations)
>  			word0 &= ~FATTR4_WORD0_FS_LOCATIONS;
> -		WRITE32(2);
> -		WRITE32(word0);
> -		WRITE32(NFSD_SUPPORTED_ATTRS_WORD1);
> +		if (!word2) {
> +			WRITE32(2);
> +			WRITE32(word0);
> +			WRITE32(word1);
> +		} else {
> +			WRITE32(3);
> +			WRITE32(word0);
> +			WRITE32(word1);
> +			WRITE32(word2);
> +		}
>  	}
>  	if (bmval0 & FATTR4_WORD0_TYPE) {
>  		if ((buflen -= 4) < 0)
> @@ -2131,6 +2180,8 @@ out_acl:
>  		}
>  		WRITE64(stat.ino);
>  	}
> +	BUG_ON(bmval2);	/* FIXME: not implemented yet */
> +
>  	*attrlenp = htonl((char *)p - (char *)attrlenp - 4);
>  	*countp = p - buffer;
>  	status = nfs_ok;
> @@ -2203,7 +2254,8 @@ nfsd4_encode_dirent_fattr(struct nfsd4_readdir *cd,
>  
>  	}
>  	nfserr = nfsd4_encode_fattr(NULL, exp, dentry, p, buflen, cd->rd_bmval,
> -					cd->rd_rqstp, ignore_crossmnt);
> +					cd->rd_rqstp, ignore_crossmnt,
> +					cd->minorversion);
>  out_put:
>  	dput(dentry);
>  	exp_put(exp);
> @@ -2369,7 +2421,7 @@ nfsd4_encode_getattr(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4
>  	buflen = resp->end - resp->p - (COMPOUND_ERR_SLACK_SPACE >> 2);
>  	nfserr = nfsd4_encode_fattr(fhp, fhp->fh_export, fhp->fh_dentry,
>  				    resp->p, &buflen, getattr->ga_bmval,
> -				    resp->rqstp, 0);
> +				    resp->rqstp, 0, resp->minorversion);
>  	if (!nfserr)
>  		resp->p += buflen;
>  	return nfserr;
> diff --git a/include/linux/nfsd/nfsd.h b/include/linux/nfsd/nfsd.h
> index ffb7897..46b8110 100644
> --- a/include/linux/nfsd/nfsd.h
> +++ b/include/linux/nfsd/nfsd.h
> @@ -321,7 +321,7 @@ extern struct timeval	nfssvc_boot;
>   *    TIME_BACKUP   (unlikely to be supported any time soon)
>   *    TIME_CREATE   (unlikely to be supported any time soon)
>   */
> -#define NFSD_SUPPORTED_ATTRS_WORD0                                                          \
> +#define NFSD4_SUPPORTED_ATTRS_WORD0                                                         \
>  (FATTR4_WORD0_SUPPORTED_ATTRS   | FATTR4_WORD0_TYPE         | FATTR4_WORD0_FH_EXPIRE_TYPE   \
>   | FATTR4_WORD0_CHANGE          | FATTR4_WORD0_SIZE         | FATTR4_WORD0_LINK_SUPPORT     \
>   | FATTR4_WORD0_SYMLINK_SUPPORT | FATTR4_WORD0_NAMED_ATTR   | FATTR4_WORD0_FSID             \
> @@ -333,7 +333,7 @@ extern struct timeval	nfssvc_boot;
>   | FATTR4_WORD0_MAXFILESIZE     | FATTR4_WORD0_MAXLINK      | FATTR4_WORD0_MAXNAME          \
>   | FATTR4_WORD0_MAXREAD         | FATTR4_WORD0_MAXWRITE     | FATTR4_WORD0_ACL)
>  
> -#define NFSD_SUPPORTED_ATTRS_WORD1                                                          \
> +#define NFSD4_SUPPORTED_ATTRS_WORD1                                                         \
>  (FATTR4_WORD1_MODE              | FATTR4_WORD1_NO_TRUNC     | FATTR4_WORD1_NUMLINKS         \
>   | FATTR4_WORD1_OWNER	        | FATTR4_WORD1_OWNER_GROUP  | FATTR4_WORD1_RAWDEV           \
>   | FATTR4_WORD1_SPACE_AVAIL     | FATTR4_WORD1_SPACE_FREE   | FATTR4_WORD1_SPACE_TOTAL      \
> @@ -341,6 +341,52 @@ extern struct timeval	nfssvc_boot;
>   | FATTR4_WORD1_TIME_DELTA   | FATTR4_WORD1_TIME_METADATA    \
>   | FATTR4_WORD1_TIME_MODIFY     | FATTR4_WORD1_TIME_MODIFY_SET | FATTR4_WORD1_MOUNTED_ON_FILEID)
>  
> +#define NFSD4_SUPPORTED_ATTRS_WORD2 0
> +
> +#if defined(CONFIG_NFSD_V4_1)

review 11-13: move the #ifdef down to the static inline implementation part.

> +#define NFSD4_1_SUPPORTED_ATTRS_WORD0 \
> +	NFSD4_SUPPORTED_ATTRS_WORD0
> +
> +#define NFSD4_1_SUPPORTED_ATTRS_WORD1 \
> +	NFSD4_SUPPORTED_ATTRS_WORD1
> +
> +#define NFSD4_1_SUPPORTED_ATTRS_WORD2 \
> +	NFSD4_SUPPORTED_ATTRS_WORD2
> +
> +static inline u32 nfsd_suppattrs0(u32 minorversion)
> +{
> +	return minorversion ? NFSD4_1_SUPPORTED_ATTRS_WORD0
> +			    : NFSD4_SUPPORTED_ATTRS_WORD0;
> +}
> +
> +static inline u32 nfsd_suppattrs1(u32 minorversion)
> +{
> +	return minorversion ? NFSD4_1_SUPPORTED_ATTRS_WORD1
> +			    : NFSD4_SUPPORTED_ATTRS_WORD1;
> +}
> +
> +static inline u32 nfsd_suppattrs2(u32 minorversion)
> +{
> +	return minorversion ? NFSD4_1_SUPPORTED_ATTRS_WORD2
> +			    : NFSD4_SUPPORTED_ATTRS_WORD2;
> +}
> +#else  /* CONFIG_NFSD_V4_1 */
> +static inline u32 nfsd_suppattrs0(u32 minorversion)
> +{
> +	return NFSD4_SUPPORTED_ATTRS_WORD0;
> +}
> +
> +static inline u32 nfsd_suppattrs1(u32 minorversion)
> +{
> +	return NFSD4_SUPPORTED_ATTRS_WORD1;
> +}
> +
> +static inline u32 nfsd_suppattrs2(u32 minorversion)
> +{
> +	return NFSD4_SUPPORTED_ATTRS_WORD2;
> +}
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  /* These will return ERR_INVAL if specified in GETATTR or READDIR. */
>  #define NFSD_WRITEONLY_ATTRS_WORD1							    \
>  (FATTR4_WORD1_TIME_ACCESS_SET   | FATTR4_WORD1_TIME_MODIFY_SET)
> @@ -351,6 +397,7 @@ extern struct timeval	nfssvc_boot;
>  #define NFSD_WRITEABLE_ATTRS_WORD1                                                          \
>  (FATTR4_WORD1_MODE              | FATTR4_WORD1_OWNER         | FATTR4_WORD1_OWNER_GROUP     \
>   | FATTR4_WORD1_TIME_ACCESS_SET | FATTR4_WORD1_TIME_MODIFY_SET)
> +#define NFSD_WRITEABLE_ATTRS_WORD2 0
>  
>  #endif /* CONFIG_NFSD_V4 */
>  
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index d78ba3c..7812e8c 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -53,6 +53,15 @@ struct nfsd4_compound_state {
>  #endif /* CONFIG_NFSD_V4_1 */
>  };
>  
> +static inline u32 nfsd4_compound_minorversion(struct nfsd4_compound_state *cs)
> +{
> +#if defined(CONFIG_NFSD_V4_1)
> +	return cs->current_ses ? 1 : 0;
> +#else  /* CONFIG_NFSD_V4_1 */
> +	return 0;
> +#endif /* CONFIG_NFSD_V4_1 */
> +}

review 11-13: need to change when nfsd4_compound_state will move into
the compound_res structure.

> +
>  struct nfsd4_change_info {
>  	u32		atomic;
>  	u32		before_ctime_sec;
> @@ -93,7 +102,7 @@ struct nfsd4_create {
>  			u32 specdata2;
>  		} dev;    /* NF4BLK, NF4CHR */
>  	} u;
> -	u32		cr_bmval[2];        /* request */
> +	u32		cr_bmval[3];        /* request */
>  	struct iattr	cr_iattr;           /* request */
>  	struct nfsd4_change_info  cr_cinfo; /* response */
>  	struct nfs4_acl *cr_acl;
> @@ -109,7 +118,7 @@ struct nfsd4_delegreturn {
>  };
>  
>  struct nfsd4_getattr {
> -	u32		ga_bmval[2];        /* request */
> +	u32		ga_bmval[3];        /* request */
>  	struct svc_fh	*ga_fhp;            /* response */
>  };
>  
> @@ -210,7 +219,7 @@ struct nfsd4_open {
>  	stateid_t       op_delegate_stateid; /* request - response */
>  	u32		op_create;     	    /* request */
>  	u32		op_createmode;      /* request */
> -	u32		op_bmval[2];        /* request */
> +	u32		op_bmval[3];        /* request */
>  	union {                             /* request */
>  		struct iattr	iattr;                      /* UNCHECKED4,GUARDED4 */
>  		nfs4_verifier	verf;                                /* EXCLUSIVE4 */
> @@ -265,7 +274,7 @@ struct nfsd4_readdir {
>  	nfs4_verifier	rd_verf;            /* request */
>  	u32		rd_dircount;        /* request */
>  	u32		rd_maxcount;        /* request */
> -	u32		rd_bmval[2];        /* request */
> +	u32		rd_bmval[3];        /* request */
>  	struct svc_rqst *rd_rqstp;          /* response */
>  	struct svc_fh * rd_fhp;             /* response */
>  
> @@ -273,6 +282,7 @@ struct nfsd4_readdir {
>  	__be32 *		buffer;
>  	int			buflen;
>  	__be32 *		offset;
> +	u32			minorversion;

review 11-13: get rid of this one

>  };
>  
>  struct nfsd4_release_lockowner {
> @@ -308,7 +318,7 @@ struct nfsd4_secinfo {
>  struct nfsd4_setattr {
>  	stateid_t	sa_stateid;         /* request */
>  	u32		sa_minorversion;    /* processing */
> -	u32		sa_bmval[2];        /* request */
> +	u32		sa_bmval[3];        /* request */
>  	struct iattr	sa_iattr;           /* request */
>  	struct nfs4_acl *sa_acl;
>  };
> @@ -334,9 +344,10 @@ struct nfsd4_setclientid_confirm {
>  
>  /* also used for NVERIFY */
>  struct nfsd4_verify {
> -	u32		ve_bmval[2];        /* request */
> +	u32		ve_bmval[3];        /* request */
>  	u32		ve_attrlen;         /* request */
>  	char *		ve_attrval;         /* request */
> +	u32		ve_minorversion;

review 11-13: get rid of this one

>  };
>  
>  struct nfsd4_write {
> @@ -479,6 +490,7 @@ struct nfsd4_compoundres {
>  	char *				tag;
>  	u32				opcnt;
>  	__be32 *			tagp; /* where to encode tag and  opcount */
> +	u32				minorversion;
>  };
>  
>  #define NFS4_SVC_XDRSIZE		sizeof(struct nfsd4_compoundargs)
> @@ -502,8 +514,9 @@ int nfs4svc_encode_compoundres(struct svc_rqst *, __be32 *,
>  void nfsd4_encode_operation(struct nfsd4_compoundres *, struct nfsd4_op *);
>  void nfsd4_encode_replay(struct nfsd4_compoundres *resp, struct nfsd4_op *op);
>  __be32 nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp,
> -		       struct dentry *dentry, __be32 *buffer, int *countp,
> -		       u32 *bmval, struct svc_rqst *, int ignore_crossmnt);
> +			  struct dentry *dentry, __be32 *buffer, int *countp,
> +			  u32 *bmval, struct svc_rqst *, int ignore_crossmnt,
> +			  u32 minorversion);
>  extern __be32 nfsd4_setclientid(struct svc_rqst *rqstp,
>  		struct nfsd4_compound_state *,
>  		struct nfsd4_setclientid *setclid);

[RFC 39/51] nfsd41: SUPPATTR_EXCLCREAT attribute

[Benny Halevy]

Return bitmask for supported EXCLUSIVE4_1 create attributes.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4xdr.c         |    9 ++++++++-
 include/linux/nfs4.h      |    2 ++
 include/linux/nfsd/nfsd.h |   16 +++++++++++++++-
 3 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 89e7b0d..227434c 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -2180,7 +2180,14 @@ out_acl:
 		}
 		WRITE64(stat.ino);
 	}
-	BUG_ON(bmval2);	/* FIXME: not implemented yet */
+#if defined(CONFIG_NFSD_V4_1)
+	if (bmval2 & FATTR4_WORD2_SUPPATTR_EXCLCREAT) {
+		WRITE32(3);
+		WRITE32(NFSD_SUPPATTR_EXCLCREAT_WORD0);
+		WRITE32(NFSD_SUPPATTR_EXCLCREAT_WORD1);
+		WRITE32(NFSD_SUPPATTR_EXCLCREAT_WORD2);
+	}
+#endif /* CONFIG_NFSD_V4_1 */
 
 	*attrlenp = htonl((char *)p - (char *)attrlenp - 4);
 	*countp = p - buffer;
diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
index be4af09..5471adf 100644
--- a/include/linux/nfs4.h
+++ b/include/linux/nfs4.h
@@ -386,6 +386,8 @@ enum lock_type4 {
 #define FATTR4_WORD0_UNIQUE_HANDLES     (1UL << 9)
 #define FATTR4_WORD0_LEASE_TIME         (1UL << 10)
 #define FATTR4_WORD0_RDATTR_ERROR       (1UL << 11)
+/* Mandatory in NFSv4.1 */
+#define FATTR4_WORD2_SUPPATTR_EXCLCREAT (1UL << 11)
 
 /* Recommended Attributes */
 #define FATTR4_WORD0_ACL                (1UL << 12)
diff --git a/include/linux/nfsd/nfsd.h b/include/linux/nfsd/nfsd.h
index 46b8110..2e763eb 100644
--- a/include/linux/nfsd/nfsd.h
+++ b/include/linux/nfsd/nfsd.h
@@ -351,7 +351,7 @@ extern struct timeval	nfssvc_boot;
 	NFSD4_SUPPORTED_ATTRS_WORD1
 
 #define NFSD4_1_SUPPORTED_ATTRS_WORD2 \
-	NFSD4_SUPPORTED_ATTRS_WORD2
+	(NFSD4_SUPPORTED_ATTRS_WORD2 | FATTR4_WORD2_SUPPATTR_EXCLCREAT)
 
 static inline u32 nfsd_suppattrs0(u32 minorversion)
 {
@@ -399,6 +399,20 @@ static inline u32 nfsd_suppattrs2(u32 minorversion)
  | FATTR4_WORD1_TIME_ACCESS_SET | FATTR4_WORD1_TIME_MODIFY_SET)
 #define NFSD_WRITEABLE_ATTRS_WORD2 0
 
+#if defined(CONFIG_NFSD_V4_1)
+#define NFSD_SUPPATTR_EXCLCREAT_WORD0 \
+	NFSD_WRITEABLE_ATTRS_WORD0
+/*
+ * we currently store the exclusive create verifier in the v_{a,m}time
+ * attributes so the client can't set these at create time using EXCLUSIVE4_1
+ */
+#define NFSD_SUPPATTR_EXCLCREAT_WORD1 \
+	(NFSD_WRITEABLE_ATTRS_WORD1 & \
+	 ~(FATTR4_WORD1_TIME_ACCESS_SET | FATTR4_WORD1_TIME_MODIFY_SET))
+#define NFSD_SUPPATTR_EXCLCREAT_WORD2 \
+	NFSD_WRITEABLE_ATTRS_WORD2
+#endif /* CONFIG_NFSD_V4_1 */
+
 #endif /* CONFIG_NFSD_V4 */
 
 #endif /* LINUX_NFSD_NFSD_H */
-- 
1.6.0.2

[RFC 40/51] nfsd41: CREATE_EXCLUSIVE4_1

[Benny Halevy]

Implement the CREATE_EXCLUSIVE4_1 open mode conforming to
http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26

This mode allows the client to atomically create a file
if it doesn't exist while setting some of its attributes.

It must be implemented if the server supports persistent
reply cache and/or pnfs.

Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c        |   15 +++++++++++++++
 fs/nfsd/nfs4xdr.c         |   21 +++++++++++++++++++++
 include/linux/nfs4.h      |    8 +++++++-
 include/linux/nfsd/xdr4.h |   10 ++++------
 4 files changed, 47 insertions(+), 7 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 5164040..9401d56 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -93,6 +93,21 @@ do_open_lookup(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nfsd4_o
 	open->op_truncate = 0;
 
 	if (open->op_create) {
+		/* FIXME: check session persistence and pnfs flags.
+		 * The nfsv4.1 spec requires the following semantics:
+		 *
+		 * Persistent   | pNFS   | Server REQUIRED | Client Allowed
+		 * Reply Cache  | server |                 |
+		 * -------------+--------+-----------------+--------------------
+		 * no           | no     | EXCLUSIVE4_1    | EXCLUSIVE4_1
+		 *              |        |                 | (SHOULD)
+		 *              |        | and EXCLUSIVE4  | or EXCLUSIVE4
+		 *              |        |                 | (SHOULD NOT)
+		 * no           | yes    | EXCLUSIVE4_1    | EXCLUSIVE4_1
+		 * yes          | no     | GUARDED4        | GUARDED4
+		 * yes          | yes    | GUARDED4        | GUARDED4
+		 */
+
 		/*
 		 * Note: create modes (UNCHECKED,GUARDED...) are the same
 		 * in NFSv4 as in v3.
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 227434c..04a9088 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -264,6 +264,14 @@ static u32 nfsd_attrmask[] = {
 	NFSD_WRITEABLE_ATTRS_WORD2
 };
 
+#if defined(CONFIG_NFSD_V4_1)
+static u32 nfsd41_ex_attrmask[] = {
+	NFSD_SUPPATTR_EXCLCREAT_WORD0,
+	NFSD_SUPPATTR_EXCLCREAT_WORD1,
+	NFSD_SUPPATTR_EXCLCREAT_WORD2
+};
+#endif /* CONFIG_NFSD_V4_1 */
+
 static __be32
 nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, u32 *writable,
 		   struct iattr *iattr, struct nfs4_acl **acl)
@@ -690,6 +698,19 @@ nfsd4_decode_open(struct nfsd4_compoundargs *argp, struct nfsd4_open *open)
 			READ_BUF(8);
 			COPYMEM(open->op_verf.data, 8);
 			break;
+#if defined(CONFIG_NFSD_V4_1)
+		case NFS4_CREATE_EXCLUSIVE4_1:
+			if (argp->minorversion < 1)
+				goto xdr_error;
+			READ_BUF(8);
+			COPYMEM(open->op_verf.data, 8);
+			status = nfsd4_decode_fattr(argp, open->op_bmval,
+				nfsd41_ex_attrmask, &open->op_iattr,
+				&open->op_acl);
+			if (status)
+				goto out;
+			break;
+#endif /* CONFIG_NFSD_V4_1 */
 		default:
 			goto xdr_error;
 		}
diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
index 5471adf..0aa4971 100644
--- a/include/linux/nfs4.h
+++ b/include/linux/nfs4.h
@@ -350,7 +350,13 @@ enum opentype4 {
 enum createmode4 {
 	NFS4_CREATE_UNCHECKED = 0,
 	NFS4_CREATE_GUARDED = 1,
-	NFS4_CREATE_EXCLUSIVE = 2
+	NFS4_CREATE_EXCLUSIVE = 2,
+	/*
+	 * New to NFSv4.1. If session is persistent,
+	 * GUARDED4 MUST be used. Otherwise, use
+	 * EXCLUSIVE4_1 instead of EXCLUSIVE4.
+	 */
+	NFS4_CREATE_EXCLUSIVE4_1 = 3
 };
 
 enum limit_by4 {
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index 7812e8c..17b0a3e 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -220,10 +220,8 @@ struct nfsd4_open {
 	u32		op_create;     	    /* request */
 	u32		op_createmode;      /* request */
 	u32		op_bmval[3];        /* request */
-	union {                             /* request */
-		struct iattr	iattr;                      /* UNCHECKED4,GUARDED4 */
-		nfs4_verifier	verf;                                /* EXCLUSIVE4 */
-	} u;
+	struct iattr	iattr;              /* UNCHECKED4, GUARDED4, EXCLUSIVE4_1 */
+	nfs4_verifier	verf;               /* EXCLUSIVE4 */
 	clientid_t	op_clientid;        /* request */
 	struct xdr_netobj op_owner;           /* request */
 	u32		op_seqid;           /* request */
@@ -238,8 +236,8 @@ struct nfsd4_open {
 	struct nfs4_stateowner *op_stateowner; /* used during processing */
 	struct nfs4_acl *op_acl;
 };
-#define op_iattr	u.iattr
-#define op_verf		u.verf
+#define op_iattr	iattr
+#define op_verf		verf
 
 struct nfsd4_open_confirm {
 	stateid_t	oc_req_stateid		/* request */;
-- 
1.6.0.2

Re: [pnfs] [RFC 40/51] nfsd41: CREATE_EXCLUSIVE4_1

[Benny Halevy]

On Nov. 10, 2008, 22:54 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> Implement the CREATE_EXCLUSIVE4_1 open mode conforming to
> http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1-26
> 
> This mode allows the client to atomically create a file
> if it doesn't exist while setting some of its attributes.
> 
> It must be implemented if the server supports persistent
> reply cache and/or pnfs.
> 
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4proc.c        |   15 +++++++++++++++
>  fs/nfsd/nfs4xdr.c         |   21 +++++++++++++++++++++
>  include/linux/nfs4.h      |    8 +++++++-
>  include/linux/nfsd/xdr4.h |   10 ++++------
>  4 files changed, 47 insertions(+), 7 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index 5164040..9401d56 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -93,6 +93,21 @@ do_open_lookup(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nfsd4_o
>  	open->op_truncate = 0;
>  
>  	if (open->op_create) {
> +		/* FIXME: check session persistence and pnfs flags.
> +		 * The nfsv4.1 spec requires the following semantics:
> +		 *
> +		 * Persistent   | pNFS   | Server REQUIRED | Client Allowed
> +		 * Reply Cache  | server |                 |
> +		 * -------------+--------+-----------------+--------------------
> +		 * no           | no     | EXCLUSIVE4_1    | EXCLUSIVE4_1
> +		 *              |        |                 | (SHOULD)
> +		 *              |        | and EXCLUSIVE4  | or EXCLUSIVE4
> +		 *              |        |                 | (SHOULD NOT)
> +		 * no           | yes    | EXCLUSIVE4_1    | EXCLUSIVE4_1
> +		 * yes          | no     | GUARDED4        | GUARDED4
> +		 * yes          | yes    | GUARDED4        | GUARDED4
> +		 */
> +
>  		/*
>  		 * Note: create modes (UNCHECKED,GUARDED...) are the same
>  		 * in NFSv4 as in v3.
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index 227434c..04a9088 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -264,6 +264,14 @@ static u32 nfsd_attrmask[] = {
>  	NFSD_WRITEABLE_ATTRS_WORD2
>  };
>  
> +#if defined(CONFIG_NFSD_V4_1)
> +static u32 nfsd41_ex_attrmask[] = {
> +	NFSD_SUPPATTR_EXCLCREAT_WORD0,
> +	NFSD_SUPPATTR_EXCLCREAT_WORD1,
> +	NFSD_SUPPATTR_EXCLCREAT_WORD2
> +};
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  static __be32
>  nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, u32 *writable,
>  		   struct iattr *iattr, struct nfs4_acl **acl)
> @@ -690,6 +698,19 @@ nfsd4_decode_open(struct nfsd4_compoundargs *argp, struct nfsd4_open *open)
>  			READ_BUF(8);
>  			COPYMEM(open->op_verf.data, 8);
>  			break;
> +#if defined(CONFIG_NFSD_V4_1)

review 11-13: get rid of #ifdef

> +		case NFS4_CREATE_EXCLUSIVE4_1:
> +			if (argp->minorversion < 1)
> +				goto xdr_error;
> +			READ_BUF(8);
> +			COPYMEM(open->op_verf.data, 8);
> +			status = nfsd4_decode_fattr(argp, open->op_bmval,
> +				nfsd41_ex_attrmask, &open->op_iattr,
> +				&open->op_acl);
> +			if (status)
> +				goto out;
> +			break;
> +#endif /* CONFIG_NFSD_V4_1 */
>  		default:
>  			goto xdr_error;
>  		}
> diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
> index 5471adf..0aa4971 100644
> --- a/include/linux/nfs4.h
> +++ b/include/linux/nfs4.h
> @@ -350,7 +350,13 @@ enum opentype4 {
>  enum createmode4 {
>  	NFS4_CREATE_UNCHECKED = 0,
>  	NFS4_CREATE_GUARDED = 1,
> -	NFS4_CREATE_EXCLUSIVE = 2
> +	NFS4_CREATE_EXCLUSIVE = 2,
> +	/*
> +	 * New to NFSv4.1. If session is persistent,
> +	 * GUARDED4 MUST be used. Otherwise, use
> +	 * EXCLUSIVE4_1 instead of EXCLUSIVE4.
> +	 */
> +	NFS4_CREATE_EXCLUSIVE4_1 = 3
>  };
>  
>  enum limit_by4 {
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index 7812e8c..17b0a3e 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -220,10 +220,8 @@ struct nfsd4_open {
>  	u32		op_create;     	    /* request */
>  	u32		op_createmode;      /* request */
>  	u32		op_bmval[3];        /* request */
> -	union {                             /* request */
> -		struct iattr	iattr;                      /* UNCHECKED4,GUARDED4 */
> -		nfs4_verifier	verf;                                /* EXCLUSIVE4 */
> -	} u;
> +	struct iattr	iattr;              /* UNCHECKED4, GUARDED4, EXCLUSIVE4_1 */
> +	nfs4_verifier	verf;               /* EXCLUSIVE4 */
>  	clientid_t	op_clientid;        /* request */
>  	struct xdr_netobj op_owner;           /* request */
>  	u32		op_seqid;           /* request */
> @@ -238,8 +236,8 @@ struct nfsd4_open {
>  	struct nfs4_stateowner *op_stateowner; /* used during processing */
>  	struct nfs4_acl *op_acl;
>  };
> -#define op_iattr	u.iattr
> -#define op_verf		u.verf
> +#define op_iattr	iattr
> +#define op_verf		verf
>  
>  struct nfsd4_open_confirm {
>  	stateid_t	oc_req_stateid		/* request */;

[RFC 41/51] sunrpc: Add deferral save and restore state callback

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

NFSv4 needs to restart deferred processing at the operation which caused the
deferral.

Add a callback and a defer_data pointer in svc_rqst to enable svc_defer to
save partial result state in the deferral request.

Add page pointer storage and resused fields to svc_deferred_req to store
the pointers to the partially processed request.

Add callbacks and a defer_data pointer in svc_deferred_request to enable
svc_deferred_recv to restore and release the partial result state.

The rq_save_state callback sets the deferral request restore_state callback and
defer_data.

The svc_deferred_req defer_data and result pages survive multiple deferral
processing caveat resetting in the save and restore callbacks.

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 include/linux/sunrpc/svc.h |   10 ++++++++++
 net/sunrpc/svc_xprt.c      |   19 +++++++++++++++----
 2 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h
index 7653dd3..d86111f 100644
--- a/include/linux/sunrpc/svc.h
+++ b/include/linux/sunrpc/svc.h
@@ -227,6 +227,9 @@ struct svc_rqst {
 	struct svc_cred		rq_cred;	/* auth info */
 	void *			rq_xprt_ctxt;	/* transport specific context ptr */
 	struct svc_deferred_req*rq_deferred;	/* deferred request we are replaying */
+	/* callback to save deferred request state */
+	void (*rq_save_state)(struct svc_rqst *, struct svc_deferred_req *);
+	void			*rq_defer_data; /* defer state data to save */
 
 	size_t			rq_xprt_hlen;	/* xprt header len */
 	struct xdr_buf		rq_arg;
@@ -335,6 +338,13 @@ struct svc_deferred_req {
 	union svc_addr_u	daddr;	/* where reply must come from */
 	struct cache_deferred_req handle;
 	size_t			xprt_hlen;
+	/* callbacks to restore and release deferred request state
+	 * set in rq_save_state */
+	void (*restore_state)(struct svc_rqst *, struct svc_deferred_req *);
+	void (*release_state)(struct svc_deferred_req *);
+	void			*defer_data; /* defer state data */
+	struct page		*respages[RPCSVC_MAXPAGES];
+	short			resused;
 	int			argslen;
 	__be32			args[0];
 };
diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
index bf5b5cd..bf12475 100644
--- a/net/sunrpc/svc_xprt.c
+++ b/net/sunrpc/svc_xprt.c
@@ -903,6 +903,8 @@ static void svc_revisit(struct cache_deferred_req *dreq, int too_many)
 	struct svc_xprt *xprt = dr->xprt;
 
 	if (too_many) {
+		if (dr->release_state)
+			dr->release_state(dr);
 		svc_xprt_put(xprt);
 		kfree(dr);
 		return;
@@ -941,10 +943,9 @@ static struct cache_deferred_req *svc_defer(struct cache_req *req)
 		size_t size;
 		/* FIXME maybe discard if size too large */
 		size = sizeof(struct svc_deferred_req) + rqstp->rq_arg.len;
-		dr = kmalloc(size, GFP_KERNEL);
+		dr = kzalloc(size, GFP_KERNEL);
 		if (dr == NULL)
-			return NULL;
-
+			goto out;
 		dr->handle.owner = rqstp->rq_server;
 		dr->prot = rqstp->rq_prot;
 		memcpy(&dr->addr, &rqstp->rq_addr, rqstp->rq_addrlen);
@@ -958,6 +959,15 @@ static struct cache_deferred_req *svc_defer(struct cache_req *req)
 		memcpy(dr->args, rqstp->rq_arg.head[0].iov_base - skip,
 		       dr->argslen << 2);
 	}
+	if (rqstp->rq_save_state)
+		rqstp->rq_save_state(rqstp, dr);
+out:
+	/* reset for each deferral */
+	rqstp->rq_defer_data = NULL;
+	rqstp->rq_save_state = NULL;
+	if (dr == NULL)
+		return NULL;
+
 	svc_xprt_get(rqstp->rq_xprt);
 	dr->xprt = rqstp->rq_xprt;
 
@@ -986,10 +996,11 @@ static int svc_deferred_recv(struct svc_rqst *rqstp)
 	rqstp->rq_xprt_hlen   = dr->xprt_hlen;
 	rqstp->rq_daddr       = dr->daddr;
 	rqstp->rq_respages    = rqstp->rq_pages;
+	if (dr->restore_state)
+		dr->restore_state(rqstp, dr);
 	return (dr->argslen<<2) - dr->xprt_hlen;
 }
 
-
 static struct svc_deferred_req *svc_deferred_dequeue(struct svc_xprt *xprt)
 {
 	struct svc_deferred_req *dr = NULL;
-- 
1.6.0.2

[RFC 42/51] nfsd: save and restore defer result pages

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

Implement the rq_save_state, resume_state, and release_state RPC deferral
callbacks.

Save the reply pages in struct svc_deferred_req.

Clear the svc_deferred_req respages in the save_state callback to setup
for another NFSD operation deferral.

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c        |   53 +++++++++++++++++++++++++++++++++++
 fs/nfsd/nfs4state.c       |   67 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/nfsd/xdr4.h |    5 +++
 3 files changed, 125 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 9401d56..72fd645 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -883,6 +883,59 @@ static struct nfsd4_compound_state *cstate_alloc(void)
 	return cstate;
 }
 
+/*
+ * RPC restore deferral state callback
+ */
+static void
+nfsd4_release_deferred_state(struct svc_deferred_req *dreq)
+{
+	dprintk("--> %s dreq %p\n", __func__, dreq);
+
+	nfsd4_clear_respages(dreq->respages, dreq->resused);
+	cstate_free(dreq->defer_data); /* kfree the cstate */
+}
+
+/*
+ * RPC restore deferral state callback
+ */
+static void
+nfsd4_restore_deferred_state(struct svc_rqst *rqstp,
+			     struct svc_deferred_req *dreq)
+{
+	dprintk("--> %s rqstp %p dreq %p\n", __func__, rqstp, dreq);
+
+	nfsd4_restore_rqst_pages(rqstp, dreq->respages, dreq->resused);
+
+	/* Reset defer_data for a NFSD deferral revisit interrupted
+	 * by a non-NFSD deferral */
+	rqstp->rq_defer_data = dreq->defer_data;
+}
+
+/*
+ * RPC save deferral state callback
+ */
+static void
+nfsd4_save_deferred_state(struct svc_rqst *rqstp,
+			  struct svc_deferred_req *dreq)
+ {
+	struct nfsd4_compound_state *cstate =
+		(struct nfsd4_compound_state *)rqstp->rq_defer_data;
+
+	dprintk("--> %s dreq %p cstate %p\n", __func__, dreq, cstate);
+
+	fh_put(&cstate->current_fh);
+	fh_put(&cstate->save_fh);
+
+	/* In case of a prior NFSD deferral processing this compound RPC */
+	nfsd4_clear_respages(dreq->respages, dreq->resused);
+
+	nfsd4_cache_rqst_pages(rqstp, dreq->respages, &dreq->resused);
+
+	dreq->defer_data = rqstp->rq_defer_data;
+	dreq->restore_state = nfsd4_restore_deferred_state;
+	dreq->release_state = nfsd4_release_deferred_state;
+}
+
 typedef __be32(*nfsd4op_func)(struct svc_rqst *, struct nfsd4_compound_state *,
 			      void *);
 
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 9377ad0..ae6de38 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -875,6 +875,73 @@ out_err:
 	return;
 }
 
+void
+nfsd4_move_pages(struct page **topages, struct page **frompages, short count)
+{
+	int page_no;
+
+	for (page_no = 0; page_no < count; page_no++) {
+		topages[page_no] = frompages[page_no];
+		if (!topages[page_no])
+			continue;
+		get_page(topages[page_no]);
+	}
+}
+
+/*
+ * Copy the result rq_respages pointers from the request
+ * to respages.
+ *
+ * Hold a reference to the result pages.
+ */
+void
+nfsd4_cache_rqst_pages(struct svc_rqst *rqstp, struct page **respages,
+		       short *resused)
+{
+	dprintk("--> %s rqstp %p\n", __func__, rqstp);
+
+	*resused = rqstp->rq_resused;
+	nfsd4_move_pages(respages, rqstp->rq_respages, rqstp->rq_resused);
+}
+
+/*
+ * Copy the page pointers from respages to the request.
+ *
+ * Deallocate the request result pages which will be replaced from respages.
+ *
+ * Hold a reference to the result pages.
+ */
+void
+nfsd4_restore_rqst_pages(struct svc_rqst *rqstp, struct page **respages,
+			 short resused)
+{
+	dprintk("--> %s rqstp %p\n", __func__, rqstp);
+
+	/* release allocated result pages to be replaced from the cache */
+	svc_free_res_pages(rqstp);
+
+	rqstp->rq_resused = resused;
+	nfsd4_move_pages(rqstp->rq_respages, respages, resused);
+}
+
+/*
+ * Dereference the result pages.
+ */
+void
+nfsd4_clear_respages(struct page **respages, short resused)
+{
+	int page_no;
+
+	dprintk("--> %s\n", __func__);
+
+	for (page_no = 0; page_no < resused; page_no++) {
+		if (!respages[page_no])
+			continue;
+		put_page(respages[page_no]);
+		respages[page_no] = NULL;
+	}
+}
+
 #if defined(CONFIG_NFSD_V4_1)
 /*
  * Set the exchange_id flags returned by the server.
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index 17b0a3e..01fb139 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -515,6 +515,11 @@ __be32 nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp,
 			  struct dentry *dentry, __be32 *buffer, int *countp,
 			  u32 *bmval, struct svc_rqst *, int ignore_crossmnt,
 			  u32 minorversion);
+extern void nfsd4_clear_respages(struct page **respages, short resused);
+extern void nfsd4_cache_rqst_pages(struct svc_rqst *rqstp,
+		struct page **respages, short *resused);
+extern void nfsd4_restore_rqst_pages(struct svc_rqst *rqstp,
+		struct page **respages, short resused);
 extern __be32 nfsd4_setclientid(struct svc_rqst *rqstp,
 		struct nfsd4_compound_state *,
 		struct nfsd4_setclientid *setclid);
-- 
1.6.0.2

[RFC 43/51] nfsd: deferral processing

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

Save the struct nfsd4_compound_state and set the save_state callback for
each request for potential deferral handling.

If an NFSv4 operation causes a deferral, the save_state callback is called
by svc_defer which saves the defer_data with the deferral, and sets the
restore_state deferral callback.

fh_put is called so that the deferral is not referencing the file handles,
allowing umount of the file system.

Use a slab cache for nfsd4_compound_state allocation.

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c        |   63 ++++++++++++++++++++++++--------------------
 fs/nfsd/nfs4state.c       |   40 ++++++++++++++++++++++++++++
 include/linux/nfsd/xdr4.h |    5 +++
 3 files changed, 79 insertions(+), 29 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 72fd645..deefba1 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -860,29 +860,6 @@ static inline void nfsd4_increment_op_stats(u32 opnum)
 		nfsdstats.nfs4_opcount[opnum]++;
 }
 
-static void cstate_free(struct nfsd4_compound_state *cstate)
-{
-	if (cstate == NULL)
-		return;
-	fh_put(&cstate->current_fh);
-	fh_put(&cstate->save_fh);
-	BUG_ON(cstate->replay_owner);
-	kfree(cstate);
-}
-
-static struct nfsd4_compound_state *cstate_alloc(void)
-{
-	struct nfsd4_compound_state *cstate;
-
-	cstate = kmalloc(sizeof(struct nfsd4_compound_state), GFP_KERNEL);
-	if (cstate == NULL)
-		return NULL;
-	fh_init(&cstate->current_fh, NFS4_FHSIZE);
-	fh_init(&cstate->save_fh, NFS4_FHSIZE);
-	cstate->replay_owner = NULL;
-	return cstate;
-}
-
 /*
  * RPC restore deferral state callback
  */
@@ -891,8 +868,7 @@ nfsd4_release_deferred_state(struct svc_deferred_req *dreq)
 {
 	dprintk("--> %s dreq %p\n", __func__, dreq);
 
-	nfsd4_clear_respages(dreq->respages, dreq->resused);
-	cstate_free(dreq->defer_data); /* kfree the cstate */
+	nfsd4_cstate_free(dreq->defer_data, dreq); /* kfree the cstate */
 }
 
 /*
@@ -961,11 +937,11 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
 		    struct nfsd4_compoundargs *args,
 		    struct nfsd4_compoundres *resp)
 {
-	struct nfsd4_op	*op = NULL;
+	struct nfsd4_op	*op;
 	struct nfsd4_operation *opdesc;
 	struct nfsd4_compound_state *cstate = NULL;
 	int		slack_bytes;
-	__be32		status;
+	__be32		status = nfs_ok;
 #if defined(CONFIG_NFSD_V4_1)
 	struct current_session *current_ses = NULL;
 #endif /* CONFIG_NFSD_V4_1 */
@@ -990,10 +966,25 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
 		goto out;
 
 	status = nfserr_resource;
-	cstate = cstate_alloc();
+	cstate = nfsd4_cstate_alloc(rqstp);
 	if (cstate == NULL)
 		goto out;
 
+	if (rqstp->rq_deferred && rqstp->rq_deferred->defer_data) {
+		dprintk("%s Resuming deferred processing\n", __func__);
+		resp->opcnt = cstate->last_op_cnt;
+		resp->p = cstate->last_op_p;
+
+		/* verify the current and save file handles that were
+		 * fh_put at deferral. */
+		fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_NOP);
+		fh_verify(rqstp, &cstate->save_fh, 0, NFSD_MAY_NOP);
+	}
+
+	/* set for potential deferral */
+	rqstp->rq_defer_data = cstate;
+	rqstp->rq_save_state = nfsd4_save_deferred_state;
+
 #if defined(CONFIG_NFSD_V4_1)
 	if (args->minorversion == 1) {
 		/* FIXME: use kmem_cache */
@@ -1007,6 +998,7 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
 
 	status = nfs_ok;
 	while (!status && resp->opcnt < args->opcnt) {
+		cstate->last_op_p = resp->p;
 		op = &args->ops[resp->opcnt++];
 
 		dprintk("nfsv4 compound op #%d/%d: %d (%s)\n",
@@ -1076,6 +1068,19 @@ encode_op:
 
 		nfsd4_increment_op_stats(op->opnum);
 	}
+	/* Done with any potential deferral */
+	rqstp->rq_defer_data = cstate;
+	rqstp->rq_save_state = NULL;
+
+	if (status == nfserr_dropit) {
+		dprintk("%s Drop it!\n", __func__);
+
+		/* svc_defer() has called the rq_save_state() callback
+		 * to cache the partially processed compound. */
+		cstate->last_op_cnt = resp->opcnt - 1;
+		return status;
+	}
+
 out_free:
 #if defined(CONFIG_NFSD_V4_1)
 	if (current_ses) {
@@ -1092,7 +1097,7 @@ out_free:
 		kfree(current_ses);
 	}
 #endif /* CONFIG_NFSD_V4_1 */
-	cstate_free(cstate);
+	nfsd4_cstate_free(cstate, rqstp->rq_deferred);
 out:
 	nfsd4_release_compoundargs(args);
 	dprintk("nfsv4 compound returned %d\n", ntohl(status));
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index ae6de38..8fd7704 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -101,6 +101,7 @@ static struct kmem_cache *stateowner_slab = NULL;
 static struct kmem_cache *file_slab = NULL;
 static struct kmem_cache *stateid_slab = NULL;
 static struct kmem_cache *deleg_slab = NULL;
+static struct kmem_cache *cstate_slab;
 
 #define BUG_ON_UNLOCKED_STATE() BUG_ON(mutex_trylock(&client_mutex))
 
@@ -480,6 +481,40 @@ free_session(struct kref *kref)
 }
 #endif /* CONFIG_NFSD_V4_1 */
 
+void
+nfsd4_cstate_free(struct nfsd4_compound_state *cstate,
+		  struct svc_deferred_req *dreq)
+{
+	if (dreq && dreq->release_state)
+		nfsd4_clear_respages(dreq->respages, dreq->resused);
+	if (cstate == NULL)
+		return;
+	fh_put(&cstate->current_fh);
+	fh_put(&cstate->save_fh);
+	BUG_ON(cstate->replay_owner);
+	kmem_cache_free(cstate_slab, cstate);
+}
+
+struct nfsd4_compound_state *
+nfsd4_cstate_alloc(struct svc_rqst *rqstp)
+{
+	struct nfsd4_compound_state *cstate;
+
+	if (rqstp->rq_deferred && rqstp->rq_deferred->defer_data) {
+		dprintk("%s Revisit deferred request\n", __func__);
+		cstate = rqstp->rq_deferred->defer_data;
+		goto out;
+	}
+	cstate = kmem_cache_alloc(cstate_slab, GFP_KERNEL);
+	if (cstate == NULL)
+		return NULL;
+	fh_init(&cstate->current_fh, NFS4_FHSIZE);
+	fh_init(&cstate->save_fh, NFS4_FHSIZE);
+	cstate->replay_owner = NULL;
+out:
+	return cstate;
+}
+
 static inline void
 renew_client(struct nfs4_client *clp)
 {
@@ -1529,6 +1564,7 @@ nfsd4_free_slabs(void)
 	nfsd4_free_slab(&file_slab);
 	nfsd4_free_slab(&stateid_slab);
 	nfsd4_free_slab(&deleg_slab);
+	nfsd4_free_slab(&cstate_slab);
 }
 
 static int
@@ -1550,6 +1586,10 @@ nfsd4_init_slabs(void)
 			sizeof(struct nfs4_delegation), 0, 0, NULL);
 	if (deleg_slab == NULL)
 		goto out_nomem;
+	cstate_slab = kmem_cache_create("nfsd4_compound_states",
+			sizeof(struct nfsd4_compound_state), 0, 0, NULL);
+	if (cstate_slab == NULL)
+		goto out_nomem;
 	return 0;
 out_nomem:
 	nfsd4_free_slabs();
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index 01fb139..a0b4f00 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -48,6 +48,8 @@ struct nfsd4_compound_state {
 	struct svc_fh current_fh;
 	struct svc_fh save_fh;
 	struct nfs4_stateowner *replay_owner;
+	__be32 *last_op_p;
+	u32 last_op_cnt;
 #if defined(CONFIG_NFSD_V4_1)
 	struct current_session *current_ses;
 #endif /* CONFIG_NFSD_V4_1 */
@@ -520,6 +522,9 @@ extern void nfsd4_cache_rqst_pages(struct svc_rqst *rqstp,
 		struct page **respages, short *resused);
 extern void nfsd4_restore_rqst_pages(struct svc_rqst *rqstp,
 		struct page **respages, short resused);
+extern struct nfsd4_compound_state *nfsd4_cstate_alloc(struct svc_rqst *rqstp);
+extern void nfsd4_cstate_free(struct nfsd4_compound_state *cstate,
+		struct svc_deferred_req *dreq);
 extern __be32 nfsd4_setclientid(struct svc_rqst *rqstp,
 		struct nfsd4_compound_state *,
 		struct nfsd4_setclientid *setclid);
-- 
1.6.0.2

[RFC 44/51] nfsd41: slab cache for current session

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

Use a separate slab cache for the current session so that an NFSv4.1 server
responding to NFSv4.0 requests uses only the compound state slab.

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c        |   36 +++++++++++++++++-------------------
 fs/nfsd/nfs4state.c       |   28 ++++++++++++++++++++++++++++
 include/linux/nfsd/xdr4.h |    2 ++
 3 files changed, 47 insertions(+), 19 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index deefba1..eaebe5b 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -942,9 +942,6 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
 	struct nfsd4_compound_state *cstate = NULL;
 	int		slack_bytes;
 	__be32		status = nfs_ok;
-#if defined(CONFIG_NFSD_V4_1)
-	struct current_session *current_ses = NULL;
-#endif /* CONFIG_NFSD_V4_1 */
 
 	resp->xbuf = &rqstp->rq_res;
 	resp->p = rqstp->rq_res.head[0].iov_base + rqstp->rq_res.head[0].iov_len;
@@ -970,6 +967,19 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
 	if (cstate == NULL)
 		goto out;
 
+#ifdef CONFIG_NFSD_V4_1
+	/* cstate->current_ses could be set revisiting a deferal */
+	if (args->minorversion == 1 && !cstate->current_ses) {
+		cstate->current_ses = nfsd41_current_session_alloc();
+		if (cstate->current_ses == NULL) {
+			/* still nfserr_resource */
+			nfsd4_cstate_free(cstate, rqstp->rq_deferred);
+			goto out;
+		}
+		cstate->current_ses->cs_slot = NULL;
+	}
+#endif /* CONFIG_NFSD_V4_1 */
+
 	if (rqstp->rq_deferred && rqstp->rq_deferred->defer_data) {
 		dprintk("%s Resuming deferred processing\n", __func__);
 		resp->opcnt = cstate->last_op_cnt;
@@ -985,17 +995,6 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
 	rqstp->rq_defer_data = cstate;
 	rqstp->rq_save_state = nfsd4_save_deferred_state;
 
-#if defined(CONFIG_NFSD_V4_1)
-	if (args->minorversion == 1) {
-		/* FIXME: use kmem_cache */
-		current_ses = kzalloc(sizeof(*current_ses), GFP_KERNEL);
-		if (current_ses == NULL)
-			goto out_free;
-	}
-	/* DM: current_ses must be NULL for minorversion 0 */
-	cstate->current_ses = current_ses;
-#endif /* CONFIG_NFSD_V4_1 */
-
 	status = nfs_ok;
 	while (!status && resp->opcnt < args->opcnt) {
 		cstate->last_op_p = resp->p;
@@ -1081,12 +1080,11 @@ encode_op:
 		return status;
 	}
 
-out_free:
 #if defined(CONFIG_NFSD_V4_1)
-	if (current_ses) {
-		struct nfs41_slot *cs_slot = current_ses->cs_slot;
+	if (cstate->current_ses) {
+		struct nfs41_slot *cs_slot = cstate->current_ses->cs_slot;
 		if (cs_slot) {
-			if (op && op->status != nfserr_dropit) {
+			if (status != nfserr_dropit) {
 				dprintk("%s: SET SLOT STATE TO AVAILABLE\n",
 					__func__);
 				nfs41_set_slot_state(cs_slot,
@@ -1094,7 +1092,7 @@ out_free:
 			}
 			nfs41_put_session(cs_slot->sl_session);
 		}
-		kfree(current_ses);
+		nfsd41_current_session_free(cstate->current_ses);
 	}
 #endif /* CONFIG_NFSD_V4_1 */
 	nfsd4_cstate_free(cstate, rqstp->rq_deferred);
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 8fd7704..ba6ab19 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -102,6 +102,9 @@ static struct kmem_cache *file_slab = NULL;
 static struct kmem_cache *stateid_slab = NULL;
 static struct kmem_cache *deleg_slab = NULL;
 static struct kmem_cache *cstate_slab;
+#ifdef CONFIG_NFSD_V4_1
+static struct kmem_cache *current_session_slab;
+#endif /* CONFIG_NFSD_V4_1 */
 
 #define BUG_ON_UNLOCKED_STATE() BUG_ON(mutex_trylock(&client_mutex))
 
@@ -479,6 +482,19 @@ free_session(struct kref *kref)
 	kfree(ses->se_slots);
 	kfree(ses);
 }
+
+void
+nfsd41_current_session_free(struct current_session *cses)
+{
+	kmem_cache_free(current_session_slab, cses);
+}
+
+struct current_session *
+nfsd41_current_session_alloc(void)
+{
+	return kmem_cache_alloc(current_session_slab, GFP_KERNEL);
+}
+
 #endif /* CONFIG_NFSD_V4_1 */
 
 void
@@ -511,6 +527,9 @@ nfsd4_cstate_alloc(struct svc_rqst *rqstp)
 	fh_init(&cstate->current_fh, NFS4_FHSIZE);
 	fh_init(&cstate->save_fh, NFS4_FHSIZE);
 	cstate->replay_owner = NULL;
+#ifdef CONFIG_NFSD_V4_1
+	cstate->current_ses = NULL;
+#endif /* CONFIG_NFSD_V4_1 */
 out:
 	return cstate;
 }
@@ -1565,6 +1584,9 @@ nfsd4_free_slabs(void)
 	nfsd4_free_slab(&stateid_slab);
 	nfsd4_free_slab(&deleg_slab);
 	nfsd4_free_slab(&cstate_slab);
+#ifdef CONFIG_NFSD_V4_1
+	nfsd4_free_slab(&current_session_slab);
+#endif /* CONFIG_NFSD_V4_1 */
 }
 
 static int
@@ -1590,6 +1612,12 @@ nfsd4_init_slabs(void)
 			sizeof(struct nfsd4_compound_state), 0, 0, NULL);
 	if (cstate_slab == NULL)
 		goto out_nomem;
+#ifdef CONFIG_NFSD_V4_1
+	current_session_slab = kmem_cache_create("nfsd4_current_sessions",
+			sizeof(struct current_session), 0, 0, NULL);
+	if (current_session_slab == NULL)
+		goto out_nomem;
+#endif /* CONFIG_NFSD_V4_1 */
 	return 0;
 out_nomem:
 	nfsd4_free_slabs();
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index a0b4f00..fc42fba 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -532,6 +532,8 @@ extern __be32 nfsd4_setclientid_confirm(struct svc_rqst *rqstp,
 		struct nfsd4_compound_state *,
 		struct nfsd4_setclientid_confirm *setclientid_confirm);
 #if defined(CONFIG_NFSD_V4_1)
+extern void nfsd41_current_session_free(struct current_session *cses);
+extern struct current_session *nfsd41_current_session_alloc(void);
 extern __be32 nfsd4_exchange_id(struct svc_rqst *rqstp,
 		struct nfsd4_compound_state *,
 		struct nfsd4_exchange_id *);
-- 
1.6.0.2

Re: [pnfs] [RFC 44/51] nfsd41: slab cache for current session

[Benny Halevy]

On Nov. 10, 2008, 22:55 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> Use a separate slab cache for the current session so that an NFSv4.1 server
> responding to NFSv4.0 requests uses only the compound state slab.
> 
> Signed-off-by: Andy Adamson<andros@netapp.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 11-12: just embed struct current_session in
nfsd4_compound_state and nfsd4_coumpoundres

> ---
>  fs/nfsd/nfs4proc.c        |   36 +++++++++++++++++-------------------
>  fs/nfsd/nfs4state.c       |   28 ++++++++++++++++++++++++++++
>  include/linux/nfsd/xdr4.h |    2 ++
>  3 files changed, 47 insertions(+), 19 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index deefba1..eaebe5b 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -942,9 +942,6 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
>  	struct nfsd4_compound_state *cstate = NULL;
>  	int		slack_bytes;
>  	__be32		status = nfs_ok;
> -#if defined(CONFIG_NFSD_V4_1)
> -	struct current_session *current_ses = NULL;
> -#endif /* CONFIG_NFSD_V4_1 */
>  
>  	resp->xbuf = &rqstp->rq_res;
>  	resp->p = rqstp->rq_res.head[0].iov_base + rqstp->rq_res.head[0].iov_len;
> @@ -970,6 +967,19 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
>  	if (cstate == NULL)
>  		goto out;
>  
> +#ifdef CONFIG_NFSD_V4_1
> +	/* cstate->current_ses could be set revisiting a deferal */
> +	if (args->minorversion == 1 && !cstate->current_ses) {
> +		cstate->current_ses = nfsd41_current_session_alloc();
> +		if (cstate->current_ses == NULL) {
> +			/* still nfserr_resource */
> +			nfsd4_cstate_free(cstate, rqstp->rq_deferred);
> +			goto out;
> +		}
> +		cstate->current_ses->cs_slot = NULL;
> +	}
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  	if (rqstp->rq_deferred && rqstp->rq_deferred->defer_data) {
>  		dprintk("%s Resuming deferred processing\n", __func__);
>  		resp->opcnt = cstate->last_op_cnt;
> @@ -985,17 +995,6 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
>  	rqstp->rq_defer_data = cstate;
>  	rqstp->rq_save_state = nfsd4_save_deferred_state;
>  
> -#if defined(CONFIG_NFSD_V4_1)
> -	if (args->minorversion == 1) {
> -		/* FIXME: use kmem_cache */
> -		current_ses = kzalloc(sizeof(*current_ses), GFP_KERNEL);
> -		if (current_ses == NULL)
> -			goto out_free;
> -	}
> -	/* DM: current_ses must be NULL for minorversion 0 */
> -	cstate->current_ses = current_ses;
> -#endif /* CONFIG_NFSD_V4_1 */
> -
>  	status = nfs_ok;
>  	while (!status && resp->opcnt < args->opcnt) {
>  		cstate->last_op_p = resp->p;
> @@ -1081,12 +1080,11 @@ encode_op:
>  		return status;
>  	}
>  
> -out_free:
>  #if defined(CONFIG_NFSD_V4_1)
> -	if (current_ses) {
> -		struct nfs41_slot *cs_slot = current_ses->cs_slot;
> +	if (cstate->current_ses) {
> +		struct nfs41_slot *cs_slot = cstate->current_ses->cs_slot;
>  		if (cs_slot) {
> -			if (op && op->status != nfserr_dropit) {
> +			if (status != nfserr_dropit) {
>  				dprintk("%s: SET SLOT STATE TO AVAILABLE\n",
>  					__func__);
>  				nfs41_set_slot_state(cs_slot,
> @@ -1094,7 +1092,7 @@ out_free:
>  			}
>  			nfs41_put_session(cs_slot->sl_session);
>  		}
> -		kfree(current_ses);
> +		nfsd41_current_session_free(cstate->current_ses);
>  	}
>  #endif /* CONFIG_NFSD_V4_1 */
>  	nfsd4_cstate_free(cstate, rqstp->rq_deferred);
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 8fd7704..ba6ab19 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -102,6 +102,9 @@ static struct kmem_cache *file_slab = NULL;
>  static struct kmem_cache *stateid_slab = NULL;
>  static struct kmem_cache *deleg_slab = NULL;
>  static struct kmem_cache *cstate_slab;
> +#ifdef CONFIG_NFSD_V4_1
> +static struct kmem_cache *current_session_slab;
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  #define BUG_ON_UNLOCKED_STATE() BUG_ON(mutex_trylock(&client_mutex))
>  
> @@ -479,6 +482,19 @@ free_session(struct kref *kref)
>  	kfree(ses->se_slots);
>  	kfree(ses);
>  }
> +
> +void
> +nfsd41_current_session_free(struct current_session *cses)
> +{
> +	kmem_cache_free(current_session_slab, cses);
> +}
> +
> +struct current_session *
> +nfsd41_current_session_alloc(void)
> +{
> +	return kmem_cache_alloc(current_session_slab, GFP_KERNEL);
> +}
> +
>  #endif /* CONFIG_NFSD_V4_1 */
>  
>  void
> @@ -511,6 +527,9 @@ nfsd4_cstate_alloc(struct svc_rqst *rqstp)
>  	fh_init(&cstate->current_fh, NFS4_FHSIZE);
>  	fh_init(&cstate->save_fh, NFS4_FHSIZE);
>  	cstate->replay_owner = NULL;
> +#ifdef CONFIG_NFSD_V4_1
> +	cstate->current_ses = NULL;
> +#endif /* CONFIG_NFSD_V4_1 */
>  out:
>  	return cstate;
>  }
> @@ -1565,6 +1584,9 @@ nfsd4_free_slabs(void)
>  	nfsd4_free_slab(&stateid_slab);
>  	nfsd4_free_slab(&deleg_slab);
>  	nfsd4_free_slab(&cstate_slab);
> +#ifdef CONFIG_NFSD_V4_1
> +	nfsd4_free_slab(&current_session_slab);
> +#endif /* CONFIG_NFSD_V4_1 */
>  }
>  
>  static int
> @@ -1590,6 +1612,12 @@ nfsd4_init_slabs(void)
>  			sizeof(struct nfsd4_compound_state), 0, 0, NULL);
>  	if (cstate_slab == NULL)
>  		goto out_nomem;
> +#ifdef CONFIG_NFSD_V4_1
> +	current_session_slab = kmem_cache_create("nfsd4_current_sessions",
> +			sizeof(struct current_session), 0, 0, NULL);
> +	if (current_session_slab == NULL)
> +		goto out_nomem;
> +#endif /* CONFIG_NFSD_V4_1 */
>  	return 0;
>  out_nomem:
>  	nfsd4_free_slabs();
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index a0b4f00..fc42fba 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -532,6 +532,8 @@ extern __be32 nfsd4_setclientid_confirm(struct svc_rqst *rqstp,
>  		struct nfsd4_compound_state *,
>  		struct nfsd4_setclientid_confirm *setclientid_confirm);
>  #if defined(CONFIG_NFSD_V4_1)
> +extern void nfsd41_current_session_free(struct current_session *cses);
> +extern struct current_session *nfsd41_current_session_alloc(void);
>  extern __be32 nfsd4_exchange_id(struct svc_rqst *rqstp,
>  		struct nfsd4_compound_state *,
>  		struct nfsd4_exchange_id *);

[RFC 45/51] nfsd41: DRC save, restore, and clear functions

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

Cache all the result pages, including the rpc header in rq_respages[0],
for a request in the slot table cache entry.

Cache the statp pointer from nfsd_dispatch which points into rq_respages[0]
just past the rpc header. When setting a cache entry, calculate and save the
length of the nfs data minus the rpc header for rq_respages[0].

When replaying a cache entry, replace the cached rpc header with the
replayed request rpc result header, unless there is not enough room in the
cached results first page. In that case, use the cached rpc header.

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c        |  106 ++++++++++++++++++++++++++++++++++++++++++++
 fs/nfsd/nfssvc.c           |    6 +++
 include/linux/nfsd/cache.h |    3 +
 include/linux/nfsd/state.h |   17 +++++--
 include/linux/nfsd/xdr4.h  |    7 +++
 5 files changed, 134 insertions(+), 5 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index ba6ab19..230ce1d 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -412,6 +412,7 @@ alloc_init_session(struct nfs4_client *clp, struct nfsd4_create_session *cses)
 	for (i = 0; i < new->se_fnumslots; i++) {
 		new->se_slots[i].sl_session = new;
 		nfs41_set_slot_state(&new->se_slots[i], NFS4_SLOT_AVAILABLE);
+		spin_lock_init(&new->se_slots[i].sl_cache_entry.ce_lock);
 	}
 
 	new->se_client = clp;
@@ -997,6 +998,111 @@ nfsd4_clear_respages(struct page **respages, short resused)
 }
 
 #if defined(CONFIG_NFSD_V4_1)
+
+void
+nfsd41_set_statp(struct svc_rqst *rqstp, __be32 *statp)
+{
+	struct nfsd4_compoundres *resp = rqstp->rq_resp;
+
+	resp->statp = statp;
+}
+
+/*
+ * Cache the reply pages, clearing the previous pages.
+ * Store the base and length of the rq_req.head[0] page
+ * of the NFSv4.1 data, just past the rpc header.
+ */
+void
+nfsd41_set_cache_entry(struct nfsd4_compoundres *resp)
+{
+	struct nfs41_cache_entry *entry =
+		&resp->current_ses->cs_slot->sl_cache_entry;
+	struct svc_rqst *rqstp = resp->rqstp;
+	struct kvec *resv = &rqstp->rq_res.head[0];
+
+	dprintk("--> %s entry %p\n", __func__, entry);
+
+	/* Don't cache a failed OP_SEQUENCE */
+	if (resp->opcnt == 1 && resp->status)
+		return;
+	spin_lock(&entry->ce_lock);
+	nfsd4_clear_respages(entry->ce_respages, entry->ce_resused);
+	nfsd4_cache_rqst_pages(rqstp, entry->ce_respages, &entry->ce_resused);
+	entry->ce_status = resp->status;
+	entry->ce_datav.iov_base = resp->statp;
+	entry->ce_datav.iov_len = resv->iov_len - ((char *)resp->statp -
+				  (char *)page_address(rqstp->rq_respages[0]));
+	spin_unlock(&entry->ce_lock);
+}
+
+/*
+ * Copy the cached NFSv4.1 reply skipping the cached rpc header into the
+ * replay result res.head[0] past the rpc header to end up with replay
+ * rpc header and cached NFSv4.1 reply.
+ */
+static int
+nfsd41_copy_replay_data(struct nfsd4_compoundres *resp,
+			struct nfs41_cache_entry *entry)
+{
+	struct svc_rqst *rqstp = resp->rqstp;
+	struct kvec *resv = &resp->rqstp->rq_res.head[0];
+	int len;
+
+	/* Current request rpc header length*/
+	len = (char *)resp->statp - (char *)page_address(rqstp->rq_respages[0]);
+	if (entry->ce_datav.iov_len + len > PAGE_SIZE) {
+		dprintk("%s v41 cached reply too large (%Zd).\n", __func__,
+			entry->ce_datav.iov_len);
+		return 0;
+	}
+	/* copy the cached reply nfsd data past the current rpc header */
+	memcpy((char *)resv->iov_base + len, entry->ce_datav.iov_base,
+		entry->ce_datav.iov_len);
+	resv->iov_len = len + entry->ce_datav.iov_len;
+	return 1;
+}
+
+/*
+ * Keep the first page of the replay. Copy the NFSv4.1 data from the first
+ * cached page.  Replace any futher replay pages from the cache.
+ */
+__be32
+nfsd41_replay_cache_entry(struct nfsd4_compoundres *resp)
+{
+	struct nfs41_cache_entry *entry =
+		&resp->current_ses->cs_slot->sl_cache_entry;
+	__be32 status;
+
+	dprintk("--> %s entry %p\n", __func__, entry);
+
+	spin_lock(&entry->ce_lock);
+
+	if (!nfsd41_copy_replay_data(resp, entry)) {
+		/*
+		 * Not enough room to use the replay rpc header, send the
+		 * cached header. Release all the allocated result pages.
+		 */
+		svc_free_res_pages(resp->rqstp);
+		nfsd4_move_pages(resp->rqstp->rq_respages, entry->ce_respages,
+			entry->ce_resused);
+	} else {
+		/* Release all but the first allocatek result page */
+
+		resp->rqstp->rq_resused--;
+		svc_free_res_pages(resp->rqstp);
+
+		nfsd4_move_pages(&resp->rqstp->rq_respages[1],
+				 &entry->ce_respages[1],
+				 entry->ce_resused - 1);
+	}
+
+	resp->rqstp->rq_resused = entry->ce_resused;
+	status = entry->ce_status;
+	spin_unlock(&entry->ce_lock);
+
+	return status;
+}
+
 /*
  * Set the exchange_id flags returned by the server.
  */
diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 07e4f5d..a9f90b6 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -543,6 +543,12 @@ nfsd_dispatch(struct svc_rqst *rqstp, __be32 *statp)
 		+ rqstp->rq_res.head[0].iov_len;
 	rqstp->rq_res.head[0].iov_len += sizeof(__be32);
 
+#ifdef CONFIG_NFSD_V4_1
+	/* NFSv4.1 DRC requires statp */
+	if (rqstp->rq_vers == 4)
+		nfsd41_set_statp(rqstp, statp);
+#endif /* CONFIG_NFSD_V4_1 */
+
 	/* Now call the procedure handler, and encode NFS status. */
 	nfserr = proc->pc_func(rqstp, rqstp->rq_argp, rqstp->rq_resp);
 	nfserr = map_new_errors(rqstp->rq_vers, nfserr);
diff --git a/include/linux/nfsd/cache.h b/include/linux/nfsd/cache.h
index 04b355c..9315699 100644
--- a/include/linux/nfsd/cache.h
+++ b/include/linux/nfsd/cache.h
@@ -75,5 +75,8 @@ int	nfsd_reply_cache_init(void);
 void	nfsd_reply_cache_shutdown(void);
 int	nfsd_cache_lookup(struct svc_rqst *, int);
 void	nfsd_cache_update(struct svc_rqst *, int, __be32 *);
+#ifdef CONFIG_NFSD_V4_1
+void	nfsd41_set_statp(struct svc_rqst *, __be32*);
+#endif /* CONFIG_NFSD_V4_1 */
 
 #endif /* NFSCACHE_H */
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index 4f37e2f..e4dc32f 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -125,15 +125,22 @@ enum {
 	NFS4_SLOT_INPROGRESS
 };
 
+struct nfs41_cache_entry {
+	spinlock_t	ce_lock;
+	__be32		ce_status;
+	struct kvec	ce_datav; /* encoded NFSv4.1 data in rq_res.head[0] */
+	struct page	*ce_respages[RPCSVC_MAXPAGES];
+	short		ce_resused;
+};
+
 /*
  * nfs41_slot
- *
- * for now, just slot sequence number - will hold DRC for this slot.
  */
 struct nfs41_slot {
-	atomic_t		sl_state;
-	struct nfs41_session	*sl_session;
-	u32			sl_seqid;
+	atomic_t			sl_state;
+	struct nfs41_session		*sl_session;
+	u32				sl_seqid;
+	struct nfs41_cache_entry	sl_cache_entry;
 };
 
 /*
diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
index fc42fba..554f2cd 100644
--- a/include/linux/nfsd/xdr4.h
+++ b/include/linux/nfsd/xdr4.h
@@ -491,6 +491,11 @@ struct nfsd4_compoundres {
 	u32				opcnt;
 	__be32 *			tagp; /* where to encode tag and  opcount */
 	u32				minorversion;
+#if defined(CONFIG_NFSD_V4_1)
+	__be32				*statp;
+	u32				status;
+	struct current_session		*current_ses;
+#endif /* CONFIG_NFSD_V4_1 */
 };
 
 #define NFS4_SVC_XDRSIZE		sizeof(struct nfsd4_compoundargs)
@@ -534,6 +539,8 @@ extern __be32 nfsd4_setclientid_confirm(struct svc_rqst *rqstp,
 #if defined(CONFIG_NFSD_V4_1)
 extern void nfsd41_current_session_free(struct current_session *cses);
 extern struct current_session *nfsd41_current_session_alloc(void);
+extern void nfsd41_set_cache_entry(struct nfsd4_compoundres *resp);
+extern __be32 nfsd41_replay_cache_entry(struct nfsd4_compoundres *resp);
 extern __be32 nfsd4_exchange_id(struct svc_rqst *rqstp,
 		struct nfsd4_compound_state *,
 		struct nfsd4_exchange_id *);
-- 
1.6.0.2

Re: [pnfs] [RFC 45/51] nfsd41: DRC save, restore, and clear functions

[Benny Halevy]

On Nov. 10, 2008, 22:56 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> Cache all the result pages, including the rpc header in rq_respages[0],
> for a request in the slot table cache entry.
> 
> Cache the statp pointer from nfsd_dispatch which points into rq_respages[0]
> just past the rpc header. When setting a cache entry, calculate and save the
> length of the nfs data minus the rpc header for rq_respages[0].
> 
> When replaying a cache entry, replace the cached rpc header with the
> replayed request rpc result header, unless there is not enough room in the
> cached results first page. In that case, use the cached rpc header.
> 
> Signed-off-by: Andy Adamson<andros@netapp.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4state.c        |  106 ++++++++++++++++++++++++++++++++++++++++++++
>  fs/nfsd/nfssvc.c           |    6 +++
>  include/linux/nfsd/cache.h |    3 +
>  include/linux/nfsd/state.h |   17 +++++--
>  include/linux/nfsd/xdr4.h  |    7 +++
>  5 files changed, 134 insertions(+), 5 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index ba6ab19..230ce1d 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -412,6 +412,7 @@ alloc_init_session(struct nfs4_client *clp, struct nfsd4_create_session *cses)
>  	for (i = 0; i < new->se_fnumslots; i++) {
>  		new->se_slots[i].sl_session = new;
>  		nfs41_set_slot_state(&new->se_slots[i], NFS4_SLOT_AVAILABLE);
> +		spin_lock_init(&new->se_slots[i].sl_cache_entry.ce_lock);
>  	}
>  
>  	new->se_client = clp;
> @@ -997,6 +998,111 @@ nfsd4_clear_respages(struct page **respages, short resused)
>  }
>  
>  #if defined(CONFIG_NFSD_V4_1)
> +
> +void
> +nfsd41_set_statp(struct svc_rqst *rqstp, __be32 *statp)
> +{
> +	struct nfsd4_compoundres *resp = rqstp->rq_resp;
> +
> +	resp->statp = statp;
> +}
> +
> +/*
> + * Cache the reply pages, clearing the previous pages.
> + * Store the base and length of the rq_req.head[0] page
> + * of the NFSv4.1 data, just past the rpc header.
> + */
> +void
> +nfsd41_set_cache_entry(struct nfsd4_compoundres *resp)
> +{
> +	struct nfs41_cache_entry *entry =
> +		&resp->current_ses->cs_slot->sl_cache_entry;
> +	struct svc_rqst *rqstp = resp->rqstp;
> +	struct kvec *resv = &rqstp->rq_res.head[0];
> +
> +	dprintk("--> %s entry %p\n", __func__, entry);
> +
> +	/* Don't cache a failed OP_SEQUENCE */
> +	if (resp->opcnt == 1 && resp->status)
> +		return;
> +	spin_lock(&entry->ce_lock);

review 11-13: we can probably get rid of the spinlock
and s/nfsd4_clear_respages/nfsd4_release_respages/

> +	nfsd4_clear_respages(entry->ce_respages, entry->ce_resused);
> +	nfsd4_cache_rqst_pages(rqstp, entry->ce_respages, &entry->ce_resused);
> +	entry->ce_status = resp->status;
> +	entry->ce_datav.iov_base = resp->statp;
> +	entry->ce_datav.iov_len = resv->iov_len - ((char *)resp->statp -
> +				  (char *)page_address(rqstp->rq_respages[0]));
> +	spin_unlock(&entry->ce_lock);
> +}
> +
> +/*
> + * Copy the cached NFSv4.1 reply skipping the cached rpc header into the
> + * replay result res.head[0] past the rpc header to end up with replay
> + * rpc header and cached NFSv4.1 reply.
> + */
> +static int
> +nfsd41_copy_replay_data(struct nfsd4_compoundres *resp,
> +			struct nfs41_cache_entry *entry)
> +{
> +	struct svc_rqst *rqstp = resp->rqstp;
> +	struct kvec *resv = &resp->rqstp->rq_res.head[0];
> +	int len;
> +
> +	/* Current request rpc header length*/
> +	len = (char *)resp->statp - (char *)page_address(rqstp->rq_respages[0]);
> +	if (entry->ce_datav.iov_len + len > PAGE_SIZE) {
> +		dprintk("%s v41 cached reply too large (%Zd).\n", __func__,
> +			entry->ce_datav.iov_len);
> +		return 0;
> +	}
> +	/* copy the cached reply nfsd data past the current rpc header */
> +	memcpy((char *)resv->iov_base + len, entry->ce_datav.iov_base,
> +		entry->ce_datav.iov_len);
> +	resv->iov_len = len + entry->ce_datav.iov_len;
> +	return 1;
> +}
> +
> +/*
> + * Keep the first page of the replay. Copy the NFSv4.1 data from the first
> + * cached page.  Replace any futher replay pages from the cache.
> + */
> +__be32
> +nfsd41_replay_cache_entry(struct nfsd4_compoundres *resp)
> +{
> +	struct nfs41_cache_entry *entry =
> +		&resp->current_ses->cs_slot->sl_cache_entry;
> +	__be32 status;
> +
> +	dprintk("--> %s entry %p\n", __func__, entry);
> +
> +	spin_lock(&entry->ce_lock);

review 11-13: no spin lock, yes state lock
do this in nfsd4_sequence under the state lock.
do nothing in nfs4svc_encode_compoundres

> +
> +	if (!nfsd41_copy_replay_data(resp, entry)) {
> +		/*
> +		 * Not enough room to use the replay rpc header, send the
> +		 * cached header. Release all the allocated result pages.
> +		 */
> +		svc_free_res_pages(resp->rqstp);
> +		nfsd4_move_pages(resp->rqstp->rq_respages, entry->ce_respages,
> +			entry->ce_resused);
> +	} else {
> +		/* Release all but the first allocatek result page */

review 11-13: s/allocatek/allocated/

> +
> +		resp->rqstp->rq_resused--;
> +		svc_free_res_pages(resp->rqstp);
> +
> +		nfsd4_move_pages(&resp->rqstp->rq_respages[1],
> +				 &entry->ce_respages[1],
> +				 entry->ce_resused - 1);
> +	}
> +
> +	resp->rqstp->rq_resused = entry->ce_resused;
> +	status = entry->ce_status;
> +	spin_unlock(&entry->ce_lock);
> +
> +	return status;
> +}
> +
>  /*
>   * Set the exchange_id flags returned by the server.
>   */
> diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
> index 07e4f5d..a9f90b6 100644
> --- a/fs/nfsd/nfssvc.c
> +++ b/fs/nfsd/nfssvc.c
> @@ -543,6 +543,12 @@ nfsd_dispatch(struct svc_rqst *rqstp, __be32 *statp)
>  		+ rqstp->rq_res.head[0].iov_len;
>  	rqstp->rq_res.head[0].iov_len += sizeof(__be32);
>  
> +#ifdef CONFIG_NFSD_V4_1
> +	/* NFSv4.1 DRC requires statp */
> +	if (rqstp->rq_vers == 4)
> +		nfsd41_set_statp(rqstp, statp);
> +#endif /* CONFIG_NFSD_V4_1 */
> +
>  	/* Now call the procedure handler, and encode NFS status. */
>  	nfserr = proc->pc_func(rqstp, rqstp->rq_argp, rqstp->rq_resp);
>  	nfserr = map_new_errors(rqstp->rq_vers, nfserr);
> diff --git a/include/linux/nfsd/cache.h b/include/linux/nfsd/cache.h
> index 04b355c..9315699 100644
> --- a/include/linux/nfsd/cache.h
> +++ b/include/linux/nfsd/cache.h
> @@ -75,5 +75,8 @@ int	nfsd_reply_cache_init(void);
>  void	nfsd_reply_cache_shutdown(void);
>  int	nfsd_cache_lookup(struct svc_rqst *, int);
>  void	nfsd_cache_update(struct svc_rqst *, int, __be32 *);
> +#ifdef CONFIG_NFSD_V4_1
> +void	nfsd41_set_statp(struct svc_rqst *, __be32*);
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  #endif /* NFSCACHE_H */
> diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
> index 4f37e2f..e4dc32f 100644
> --- a/include/linux/nfsd/state.h
> +++ b/include/linux/nfsd/state.h
> @@ -125,15 +125,22 @@ enum {
>  	NFS4_SLOT_INPROGRESS
>  };
>  
> +struct nfs41_cache_entry {
> +	spinlock_t	ce_lock;
> +	__be32		ce_status;
> +	struct kvec	ce_datav; /* encoded NFSv4.1 data in rq_res.head[0] */
> +	struct page	*ce_respages[RPCSVC_MAXPAGES];
> +	short		ce_resused;
> +};
> +
>  /*
>   * nfs41_slot
> - *
> - * for now, just slot sequence number - will hold DRC for this slot.
>   */
>  struct nfs41_slot {
> -	atomic_t		sl_state;
> -	struct nfs41_session	*sl_session;
> -	u32			sl_seqid;
> +	atomic_t			sl_state;
> +	struct nfs41_session		*sl_session;
> +	u32				sl_seqid;
> +	struct nfs41_cache_entry	sl_cache_entry;
>  };
>  
>  /*
> diff --git a/include/linux/nfsd/xdr4.h b/include/linux/nfsd/xdr4.h
> index fc42fba..554f2cd 100644
> --- a/include/linux/nfsd/xdr4.h
> +++ b/include/linux/nfsd/xdr4.h
> @@ -491,6 +491,11 @@ struct nfsd4_compoundres {
>  	u32				opcnt;
>  	__be32 *			tagp; /* where to encode tag and  opcount */
>  	u32				minorversion;
> +#if defined(CONFIG_NFSD_V4_1)
> +	__be32				*statp;
> +	u32				status;
> +	struct current_session		*current_ses;
> +#endif /* CONFIG_NFSD_V4_1 */
>  };
>  
>  #define NFS4_SVC_XDRSIZE		sizeof(struct nfsd4_compoundargs)
> @@ -534,6 +539,8 @@ extern __be32 nfsd4_setclientid_confirm(struct svc_rqst *rqstp,
>  #if defined(CONFIG_NFSD_V4_1)
>  extern void nfsd41_current_session_free(struct current_session *cses);
>  extern struct current_session *nfsd41_current_session_alloc(void);
> +extern void nfsd41_set_cache_entry(struct nfsd4_compoundres *resp);
> +extern __be32 nfsd41_replay_cache_entry(struct nfsd4_compoundres *resp);
>  extern __be32 nfsd4_exchange_id(struct svc_rqst *rqstp,
>  		struct nfsd4_compound_state *,
>  		struct nfsd4_exchange_id *);

[RFC 46/51] nfsd41: nfsd nfsd4_sequence DRC logic

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

Create a new internal error nfserr_replay_cache for the nfs41 DRC.

Refactor nfsd4_sequence processing to enforce error and replay behaviour.

Move seqid logic into check_slot_seqid
In preparation for use with create session slot table.

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c       |   84 ++++++++++++++++++++------------------------
 include/linux/nfs.h       |    1 +
 include/linux/nfsd/nfsd.h |    1 +
 3 files changed, 40 insertions(+), 46 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 230ce1d..a1a9e0a 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -1222,6 +1222,32 @@ error:
 	return status;
 }
 
+static int
+check_slot_seqid(u32 seqid, struct nfs41_slot *slot)
+{
+	dprintk("%s enter. seqid %d slot->sl_seqid %d\n", __func__, seqid,
+		slot->sl_seqid);
+
+	/* The slot is in use, and no response has been sent. */
+	if (nfs41_get_slot_state(slot) == NFS4_SLOT_INPROGRESS) {
+		if (seqid == slot->sl_seqid)
+			return nfserr_jukebox;
+		else
+			return nfserr_seq_misordered;
+	}
+	/* Normal */
+	if (likely(seqid == slot->sl_seqid + 1))
+		return nfs_ok;
+	/* Replay */
+	if (seqid == slot->sl_seqid)
+		return nfserr_replay_cache;
+	/* Wraparound */
+	if (seqid == 1 && (slot->sl_seqid + 1) == 0)
+		return nfs_ok;
+	/* Misordered replay or misordered new request */
+	return nfserr_seq_misordered;
+}
+
 __be32
 nfsd4_create_session(struct svc_rqst *rqstp,
 		     struct nfsd4_compound_state *cstate,
@@ -1337,24 +1363,6 @@ out:
 	return status;
 }
 
-static int
-check_slot_seqid(u32 seqid, struct nfs41_slot *slot)
-{
-	dprintk("%s enter. seqid %d slot->sl_seqid %d\n", __func__, seqid,
-		slot->sl_seqid);
-	/* Normal */
-	if (likely(seqid == slot->sl_seqid + 1))
-		return nfs_ok;
-	/* Replay */
-	if (seqid == slot->sl_seqid)
-		return NFSERR_REPLAY_ME;
-	/* Wraparound */
-	if (seqid == 1 && (slot->sl_seqid + 1) == 0)
-		return nfs_ok;
-	/* Misordered replay or misordered new request */
-	return nfserr_seq_misordered;
-}
-
 __be32
 nfsd4_sequence(struct svc_rqst *r,
 	       struct nfsd4_compound_state *cstate,
@@ -1381,31 +1389,26 @@ nfsd4_sequence(struct svc_rqst *r,
 	slot = &elem->se_slots[seq->slotid];
 	dprintk("%s: slotid %d\n", __func__, seq->slotid);
 
-	/* Server post op_sequence compound processing had an upcall which
-	 * resulted in replaying the compound processing including the
-	 * already processed op_sequence. Set current_session
-	 * but don't bump slot->sl_seqid which was incremented in successful
-	 * op_sequence processing prior to upcall.
-	 */
-	if (nfs41_get_slot_state(slot) == NFS4_SLOT_INPROGRESS) {
-		dprintk("%s: NFS4_SLOT_INPROGRESS. set current_session\n",
-			__func__);
-		goto set_curr_ses;
-	}
-
 	status = check_slot_seqid(seq->seqid, slot);
-	if (status == NFSERR_REPLAY_ME)
-		goto replay;
-	else if (status)
+	if (status == nfserr_replay_cache)
+		goto replay_cache;
+	if (status)
 		goto out;
 
-	/* Success! bump slot seqid and renew clientid */
+	/* Success! bump slot seqid */
 	slot->sl_seqid = seq->seqid;
+
+	/* FIXME: for now just initialize target_highest_slotid and flags
+	 * response fields */
+	seq->target_maxslots = seq->maxslots;
+	seq->status_flags = 0;
+
+replay_cache:
+	/* renew the clientid on success and on replay */
 	renew_client(elem->se_client);
 	dprintk("%s: set NFS4_SLOT_INPROGRESS\n", __func__);
 	nfs41_set_slot_state(slot, NFS4_SLOT_INPROGRESS);
 
-set_curr_ses:
 	/* Set current_session. hold reference until done processing compound.
 	 * nfs41_put_session called only if cs_slot is set
 	 */
@@ -1414,21 +1417,10 @@ set_curr_ses:
 	c_ses->cs_slot = slot;
 	nfs41_get_session(slot->sl_session);
 
-	/* FIXME: for now just initialize target_highest_slotid and flags
-	 * response fields */
-	seq->target_maxslots = seq->maxslots;
-	seq->status_flags = 0;
-
-	status = nfs_ok;
 out:
 	dprintk("%s: return %d\n", __func__, ntohl(status));
 	nfs4_unlock_state();
 	return status;
-replay:
-	dprintk("%s: REPLAY - AKKKK! no code yet! return BAD SESSION\n",
-		__func__);
-	status = nfserr_badsession;
-	goto out;
 }
 #endif /* CONFIG_NFSD_V4_1 */
 
diff --git a/include/linux/nfs.h b/include/linux/nfs.h
index 4dd83f2..b148b16 100644
--- a/include/linux/nfs.h
+++ b/include/linux/nfs.h
@@ -127,6 +127,7 @@
 	NFSERR_SEQUENCE_POS = 10064,		/* v4.1 */
 
 	NFSERR_REPLAY_ME = 11001,	/* linux internal */
+	NFSERR_REPLAY_CACHE = 11002,		/* v4.1 linux internal */
 };
 
 /* NFSv2 file types - beware, these are not the same in NFSv3 */
diff --git a/include/linux/nfsd/nfsd.h b/include/linux/nfsd/nfsd.h
index 2e763eb..c8bbe94 100644
--- a/include/linux/nfsd/nfsd.h
+++ b/include/linux/nfsd/nfsd.h
@@ -255,6 +255,7 @@ void		nfsd_lockd_shutdown(void);
 #define	nfserr_locked		__constant_htonl(NFSERR_LOCKED)
 #define	nfserr_wrongsec		__constant_htonl(NFSERR_WRONGSEC)
 #define	nfserr_replay_me	__constant_htonl(NFSERR_REPLAY_ME)
+#define	nfserr_replay_cache	__constant_htonl(NFSERR_REPLAY_CACHE)
 #define	nfserr_badiomode	__constant_htonl(NFSERR_BADIOMODE)
 #define	nfserr_badlayout	__constant_htonl(NFSERR_BADLAYOUT)
 #define nfserr_bad_session_digest	__constant_htonl(NFSERR_BAD_SESSION_DIGEST)
-- 
1.6.0.2

[RFC 47/51] nfsd41: enforce NFS4ERR_SEQUENCE_POS operation order rules

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c |   30 ++++++++++++++++++++++--------
 1 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index eaebe5b..4a6ec7e 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -914,14 +914,18 @@ nfsd4_save_deferred_state(struct svc_rqst *rqstp,
 
 typedef __be32(*nfsd4op_func)(struct svc_rqst *, struct nfsd4_compound_state *,
 			      void *);
+enum nfsd4_op_flags {
+/* Most ops require a valid current filehandle; a few don't: */
+	ALLOWED_WITHOUT_FH = 1 << 0,
+/* GETATTR and ops not listed as returning NFS4ERR_MOVED: */
+	ALLOWED_ON_ABSENT_FS = 2 << 0,
+/* NFSv4.1 insists on some operations being first in a compound. */
+	ALLOWED_AS_FIRST_OP = 3 << 0,
+};
 
 struct nfsd4_operation {
 	nfsd4op_func op_func;
 	u32 op_flags;
-/* Most ops require a valid current filehandle; a few don't: */
-#define ALLOWED_WITHOUT_FH 1
-/* GETATTR and ops not listed as returning NFS4ERR_MOVED: */
-#define ALLOWED_ON_ABSENT_FS 2
 	char *op_name;
 };
 
@@ -1037,6 +1041,16 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
 			op->status = nfserr_moved;
 			goto encode_op;
 		}
+#ifdef CONFIG_NFSD_V4_1
+		if (resp->minorversion == 1) {
+			if ((resp->opcnt == 1 &&
+			    !(opdesc->op_flags & ALLOWED_AS_FIRST_OP)) ||
+			    (resp->opcnt > 1 && (op->opnum == OP_SEQUENCE))) {
+				op->status = nfserr_sequence_pos;
+				goto encode_op;
+			}
+		}
+#endif /* CONFIG_NFSD_V4_1 */
 
 		if (opdesc->op_func)
 			op->status = opdesc->op_func(rqstp, cstate, &op->u);
@@ -1261,22 +1275,22 @@ static struct nfsd4_operation nfsd4_ops[NFSD4_LAST_OP+1] = {
 #if defined(CONFIG_NFSD_V4_1)
 	[OP_EXCHANGE_ID] = {
 		.op_func = (nfsd4op_func)nfsd4_exchange_id,
-		.op_flags = ALLOWED_WITHOUT_FH,
+		.op_flags = ALLOWED_WITHOUT_FH | ALLOWED_AS_FIRST_OP,
 		.op_name = "OP_EXCHANGE_ID",
 	},
 	[OP_CREATE_SESSION] = {
 		.op_func = (nfsd4op_func)nfsd4_create_session,
-		.op_flags = ALLOWED_WITHOUT_FH,
+		.op_flags = ALLOWED_WITHOUT_FH | ALLOWED_AS_FIRST_OP,
 		.op_name = "OP_CREATE_SESSION",
 	},
 	[OP_DESTROY_SESSION] = {
 		.op_func = (nfsd4op_func)nfsd4_destroy_session,
-		.op_flags = ALLOWED_WITHOUT_FH,
+		.op_flags = ALLOWED_WITHOUT_FH | ALLOWED_AS_FIRST_OP,
 		.op_name = "OP_DESTROY_SESSION",
 	},
 	[OP_SEQUENCE] = {
 		.op_func = (nfsd4op_func)nfsd4_sequence,
-		.op_flags = ALLOWED_WITHOUT_FH,
+		.op_flags = ALLOWED_WITHOUT_FH | ALLOWED_AS_FIRST_OP,
 		.op_name = "OP_SEQUENCE",
 	},
 #endif /* CONFIG_NFSD_V4_1 */
-- 
1.6.0.2

Re: [pnfs] [RFC 47/51] nfsd41: enforce NFS4ERR_SEQUENCE_POS operation order rules

[Benny Halevy]

On Nov. 10, 2008, 22:56 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> Signed-off-by: Andy Adamson<andros@netapp.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4proc.c |   30 ++++++++++++++++++++++--------
>  1 files changed, 22 insertions(+), 8 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index eaebe5b..4a6ec7e 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -914,14 +914,18 @@ nfsd4_save_deferred_state(struct svc_rqst *rqstp,
>  
>  typedef __be32(*nfsd4op_func)(struct svc_rqst *, struct nfsd4_compound_state *,
>  			      void *);
> +enum nfsd4_op_flags {
> +/* Most ops require a valid current filehandle; a few don't: */
> +	ALLOWED_WITHOUT_FH = 1 << 0,
> +/* GETATTR and ops not listed as returning NFS4ERR_MOVED: */
> +	ALLOWED_ON_ABSENT_FS = 2 << 0,
> +/* NFSv4.1 insists on some operations being first in a compound. */
> +	ALLOWED_AS_FIRST_OP = 3 << 0,
> +};
>  
>  struct nfsd4_operation {
>  	nfsd4op_func op_func;
>  	u32 op_flags;
> -/* Most ops require a valid current filehandle; a few don't: */
> -#define ALLOWED_WITHOUT_FH 1
> -/* GETATTR and ops not listed as returning NFS4ERR_MOVED: */
> -#define ALLOWED_ON_ABSENT_FS 2
>  	char *op_name;
>  };
>  
> @@ -1037,6 +1041,16 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
>  			op->status = nfserr_moved;
>  			goto encode_op;
>  		}
> +#ifdef CONFIG_NFSD_V4_1
> +		if (resp->minorversion == 1) {
> +			if ((resp->opcnt == 1 &&
> +			    !(opdesc->op_flags & ALLOWED_AS_FIRST_OP)) ||
> +			    (resp->opcnt > 1 && (op->opnum == OP_SEQUENCE))) {

review 11-13: get rid of #ifdef here and in enum nfs_opnum4
move the first part out of the processing loop by peeking at the first op.
and (resp->opcnt > 1) can be done inside nfsd4_sequence.

> +				op->status = nfserr_sequence_pos;
> +				goto encode_op;
> +			}
> +		}
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  		if (opdesc->op_func)
>  			op->status = opdesc->op_func(rqstp, cstate, &op->u);
> @@ -1261,22 +1275,22 @@ static struct nfsd4_operation nfsd4_ops[NFSD4_LAST_OP+1] = {
>  #if defined(CONFIG_NFSD_V4_1)
>  	[OP_EXCHANGE_ID] = {
>  		.op_func = (nfsd4op_func)nfsd4_exchange_id,
> -		.op_flags = ALLOWED_WITHOUT_FH,
> +		.op_flags = ALLOWED_WITHOUT_FH | ALLOWED_AS_FIRST_OP,
>  		.op_name = "OP_EXCHANGE_ID",
>  	},
>  	[OP_CREATE_SESSION] = {
>  		.op_func = (nfsd4op_func)nfsd4_create_session,
> -		.op_flags = ALLOWED_WITHOUT_FH,
> +		.op_flags = ALLOWED_WITHOUT_FH | ALLOWED_AS_FIRST_OP,
>  		.op_name = "OP_CREATE_SESSION",
>  	},
>  	[OP_DESTROY_SESSION] = {
>  		.op_func = (nfsd4op_func)nfsd4_destroy_session,
> -		.op_flags = ALLOWED_WITHOUT_FH,
> +		.op_flags = ALLOWED_WITHOUT_FH | ALLOWED_AS_FIRST_OP,
>  		.op_name = "OP_DESTROY_SESSION",
>  	},
>  	[OP_SEQUENCE] = {
>  		.op_func = (nfsd4op_func)nfsd4_sequence,
> -		.op_flags = ALLOWED_WITHOUT_FH,
> +		.op_flags = ALLOWED_WITHOUT_FH | ALLOWED_AS_FIRST_OP,
>  		.op_name = "OP_SEQUENCE",
>  	},
>  #endif /* CONFIG_NFSD_V4_1 */

[RFC 48/51] nfsd41: nfsd DRC logic

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

Pass the current session to nfs4svc_encode_compound res via struct
nfsd4_compoundres and replay or set an NFSv4.1 DRC entry.

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4proc.c |   16 ++++------------
 fs/nfsd/nfs4xdr.c  |   26 ++++++++++++++++++++++++++
 2 files changed, 30 insertions(+), 12 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 4a6ec7e..5a3a08d 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1095,18 +1095,10 @@ encode_op:
 	}
 
 #if defined(CONFIG_NFSD_V4_1)
-	if (cstate->current_ses) {
-		struct nfs41_slot *cs_slot = cstate->current_ses->cs_slot;
-		if (cs_slot) {
-			if (status != nfserr_dropit) {
-				dprintk("%s: SET SLOT STATE TO AVAILABLE\n",
-					__func__);
-				nfs41_set_slot_state(cs_slot,
-						     NFS4_SLOT_AVAILABLE);
-			}
-			nfs41_put_session(cs_slot->sl_session);
-		}
-		nfsd41_current_session_free(cstate->current_ses);
+	if (resp->minorversion == 1) {
+		/* Setup the compoundres nfs4svc_encode_compoundres */
+		resp->current_ses = cstate->current_ses;
+		resp->status = status;
 	}
 #endif /* CONFIG_NFSD_V4_1 */
 	nfsd4_cstate_free(cstate, rqstp->rq_deferred);
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 04a9088..4b4f5fb 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -3302,6 +3302,32 @@ nfs4svc_encode_compoundres(struct svc_rqst *rqstp, __be32 *p, struct nfsd4_compo
 		iov = &rqstp->rq_res.head[0];
 	iov->iov_len = ((char*)resp->p) - (char*)iov->iov_base;
 	BUG_ON(iov->iov_len > PAGE_SIZE);
+#ifdef CONFIG_NFSD_V4_1
+	if (resp->minorversion == 1) {
+		/*
+		* Always have a current session.
+		* The cs_slot is not set on SEQUENCE error, EXCHANGE_ID,
+		* solo CREATE_SESSION, DESTROY_SESSION compounds.
+		*/
+		if (!resp->current_ses || !resp->current_ses->cs_slot) {
+			dprintk("%s NO CS_SLOT. current_ses %p\n", __func__,
+				resp->current_ses);
+			goto free_current_ses;
+		}
+
+		if (resp->status == nfserr_replay_cache)
+			nfsd41_replay_cache_entry(resp);
+		else
+			nfsd41_set_cache_entry(resp);
+
+		dprintk("%s: SET SLOT STATE TO AVAILABLE\n", __func__);
+		nfs41_set_slot_state(resp->current_ses->cs_slot,
+				     NFS4_SLOT_AVAILABLE);
+		nfs41_put_session(resp->current_ses->cs_slot->sl_session);
+free_current_ses:
+		nfsd41_current_session_free(resp->current_ses);
+}
+#endif /* CONFIG_NFSD_V4_1 */
 	return 1;
 }
 
-- 
1.6.0.2

[RFC 49/51] nfsd41: clear DRC cache on free_session

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index a1a9e0a..cefb605 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -478,8 +478,15 @@ void
 free_session(struct kref *kref)
 {
 	struct nfs41_session *ses;
+	int i;
 
 	ses = container_of(kref, struct nfs41_session, se_ref);
+	for (i = 0; i < ses->se_fnumslots; i++) {
+		struct nfs41_cache_entry *e = &ses->se_slots[i].sl_cache_entry;
+		spin_lock(&e->ce_lock);
+		nfsd4_clear_respages(e->ce_respages, e->ce_resused);
+		spin_unlock(&e->ce_lock);
+	}
 	kfree(ses->se_slots);
 	kfree(ses);
 }
-- 
1.6.0.2

[RFC 50/51] nfsd41: Add a create session replay cache

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

Replace the nfs4_client cl_seqid field with a single struct nfs41_slot used
for the create session replay cache.

The CREATE_SESSION slot sets the sl_session pointer to NULL. Otherwise, the
slot and it's replay cache are used just like the session slots.

Fix unconfirmed create_session replay response by initializing the
create_session slot sequence id to 0.

A future patch will set the CREATE_SESSION cache when a SEQUENCE operation
preceeds the CREATE_SESSION operation. This compound is currently only cached
in the session slot table.

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c        |   45 +++++++++++++++++++++++++++++--------------
 fs/nfsd/nfs4xdr.c          |   10 +++++---
 include/linux/nfsd/state.h |    2 +-
 3 files changed, 37 insertions(+), 20 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index cefb605..0d8fde7 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -610,6 +610,8 @@ free_client(struct nfs4_client *clp)
 #if defined(CONFIG_NFSD_V4_1)
 	if (clp->cl_cb_xprt)
 		svc_xprt_put(clp->cl_cb_xprt);
+	nfsd4_clear_respages(clp->cl_slot.sl_cache_entry.ce_respages,
+			     clp->cl_slot.sl_cache_entry.ce_resused);
 #endif /* CONFIG_NFSD_V4_1 */
 	if (clp->cl_cred.cr_group_info)
 		put_group_info(clp->cl_cred.cr_group_info);
@@ -686,6 +688,8 @@ static struct nfs4_client *create_client(struct xdr_netobj name, char *recdir)
 	INIT_LIST_HEAD(&clp->cl_openowners);
 	INIT_LIST_HEAD(&clp->cl_delegations);
 #if defined(CONFIG_NFSD_V4_1)
+	nfs41_set_slot_state(&clp->cl_slot, NFS4_SLOT_AVAILABLE);
+	spin_lock_init(&clp->cl_slot.sl_cache_entry.ce_lock);
 	INIT_LIST_HEAD(&clp->cl_sessions);
 	mutex_init(&clp->cl_cb_mutex);
 #endif /* CONFIG_NFSD_V4_1 */
@@ -1022,15 +1026,16 @@ nfsd41_set_statp(struct svc_rqst *rqstp, __be32 *statp)
 void
 nfsd41_set_cache_entry(struct nfsd4_compoundres *resp)
 {
-	struct nfs41_cache_entry *entry =
-		&resp->current_ses->cs_slot->sl_cache_entry;
+	struct nfs41_slot *slot = resp->current_ses->cs_slot;
+	struct nfs41_cache_entry *entry = &slot->sl_cache_entry;
 	struct svc_rqst *rqstp = resp->rqstp;
 	struct kvec *resv = &rqstp->rq_res.head[0];
 
 	dprintk("--> %s entry %p\n", __func__, entry);
 
-	/* Don't cache a failed OP_SEQUENCE */
-	if (resp->opcnt == 1 && resp->status)
+	/* Don't cache a failed OP_SEQUENCE. The CREATE_SESSION slot has NULL
+	 * sl_session pointer */
+	if (resp->opcnt == 1 && resp->status && slot->sl_session)
 		return;
 	spin_lock(&entry->ce_lock);
 	nfsd4_clear_respages(entry->ce_respages, entry->ce_resused);
@@ -1215,11 +1220,12 @@ out_copy:
 	clid->clientid.cl_boot = new->cl_clientid.cl_boot;
 	clid->clientid.cl_id = new->cl_clientid.cl_id;
 
-	new->cl_seqid = clid->seqid = 1;
+	new->cl_slot.sl_seqid = 0;
+	clid->seqid = 1;
 	nfsd4_set_ex_flags(new, clid);
 
 	dprintk("nfsd4_exchange_id seqid %d flags %x\n",
-		new->cl_seqid, new->cl_exchange_flags);
+		new->cl_slot.sl_seqid, new->cl_exchange_flags);
 	status = nfs_ok;
 
 out:
@@ -1262,6 +1268,8 @@ nfsd4_create_session(struct svc_rqst *rqstp,
 {
 	u32 ip_addr = svc_addr_in(rqstp)->sin_addr.s_addr;
 	struct nfs4_client *conf, *unconf;
+	struct current_session *c_ses = cstate->current_ses;
+	struct nfs41_slot *slot = NULL;
 	__u32   max_blocksize = svc_max_payload(rqstp);
 	int status = 0;
 
@@ -1277,19 +1285,20 @@ nfsd4_create_session(struct svc_rqst *rqstp,
 		goto out;
 	}
 	if (conf) {
-		status = nfs_ok;
-		if (conf->cl_seqid == cr_ses->seqid) {
+		slot = &conf->cl_slot;
+		status = check_slot_seqid(cr_ses->seqid, slot);
+		if (status == nfserr_replay_cache) {
 			dprintk("Got a create_session replay! seqid= %d\n",
-				conf->cl_seqid);
+				slot->sl_seqid);
 			goto out_replay;
-		} else if (cr_ses->seqid != conf->cl_seqid + 1) {
+		} else if (cr_ses->seqid != conf->cl_slot.sl_seqid + 1) {
 			status = nfserr_seq_misordered;
 			dprintk("Sequence misordered!\n");
 			dprintk("Expected seqid= %d but got seqid= %d\n",
-				conf->cl_seqid, cr_ses->seqid);
+				slot->sl_seqid, cr_ses->seqid);
 			goto out;
 		}
-		conf->cl_seqid++;
+		conf->cl_slot.sl_seqid++;
 	} else if (unconf) {
 		if (!same_creds(&unconf->cl_cred, &rqstp->rq_cred) ||
 		    (ip_addr != unconf->cl_addr)) {
@@ -1297,11 +1306,15 @@ nfsd4_create_session(struct svc_rqst *rqstp,
 			goto out;
 		}
 
-		if (unconf->cl_seqid != cr_ses->seqid) {
+		slot = &unconf->cl_slot;
+		status = check_slot_seqid(cr_ses->seqid, slot);
+		if (status) {
+			/* an unconfirmed replay returns misordered */
 			status = nfserr_seq_misordered;
 			goto out;
 		}
 
+		slot->sl_seqid++; /* from 0 to 1 */
 		move_to_confirmed(unconf);
 
 		/*
@@ -1322,9 +1335,8 @@ nfsd4_create_session(struct svc_rqst *rqstp,
 
 	status = alloc_init_session(conf, cr_ses);
 
-out_replay:
 	memcpy(cr_ses->sessionid, conf->cl_sessionid, 16);
-	cr_ses->seqid = conf->cl_seqid;
+	cr_ses->seqid = slot->sl_seqid;
 	cr_ses->fore_channel.maxreq_sz = max_blocksize;
 	cr_ses->fore_channel.maxresp_sz = max_blocksize;
 	cr_ses->fore_channel.maxresp_cached = max_blocksize;
@@ -1332,6 +1344,9 @@ out_replay:
 	cr_ses->back_channel.maxresp_sz = max_blocksize;
 	cr_ses->back_channel.maxresp_cached = max_blocksize;
 
+out_replay:
+	nfs41_set_slot_state(slot, NFS4_SLOT_INPROGRESS);
+	c_ses->cs_slot = slot;
 out:
 	nfs4_unlock_state();
 	dprintk("%s returns %d\n", __func__, ntohl(status));
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 4b4f5fb..9799ea1 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -3304,10 +3304,11 @@ nfs4svc_encode_compoundres(struct svc_rqst *rqstp, __be32 *p, struct nfsd4_compo
 	BUG_ON(iov->iov_len > PAGE_SIZE);
 #ifdef CONFIG_NFSD_V4_1
 	if (resp->minorversion == 1) {
+		struct nfs41_slot *slot;
 		/*
 		* Always have a current session.
 		* The cs_slot is not set on SEQUENCE error, EXCHANGE_ID,
-		* solo CREATE_SESSION, DESTROY_SESSION compounds.
+		* and solo DESTROY_SESSION compounds.
 		*/
 		if (!resp->current_ses || !resp->current_ses->cs_slot) {
 			dprintk("%s NO CS_SLOT. current_ses %p\n", __func__,
@@ -3321,9 +3322,10 @@ nfs4svc_encode_compoundres(struct svc_rqst *rqstp, __be32 *p, struct nfsd4_compo
 			nfsd41_set_cache_entry(resp);
 
 		dprintk("%s: SET SLOT STATE TO AVAILABLE\n", __func__);
-		nfs41_set_slot_state(resp->current_ses->cs_slot,
-				     NFS4_SLOT_AVAILABLE);
-		nfs41_put_session(resp->current_ses->cs_slot->sl_session);
+		slot = resp->current_ses->cs_slot;
+		nfs41_set_slot_state(slot, NFS4_SLOT_AVAILABLE);
+		if (slot->sl_session)
+			nfs41_put_session(slot->sl_session);
 free_current_ses:
 		nfsd41_current_session_free(resp->current_ses);
 }
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index e4dc32f..a336687 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -233,7 +233,7 @@ struct nfs4_client {
 	atomic_t		cl_count;	/* ref count */
 	u32			cl_firststate;	/* recovery dir creation */
 #if defined(CONFIG_NFSD_V4_1)
-	u32			cl_seqid;	/* seqid for create_session */
+	struct nfs41_slot	cl_slot;	/* create_session slot */
 	u32			cl_exchange_flags;
 	nfs41_sessionid		cl_sessionid;
 
-- 
1.6.0.2

Re: [pnfs] [RFC 50/51] nfsd41: Add a create session replay cache

[Benny Halevy]

On Nov. 10, 2008, 22:57 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> Replace the nfs4_client cl_seqid field with a single struct nfs41_slot used
> for the create session replay cache.
> 
> The CREATE_SESSION slot sets the sl_session pointer to NULL. Otherwise, the
> slot and it's replay cache are used just like the session slots.
> 
> Fix unconfirmed create_session replay response by initializing the
> create_session slot sequence id to 0.
> 
> A future patch will set the CREATE_SESSION cache when a SEQUENCE operation
> preceeds the CREATE_SESSION operation. This compound is currently only cached
> in the session slot table.
> 
> Signed-off-by: Andy Adamson<andros@netapp.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>
> ---
>  fs/nfsd/nfs4state.c        |   45 +++++++++++++++++++++++++++++--------------
>  fs/nfsd/nfs4xdr.c          |   10 +++++---
>  include/linux/nfsd/state.h |    2 +-
>  3 files changed, 37 insertions(+), 20 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index cefb605..0d8fde7 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -610,6 +610,8 @@ free_client(struct nfs4_client *clp)
>  #if defined(CONFIG_NFSD_V4_1)
>  	if (clp->cl_cb_xprt)
>  		svc_xprt_put(clp->cl_cb_xprt);
> +	nfsd4_clear_respages(clp->cl_slot.sl_cache_entry.ce_respages,
> +			     clp->cl_slot.sl_cache_entry.ce_resused);
>  #endif /* CONFIG_NFSD_V4_1 */
>  	if (clp->cl_cred.cr_group_info)
>  		put_group_info(clp->cl_cred.cr_group_info);
> @@ -686,6 +688,8 @@ static struct nfs4_client *create_client(struct xdr_netobj name, char *recdir)
>  	INIT_LIST_HEAD(&clp->cl_openowners);
>  	INIT_LIST_HEAD(&clp->cl_delegations);
>  #if defined(CONFIG_NFSD_V4_1)
> +	nfs41_set_slot_state(&clp->cl_slot, NFS4_SLOT_AVAILABLE);
> +	spin_lock_init(&clp->cl_slot.sl_cache_entry.ce_lock);
>  	INIT_LIST_HEAD(&clp->cl_sessions);
>  	mutex_init(&clp->cl_cb_mutex);
>  #endif /* CONFIG_NFSD_V4_1 */
> @@ -1022,15 +1026,16 @@ nfsd41_set_statp(struct svc_rqst *rqstp, __be32 *statp)
>  void
>  nfsd41_set_cache_entry(struct nfsd4_compoundres *resp)
>  {
> -	struct nfs41_cache_entry *entry =
> -		&resp->current_ses->cs_slot->sl_cache_entry;
> +	struct nfs41_slot *slot = resp->current_ses->cs_slot;
> +	struct nfs41_cache_entry *entry = &slot->sl_cache_entry;
>  	struct svc_rqst *rqstp = resp->rqstp;
>  	struct kvec *resv = &rqstp->rq_res.head[0];
>  
>  	dprintk("--> %s entry %p\n", __func__, entry);
>  
> -	/* Don't cache a failed OP_SEQUENCE */
> -	if (resp->opcnt == 1 && resp->status)
> +	/* Don't cache a failed OP_SEQUENCE. The CREATE_SESSION slot has NULL
> +	 * sl_session pointer */

review 11-13: (The solo CREATE_SESSION ...) in parenthesis.

> +	if (resp->opcnt == 1 && resp->status && slot->sl_session)
>  		return;
>  	spin_lock(&entry->ce_lock);
>  	nfsd4_clear_respages(entry->ce_respages, entry->ce_resused);
> @@ -1215,11 +1220,12 @@ out_copy:
>  	clid->clientid.cl_boot = new->cl_clientid.cl_boot;
>  	clid->clientid.cl_id = new->cl_clientid.cl_id;
>  
> -	new->cl_seqid = clid->seqid = 1;
> +	new->cl_slot.sl_seqid = 0;
> +	clid->seqid = 1;
>  	nfsd4_set_ex_flags(new, clid);
>  
>  	dprintk("nfsd4_exchange_id seqid %d flags %x\n",
> -		new->cl_seqid, new->cl_exchange_flags);
> +		new->cl_slot.sl_seqid, new->cl_exchange_flags);
>  	status = nfs_ok;
>  
>  out:
> @@ -1262,6 +1268,8 @@ nfsd4_create_session(struct svc_rqst *rqstp,
>  {
>  	u32 ip_addr = svc_addr_in(rqstp)->sin_addr.s_addr;
>  	struct nfs4_client *conf, *unconf;
> +	struct current_session *c_ses = cstate->current_ses;
> +	struct nfs41_slot *slot = NULL;
>  	__u32   max_blocksize = svc_max_payload(rqstp);
>  	int status = 0;
>  
> @@ -1277,19 +1285,20 @@ nfsd4_create_session(struct svc_rqst *rqstp,
>  		goto out;
>  	}
>  	if (conf) {
> -		status = nfs_ok;
> -		if (conf->cl_seqid == cr_ses->seqid) {
> +		slot = &conf->cl_slot;
> +		status = check_slot_seqid(cr_ses->seqid, slot);
> +		if (status == nfserr_replay_cache) {
>  			dprintk("Got a create_session replay! seqid= %d\n",
> -				conf->cl_seqid);
> +				slot->sl_seqid);
>  			goto out_replay;

review 11-13: change this the same way as for SEQUENCE

> -		} else if (cr_ses->seqid != conf->cl_seqid + 1) {
> +		} else if (cr_ses->seqid != conf->cl_slot.sl_seqid + 1) {
>  			status = nfserr_seq_misordered;
>  			dprintk("Sequence misordered!\n");
>  			dprintk("Expected seqid= %d but got seqid= %d\n",
> -				conf->cl_seqid, cr_ses->seqid);
> +				slot->sl_seqid, cr_ses->seqid);
>  			goto out;
>  		}
> -		conf->cl_seqid++;
> +		conf->cl_slot.sl_seqid++;
>  	} else if (unconf) {
>  		if (!same_creds(&unconf->cl_cred, &rqstp->rq_cred) ||
>  		    (ip_addr != unconf->cl_addr)) {
> @@ -1297,11 +1306,15 @@ nfsd4_create_session(struct svc_rqst *rqstp,
>  			goto out;
>  		}
>  
> -		if (unconf->cl_seqid != cr_ses->seqid) {
> +		slot = &unconf->cl_slot;
> +		status = check_slot_seqid(cr_ses->seqid, slot);
> +		if (status) {
> +			/* an unconfirmed replay returns misordered */
>  			status = nfserr_seq_misordered;
>  			goto out;
>  		}
>  
> +		slot->sl_seqid++; /* from 0 to 1 */
>  		move_to_confirmed(unconf);
>  
>  		/*
> @@ -1322,9 +1335,8 @@ nfsd4_create_session(struct svc_rqst *rqstp,
>  
>  	status = alloc_init_session(conf, cr_ses);
>  
> -out_replay:
>  	memcpy(cr_ses->sessionid, conf->cl_sessionid, 16);
> -	cr_ses->seqid = conf->cl_seqid;
> +	cr_ses->seqid = slot->sl_seqid;
>  	cr_ses->fore_channel.maxreq_sz = max_blocksize;
>  	cr_ses->fore_channel.maxresp_sz = max_blocksize;
>  	cr_ses->fore_channel.maxresp_cached = max_blocksize;
> @@ -1332,6 +1344,9 @@ out_replay:
>  	cr_ses->back_channel.maxresp_sz = max_blocksize;
>  	cr_ses->back_channel.maxresp_cached = max_blocksize;
>  
> +out_replay:
> +	nfs41_set_slot_state(slot, NFS4_SLOT_INPROGRESS);
> +	c_ses->cs_slot = slot;
>  out:
>  	nfs4_unlock_state();
>  	dprintk("%s returns %d\n", __func__, ntohl(status));
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index 4b4f5fb..9799ea1 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -3304,10 +3304,11 @@ nfs4svc_encode_compoundres(struct svc_rqst *rqstp, __be32 *p, struct nfsd4_compo
>  	BUG_ON(iov->iov_len > PAGE_SIZE);
>  #ifdef CONFIG_NFSD_V4_1
>  	if (resp->minorversion == 1) {
> +		struct nfs41_slot *slot;
>  		/*
>  		* Always have a current session.
>  		* The cs_slot is not set on SEQUENCE error, EXCHANGE_ID,
> -		* solo CREATE_SESSION, DESTROY_SESSION compounds.
> +		* and solo DESTROY_SESSION compounds.
>  		*/
>  		if (!resp->current_ses || !resp->current_ses->cs_slot) {
>  			dprintk("%s NO CS_SLOT. current_ses %p\n", __func__,
> @@ -3321,9 +3322,10 @@ nfs4svc_encode_compoundres(struct svc_rqst *rqstp, __be32 *p, struct nfsd4_compo
>  			nfsd41_set_cache_entry(resp);
>  
>  		dprintk("%s: SET SLOT STATE TO AVAILABLE\n", __func__);
> -		nfs41_set_slot_state(resp->current_ses->cs_slot,
> -				     NFS4_SLOT_AVAILABLE);
> -		nfs41_put_session(resp->current_ses->cs_slot->sl_session);
> +		slot = resp->current_ses->cs_slot;
> +		nfs41_set_slot_state(slot, NFS4_SLOT_AVAILABLE);
> +		if (slot->sl_session)
> +			nfs41_put_session(slot->sl_session);
>  free_current_ses:
>  		nfsd41_current_session_free(resp->current_ses);
>  }
> diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
> index e4dc32f..a336687 100644
> --- a/include/linux/nfsd/state.h
> +++ b/include/linux/nfsd/state.h
> @@ -233,7 +233,7 @@ struct nfs4_client {
>  	atomic_t		cl_count;	/* ref count */
>  	u32			cl_firststate;	/* recovery dir creation */
>  #if defined(CONFIG_NFSD_V4_1)
> -	u32			cl_seqid;	/* seqid for create_session */
> +	struct nfs41_slot	cl_slot;	/* create_session slot */
>  	u32			cl_exchange_flags;
>  	nfs41_sessionid		cl_sessionid;
>  

[RFC 51/51] nfsd41: print DRC statistics

[Benny Halevy]

From: Andy Adamson <andros@netapp.com>

Add page-use counters to the cache entry. Add a proc file to trigger printing
the session, slot, and DRC statistics for a client ipv4 address.

To print the DRC stats for an IP address:
  echo "192.168.1.106" > /proc/fs/nfsd/nfsv41_DRC_printstats

To reset all slot counters on all clients:
  echo "0.0.0.0" > /proc/fs/nfsd/nfsv41_DRC_printstats

Signed-off-by: Andy Adamson<andros@netapp.com>
Signed-off-by: Benny Halevy <bhalevy@panasas.com>
---
 fs/nfsd/nfs4state.c        |   95 ++++++++++++++++++++++++++++++++++++++++++++
 fs/nfsd/nfsctl.c           |   52 ++++++++++++++++++++++++
 include/linux/nfsd/nfsd.h  |    8 ++++
 include/linux/nfsd/state.h |    4 ++
 4 files changed, 159 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 0d8fde7..3a8d5c7 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -1018,6 +1018,100 @@ nfsd41_set_statp(struct svc_rqst *rqstp, __be32 *statp)
 	resp->statp = statp;
 }
 
+static void
+nfsd4_print_slotstats(struct nfs41_slot *slot)
+{
+	struct nfs41_cache_entry *e = &slot->sl_cache_entry;
+
+	printk(KERN_INFO " sequence number               %d\n", slot->sl_seqid);
+	printk(KERN_INFO " pages cached                  %d\n", e->ce_resused);
+	printk(KERN_INFO " total pages cached            %lu\n",
+		e->ce_total_pages);
+	printk(KERN_INFO " lowest # pages per request    %lu\n",
+		e->ce_lowest_pages);
+	printk(KERN_INFO " higest # pages per request    %lu\n",
+		e->ce_highest_pages);
+}
+
+static void
+nfsd4_reset_slotstats(struct nfs41_slot *slot)
+{
+	struct nfs41_cache_entry *e = &slot->sl_cache_entry;
+
+	e->ce_total_pages = e->ce_lowest_pages = e->ce_highest_pages = 0;
+}
+
+static void
+nfsd4_drc_printstats(struct nfs4_client *clp, int reset)
+{
+	struct nfs41_session *ses;
+	struct nfsd_sessionid *id;
+	int i;
+
+	printk(KERN_INFO "\n----- DRC Stats for %u.%u.%u.%u -----\n",
+		NIPQUAD(clp->cl_addr));
+	if (reset)
+		printk(KERN_INFO "         ------ RESET ------\n");
+	list_for_each_entry(ses, &clp->cl_sessions, se_perclnt) {
+		id = (struct nfsd_sessionid *)ses;
+		printk(KERN_INFO " -- Sessionid:[clientid] %x%x [boot]"
+			"%x [seq] %x -- \n",
+			id->clientid.cl_boot, id->clientid.cl_id,
+			id->boot_time, id->sequence);
+		printk(KERN_INFO "   Forward Channel\n\n");
+		printk(KERN_INFO "     headerpad          %d\n",
+			ses->se_fheaderpad_sz);
+		printk(KERN_INFO "     maxreq size        %d\n",
+			ses->se_fmaxreq_sz);
+		printk(KERN_INFO "     maxresp size       %d\n",
+			ses->se_fmaxresp_sz);
+		printk(KERN_INFO "     maxresp (numslots) %d\n",
+			ses->se_fmaxresp_cached);
+		printk(KERN_INFO "     maxops             %d\n",
+			ses->se_fmaxops);
+		for (i = 0; i < ses->se_forward.ch_maxreqs; i++) {
+			printk(KERN_INFO "\n----- Slot # %d -------\n", i);
+			if (reset)
+				nfsd4_reset_slotstats(&ses->se_slots[i]);
+			else
+				nfsd4_print_slotstats(&ses->se_slots[i]);
+		}
+	}
+	printk(KERN_INFO "\n\n   Create Session Slot\n\n");
+	nfsd4_print_slotstats(&clp->cl_slot);
+	printk(KERN_INFO "\n--------- DRC Slot Stats End ---------\n\n");
+}
+
+void
+nfsd4_do_drcstats(__be32 ip4addr)
+{
+	struct nfs4_client *clp = NULL;
+	int i;
+
+	for (i = 0; i < CLIENT_HASH_SIZE; i++) {
+		list_for_each_entry(clp, &conf_id_hashtbl[i], cl_idhash) {
+			if (ip4addr == 0)
+				nfsd4_drc_printstats(clp, 1);
+			else if (!memcmp(&clp->cl_addr, &ip4addr,
+							sizeof(__be32))) {
+				nfsd4_drc_printstats(clp, 0);
+				break;
+			}
+		}
+	}
+}
+
+static void
+nfs4_set_drc_stats(struct svc_rqst *rqstp, struct nfs41_cache_entry *entry)
+{
+	if (entry->ce_lowest_pages == 0 ||
+	    rqstp->rq_resused < entry->ce_lowest_pages)
+		entry->ce_lowest_pages = rqstp->rq_resused;
+	if (rqstp->rq_resused > entry->ce_highest_pages)
+		entry->ce_highest_pages = rqstp->rq_resused;
+	entry->ce_total_pages += rqstp->rq_resused;
+}
+
 /*
  * Cache the reply pages, clearing the previous pages.
  * Store the base and length of the rq_req.head[0] page
@@ -1038,6 +1132,7 @@ nfsd41_set_cache_entry(struct nfsd4_compoundres *resp)
 	if (resp->opcnt == 1 && resp->status && slot->sl_session)
 		return;
 	spin_lock(&entry->ce_lock);
+	nfs4_set_drc_stats(rqstp, entry);
 	nfsd4_clear_respages(entry->ce_respages, entry->ce_resused);
 	nfsd4_cache_rqst_pages(rqstp, entry->ce_respages, &entry->ce_resused);
 	entry->ce_status = resp->status;
diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index e3f9783..e7f850a 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -71,6 +71,9 @@ enum {
 	NFSD_Leasetime,
 	NFSD_RecoveryDir,
 #endif
+#ifdef CONFIG_NFSD_V4_1
+	NFSD_DRC_print_stats,
+#endif /* CONFIG_NFSD_V4_1 */
 };
 
 /*
@@ -93,6 +96,9 @@ static ssize_t write_maxblksize(struct file *file, char *buf, size_t size);
 static ssize_t write_leasetime(struct file *file, char *buf, size_t size);
 static ssize_t write_recoverydir(struct file *file, char *buf, size_t size);
 #endif
+#ifdef CONFIG_NFSD_V4_1
+static ssize_t write_drcstats(struct file *file, char *buf, size_t size);
+#endif /* CONFIG_NFSD_V4_1 */
 
 static ssize_t failover_unlock_ip(struct file *file, char *buf, size_t size);
 static ssize_t failover_unlock_fs(struct file *file, char *buf, size_t size);
@@ -117,6 +123,9 @@ static ssize_t (*write_op[])(struct file *, char *, size_t) = {
 	[NFSD_Leasetime] = write_leasetime,
 	[NFSD_RecoveryDir] = write_recoverydir,
 #endif
+#ifdef CONFIG_NFSD_V4_1
+	[NFSD_DRC_print_stats] = write_drcstats,
+#endif /* CONFIG_NFSD_V4_1 */
 };
 
 static ssize_t nfsctl_transaction_write(struct file *file, const char __user *buf, size_t size, loff_t *pos)
@@ -727,6 +736,45 @@ static ssize_t write_maxblksize(struct file *file, char *buf, size_t size)
 	return sprintf(buf, "%d\n", nfsd_max_blksize);
 }
 
+#ifdef CONFIG_NFSD_V4_1
+static ssize_t __write_drcstats(struct file *file, char *buf, size_t size)
+{
+	char *mesg = buf;
+	char *ip4addr = NULL;
+	int len;
+	__be32 cl_addr = 0;
+	u8 *addr = (u8 *)&cl_addr;
+
+	if (size > 0) {
+		if (size > PATH_MAX || buf[size-1] != '\n')
+			return -EINVAL;
+		buf[size-1] = 0;
+
+		ip4addr = mesg;
+		len = qword_get(&mesg, ip4addr, size);
+		if (len <= 0)
+			return -EINVAL;
+		if (!(in4_pton(ip4addr, len, addr, '\0', NULL)))
+			return -EINVAL;
+		nfsd4_do_drcstats(cl_addr);
+	}
+	if (!ip4addr)
+		return 0;
+	sprintf(buf, "%s\n", ip4addr);
+	return strlen(buf);
+}
+
+static ssize_t write_drcstats(struct file *file, char *buf, size_t size)
+{
+	ssize_t rv;
+
+	mutex_lock(&nfsd_mutex);
+	rv = __write_drcstats(file, buf, size);
+	mutex_unlock(&nfsd_mutex);
+	return rv;
+}
+
+#endif /* CONFIG_NFSD_V4_1 */
 #ifdef CONFIG_NFSD_V4
 extern time_t nfs4_leasetime(void);
 
@@ -830,6 +878,10 @@ static int nfsd_fill_super(struct super_block * sb, void * data, int silent)
 		[NFSD_Leasetime] = {"nfsv4leasetime", &transaction_ops, S_IWUSR|S_IRUSR},
 		[NFSD_RecoveryDir] = {"nfsv4recoverydir", &transaction_ops, S_IWUSR|S_IRUSR},
 #endif
+#ifdef CONFIG_NFSD_V4_1
+		[NFSD_DRC_print_stats] =
+		   {"nfsv41_DRC_printstats", &transaction_ops, S_IWUSR|S_IRUSR},
+#endif /* CONFIG_NFSD_V4_1 */
 		/* last one */ {""}
 	};
 	return simple_fill_super(sb, 0x6e667364, nfsd_files);
diff --git a/include/linux/nfsd/nfsd.h b/include/linux/nfsd/nfsd.h
index c8bbe94..e31a4f6 100644
--- a/include/linux/nfsd/nfsd.h
+++ b/include/linux/nfsd/nfsd.h
@@ -181,6 +181,14 @@ static inline void nfs4_reset_lease(time_t leasetime) { }
 static inline int nfs4_reset_recoverydir(char *recdir) { return 0; }
 #endif
 
+#ifdef CONFIG_NFSD_V4_1
+void nfsd4_do_drcstats(__be32 ip4addr);
+#else
+static inline void nfsd4_do_drcstats(__be32 client_ip4addr) {}
+#endif /* CONFIG_NFSD_V4_1 */
+
+
+
 /*
  * lockd binding
  */
diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
index a336687..03aaba4 100644
--- a/include/linux/nfsd/state.h
+++ b/include/linux/nfsd/state.h
@@ -131,6 +131,10 @@ struct nfs41_cache_entry {
 	struct kvec	ce_datav; /* encoded NFSv4.1 data in rq_res.head[0] */
 	struct page	*ce_respages[RPCSVC_MAXPAGES];
 	short		ce_resused;
+	/* slot page stats */
+	unsigned long	ce_total_pages;
+	unsigned long	ce_lowest_pages;
+	unsigned long	ce_highest_pages;
 };
 
 /*
-- 
1.6.0.2

Re: [pnfs] [RFC 51/51] nfsd41: print DRC statistics

[Benny Halevy]

On Nov. 10, 2008, 22:58 +0200, Benny Halevy <bhalevy@panasas.com> wrote:
> From: Andy Adamson <andros@netapp.com>
> 
> Add page-use counters to the cache entry. Add a proc file to trigger printing
> the session, slot, and DRC statistics for a client ipv4 address.
> 
> To print the DRC stats for an IP address:
>   echo "192.168.1.106" > /proc/fs/nfsd/nfsv41_DRC_printstats
> 
> To reset all slot counters on all clients:
>   echo "0.0.0.0" > /proc/fs/nfsd/nfsv41_DRC_printstats
> 
> Signed-off-by: Andy Adamson<andros@netapp.com>
> Signed-off-by: Benny Halevy <bhalevy@panasas.com>

review 11-13: Bruce says leave it out for now.

> ---
>  fs/nfsd/nfs4state.c        |   95 ++++++++++++++++++++++++++++++++++++++++++++
>  fs/nfsd/nfsctl.c           |   52 ++++++++++++++++++++++++
>  include/linux/nfsd/nfsd.h  |    8 ++++
>  include/linux/nfsd/state.h |    4 ++
>  4 files changed, 159 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 0d8fde7..3a8d5c7 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -1018,6 +1018,100 @@ nfsd41_set_statp(struct svc_rqst *rqstp, __be32 *statp)
>  	resp->statp = statp;
>  }
>  
> +static void
> +nfsd4_print_slotstats(struct nfs41_slot *slot)
> +{
> +	struct nfs41_cache_entry *e = &slot->sl_cache_entry;
> +
> +	printk(KERN_INFO " sequence number               %d\n", slot->sl_seqid);
> +	printk(KERN_INFO " pages cached                  %d\n", e->ce_resused);
> +	printk(KERN_INFO " total pages cached            %lu\n",
> +		e->ce_total_pages);
> +	printk(KERN_INFO " lowest # pages per request    %lu\n",
> +		e->ce_lowest_pages);
> +	printk(KERN_INFO " higest # pages per request    %lu\n",
> +		e->ce_highest_pages);
> +}
> +
> +static void
> +nfsd4_reset_slotstats(struct nfs41_slot *slot)
> +{
> +	struct nfs41_cache_entry *e = &slot->sl_cache_entry;
> +
> +	e->ce_total_pages = e->ce_lowest_pages = e->ce_highest_pages = 0;
> +}
> +
> +static void
> +nfsd4_drc_printstats(struct nfs4_client *clp, int reset)
> +{
> +	struct nfs41_session *ses;
> +	struct nfsd_sessionid *id;
> +	int i;
> +
> +	printk(KERN_INFO "\n----- DRC Stats for %u.%u.%u.%u -----\n",
> +		NIPQUAD(clp->cl_addr));
> +	if (reset)
> +		printk(KERN_INFO "         ------ RESET ------\n");
> +	list_for_each_entry(ses, &clp->cl_sessions, se_perclnt) {
> +		id = (struct nfsd_sessionid *)ses;
> +		printk(KERN_INFO " -- Sessionid:[clientid] %x%x [boot]"
> +			"%x [seq] %x -- \n",
> +			id->clientid.cl_boot, id->clientid.cl_id,
> +			id->boot_time, id->sequence);
> +		printk(KERN_INFO "   Forward Channel\n\n");
> +		printk(KERN_INFO "     headerpad          %d\n",
> +			ses->se_fheaderpad_sz);
> +		printk(KERN_INFO "     maxreq size        %d\n",
> +			ses->se_fmaxreq_sz);
> +		printk(KERN_INFO "     maxresp size       %d\n",
> +			ses->se_fmaxresp_sz);
> +		printk(KERN_INFO "     maxresp (numslots) %d\n",
> +			ses->se_fmaxresp_cached);
> +		printk(KERN_INFO "     maxops             %d\n",
> +			ses->se_fmaxops);
> +		for (i = 0; i < ses->se_forward.ch_maxreqs; i++) {
> +			printk(KERN_INFO "\n----- Slot # %d -------\n", i);
> +			if (reset)
> +				nfsd4_reset_slotstats(&ses->se_slots[i]);
> +			else
> +				nfsd4_print_slotstats(&ses->se_slots[i]);
> +		}
> +	}
> +	printk(KERN_INFO "\n\n   Create Session Slot\n\n");
> +	nfsd4_print_slotstats(&clp->cl_slot);
> +	printk(KERN_INFO "\n--------- DRC Slot Stats End ---------\n\n");
> +}
> +
> +void
> +nfsd4_do_drcstats(__be32 ip4addr)
> +{
> +	struct nfs4_client *clp = NULL;
> +	int i;
> +
> +	for (i = 0; i < CLIENT_HASH_SIZE; i++) {
> +		list_for_each_entry(clp, &conf_id_hashtbl[i], cl_idhash) {
> +			if (ip4addr == 0)
> +				nfsd4_drc_printstats(clp, 1);
> +			else if (!memcmp(&clp->cl_addr, &ip4addr,
> +							sizeof(__be32))) {
> +				nfsd4_drc_printstats(clp, 0);
> +				break;
> +			}
> +		}
> +	}
> +}
> +
> +static void
> +nfs4_set_drc_stats(struct svc_rqst *rqstp, struct nfs41_cache_entry *entry)
> +{
> +	if (entry->ce_lowest_pages == 0 ||
> +	    rqstp->rq_resused < entry->ce_lowest_pages)
> +		entry->ce_lowest_pages = rqstp->rq_resused;
> +	if (rqstp->rq_resused > entry->ce_highest_pages)
> +		entry->ce_highest_pages = rqstp->rq_resused;
> +	entry->ce_total_pages += rqstp->rq_resused;
> +}
> +
>  /*
>   * Cache the reply pages, clearing the previous pages.
>   * Store the base and length of the rq_req.head[0] page
> @@ -1038,6 +1132,7 @@ nfsd41_set_cache_entry(struct nfsd4_compoundres *resp)
>  	if (resp->opcnt == 1 && resp->status && slot->sl_session)
>  		return;
>  	spin_lock(&entry->ce_lock);
> +	nfs4_set_drc_stats(rqstp, entry);
>  	nfsd4_clear_respages(entry->ce_respages, entry->ce_resused);
>  	nfsd4_cache_rqst_pages(rqstp, entry->ce_respages, &entry->ce_resused);
>  	entry->ce_status = resp->status;
> diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
> index e3f9783..e7f850a 100644
> --- a/fs/nfsd/nfsctl.c
> +++ b/fs/nfsd/nfsctl.c
> @@ -71,6 +71,9 @@ enum {
>  	NFSD_Leasetime,
>  	NFSD_RecoveryDir,
>  #endif
> +#ifdef CONFIG_NFSD_V4_1
> +	NFSD_DRC_print_stats,
> +#endif /* CONFIG_NFSD_V4_1 */
>  };
>  
>  /*
> @@ -93,6 +96,9 @@ static ssize_t write_maxblksize(struct file *file, char *buf, size_t size);
>  static ssize_t write_leasetime(struct file *file, char *buf, size_t size);
>  static ssize_t write_recoverydir(struct file *file, char *buf, size_t size);
>  #endif
> +#ifdef CONFIG_NFSD_V4_1
> +static ssize_t write_drcstats(struct file *file, char *buf, size_t size);
> +#endif /* CONFIG_NFSD_V4_1 */
>  
>  static ssize_t failover_unlock_ip(struct file *file, char *buf, size_t size);
>  static ssize_t failover_unlock_fs(struct file *file, char *buf, size_t size);
> @@ -117,6 +123,9 @@ static ssize_t (*write_op[])(struct file *, char *, size_t) = {
>  	[NFSD_Leasetime] = write_leasetime,
>  	[NFSD_RecoveryDir] = write_recoverydir,
>  #endif
> +#ifdef CONFIG_NFSD_V4_1
> +	[NFSD_DRC_print_stats] = write_drcstats,
> +#endif /* CONFIG_NFSD_V4_1 */
>  };
>  
>  static ssize_t nfsctl_transaction_write(struct file *file, const char __user *buf, size_t size, loff_t *pos)
> @@ -727,6 +736,45 @@ static ssize_t write_maxblksize(struct file *file, char *buf, size_t size)
>  	return sprintf(buf, "%d\n", nfsd_max_blksize);
>  }
>  
> +#ifdef CONFIG_NFSD_V4_1
> +static ssize_t __write_drcstats(struct file *file, char *buf, size_t size)
> +{
> +	char *mesg = buf;
> +	char *ip4addr = NULL;
> +	int len;
> +	__be32 cl_addr = 0;
> +	u8 *addr = (u8 *)&cl_addr;
> +
> +	if (size > 0) {
> +		if (size > PATH_MAX || buf[size-1] != '\n')
> +			return -EINVAL;
> +		buf[size-1] = 0;
> +
> +		ip4addr = mesg;
> +		len = qword_get(&mesg, ip4addr, size);
> +		if (len <= 0)
> +			return -EINVAL;
> +		if (!(in4_pton(ip4addr, len, addr, '\0', NULL)))
> +			return -EINVAL;
> +		nfsd4_do_drcstats(cl_addr);
> +	}
> +	if (!ip4addr)
> +		return 0;
> +	sprintf(buf, "%s\n", ip4addr);
> +	return strlen(buf);
> +}
> +
> +static ssize_t write_drcstats(struct file *file, char *buf, size_t size)
> +{
> +	ssize_t rv;
> +
> +	mutex_lock(&nfsd_mutex);
> +	rv = __write_drcstats(file, buf, size);
> +	mutex_unlock(&nfsd_mutex);
> +	return rv;
> +}
> +
> +#endif /* CONFIG_NFSD_V4_1 */
>  #ifdef CONFIG_NFSD_V4
>  extern time_t nfs4_leasetime(void);
>  
> @@ -830,6 +878,10 @@ static int nfsd_fill_super(struct super_block * sb, void * data, int silent)
>  		[NFSD_Leasetime] = {"nfsv4leasetime", &transaction_ops, S_IWUSR|S_IRUSR},
>  		[NFSD_RecoveryDir] = {"nfsv4recoverydir", &transaction_ops, S_IWUSR|S_IRUSR},
>  #endif
> +#ifdef CONFIG_NFSD_V4_1
> +		[NFSD_DRC_print_stats] =
> +		   {"nfsv41_DRC_printstats", &transaction_ops, S_IWUSR|S_IRUSR},
> +#endif /* CONFIG_NFSD_V4_1 */
>  		/* last one */ {""}
>  	};
>  	return simple_fill_super(sb, 0x6e667364, nfsd_files);
> diff --git a/include/linux/nfsd/nfsd.h b/include/linux/nfsd/nfsd.h
> index c8bbe94..e31a4f6 100644
> --- a/include/linux/nfsd/nfsd.h
> +++ b/include/linux/nfsd/nfsd.h
> @@ -181,6 +181,14 @@ static inline void nfs4_reset_lease(time_t leasetime) { }
>  static inline int nfs4_reset_recoverydir(char *recdir) { return 0; }
>  #endif
>  
> +#ifdef CONFIG_NFSD_V4_1
> +void nfsd4_do_drcstats(__be32 ip4addr);
> +#else
> +static inline void nfsd4_do_drcstats(__be32 client_ip4addr) {}
> +#endif /* CONFIG_NFSD_V4_1 */
> +
> +
> +
>  /*
>   * lockd binding
>   */
> diff --git a/include/linux/nfsd/state.h b/include/linux/nfsd/state.h
> index a336687..03aaba4 100644
> --- a/include/linux/nfsd/state.h
> +++ b/include/linux/nfsd/state.h
> @@ -131,6 +131,10 @@ struct nfs41_cache_entry {
>  	struct kvec	ce_datav; /* encoded NFSv4.1 data in rq_res.head[0] */
>  	struct page	*ce_respages[RPCSVC_MAXPAGES];
>  	short		ce_resused;
> +	/* slot page stats */
> +	unsigned long	ce_total_pages;
> +	unsigned long	ce_lowest_pages;
> +	unsigned long	ce_highest_pages;
>  };
>  
>  /*