[PATCH 0/4] sunrpc/nfs: better handling for expired krb5 credentials

[PATCH 0/4] sunrpc/nfs: better handling for expired krb5 credentials

[Jeff Layton]

When someone deploys kerberized NFS, they usually will quickly run
across a major problem. As soon as their credentials expire, all RPCs
start failing with -EACCES errors. This makes it really difficult to
have any sort of long-running job since you have to proactively kinit
before your TGT expires. If you miss doing so, then your job may start
getting errors unexpectedly.

This patchset represents a first pass at fixing this. The idea here is
to distinguish between the situation where someone has an expired
credential cache and someone that has no credential cache at all. In the
latter case, we want to have the RPC return -EACCES (just like it does
today), in the former case we want to return a different error that will
make the NFS layer delay and retry the call instead of erroring out
(-EKEYEXPIRED).

This patchset is for the kernel patches. To make this work, gssd will
also need to be fixed to send different errors in these situations. That
patch will follow this set.

Jeff Layton (4):
  sunrpc: parse and return errors reported by gssd
  nfs4: handle -EKEYEXPIRED errors from RPC layer
  nfs: handle NFSv3 -EKEYEXPIRED errors as we would -EJUKEBOX
  nfs: handle NFSv2 -EKEYEXPIRED returns from RPC layer appropriately

 fs/nfs/nfs3proc.c              |    9 ++++---
 fs/nfs/nfs4proc.c              |   11 ++++++++-
 fs/nfs/nfs4state.c             |    1 +
 fs/nfs/proc.c                  |   41 ++++++++++++++++++++++++++++++++++++++++
 net/sunrpc/auth_gss/auth_gss.c |   11 ++++++++-
 5 files changed, 65 insertions(+), 8 deletions(-)

_______________________________________________
NFSv4 mailing list
NFSv4@linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

[PATCH 1/4] sunrpc: parse and return errors reported by gssd

[Jeff Layton]

The kernel currently ignores any error code sent by gssd and always
considers it to be -EACCES. In order to better handle the situation of
an expired KRB5 TGT, the kernel needs to be able to parse and deal with
the errors that gssd sends. Aside from -EACCES the only error we care
about is -EKEYEXPIRED, which we're using to indicate that the upper
layers should retry the call a little later.

To maintain backward compatibility with older gssd's, any error other
than -EKEYEXPIRED is interpreted as -EACCES.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 net/sunrpc/auth_gss/auth_gss.c |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/net/sunrpc/auth_gss/auth_gss.c b/net/sunrpc/auth_gss/auth_gss.c
index 66cb89c..11f2915 100644
--- a/net/sunrpc/auth_gss/auth_gss.c
+++ b/net/sunrpc/auth_gss/auth_gss.c
@@ -206,8 +206,14 @@ gss_fill_context(const void *p, const void *end, struct gss_cl_ctx *ctx, struct
 	ctx->gc_win = window_size;
 	/* gssd signals an error by passing ctx->gc_win = 0: */
 	if (ctx->gc_win == 0) {
-		/* in which case, p points to  an error code which we ignore */
-		p = ERR_PTR(-EACCES);
+		/*
+		 * in which case, p points to an error code. Anything other
+		 * than -EKEYEXPIRED gets converted to -EACCES.
+		 */
+		p = simple_get_bytes(p, end, &ret, sizeof(ret));
+		if (!IS_ERR(p))
+			p = (ret == -EKEYEXPIRED) ? ERR_PTR(-EKEYEXPIRED) :
+						    ERR_PTR(-EACCES);
 		goto err;
 	}
 	/* copy the opaque wire context */
@@ -646,6 +652,7 @@ gss_pipe_downcall(struct file *filp, const char __user *src, size_t mlen)
 		err = PTR_ERR(p);
 		switch (err) {
 		case -EACCES:
+		case -EKEYEXPIRED:
 			gss_msg->msg.errno = err;
 			err = mlen;
 			break;
-- 
1.6.5.2

Re: [PATCH 1/4] sunrpc: parse and return errors reported by gssd

[Trond Myklebust]

On Thu, 2010-01-07 at 09:42 -0500, Jeff Layton wrote:

> @@ -646,6 +652,7 @@ gss_pipe_downcall(struct file *filp, const char __user *src, size_t mlen)
>  		err = PTR_ERR(p);
>  		switch (err) {
>  		case -EACCES:
> +		case -EKEYEXPIRED:
>  			gss_msg->msg.errno = err;
>  			err = mlen;
>  			break;

              ^^^^^^^^^^^^^^^^^^^^^ This doesn't appear to want to apply
to mainline. Did you perhaps skip a patch?

Trond

Re: [PATCH 1/4] sunrpc: parse and return errors reported by gssd

[Jeff Layton]

On Thu, 07 Jan 2010 14:11:56 -0500
Trond Myklebust <Trond.Myklebust@netapp.com> wrote:

> On Thu, 2010-01-07 at 09:42 -0500, Jeff Layton wrote:
> 
> > @@ -646,6 +652,7 @@ gss_pipe_downcall(struct file *filp, const char __user *src, size_t mlen)
> >  		err = PTR_ERR(p);
> >  		switch (err) {
> >  		case -EACCES:
> > +		case -EKEYEXPIRED:
> >  			gss_msg->msg.errno = err;
> >  			err = mlen;
> >  			break;
> 
>               ^^^^^^^^^^^^^^^^^^^^^ This doesn't appear to want to apply
> to mainline. Did you perhaps skip a patch?
> 
> Trond

Sorry, I should have made it clear...

This set is based on your linux-next tree. Specifically, it depends on
commit 486bad2e40e938cd68fd853b7a9fa3115a9d3a4a in your tree:

    sunrpc: on successful gss error pipe write, don't return error

...I didn't repost that one since I figured you already had it queued
up for 2.6.34.

Thanks,
-- 
Jeff Layton <jlayton@redhat.com>

Re: [PATCH 1/4] sunrpc: parse and return errors reported by gssd

[Trond Myklebust]

On Thu, 2010-01-07 at 14:21 -0500, Jeff Layton wrote: 
> On Thu, 07 Jan 2010 14:11:56 -0500
> Trond Myklebust <Trond.Myklebust@netapp.com> wrote:
> 
> > On Thu, 2010-01-07 at 09:42 -0500, Jeff Layton wrote:
> > 
> > > @@ -646,6 +652,7 @@ gss_pipe_downcall(struct file *filp, const char __user *src, size_t mlen)
> > >  		err = PTR_ERR(p);
> > >  		switch (err) {
> > >  		case -EACCES:
> > > +		case -EKEYEXPIRED:
> > >  			gss_msg->msg.errno = err;
> > >  			err = mlen;
> > >  			break;
> > 
> >               ^^^^^^^^^^^^^^^^^^^^^ This doesn't appear to want to apply
> > to mainline. Did you perhaps skip a patch?
> > 
> > Trond
> 
> Sorry, I should have made it clear...
> 
> This set is based on your linux-next tree. Specifically, it depends on
> commit 486bad2e40e938cd68fd853b7a9fa3115a9d3a4a in your tree:
> 
>     sunrpc: on successful gss error pipe write, don't return error
> 
> ...I didn't repost that one since I figured you already had it queued
> up for 2.6.34.
> 
> Thanks,

Oh, duh... No problem, I'll rebase onto that branch...

Trond

[PATCH 2/4] nfs4: handle -EKEYEXPIRED errors from RPC layer

[Jeff Layton]

If a KRB5 TGT ticket expires, we don't want to return an error
immediatel. If someone has a long running job and just forgets to run
"kinit" in time then this will make it fail.

Instead, we want to treat this situation as we would NFS4ERR_DELAY and
retry the upcall after delaying a bit with an exponential backoff.

This patch just makes any place that would handle NFS4ERR_DELAY also
handle -EKEYEXPIRED the same way. In the future, we may want to be more
sophisticated however and handle hard vs. soft mounts differently, or
specify some upper limit on how long we'll wait for a new TGT to be
acquired.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 fs/nfs/nfs4proc.c  |   11 +++++++++--
 fs/nfs/nfs4state.c |    1 +
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 198d51d..2e2dda2 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -285,6 +285,7 @@ static int nfs4_handle_exception(const struct nfs_server *server, int errorcode,
 			}
 		case -NFS4ERR_GRACE:
 		case -NFS4ERR_DELAY:
+		case -EKEYEXPIRED:
 			ret = nfs4_delay(server->client, &exception->timeout);
 			if (ret != 0)
 				break;
@@ -1161,7 +1162,7 @@ static int nfs4_do_open_reclaim(struct nfs_open_context *ctx, struct nfs4_state
 	int err;
 	do {
 		err = _nfs4_do_open_reclaim(ctx, state);
-		if (err != -NFS4ERR_DELAY)
+		if (err != -NFS4ERR_DELAY && err != -EKEYEXPIRED)
 			break;
 		nfs4_handle_exception(server, err, &exception);
 	} while (exception.retry);
@@ -1580,6 +1581,7 @@ static int nfs4_do_open_expired(struct nfs_open_context *ctx, struct nfs4_state
 			goto out;
 		case -NFS4ERR_GRACE:
 		case -NFS4ERR_DELAY:
+		case -EKEYEXPIRED:
 			nfs4_handle_exception(server, err, &exception);
 			err = 0;
 		}
@@ -3449,6 +3451,7 @@ _nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,
 			if (server)
 				nfs_inc_server_stats(server, NFSIOS_DELAY);
 		case -NFS4ERR_GRACE:
+		case -EKEYEXPIRED:
 			rpc_delay(task, NFS4_POLL_RETRY_MAX);
 			task->tk_status = 0;
 			return -EAGAIN;
@@ -3554,6 +3557,7 @@ int nfs4_proc_setclientid_confirm(struct nfs_client *clp, struct rpc_cred *cred)
 			case -NFS4ERR_RESOURCE:
 				/* The IBM lawyers misread another document! */
 			case -NFS4ERR_DELAY:
+			case -EKEYEXPIRED:
 				err = nfs4_delay(clp->cl_rpcclient, &timeout);
 		}
 	} while (err == 0);
@@ -4144,7 +4148,7 @@ static int nfs4_lock_reclaim(struct nfs4_state *state, struct file_lock *request
 		if (test_bit(NFS_DELEGATED_STATE, &state->flags) != 0)
 			return 0;
 		err = _nfs4_do_setlk(state, F_SETLK, request, NFS_LOCK_RECLAIM);
-		if (err != -NFS4ERR_DELAY)
+		if (err != -NFS4ERR_DELAY && err != -EKEYEXPIRED)
 			break;
 		nfs4_handle_exception(server, err, &exception);
 	} while (exception.retry);
@@ -4169,6 +4173,7 @@ static int nfs4_lock_expired(struct nfs4_state *state, struct file_lock *request
 			goto out;
 		case -NFS4ERR_GRACE:
 		case -NFS4ERR_DELAY:
+		case -EKEYEXPIRED:
 			nfs4_handle_exception(server, err, &exception);
 			err = 0;
 		}
@@ -4317,6 +4322,7 @@ int nfs4_lock_delegation_recall(struct nfs4_state *state, struct file_lock *fl)
 				err = 0;
 				goto out;
 			case -NFS4ERR_DELAY:
+			case -EKEYEXPIRED:
 				break;
 		}
 		err = nfs4_handle_exception(server, err, &exception);
@@ -4516,6 +4522,7 @@ static void nfs4_get_lease_time_done(struct rpc_task *task, void *calldata)
 	switch (task->tk_status) {
 	case -NFS4ERR_DELAY:
 	case -NFS4ERR_GRACE:
+	case -EKEYEXPIRED:
 		dprintk("%s Retry: tk_status %d\n", __func__, task->tk_status);
 		rpc_delay(task, NFS4_POLL_RETRY_MIN);
 		task->tk_status = 0;
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 6d263ed..423a4aa 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1314,6 +1314,7 @@ static void nfs4_set_lease_expired(struct nfs_client *clp, int status)
 		case -NFS4ERR_DELAY:
 		case -NFS4ERR_CLID_INUSE:
 		case -EAGAIN:
+		case -EKEYEXPIRED:
 			break;
 
 		case -NFS4ERR_NOT_SAME: /* FixMe: implement recovery
-- 
1.6.5.2

[PATCH 3/4] nfs: handle NFSv3 -EKEYEXPIRED errors as we would -EJUKEBOX

[Jeff Layton]

We're using -EKEYEXPIRED to indicate that a krb5 credcache contains an
expired ticket and that we should have the NFS layer retry the RPC call
instead of returning an error back to the caller. Handle this as we
would an -EJUKEBOX error return.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 fs/nfs/nfs3proc.c |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/fs/nfs/nfs3proc.c b/fs/nfs/nfs3proc.c
index 3f8881d..24992f0 100644
--- a/fs/nfs/nfs3proc.c
+++ b/fs/nfs/nfs3proc.c
@@ -22,14 +22,14 @@
 
 #define NFSDBG_FACILITY		NFSDBG_PROC
 
-/* A wrapper to handle the EJUKEBOX error message */
+/* A wrapper to handle the EJUKEBOX and EKEYEXPIRED error messages */
 static int
 nfs3_rpc_wrapper(struct rpc_clnt *clnt, struct rpc_message *msg, int flags)
 {
 	int res;
 	do {
 		res = rpc_call_sync(clnt, msg, flags);
-		if (res != -EJUKEBOX)
+		if (res != -EJUKEBOX && res != -EKEYEXPIRED)
 			break;
 		schedule_timeout_killable(NFS_JUKEBOX_RETRY_TIME);
 		res = -ERESTARTSYS;
@@ -42,9 +42,10 @@ nfs3_rpc_wrapper(struct rpc_clnt *clnt, struct rpc_message *msg, int flags)
 static int
 nfs3_async_handle_jukebox(struct rpc_task *task, struct inode *inode)
 {
-	if (task->tk_status != -EJUKEBOX)
+	if (task->tk_status != -EJUKEBOX && task->tk_status != -EKEYEXPIRED)
 		return 0;
-	nfs_inc_stats(inode, NFSIOS_DELAY);
+	if (task->tk_status == -EJUKEBOX)
+		nfs_inc_stats(inode, NFSIOS_DELAY);
 	task->tk_status = 0;
 	rpc_restart_call(task);
 	rpc_delay(task, NFS_JUKEBOX_RETRY_TIME);
-- 
1.6.5.2

[PATCH 4/4] nfs: handle NFSv2 -EKEYEXPIRED returns from RPC layer appropriately

[Jeff Layton]

Add a wrapper around rpc_call_sync that handles -EKEYEXPIRED errors from
the RPC layer as it would an -EJUKEBOX error if NFSv2 had such a thing.
Also, add a handler for that error for async calls that makes it
resubmit the RPC on -EKEYEXPIRED.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 fs/nfs/proc.c |   41 +++++++++++++++++++++++++++++++++++++++++
 1 files changed, 41 insertions(+), 0 deletions(-)

diff --git a/fs/nfs/proc.c b/fs/nfs/proc.c
index ef58385..2b6b6de 100644
--- a/fs/nfs/proc.c
+++ b/fs/nfs/proc.c
@@ -47,6 +47,39 @@
 #define NFSDBG_FACILITY		NFSDBG_PROC
 
 /*
+ * wrapper to handle the -EKEYEXPIRED error message. This should generally
+ * only happen if using krb5 auth and a user's TGT expires. NFSv2 doesn't
+ * support the NFSERR_JUKEBOX error code, but we handle this situation in the
+ * same way that we handle that error with NFSv3.
+ */
+static int
+nfs_rpc_wrapper(struct rpc_clnt *clnt, struct rpc_message *msg, int flags)
+{
+	int res;
+	do {
+		res = rpc_call_sync(clnt, msg, flags);
+		if (res != -EKEYEXPIRED)
+			break;
+		schedule_timeout_killable(NFS_JUKEBOX_RETRY_TIME);
+		res = -ERESTARTSYS;
+	} while (!fatal_signal_pending(current));
+	return res;
+}
+
+#define rpc_call_sync(clnt, msg, flags)	nfs_rpc_wrapper(clnt, msg, flags)
+
+static int
+nfs_async_handle_expired_key(struct rpc_task *task, struct inode *inode)
+{
+	if (task->tk_status != -EKEYEXPIRED)
+		return 0;
+	task->tk_status = 0;
+	rpc_restart_call(task);
+	rpc_delay(task, NFS_JUKEBOX_RETRY_TIME);
+	return 1;
+}
+
+/*
  * Bare-bones access to getattr: this is for nfs_read_super.
  */
 static int
@@ -307,6 +340,8 @@ nfs_proc_unlink_setup(struct rpc_message *msg, struct inode *dir)
 
 static int nfs_proc_unlink_done(struct rpc_task *task, struct inode *dir)
 {
+	if (nfs_async_handle_expired_key(task, dir))
+		return 0;
 	nfs_mark_for_revalidate(dir);
 	return 1;
 }
@@ -560,6 +595,9 @@ nfs_proc_pathconf(struct nfs_server *server, struct nfs_fh *fhandle,
 
 static int nfs_read_done(struct rpc_task *task, struct nfs_read_data *data)
 {
+	if (nfs_async_handle_expired_key(task, data->inode))
+		return -EAGAIN;
+
 	nfs_invalidate_atime(data->inode);
 	if (task->tk_status >= 0) {
 		nfs_refresh_inode(data->inode, data->res.fattr);
@@ -579,6 +617,9 @@ static void nfs_proc_read_setup(struct nfs_read_data *data, struct rpc_message *
 
 static int nfs_write_done(struct rpc_task *task, struct nfs_write_data *data)
 {
+	if (nfs_async_handle_expired_key(task, data->inode))
+		return -EAGAIN;
+
 	if (task->tk_status >= 0)
 		nfs_post_op_update_inode_force_wcc(data->inode, data->res.fattr);
 	return 0;
-- 
1.6.5.2

Re: [PATCH 4/4] nfs: handle NFSv2 -EKEYEXPIRED returns from RPC layer appropriately

[Jeff Layton]

On Thu,  7 Jan 2010 09:42:05 -0500
Jeff Layton <jlayton@redhat.com> wrote:

> Add a wrapper around rpc_call_sync that handles -EKEYEXPIRED errors from
> the RPC layer as it would an -EJUKEBOX error if NFSv2 had such a thing.
> Also, add a handler for that error for async calls that makes it
> resubmit the RPC on -EKEYEXPIRED.
> 
> Signed-off-by: Jeff Layton <jlayton@redhat.com>
> ---
>  fs/nfs/proc.c |   41 +++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 41 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/nfs/proc.c b/fs/nfs/proc.c
> index ef58385..2b6b6de 100644
> --- a/fs/nfs/proc.c
> +++ b/fs/nfs/proc.c
> @@ -47,6 +47,39 @@
>  #define NFSDBG_FACILITY		NFSDBG_PROC
>  
>  /*
> + * wrapper to handle the -EKEYEXPIRED error message. This should generally
> + * only happen if using krb5 auth and a user's TGT expires. NFSv2 doesn't
> + * support the NFSERR_JUKEBOX error code, but we handle this situation in the
> + * same way that we handle that error with NFSv3.
> + */
> +static int
> +nfs_rpc_wrapper(struct rpc_clnt *clnt, struct rpc_message *msg, int flags)
> +{
> +	int res;
> +	do {
> +		res = rpc_call_sync(clnt, msg, flags);
> +		if (res != -EKEYEXPIRED)
> +			break;
> +		schedule_timeout_killable(NFS_JUKEBOX_RETRY_TIME);
> +		res = -ERESTARTSYS;
> +	} while (!fatal_signal_pending(current));
> +	return res;
> +}
> +
> +#define rpc_call_sync(clnt, msg, flags)	nfs_rpc_wrapper(clnt, msg, flags)
> +
> +static int
> +nfs_async_handle_expired_key(struct rpc_task *task, struct inode *inode)
							^^^^^^^^^^^^^^^^
I did just notice something silly. The inode parameter here is unused
and can be eliminated. I'll plan to drop that from later versions of
this set.

> +{
> +	if (task->tk_status != -EKEYEXPIRED)
> +		return 0;
> +	task->tk_status = 0;
> +	rpc_restart_call(task);
> +	rpc_delay(task, NFS_JUKEBOX_RETRY_TIME);
> +	return 1;
> +}
> +
> +/*
>   * Bare-bones access to getattr: this is for nfs_read_super.
>   */
>  static int
> @@ -307,6 +340,8 @@ nfs_proc_unlink_setup(struct rpc_message *msg, struct inode *dir)
>  
>  static int nfs_proc_unlink_done(struct rpc_task *task, struct inode *dir)
>  {
> +	if (nfs_async_handle_expired_key(task, dir))
> +		return 0;
>  	nfs_mark_for_revalidate(dir);
>  	return 1;
>  }
> @@ -560,6 +595,9 @@ nfs_proc_pathconf(struct nfs_server *server, struct nfs_fh *fhandle,
>  
>  static int nfs_read_done(struct rpc_task *task, struct nfs_read_data *data)
>  {
> +	if (nfs_async_handle_expired_key(task, data->inode))
> +		return -EAGAIN;
> +
>  	nfs_invalidate_atime(data->inode);
>  	if (task->tk_status >= 0) {
>  		nfs_refresh_inode(data->inode, data->res.fattr);
> @@ -579,6 +617,9 @@ static void nfs_proc_read_setup(struct nfs_read_data *data, struct rpc_message *
>  
>  static int nfs_write_done(struct rpc_task *task, struct nfs_write_data *data)
>  {
> +	if (nfs_async_handle_expired_key(task, data->inode))
> +		return -EAGAIN;
> +
>  	if (task->tk_status >= 0)
>  		nfs_post_op_update_inode_force_wcc(data->inode, data->res.fattr);
>  	return 0;


-- 
Jeff Layton <jlayton@poochiereds.net>

Re: [PATCH 4/4] nfs: handle NFSv2 -EKEYEXPIRED returns from RPC layer appropriately

[Jeff Layton]

On Thu,  7 Jan 2010 09:42:05 -0500
Jeff Layton <jlayton@redhat.com> wrote:

> +
> +static int
> +nfs_async_handle_expired_key(struct rpc_task *task, struct inode *inode)

As I mentioned yesterday, the inode pointer here is unused and can be
removed. An updated patch follows:

---------------------[snip]----------------------

nfs: handle NFSv2 -EKEYEXPIRED returns from RPC layer appropriately

Add a wrapper around rpc_call_sync that handles -EKEYEXPIRED errors from
the RPC layer as it would an -EJUKEBOX error if NFSv2 had such a thing.
Also, add a handler for that error for async calls that makes it
resubmit the RPC on -EKEYEXPIRED.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 fs/nfs/proc.c |   41 +++++++++++++++++++++++++++++++++++++++++
 1 files changed, 41 insertions(+), 0 deletions(-)

diff --git a/fs/nfs/proc.c b/fs/nfs/proc.c
index ef58385..c752d94 100644
--- a/fs/nfs/proc.c
+++ b/fs/nfs/proc.c
@@ -47,6 +47,39 @@
 #define NFSDBG_FACILITY		NFSDBG_PROC
 
 /*
+ * wrapper to handle the -EKEYEXPIRED error message. This should generally
+ * only happen if using krb5 auth and a user's TGT expires. NFSv2 doesn't
+ * support the NFSERR_JUKEBOX error code, but we handle this situation in the
+ * same way that we handle that error with NFSv3.
+ */
+static int
+nfs_rpc_wrapper(struct rpc_clnt *clnt, struct rpc_message *msg, int flags)
+{
+	int res;
+	do {
+		res = rpc_call_sync(clnt, msg, flags);
+		if (res != -EKEYEXPIRED)
+			break;
+		schedule_timeout_killable(NFS_JUKEBOX_RETRY_TIME);
+		res = -ERESTARTSYS;
+	} while (!fatal_signal_pending(current));
+	return res;
+}
+
+#define rpc_call_sync(clnt, msg, flags)	nfs_rpc_wrapper(clnt, msg, flags)
+
+static int
+nfs_async_handle_expired_key(struct rpc_task *task)
+{
+	if (task->tk_status != -EKEYEXPIRED)
+		return 0;
+	task->tk_status = 0;
+	rpc_restart_call(task);
+	rpc_delay(task, NFS_JUKEBOX_RETRY_TIME);
+	return 1;
+}
+
+/*
  * Bare-bones access to getattr: this is for nfs_read_super.
  */
 static int
@@ -307,6 +340,8 @@ nfs_proc_unlink_setup(struct rpc_message *msg, struct inode *dir)
 
 static int nfs_proc_unlink_done(struct rpc_task *task, struct inode *dir)
 {
+	if (nfs_async_handle_expired_key(task))
+		return 0;
 	nfs_mark_for_revalidate(dir);
 	return 1;
 }
@@ -560,6 +595,9 @@ nfs_proc_pathconf(struct nfs_server *server, struct nfs_fh *fhandle,
 
 static int nfs_read_done(struct rpc_task *task, struct nfs_read_data *data)
 {
+	if (nfs_async_handle_expired_key(task))
+		return -EAGAIN;
+
 	nfs_invalidate_atime(data->inode);
 	if (task->tk_status >= 0) {
 		nfs_refresh_inode(data->inode, data->res.fattr);
@@ -579,6 +617,9 @@ static void nfs_proc_read_setup(struct nfs_read_data *data, struct rpc_message *
 
 static int nfs_write_done(struct rpc_task *task, struct nfs_write_data *data)
 {
+	if (nfs_async_handle_expired_key(task))
+		return -EAGAIN;
+
 	if (task->tk_status >= 0)
 		nfs_post_op_update_inode_force_wcc(data->inode, data->res.fattr);
 	return 0;
-- 
1.6.5.2