From: Trond Myklebust <trondmy@gmail.com>
To: Chuck Lever <chucklever@gmail.com>
Cc: Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH 20/27] SUNRPC: Treat the task and request as separate in the xprt_ops->send_request()
Date: Mon, 03 Sep 2018 13:47:56 -0400 [thread overview]
Message-ID: <0b84ef010e4b3e4d156b6b7ac60c5fb0c2a48627.camel@gmail.com> (raw)
In-Reply-To: <C7D44AFD-1BFF-4FB9-A4AB-AF60F836DA34@gmail.com>
On Mon, 2018-09-03 at 13:28 -0400, Chuck Lever wrote:
> > On Sep 3, 2018, at 11:29 AM, Trond Myklebust <trondmy@gmail.com>
> > wrote:
> >
> > When we shift to using the transmit queue, then the task that holds
> > the
> > write lock will not necessarily be the same as the one being
> > transmitted.
>
> Can you pass in just the rpc_rqst? Then @task would be available as
> xprt->snd_task.
No. The point is that when we add queueing, @req->rq_task will usually
be asleep on xprt->sending. Only the task that holds the XPRT_LOCK is
awake, and so it is the one that needs to be put asleep when we hit a
socket nospace/EAGAIN error.
We _could_ potentially avoid passing in the task at all, and just have
xs_nospace() put xprt->snd_task to sleep, but that would skirt the edge
of a transport layering violation.
>
> > Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> > ---
> > include/linux/sunrpc/xprt.h | 2 +-
> > net/sunrpc/xprt.c | 2 +-
> > net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 3 +--
> > net/sunrpc/xprtrdma/transport.c | 5 ++--
> > net/sunrpc/xprtsock.c | 27 +++++++++++------
> > -----
> > 5 files changed, 18 insertions(+), 21 deletions(-)
> >
> > diff --git a/include/linux/sunrpc/xprt.h
> > b/include/linux/sunrpc/xprt.h
> > index 81a6c2c8dfc7..6d91acfe0644 100644
> > --- a/include/linux/sunrpc/xprt.h
> > +++ b/include/linux/sunrpc/xprt.h
> > @@ -140,7 +140,7 @@ struct rpc_xprt_ops {
> > void (*connect)(struct rpc_xprt *xprt, struct
> > rpc_task *task);
> > int (*buf_alloc)(struct rpc_task *task);
> > void (*buf_free)(struct rpc_task *task);
> > - int (*send_request)(struct rpc_task *task);
> > + int (*send_request)(struct rpc_rqst *req, struct
> > rpc_task *task);
> > void (*set_retrans_timeout)(struct rpc_task
> > *task);
> > void (*timer)(struct rpc_xprt *xprt, struct
> > rpc_task *task);
> > void (*release_request)(struct rpc_task *task);
> > diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
> > index defc6167570a..3efcb69af526 100644
> > --- a/net/sunrpc/xprt.c
> > +++ b/net/sunrpc/xprt.c
> > @@ -1170,7 +1170,7 @@ void xprt_transmit(struct rpc_task *task)
> > }
> >
> > connect_cookie = xprt->connect_cookie;
> > - status = xprt->ops->send_request(task);
> > + status = xprt->ops->send_request(req, task);
> > trace_xprt_transmit(xprt, req->rq_xid, status);
> > if (status != 0) {
> > task->tk_status = status;
> > diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
> > b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
> > index 09b12b7568fe..d1618c70edb4 100644
> > --- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
> > +++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c
> > @@ -215,9 +215,8 @@ rpcrdma_bc_send_request(struct svcxprt_rdma
> > *rdma, struct rpc_rqst *rqst)
> > * connection.
> > */
> > static int
> > -xprt_rdma_bc_send_request(struct rpc_task *task)
> > +xprt_rdma_bc_send_request(struct rpc_rqst *rqst, struct rpc_task
> > *task)
> > {
> > - struct rpc_rqst *rqst = task->tk_rqstp;
> > struct svc_xprt *sxprt = rqst->rq_xprt->bc_xprt;
> > struct svcxprt_rdma *rdma;
> > int ret;
> > diff --git a/net/sunrpc/xprtrdma/transport.c
> > b/net/sunrpc/xprtrdma/transport.c
> > index 143ce2579ba9..fa684bf4d090 100644
> > --- a/net/sunrpc/xprtrdma/transport.c
> > +++ b/net/sunrpc/xprtrdma/transport.c
> > @@ -706,9 +706,8 @@ xprt_rdma_free(struct rpc_task *task)
> > * sent. Do not try to send this message again.
> > */
> > static int
> > -xprt_rdma_send_request(struct rpc_task *task)
> > +xprt_rdma_send_request(struct rpc_rqst *rqst, struct rpc_task
> > *task)
>
> Nit: The function's documenting comment will need to be updated.
>
>
> > {
> > - struct rpc_rqst *rqst = task->tk_rqstp;
> > struct rpc_xprt *xprt = rqst->rq_xprt;
> > struct rpcrdma_req *req = rpcr_to_rdmar(rqst);
> > struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt);
> > @@ -741,7 +740,7 @@ xprt_rdma_send_request(struct rpc_task *task)
> > /* An RPC with no reply will throw off credit accounting,
> > * so drop the connection to reset the credit grant.
> > */
> > - if (!rpc_reply_expected(task))
> > + if (!rpc_reply_expected(rqst->rq_task))
> > goto drop_connection;
> > return 0;
> >
> > diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
> > index 8d6404259ff9..b8143eded4af 100644
> > --- a/net/sunrpc/xprtsock.c
> > +++ b/net/sunrpc/xprtsock.c
> > @@ -449,12 +449,12 @@ static void xs_nospace_callback(struct
> > rpc_task *task)
> >
> > /**
> > * xs_nospace - place task on wait queue if transmit was incomplete
> > + * @req: pointer to RPC request
> > * @task: task to put to sleep
> > *
> > */
> > -static int xs_nospace(struct rpc_task *task)
> > +static int xs_nospace(struct rpc_rqst *req, struct rpc_task *task)
> > {
> > - struct rpc_rqst *req = task->tk_rqstp;
> > struct rpc_xprt *xprt = req->rq_xprt;
> > struct sock_xprt *transport = container_of(xprt, struct
> > sock_xprt, xprt);
> > struct sock *sk = transport->inet;
> > @@ -513,6 +513,7 @@ static inline void
> > xs_encode_stream_record_marker(struct xdr_buf *buf)
> >
> > /**
> > * xs_local_send_request - write an RPC request to an AF_LOCAL
> > socket
> > + * @req: pointer to RPC request
> > * @task: RPC task that manages the state of an RPC request
> > *
> > * Return values:
> > @@ -522,9 +523,8 @@ static inline void
> > xs_encode_stream_record_marker(struct xdr_buf *buf)
> > * ENOTCONN: Caller needs to invoke connect logic then call
> > again
> > * other: Some other error occured, the request was not
> > sent
> > */
> > -static int xs_local_send_request(struct rpc_task *task)
> > +static int xs_local_send_request(struct rpc_rqst *req, struct
> > rpc_task *task)
> > {
> > - struct rpc_rqst *req = task->tk_rqstp;
> > struct rpc_xprt *xprt = req->rq_xprt;
> > struct sock_xprt *transport =
> > container_of(xprt, struct sock_xprt,
> > xprt);
> > @@ -569,7 +569,7 @@ static int xs_local_send_request(struct
> > rpc_task *task)
> > case -ENOBUFS:
> > break;
> > case -EAGAIN:
> > - status = xs_nospace(task);
> > + status = xs_nospace(req, task);
> > break;
> > default:
> > dprintk("RPC: sendmsg returned unrecognized error
> > %d\n",
> > @@ -585,6 +585,7 @@ static int xs_local_send_request(struct
> > rpc_task *task)
> >
> > /**
> > * xs_udp_send_request - write an RPC request to a UDP socket
> > + * @req: pointer to RPC request
> > * @task: address of RPC task that manages the state of an RPC
> > request
> > *
> > * Return values:
> > @@ -594,9 +595,8 @@ static int xs_local_send_request(struct
> > rpc_task *task)
> > * ENOTCONN: Caller needs to invoke connect logic then call
> > again
> > * other: Some other error occurred, the request was not
> > sent
> > */
> > -static int xs_udp_send_request(struct rpc_task *task)
> > +static int xs_udp_send_request(struct rpc_rqst *req, struct
> > rpc_task *task)
> > {
> > - struct rpc_rqst *req = task->tk_rqstp;
> > struct rpc_xprt *xprt = req->rq_xprt;
> > struct sock_xprt *transport = container_of(xprt, struct
> > sock_xprt, xprt);
> > struct xdr_buf *xdr = &req->rq_snd_buf;
> > @@ -638,7 +638,7 @@ static int xs_udp_send_request(struct rpc_task
> > *task)
> > /* Should we call xs_close() here? */
> > break;
> > case -EAGAIN:
> > - status = xs_nospace(task);
> > + status = xs_nospace(req, task);
> > break;
> > case -ENETUNREACH:
> > case -ENOBUFS:
> > @@ -658,6 +658,7 @@ static int xs_udp_send_request(struct rpc_task
> > *task)
> >
> > /**
> > * xs_tcp_send_request - write an RPC request to a TCP socket
> > + * @req: pointer to RPC request
> > * @task: address of RPC task that manages the state of an RPC
> > request
> > *
> > * Return values:
> > @@ -670,9 +671,8 @@ static int xs_udp_send_request(struct rpc_task
> > *task)
> > * XXX: In the case of soft timeouts, should we eventually give up
> > * if sendmsg is not able to make progress?
> > */
> > -static int xs_tcp_send_request(struct rpc_task *task)
> > +static int xs_tcp_send_request(struct rpc_rqst *req, struct
> > rpc_task *task)
> > {
> > - struct rpc_rqst *req = task->tk_rqstp;
> > struct rpc_xprt *xprt = req->rq_xprt;
> > struct sock_xprt *transport = container_of(xprt, struct
> > sock_xprt, xprt);
> > struct xdr_buf *xdr = &req->rq_snd_buf;
> > @@ -697,7 +697,7 @@ static int xs_tcp_send_request(struct rpc_task
> > *task)
> > * completes while the socket holds a reference to the pages,
> > * then we may end up resending corrupted data.
> > */
> > - if (task->tk_flags & RPC_TASK_SENT)
> > + if (req->rq_task->tk_flags & RPC_TASK_SENT)
> > zerocopy = false;
> >
> > if (test_bit(XPRT_SOCK_UPD_TIMEOUT, &transport->sock_state))
> > @@ -761,7 +761,7 @@ static int xs_tcp_send_request(struct rpc_task
> > *task)
> > /* Should we call xs_close() here? */
> > break;
> > case -EAGAIN:
> > - status = xs_nospace(task);
> > + status = xs_nospace(req, task);
> > break;
> > case -ECONNRESET:
> > case -ECONNREFUSED:
> > @@ -2706,9 +2706,8 @@ static int bc_sendto(struct rpc_rqst *req)
> > /*
> > * The send routine. Borrows from svc_send
> > */
> > -static int bc_send_request(struct rpc_task *task)
> > +static int bc_send_request(struct rpc_rqst *req, struct rpc_task
> > *task)
> > {
> > - struct rpc_rqst *req = task->tk_rqstp;
> > struct svc_xprt *xprt;
> > int len;
> >
> > --
> > 2.17.1
> >
>
> --
> Chuck Lever
> chucklever@gmail.com
>
>
>
next prev parent reply other threads:[~2018-09-03 22:09 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-09-03 15:29 [PATCH 00/27] Convert RPC client transmission to a queued model Trond Myklebust
2018-09-03 15:29 ` [PATCH 01/27] SUNRPC: Clean up initialisation of the struct rpc_rqst Trond Myklebust
2018-09-03 15:29 ` [PATCH 02/27] SUNRPC: If there is no reply expected, bail early from call_decode Trond Myklebust
2018-09-03 15:29 ` [PATCH 03/27] SUNRPC: The transmitted message must lie in the RPCSEC window of validity Trond Myklebust
2018-09-03 15:29 ` [PATCH 04/27] SUNRPC: Simplify identification of when the message send/receive is complete Trond Myklebust
2018-09-03 15:29 ` [PATCH 05/27] SUNRPC: Avoid holding locks across the XDR encoding of the RPC message Trond Myklebust
2018-09-03 15:29 ` [PATCH 06/27] SUNRPC: Rename TCP receive-specific state variables Trond Myklebust
2018-09-03 15:29 ` [PATCH 07/27] SUNRPC: Move reset of TCP state variables into the reconnect code Trond Myklebust
2018-09-03 15:29 ` [PATCH 08/27] SUNRPC: Add socket transmit queue offset tracking Trond Myklebust
2018-09-03 15:29 ` [PATCH 09/27] SUNRPC: Simplify dealing with aborted partially transmitted messages Trond Myklebust
2018-09-03 15:29 ` [PATCH 10/27] SUNRPC: Refactor the transport request pinning Trond Myklebust
2018-09-03 15:29 ` [PATCH 11/27] SUNRPC: Add a helper to wake up a sleeping rpc_task and set its status Trond Myklebust
2018-09-03 15:29 ` [PATCH 12/27] SUNRPC: Don't wake queued RPC calls multiple times in xprt_transmit Trond Myklebust
2018-09-03 15:29 ` [PATCH 13/27] SUNRPC: Rename xprt->recv_lock to xprt->queue_lock Trond Myklebust
2018-09-03 15:29 ` [PATCH 14/27] SUNRPC: Refactor xprt_transmit() to remove the reply queue code Trond Myklebust
2018-09-03 15:29 ` [PATCH 15/27] SUNRPC: Refactor xprt_transmit() to remove wait for reply code Trond Myklebust
2018-09-03 15:29 ` [PATCH 16/27] SUNRPC: Minor cleanup for call_transmit() Trond Myklebust
2018-09-03 15:29 ` [PATCH 17/27] SUNRPC: Distinguish between the slot allocation list and receive queue Trond Myklebust
2018-09-03 15:29 ` [PATCH 18/27] NFS: Add a transmission queue for RPC requests Trond Myklebust
2018-09-03 15:29 ` [PATCH 19/27] SUNRPC: Refactor RPC call encoding Trond Myklebust
2018-09-03 15:29 ` [PATCH 20/27] SUNRPC: Treat the task and request as separate in the xprt_ops->send_request() Trond Myklebust
2018-09-03 15:29 ` [PATCH 21/27] SUNRPC: Don't reset the request 'bytes_sent' counter when releasing XPRT_LOCK Trond Myklebust
2018-09-03 15:29 ` [PATCH 22/27] SUNRPC: Simplify xprt_prepare_transmit() Trond Myklebust
2018-09-03 15:29 ` [PATCH 23/27] SUNRPC: Move RPC retransmission stat counter to xprt_transmit() Trond Myklebust
2018-09-03 15:29 ` [PATCH 24/27] SUNRPC: Fix up the back channel transmit Trond Myklebust
2018-09-03 15:29 ` [PATCH 25/27] SUNRPC: Allow calls to xprt_transmit() to drain the entire transmit queue Trond Myklebust
2018-09-03 15:29 ` [PATCH 26/27] SUNRPC: Queue the request for transmission immediately after encoding Trond Myklebust
2018-09-03 15:29 ` [PATCH 27/27] SUNRPC: Convert the xprt->sending queue back to an ordinary wait queue Trond Myklebust
2018-09-03 17:31 ` [PATCH 23/27] SUNRPC: Move RPC retransmission stat counter to xprt_transmit() Chuck Lever
2018-09-03 17:49 ` Trond Myklebust
2018-09-03 17:28 ` [PATCH 20/27] SUNRPC: Treat the task and request as separate in the xprt_ops->send_request() Chuck Lever
2018-09-03 17:47 ` Trond Myklebust [this message]
2018-09-03 17:49 ` Chuck Lever
2018-09-03 17:11 ` [PATCH 05/27] SUNRPC: Avoid holding locks across the XDR encoding of the RPC message Chuck Lever
2018-09-03 17:40 ` Trond Myklebust
2018-09-03 17:15 ` [PATCH 04/27] SUNRPC: Simplify identification of when the message send/receive is complete Chuck Lever
2018-09-04 15:46 ` [PATCH 03/27] SUNRPC: The transmitted message must lie in the RPCSEC window of validity J. Bruce Fields
2018-09-03 17:41 ` [PATCH 00/27] Convert RPC client transmission to a queued model Chuck Lever
2018-09-03 17:55 ` Trond Myklebust
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0b84ef010e4b3e4d156b6b7ac60c5fb0c2a48627.camel@gmail.com \
--to=trondmy@gmail.com \
--cc=chucklever@gmail.com \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).