[PATCH] SUNRPC: Use MSG_SENDPAGE_NOTLAST when calling sendpage()

[PATCH] SUNRPC: Use MSG_SENDPAGE_NOTLAST when calling sendpage()

[Trond Myklebust]

If we're sending more pages via kernel_sendpage(), then set
MSG_SENDPAGE_NOTLAST.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 net/sunrpc/svcsock.c  | 2 +-
 net/sunrpc/xprtsock.c | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index 0c8120229a03..48923730722d 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -181,7 +181,7 @@ int svc_send_common(struct socket *sock, struct xdr_buf *xdr,
 	struct page	**ppage = xdr->pages;
 	size_t		base = xdr->page_base;
 	unsigned int	pglen = xdr->page_len;
-	unsigned int	flags = MSG_MORE;
+	unsigned int	flags = MSG_MORE | MSG_SENDPAGE_NOTLAST;
 	int		slen;
 	int		len = 0;
 
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 1471ecceabf9..e71aff251ac1 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -360,8 +360,10 @@ static int xs_send_pagedata(struct socket *sock, struct xdr_buf *xdr, unsigned i
 		int flags = XS_SENDMSG_FLAGS;
 
 		remainder -= len;
-		if (remainder != 0 || more)
+		if (more)
 			flags |= MSG_MORE;
+		if (remainder != 0)
+			flags |= MSG_SENDPAGE_NOTLAST | MSG_MORE;
 		err = do_sendpage(sock, *ppage, base, len, flags);
 		if (remainder == 0 || err != len)
 			break;
-- 
2.4.3

Re: [PATCH] SUNRPC: Use MSG_SENDPAGE_NOTLAST when calling sendpage()

[Bruce Fields]

On Wed, Oct 07, 2015 at 01:54:42PM -0400, Trond Myklebust wrote:
> If we're sending more pages via kernel_sendpage(), then set
> MSG_SENDPAGE_NOTLAST.

Given that we've survived without this a while, I assume this is a hint
that may provide better performance in some cases (have you seen that?)
and that it doesn't have to be 100% correct?

In which case, looks fine, will queue up for 4.4....

--b.

> 
> Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
> ---
>  net/sunrpc/svcsock.c  | 2 +-
>  net/sunrpc/xprtsock.c | 4 +++-
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
> index 0c8120229a03..48923730722d 100644
> --- a/net/sunrpc/svcsock.c
> +++ b/net/sunrpc/svcsock.c
> @@ -181,7 +181,7 @@ int svc_send_common(struct socket *sock, struct xdr_buf *xdr,
>  	struct page	**ppage = xdr->pages;
>  	size_t		base = xdr->page_base;
>  	unsigned int	pglen = xdr->page_len;
> -	unsigned int	flags = MSG_MORE;
> +	unsigned int	flags = MSG_MORE | MSG_SENDPAGE_NOTLAST;
>  	int		slen;
>  	int		len = 0;
>  
> diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
> index 1471ecceabf9..e71aff251ac1 100644
> --- a/net/sunrpc/xprtsock.c
> +++ b/net/sunrpc/xprtsock.c
> @@ -360,8 +360,10 @@ static int xs_send_pagedata(struct socket *sock, struct xdr_buf *xdr, unsigned i
>  		int flags = XS_SENDMSG_FLAGS;
>  
>  		remainder -= len;
> -		if (remainder != 0 || more)
> +		if (more)
>  			flags |= MSG_MORE;
> +		if (remainder != 0)
> +			flags |= MSG_SENDPAGE_NOTLAST | MSG_MORE;
>  		err = do_sendpage(sock, *ppage, base, len, flags);
>  		if (remainder == 0 || err != len)
>  			break;
> -- 
> 2.4.3

Re: [PATCH] SUNRPC: Use MSG_SENDPAGE_NOTLAST when calling sendpage()

[Trond Myklebust]

On Wed, Oct 7, 2015 at 4:08 PM, Bruce Fields <bfields@fieldses.org> wrote:
> On Wed, Oct 07, 2015 at 01:54:42PM -0400, Trond Myklebust wrote:
>> If we're sending more pages via kernel_sendpage(), then set
>> MSG_SENDPAGE_NOTLAST.
>
> Given that we've survived without this a while, I assume this is a hint
> that may provide better performance in some cases (have you seen that?)
> and that it doesn't have to be 100% correct?

See the original motivation in:

commit 2f5338442425 ("tcp: allow splice() to build full TSO packets")

and the later bugfix (which actually introduces MSG_SENDPAGE_NOTLAST):

commit 35f9c09fe9c7 ("tcp: tcp_sendpages() should call tcp_push() once")

> In which case, looks fine, will queue up for 4.4....

Sure. It's a minor performance enhancement, so there is no immediate rush.

Re: [PATCH] SUNRPC: Use MSG_SENDPAGE_NOTLAST when calling sendpage()

[Trond Myklebust]

On Wed, Oct 7, 2015 at 7:17 PM, Trond Myklebust
<trond.myklebust@primarydata.com> wrote:
> On Wed, Oct 7, 2015 at 4:08 PM, Bruce Fields <bfields@fieldses.org> wrote:
>> On Wed, Oct 07, 2015 at 01:54:42PM -0400, Trond Myklebust wrote:
>>> If we're sending more pages via kernel_sendpage(), then set
>>> MSG_SENDPAGE_NOTLAST.
>>
>> Given that we've survived without this a while, I assume this is a hint
>> that may provide better performance in some cases (have you seen that?)
>> and that it doesn't have to be 100% correct?
>
> See the original motivation in:
>
> commit 2f5338442425 ("tcp: allow splice() to build full TSO packets")
>
> and the later bugfix (which actually introduces MSG_SENDPAGE_NOTLAST):
>
> commit 35f9c09fe9c7 ("tcp: tcp_sendpages() should call tcp_push() once")
>
>> In which case, looks fine, will queue up for 4.4....
>
> Sure. It's a minor performance enhancement, so there is no immediate rush.

Actually, please let me split this into 2 subpatches; one for the
sunrpc server code and one for the client. It would make it easier for
me if I could keep the client performance stuff in one branch.

Re: [PATCH] SUNRPC: Use MSG_SENDPAGE_NOTLAST when calling sendpage()

[Bruce Fields]

On Thu, Oct 08, 2015 at 08:35:02AM -0400, Trond Myklebust wrote:
> On Wed, Oct 7, 2015 at 7:17 PM, Trond Myklebust
> <trond.myklebust@primarydata.com> wrote:
> > On Wed, Oct 7, 2015 at 4:08 PM, Bruce Fields <bfields@fieldses.org> wrote:
> >> On Wed, Oct 07, 2015 at 01:54:42PM -0400, Trond Myklebust wrote:
> >>> If we're sending more pages via kernel_sendpage(), then set
> >>> MSG_SENDPAGE_NOTLAST.
> >>
> >> Given that we've survived without this a while, I assume this is a hint
> >> that may provide better performance in some cases (have you seen that?)
> >> and that it doesn't have to be 100% correct?
> >
> > See the original motivation in:
> >
> > commit 2f5338442425 ("tcp: allow splice() to build full TSO packets")
> >
> > and the later bugfix (which actually introduces MSG_SENDPAGE_NOTLAST):
> >
> > commit 35f9c09fe9c7 ("tcp: tcp_sendpages() should call tcp_push() once")
> >
> >> In which case, looks fine, will queue up for 4.4....
> >
> > Sure. It's a minor performance enhancement, so there is no immediate rush.
> 
> Actually, please let me split this into 2 subpatches; one for the
> sunrpc server code and one for the client. It would make it easier for
> me if I could keep the client performance stuff in one branch.

Sure, makes sense.--b.

[PATCH v2 1/2] SUNRPC: Use MSG_SENDPAGE_NOTLAST in xs_send_pagedata()

[Trond Myklebust]

If we're sending more than one page via kernel_sendpage(), then set
MSG_SENDPAGE_NOTLAST between the pages so that we don't send suboptimal
frames (see commit 2f5338442425 and commit 35f9c09fe9c7).

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 net/sunrpc/xprtsock.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 1471ecceabf9..e71aff251ac1 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -360,8 +360,10 @@ static int xs_send_pagedata(struct socket *sock, struct xdr_buf *xdr, unsigned i
 		int flags = XS_SENDMSG_FLAGS;
 
 		remainder -= len;
-		if (remainder != 0 || more)
+		if (more)
 			flags |= MSG_MORE;
+		if (remainder != 0)
+			flags |= MSG_SENDPAGE_NOTLAST | MSG_MORE;
 		err = do_sendpage(sock, *ppage, base, len, flags);
 		if (remainder == 0 || err != len)
 			break;
-- 
2.4.3

[PATCH v2 2/2] SUNRPC: Use MSG_SENDPAGE_NOTLAST in svc_send_common()

[Trond Myklebust]

Allow the NFS server to send large amounts of data in page form
without forcing early transmit by do_tcp_sendpages(). See
commit 2f5338442425 and commit 35f9c09fe9c7.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 net/sunrpc/svcsock.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index 0c8120229a03..48923730722d 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -181,7 +181,7 @@ int svc_send_common(struct socket *sock, struct xdr_buf *xdr,
 	struct page	**ppage = xdr->pages;
 	size_t		base = xdr->page_base;
 	unsigned int	pglen = xdr->page_len;
-	unsigned int	flags = MSG_MORE;
+	unsigned int	flags = MSG_MORE | MSG_SENDPAGE_NOTLAST;
 	int		slen;
 	int		len = 0;
 
-- 
2.4.3