[PATCH v2] sunrpc: Use GFP_NOFS to prevent use of current->task_frag.

[PATCH v2] sunrpc: Use GFP_NOFS to prevent use of current->task_frag.

[Guillaume Nault]

Commit a1231fda7e94 ("SUNRPC: Set memalloc_nofs_save() on all
rpciod/xprtiod jobs") stopped setting sk->sk_allocation explicitly in
favor of using memalloc_nofs_save()/memalloc_nofs_restore() critical
sections.

However, ->sk_allocation isn't used just by the memory allocator.
In particular, sk_page_frag() uses it to figure out if it can return
the page_frag from current or if it has to use the socket one.
With ->sk_allocation set to the default GFP_KERNEL, sk_page_frag() now
returns current->page_frag, which might already be in use in the
current context if the call happens during memory reclaim.

Fix this by setting ->sk_allocation to GFP_NOFS.
Note that we can't just instruct sk_page_frag() to look at
current->flags, because it could generate a cache miss, thus slowing
down the TCP fast path.

This is similar to the problems fixed by the following two commits:
  * cifs: commit dacb5d8875cc ("tcp: fix page frag corruption on page
    fault").
  * nbd: commit 20eb4f29b602 ("net: fix sk_page_frag() recursion from
    memory reclaim").

Fixes: a1231fda7e94 ("SUNRPC: Set memalloc_nofs_save() on all rpciod/xprtiod jobs")
Link: https://lore.kernel.org/netdev/b4d8cb09c913d3e34f853736f3f5628abfd7f4b6.1656699567.git.gnault@redhat.com/
Acked-by: Paolo Abeni <pabeni@redhat.com>
Reviewed-by: Benjamin Coddington <bcodding@redhat.com>
Signed-off-by: Guillaume Nault <gnault@redhat.com>
---
v2: - Post only to linux-nfs to make it clearer which is the target tree.
    - Incorporate already collected acked-by and reviewed-by tags.

 net/sunrpc/xprtsock.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index e976007f4fd0..1bd3048d43ae 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -1882,6 +1882,7 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
 		sk->sk_write_space = xs_udp_write_space;
 		sk->sk_state_change = xs_local_state_change;
 		sk->sk_error_report = xs_error_report;
+		sk->sk_allocation = GFP_NOFS;
 
 		xprt_clear_connected(xprt);
 
@@ -2083,6 +2084,7 @@ static void xs_udp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)
 		sk->sk_user_data = xprt;
 		sk->sk_data_ready = xs_data_ready;
 		sk->sk_write_space = xs_udp_write_space;
+		sk->sk_allocation = GFP_NOFS;
 
 		xprt_set_connected(xprt);
 
@@ -2250,6 +2252,7 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)
 		sk->sk_state_change = xs_tcp_state_change;
 		sk->sk_write_space = xs_tcp_write_space;
 		sk->sk_error_report = xs_error_report;
+		sk->sk_allocation = GFP_NOFS;
 
 		/* socket options */
 		sock_reset_flag(sk, SOCK_LINGER);
-- 
2.21.3

Re: [PATCH v2] sunrpc: Use GFP_NOFS to prevent use of current->task_frag.

[Christoph Hellwig]

On Mon, Oct 10, 2022 at 01:41:57PM +0200, Guillaume Nault wrote:
> However, ->sk_allocation isn't used just by the memory allocator.
> In particular, sk_page_frag() uses it to figure out if it can return
> the page_frag from current or if it has to use the socket one.

Well, that just means sk_page_frag really needs to look at
PF_MEMALLOC_* as well.  So instead of reverting the proper change
please fix that.  A helper that looks at sk_allocation and
current->flags is probably the right way to deal with that.

Re: [PATCH v2] sunrpc: Use GFP_NOFS to prevent use of current->task_frag.

[Guillaume Nault]

On Mon, Oct 10, 2022 at 07:55:29AM -0700, Christoph Hellwig wrote:
> On Mon, Oct 10, 2022 at 01:41:57PM +0200, Guillaume Nault wrote:
> > However, ->sk_allocation isn't used just by the memory allocator.
> > In particular, sk_page_frag() uses it to figure out if it can return
> > the page_frag from current or if it has to use the socket one.
> 
> Well, that just means sk_page_frag really needs to look at
> PF_MEMALLOC_* as well.  So instead of reverting the proper change
> please fix that.  A helper that looks at sk_allocation and
> current->flags is probably the right way to deal with that.

That's what my RFC patch did. It was rejected because reading
current->flags may incur a cache miss thus slowing down TCP fast path.
See the discussion in the Link tag:
https://lore.kernel.org/netdev/b4d8cb09c913d3e34f853736f3f5628abfd7f4b6.1656699567.git.gnault@redhat.com/

Re: [PATCH v2] sunrpc: Use GFP_NOFS to prevent use of current->task_frag.

[Christoph Hellwig]

On Mon, Oct 10, 2022 at 06:56:50PM +0200, Guillaume Nault wrote:
> That's what my RFC patch did. It was rejected because reading
> current->flags may incur a cache miss thus slowing down TCP fast path.
> See the discussion in the Link tag:
> https://lore.kernel.org/netdev/b4d8cb09c913d3e34f853736f3f5628abfd7f4b6.1656699567.git.gnault@redhat.com/

As GFP_NOFS/NOIO are on their way out the networking people will have to
do this anyway.

Re: [PATCH v2] sunrpc: Use GFP_NOFS to prevent use of current->task_frag.

[Guillaume Nault]

On Mon, Oct 10, 2022 at 11:18:19PM -0700, Christoph Hellwig wrote:
> On Mon, Oct 10, 2022 at 06:56:50PM +0200, Guillaume Nault wrote:
> > That's what my RFC patch did. It was rejected because reading
> > current->flags may incur a cache miss thus slowing down TCP fast path.
> > See the discussion in the Link tag:
> > https://lore.kernel.org/netdev/b4d8cb09c913d3e34f853736f3f5628abfd7f4b6.1656699567.git.gnault@redhat.com/
> 
> As GFP_NOFS/NOIO are on their way out the networking people will have to
> do this anyway.

We can always think of a nicer solution in the future. But right now we
have a real bug to fix.

Commit a1231fda7e94 ("SUNRPC: Set memalloc_nofs_save() on all
rpciod/xprtiod jobs") introduces a bug that crashes the kernel. I can't
see anything wrong with a partial revert.

Re: [PATCH v2] sunrpc: Use GFP_NOFS to prevent use of current->task_frag.

[Trond Myklebust]

On Tue, 2022-10-11 at 17:00 +0200, Guillaume Nault wrote:
> On Mon, Oct 10, 2022 at 11:18:19PM -0700, Christoph Hellwig wrote:
> > On Mon, Oct 10, 2022 at 06:56:50PM +0200, Guillaume Nault wrote:
> > > That's what my RFC patch did. It was rejected because reading
> > > current->flags may incur a cache miss thus slowing down TCP fast
> > > path.
> > > See the discussion in the Link tag:
> > > https://lore.kernel.org/netdev/b4d8cb09c913d3e34f853736f3f5628abfd7f4b6.1656699567.git.gnault@redhat.com/
> > 
> > As GFP_NOFS/NOIO are on their way out the networking people will
> > have to
> > do this anyway.
> 
> We can always think of a nicer solution in the future. But right now
> we
> have a real bug to fix.
> 
> Commit a1231fda7e94 ("SUNRPC: Set memalloc_nofs_save() on all
> rpciod/xprtiod jobs") introduces a bug that crashes the kernel. I
> can't
> see anything wrong with a partial revert.
> 

How about instead just adding a dedicated flag to the socket that
switches between the two page_frag modes?

That would remain future proofed, and it would give kernel users a
lever with which to do the right thing without unnecessarily
constraining the allocation modes.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: [PATCH v2] sunrpc: Use GFP_NOFS to prevent use of current->task_frag.

[Guillaume Nault]

On Tue, Oct 11, 2022 at 11:57:53AM -0400, Trond Myklebust wrote:
> On Tue, 2022-10-11 at 17:00 +0200, Guillaume Nault wrote:
> > On Mon, Oct 10, 2022 at 11:18:19PM -0700, Christoph Hellwig wrote:
> > > On Mon, Oct 10, 2022 at 06:56:50PM +0200, Guillaume Nault wrote:
> > > > That's what my RFC patch did. It was rejected because reading
> > > > current->flags may incur a cache miss thus slowing down TCP fast
> > > > path.
> > > > See the discussion in the Link tag:
> > > > https://lore.kernel.org/netdev/b4d8cb09c913d3e34f853736f3f5628abfd7f4b6.1656699567.git.gnault@redhat.com/
> > > 
> > > As GFP_NOFS/NOIO are on their way out the networking people will
> > > have to
> > > do this anyway.
> > 
> > We can always think of a nicer solution in the future. But right now
> > we
> > have a real bug to fix.
> > 
> > Commit a1231fda7e94 ("SUNRPC: Set memalloc_nofs_save() on all
> > rpciod/xprtiod jobs") introduces a bug that crashes the kernel. I
> > can't
> > see anything wrong with a partial revert.
> > 
> 
> How about instead just adding a dedicated flag to the socket that
> switches between the two page_frag modes?
> 
> That would remain future proofed, and it would give kernel users a
> lever with which to do the right thing without unnecessarily
> constraining the allocation modes.

The problem is to find a hole in struct sock, in a cacheline that
wouldn't incur a cache miss.

So far the best option seems to add a special flag in sk_allocation,
which would serve the same purpose as GFP_NOFS in sk_page_frag(). But
that'd mean clearing this bit every time sk_allocation is used for
memory allocation. Not really something I'd propose for stable trees.

> -- 
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> trond.myklebust@hammerspace.com
> 
> 

Re: [PATCH v2] sunrpc: Use GFP_NOFS to prevent use of current->task_frag.

[Guillaume Nault]

[Adding netdev and Eric, who commented on the original RFC.]

On Tue, Oct 11, 2022 at 11:14:36PM +0200, Guillaume Nault wrote:
> On Tue, Oct 11, 2022 at 11:57:53AM -0400, Trond Myklebust wrote:
> > On Tue, 2022-10-11 at 17:00 +0200, Guillaume Nault wrote:
> > > On Mon, Oct 10, 2022 at 11:18:19PM -0700, Christoph Hellwig wrote:
> > > > On Mon, Oct 10, 2022 at 06:56:50PM +0200, Guillaume Nault wrote:
> > > > > That's what my RFC patch did. It was rejected because reading
> > > > > current->flags may incur a cache miss thus slowing down TCP fast
> > > > > path.
> > > > > See the discussion in the Link tag:
> > > > > https://lore.kernel.org/netdev/b4d8cb09c913d3e34f853736f3f5628abfd7f4b6.1656699567.git.gnault@redhat.com/
> > > > 
> > > > As GFP_NOFS/NOIO are on their way out the networking people will
> > > > have to
> > > > do this anyway.
> > > 
> > > We can always think of a nicer solution in the future. But right now
> > > we
> > > have a real bug to fix.
> > > 
> > > Commit a1231fda7e94 ("SUNRPC: Set memalloc_nofs_save() on all
> > > rpciod/xprtiod jobs") introduces a bug that crashes the kernel. I
> > > can't
> > > see anything wrong with a partial revert.
> > > 
> > 
> > How about instead just adding a dedicated flag to the socket that
> > switches between the two page_frag modes?
> > 
> > That would remain future proofed, and it would give kernel users a
> > lever with which to do the right thing without unnecessarily
> > constraining the allocation modes.
> 
> The problem is to find a hole in struct sock, in a cacheline that
> wouldn't incur a cache miss.

Okay, so I have this patch that adds a flag in struct sock. The cache
line is shared with ->sk_shutdown and should be hot as ->sk_shutdown is
is tested just before the while() loop in tcp_sendmsg_locked().

Still, that looks like net-next material to me. Reverting sunrpc to use
GFP_NOFS looks better for an immediate bug fix.

------------ >8 ------------
net: Introduce sk_use_task_frag in struct sock.

Sockets that can be used while recursing into memory reclaim, like
those used by network block devices and file systems, mustn't use
current->task_frag: if the current process is already using it, then
the inner memory reclaim call would corrupt the task_frag structure.

To avoid this, sk_page_frag() uses ->sk_allocation to detect sockets
that mustn't use current->task_frag, assuming that those used during
memory reclaim had their allocation constraints reflected in
->sk_allocation.

This unfortunately doesn't cover all cases: in an attempt to remove all
usage of GFP_NOFS and GFP_NOIO, sunrpc stopped setting these flags in
->sk_allocation, and used memalloc_nofs critical sections instead.
This breaks the sk_page_frag() heuristic since the allocation
constraints are now stored in current->flags, which sk_page_frag()
can't read without risking triggering a cache miss and slowing down
TCP's fast path.

This patch creates a new field in struct sock, named sk_use_task_frag,
which sockets with memory reclaim constraints can set to false if they
can't safely use current->task_frag. In such cases, sk_page_frag() now
always returns the socket's page_frag (->sk_frag). The first user is
sunrpc, which needs to avoid using current->task_frag but can keep
->sk_allocation set to GFP_KERNEL otherwise.

Eventually, it might be possible to simplify sk_page_frag() by only
testing ->sk_use_task_frag and avoid relying on the ->sk_allocation
heuristic entirely (assuming other sockets will set ->sk_use_task_frag
according to their constraints in the future).

The new ->sk_use_task_frag field is placed in a hole in struct sock and
belongs to a cache line shared with ->sk_shutdown. Therefore it should
be hot and shouldn't have negative performance impacts on TCP's fast
path (sk_shutdown is tested just before the while() loop in
tcp_sendmsg_locked()).

Fixes: a1231fda7e94 ("SUNRPC: Set memalloc_nofs_save() on all rpciod/xprtiod jobs")
Link: https://lore.kernel.org/netdev/b4d8cb09c913d3e34f853736f3f5628abfd7f4b6.1656699567.git.gnault@redhat.com/

diff --git a/include/net/sock.h b/include/net/sock.h
index 08038a385ef2..bd3eef3afb92 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -318,6 +318,9 @@ struct sk_filter;
   *	@sk_stamp: time stamp of last packet received
   *	@sk_stamp_seq: lock for accessing sk_stamp on 32 bit architectures only
   *	@sk_tsflags: SO_TIMESTAMPING flags
+  *	@sk_use_task_frag: allow sk_page_frag() to use current->task_frag.
+                           Sockets that can be used under memory reclaim should
+                           set this to false.
   *	@sk_bind_phc: SO_TIMESTAMPING bind PHC index of PTP virtual clock
   *	              for timestamping
   *	@sk_tskey: counter to disambiguate concurrent tstamp requests
@@ -505,6 +508,7 @@ struct sock {
 #endif
 	u16			sk_tsflags;
 	u8			sk_shutdown;
+	bool			sk_use_task_frag;
 	atomic_t		sk_tskey;
 	atomic_t		sk_zckey;
 
@@ -2554,14 +2558,17 @@ static inline void sk_stream_moderate_sndbuf(struct sock *sk)
  * socket operations and end up recursing into sk_page_frag()
  * while it's already in use: explicitly avoid task page_frag
  * usage if the caller is potentially doing any of them.
- * This assumes that page fault handlers use the GFP_NOFS flags.
+ * This assumes that page fault handlers use the GFP_NOFS flags or
+ * explicitely disable sk_use_task_frag.
  *
  * Return: a per task page_frag if context allows that,
  * otherwise a per socket one.
  */
 static inline struct page_frag *sk_page_frag(struct sock *sk)
 {
-	if ((sk->sk_allocation & (__GFP_DIRECT_RECLAIM | __GFP_MEMALLOC | __GFP_FS)) ==
+	if (sk->sk_use_task_frag &&
+	    (sk->sk_allocation & (__GFP_DIRECT_RECLAIM | __GFP_MEMALLOC |
+				  __GFP_FS)) ==
 	    (__GFP_DIRECT_RECLAIM | __GFP_FS))
 		return &current->task_frag;
 
diff --git a/net/core/sock.c b/net/core/sock.c
index a3ba0358c77c..cc113500d442 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -3368,6 +3368,7 @@ void sock_init_data(struct socket *sock, struct sock *sk)
 	sk->sk_rcvbuf		=	READ_ONCE(sysctl_rmem_default);
 	sk->sk_sndbuf		=	READ_ONCE(sysctl_wmem_default);
 	sk->sk_state		=	TCP_CLOSE;
+	sk->sk_use_task_frag	=	true;
 	sk_set_socket(sk, sock);
 
 	sock_set_flag(sk, SOCK_ZAPPED);
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index e976007f4fd0..d3170b753dfc 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -1882,6 +1882,7 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
 		sk->sk_write_space = xs_udp_write_space;
 		sk->sk_state_change = xs_local_state_change;
 		sk->sk_error_report = xs_error_report;
+		sk->sk_use_task_frag = false;
 
 		xprt_clear_connected(xprt);
 
@@ -2083,6 +2084,7 @@ static void xs_udp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)
 		sk->sk_user_data = xprt;
 		sk->sk_data_ready = xs_data_ready;
 		sk->sk_write_space = xs_udp_write_space;
+		sk->sk_use_task_frag = false;
 
 		xprt_set_connected(xprt);
 
@@ -2250,6 +2252,7 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)
 		sk->sk_state_change = xs_tcp_state_change;
 		sk->sk_write_space = xs_tcp_write_space;
 		sk->sk_error_report = xs_error_report;
+		sk->sk_use_task_frag = false;
 
 		/* socket options */
 		sock_reset_flag(sk, SOCK_LINGER);

Re: [PATCH v2] sunrpc: Use GFP_NOFS to prevent use of current->task_frag.

[Guillaume Nault]

On Thu, Oct 13, 2022 at 02:18:37PM +0200, Guillaume Nault wrote:
> Still, that looks like net-next material to me. Reverting sunrpc to use
> GFP_NOFS looks better for an immediate bug fix.

Could we please move forward with this patch? This bug really needs to be
fixed. So please let's either revert to GFP_NOFS or actively work on a
different solution.

Re: [PATCH v2] sunrpc: Use GFP_NOFS to prevent use of current->task_frag.

[Benjamin Coddington]

On 25 Oct 2022, at 7:15, Guillaume Nault wrote:

> On Thu, Oct 13, 2022 at 02:18:37PM +0200, Guillaume Nault wrote:
>> Still, that looks like net-next material to me. Reverting sunrpc to use
>> GFP_NOFS looks better for an immediate bug fix.
>
> Could we please move forward with this patch? This bug really needs to be
> fixed. So please let's either revert to GFP_NOFS or actively work on a
> different solution.

Hey Trond, Anna - I think the right thing to do here is send this patch to
-stable as stable-only while we try to get this fixed up.  Any objections to
that?

Ben