[PATCH] net: use __GFP_NORETRY for high order allocations

[PATCH] net: use __GFP_NORETRY for high order allocations

[Eric Dumazet]

From: Eric Dumazet <edumazet@google.com>

sock_alloc_send_pskb() & sk_page_frag_refill()
have a loop trying high order allocations to prepare
skb with low number of fragments as this increases performance.

Problem is that under memory pressure/fragmentation, this can
trigger OOM while the intent was only to try the high order
allocations, then fallback to order-0 allocations.

We had various reports from unexpected regressions.

According to David, setting __GFP_NORETRY should be fine,
as the asynchronous compaction is still enabled, and this
will prevent OOM from kicking as in :

CFSClientEventm invoked oom-killer: gfp_mask=0x42d0, order=3, oom_adj=0,
oom_score_adj=0, oom_score_badness=2 (enabled),memcg_scoring=disabled
CFSClientEventm 

Call Trace:
 [<ffffffff8043766c>] dump_header+0xe1/0x23e
 [<ffffffff80437a02>] oom_kill_process+0x6a/0x323
 [<ffffffff80438443>] out_of_memory+0x4b3/0x50d
 [<ffffffff8043a4a6>] __alloc_pages_may_oom+0xa2/0xc7
 [<ffffffff80236f42>] __alloc_pages_nodemask+0x1002/0x17f0
 [<ffffffff8024bd23>] alloc_pages_current+0x103/0x2b0
 [<ffffffff8028567f>] sk_page_frag_refill+0x8f/0x160
 [<ffffffff80295fa0>] tcp_sendmsg+0x560/0xee0
 [<ffffffff802a5037>] inet_sendmsg+0x67/0x100
 [<ffffffff80283c9c>] __sock_sendmsg_nosec+0x6c/0x90
 [<ffffffff80283e85>] sock_sendmsg+0xc5/0xf0
 [<ffffffff802847b6>] __sys_sendmsg+0x136/0x430
 [<ffffffff80284ec8>] sys_sendmsg+0x88/0x110
 [<ffffffff80711472>] system_call_fastpath+0x16/0x1b
Out of Memory: Kill process 2856 (bash) score 9999 or sacrifice child

Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: David Rientjes <rientjes@google.com>
---
 net/core/sock.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/net/core/sock.c b/net/core/sock.c
index 0c127dcdf6a8..5b6a9431b017 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1775,7 +1775,9 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
 			while (order) {
 				if (npages >= 1 << order) {
 					page = alloc_pages(sk->sk_allocation |
-							   __GFP_COMP | __GFP_NOWARN,
+							   __GFP_COMP |
+							   __GFP_NOWARN |
+							   __GFP_NORETRY,
 							   order);
 					if (page)
 						goto fill_page;
@@ -1845,7 +1847,7 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t prio)
 		gfp_t gfp = prio;
 
 		if (order)
-			gfp |= __GFP_COMP | __GFP_NOWARN;
+			gfp |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY;
 		pfrag->page = alloc_pages(gfp, order);
 		if (likely(pfrag->page)) {
 			pfrag->offset = 0;

Re: [PATCH] net: use __GFP_NORETRY for high order allocations

[Joe Perches]

On Thu, 2014-02-06 at 10:42 -0800, Eric Dumazet wrote:
> sock_alloc_send_pskb() & sk_page_frag_refill()
> have a loop trying high order allocations to prepare
> skb with low number of fragments as this increases performance.
> 
> Problem is that under memory pressure/fragmentation, this can
> trigger OOM while the intent was only to try the high order
> allocations, then fallback to order-0 allocations.
[]
> Call Trace:
>  [<ffffffff8043766c>] dump_header+0xe1/0x23e
>  [<ffffffff80437a02>] oom_kill_process+0x6a/0x323
>  [<ffffffff80438443>] out_of_memory+0x4b3/0x50d
>  [<ffffffff8043a4a6>] __alloc_pages_may_oom+0xa2/0xc7
>  [<ffffffff80236f42>] __alloc_pages_nodemask+0x1002/0x17f0
>  [<ffffffff8024bd23>] alloc_pages_current+0x103/0x2b0
>  [<ffffffff8028567f>] sk_page_frag_refill+0x8f/0x160
[]
> diff --git a/net/core/sock.c b/net/core/sock.c
[]
> @@ -1775,7 +1775,9 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
>  			while (order) {
>  				if (npages >= 1 << order) {
>  					page = alloc_pages(sk->sk_allocation |
> -							   __GFP_COMP | __GFP_NOWARN,
> +							   __GFP_COMP |
> +							   __GFP_NOWARN |
> +							   __GFP_NORETRY,
>  							   order);
>  					if (page)
>  						goto fill_page;
> @@ -1845,7 +1847,7 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t prio)
>  		gfp_t gfp = prio;
>  
>  		if (order)
> -			gfp |= __GFP_COMP | __GFP_NOWARN;
> +			gfp |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY;

Perhaps add __GFP_THISNODE too ?

Re: [PATCH] net: use __GFP_NORETRY for high order allocations

[Eric Dumazet]

On Thu, 2014-02-06 at 12:24 -0800, Joe Perches wrote:

> Perhaps add __GFP_THISNODE too ?

Why ?

Re: [PATCH] net: use __GFP_NORETRY for high order allocations

[David Rientjes]

On Thu, 6 Feb 2014, Joe Perches wrote:

> On Thu, 2014-02-06 at 10:42 -0800, Eric Dumazet wrote:
> > sock_alloc_send_pskb() & sk_page_frag_refill()
> > have a loop trying high order allocations to prepare
> > skb with low number of fragments as this increases performance.
> > 
> > Problem is that under memory pressure/fragmentation, this can
> > trigger OOM while the intent was only to try the high order
> > allocations, then fallback to order-0 allocations.
> []
> > Call Trace:
> >  [<ffffffff8043766c>] dump_header+0xe1/0x23e
> >  [<ffffffff80437a02>] oom_kill_process+0x6a/0x323
> >  [<ffffffff80438443>] out_of_memory+0x4b3/0x50d
> >  [<ffffffff8043a4a6>] __alloc_pages_may_oom+0xa2/0xc7
> >  [<ffffffff80236f42>] __alloc_pages_nodemask+0x1002/0x17f0
> >  [<ffffffff8024bd23>] alloc_pages_current+0x103/0x2b0
> >  [<ffffffff8028567f>] sk_page_frag_refill+0x8f/0x160
> []
> > diff --git a/net/core/sock.c b/net/core/sock.c
> []
> > @@ -1775,7 +1775,9 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
> >  			while (order) {
> >  				if (npages >= 1 << order) {
> >  					page = alloc_pages(sk->sk_allocation |
> > -							   __GFP_COMP | __GFP_NOWARN,
> > +							   __GFP_COMP |
> > +							   __GFP_NOWARN |
> > +							   __GFP_NORETRY,
> >  							   order);
> >  					if (page)
> >  						goto fill_page;
> > @@ -1845,7 +1847,7 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t prio)
> >  		gfp_t gfp = prio;
> >  
> >  		if (order)
> > -			gfp |= __GFP_COMP | __GFP_NOWARN;
> > +			gfp |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY;
> 
> Perhaps add __GFP_THISNODE too ?
> 

How does __GFP_THISNODE have anything to do with avoiding oom killing due 
to high-order fragmentation?  If they absolutely require local memory to 
currnet's cpu node then that would make sense, but the fallback still 
allocates order-0 memory remotely and with __GFP_THISNODE on this attempt 
we wouldn't even attempt remote reclaim.

Re: [PATCH] net: use __GFP_NORETRY for high order allocations

[Joe Perches]

On Thu, 2014-02-06 at 13:03 -0800, David Rientjes wrote:
> On Thu, 6 Feb 2014, Joe Perches wrote:
> 
> > On Thu, 2014-02-06 at 10:42 -0800, Eric Dumazet wrote:
> > > sock_alloc_send_pskb() & sk_page_frag_refill()
> > > have a loop trying high order allocations to prepare
> > > skb with low number of fragments as this increases performance.
> > > 
> > > Problem is that under memory pressure/fragmentation, this can
> > > trigger OOM while the intent was only to try the high order
> > > allocations, then fallback to order-0 allocations.
> > []
> > > Call Trace:
> > >  [<ffffffff8043766c>] dump_header+0xe1/0x23e
> > >  [<ffffffff80437a02>] oom_kill_process+0x6a/0x323
> > >  [<ffffffff80438443>] out_of_memory+0x4b3/0x50d
> > >  [<ffffffff8043a4a6>] __alloc_pages_may_oom+0xa2/0xc7
> > >  [<ffffffff80236f42>] __alloc_pages_nodemask+0x1002/0x17f0
> > >  [<ffffffff8024bd23>] alloc_pages_current+0x103/0x2b0
> > >  [<ffffffff8028567f>] sk_page_frag_refill+0x8f/0x160
> > []
> > > diff --git a/net/core/sock.c b/net/core/sock.c
> > []
> > > @@ -1775,7 +1775,9 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
> > >  			while (order) {
> > >  				if (npages >= 1 << order) {
> > >  					page = alloc_pages(sk->sk_allocation |
> > > -							   __GFP_COMP | __GFP_NOWARN,
> > > +							   __GFP_COMP |
> > > +							   __GFP_NOWARN |
> > > +							   __GFP_NORETRY,
> > >  							   order);
> > >  					if (page)
> > >  						goto fill_page;
> > > @@ -1845,7 +1847,7 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t prio)
> > >  		gfp_t gfp = prio;
> > >  
> > >  		if (order)
> > > -			gfp |= __GFP_COMP | __GFP_NOWARN;
> > > +			gfp |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY;
> > 
> > Perhaps add __GFP_THISNODE too ?
> > 
> 
> How does __GFP_THISNODE have anything to do with avoiding oom killing due 
> to high-order fragmentation?

I don't think it does.

> If they absolutely require local memory to 
> currnet's cpu node then that would make sense,

I presumed THISNODE would be used only with NORETRY

> but the fallback still 
> allocates order-0 memory remotely and with __GFP_THISNODE on this attempt 
> we wouldn't even attempt remote reclaim.
any other alloc attempt could work on other cpus.

It was just a thought, ignore it if it's a dumb thought.

Re: [PATCH] net: use __GFP_NORETRY for high order allocations

[David Rientjes]

On Thu, 6 Feb 2014, Joe Perches wrote:

> > If they absolutely require local memory to 
> > currnet's cpu node then that would make sense,
> 
> I presumed THISNODE would be used only with NORETRY
> 

Unfortunately, that would avoid attempting to defragment or reclaim 
remotely for the order-3 allocation and only allocate remotely for 
order-0.

So we have to make a decision here what is more important: allocating 
high-order memory or local memory to current's cpu at this given time?

The policy of transparent hugepages, for example, is always to allocate 
order-9 memory remotely instead of falling back to allocating a smallpage 
locally.

The original code never cared about local vs remote memory, so I don't 
think there's any restriction to allocating remotely.  I also don't know 
many allocators that do __GFP_THISNODE and care about local memory to 
current's cpu at this point in time, usually users of that flag are 
allocating memory that must be local to a specific node (for things like 
kmem_cache_alloc_node()).

Re: [PATCH] net: use __GFP_NORETRY for high order allocations

[Eric W. Biederman]

Eric Dumazet <eric.dumazet@gmail.com> writes:

> From: Eric Dumazet <edumazet@google.com>
>
> sock_alloc_send_pskb() & sk_page_frag_refill()
> have a loop trying high order allocations to prepare
> skb with low number of fragments as this increases performance.
>
> Problem is that under memory pressure/fragmentation, this can
> trigger OOM while the intent was only to try the high order
> allocations, then fallback to order-0 allocations.
>
> We had various reports from unexpected regressions.
>
> According to David, setting __GFP_NORETRY should be fine,
> as the asynchronous compaction is still enabled, and this
> will prevent OOM from kicking as in :
>
> CFSClientEventm invoked oom-killer: gfp_mask=0x42d0, order=3, oom_adj=0,
> oom_score_adj=0, oom_score_badness=2 (enabled),memcg_scoring=disabled
> CFSClientEventm 
>
> Call Trace:
>  [<ffffffff8043766c>] dump_header+0xe1/0x23e
>  [<ffffffff80437a02>] oom_kill_process+0x6a/0x323
>  [<ffffffff80438443>] out_of_memory+0x4b3/0x50d
>  [<ffffffff8043a4a6>] __alloc_pages_may_oom+0xa2/0xc7
>  [<ffffffff80236f42>] __alloc_pages_nodemask+0x1002/0x17f0
>  [<ffffffff8024bd23>] alloc_pages_current+0x103/0x2b0
>  [<ffffffff8028567f>] sk_page_frag_refill+0x8f/0x160
>  [<ffffffff80295fa0>] tcp_sendmsg+0x560/0xee0
>  [<ffffffff802a5037>] inet_sendmsg+0x67/0x100
>  [<ffffffff80283c9c>] __sock_sendmsg_nosec+0x6c/0x90
>  [<ffffffff80283e85>] sock_sendmsg+0xc5/0xf0
>  [<ffffffff802847b6>] __sys_sendmsg+0x136/0x430
>  [<ffffffff80284ec8>] sys_sendmsg+0x88/0x110
>  [<ffffffff80711472>] system_call_fastpath+0x16/0x1b
> Out of Memory: Kill process 2856 (bash) score 9999 or sacrifice child
>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Acked-by: David Rientjes <rientjes@google.com>
Acked-by: "Eric W. Biederman" <ebiederm@xmission.com>

I have seen similiar order 3 allocation failures as well and reached the
same conclusion that __GFP_NORETRY was the solution.

Eric
> ---
>  net/core/sock.c |    6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/net/core/sock.c b/net/core/sock.c
> index 0c127dcdf6a8..5b6a9431b017 100644
> --- a/net/core/sock.c
> +++ b/net/core/sock.c
> @@ -1775,7 +1775,9 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
>  			while (order) {
>  				if (npages >= 1 << order) {
>  					page = alloc_pages(sk->sk_allocation |
> -							   __GFP_COMP | __GFP_NOWARN,
> +							   __GFP_COMP |
> +							   __GFP_NOWARN |
> +							   __GFP_NORETRY,
>  							   order);
>  					if (page)
>  						goto fill_page;
> @@ -1845,7 +1847,7 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t prio)
>  		gfp_t gfp = prio;
>  
>  		if (order)
> -			gfp |= __GFP_COMP | __GFP_NOWARN;
> +			gfp |= __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY;
>  		pfrag->page = alloc_pages(gfp, order);
>  		if (likely(pfrag->page)) {
>  			pfrag->offset = 0;
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] net: use __GFP_NORETRY for high order allocations

[Eric Dumazet]

On Thu, 2014-02-06 at 18:03 -0800, Eric W. Biederman wrote:
> Eric Dumazet <eric.dumazet@gmail.com> writes:
> 
> > From: Eric Dumazet <edumazet@google.com>
> >
> > sock_alloc_send_pskb() & sk_page_frag_refill()
> > have a loop trying high order allocations to prepare
> > skb with low number of fragments as this increases performance.
> >
> > Problem is that under memory pressure/fragmentation, this can
> > trigger OOM while the intent was only to try the high order
> > allocations, then fallback to order-0 allocations.
> >
> > We had various reports from unexpected regressions.
> >
> > According to David, setting __GFP_NORETRY should be fine,
> > as the asynchronous compaction is still enabled, and this
> > will prevent OOM from kicking as in :
> >
> > CFSClientEventm invoked oom-killer: gfp_mask=0x42d0, order=3, oom_adj=0,
> > oom_score_adj=0, oom_score_badness=2 (enabled),memcg_scoring=disabled
> > CFSClientEventm 
> >
> > Call Trace:
> >  [<ffffffff8043766c>] dump_header+0xe1/0x23e
> >  [<ffffffff80437a02>] oom_kill_process+0x6a/0x323
> >  [<ffffffff80438443>] out_of_memory+0x4b3/0x50d
> >  [<ffffffff8043a4a6>] __alloc_pages_may_oom+0xa2/0xc7
> >  [<ffffffff80236f42>] __alloc_pages_nodemask+0x1002/0x17f0
> >  [<ffffffff8024bd23>] alloc_pages_current+0x103/0x2b0
> >  [<ffffffff8028567f>] sk_page_frag_refill+0x8f/0x160
> >  [<ffffffff80295fa0>] tcp_sendmsg+0x560/0xee0
> >  [<ffffffff802a5037>] inet_sendmsg+0x67/0x100
> >  [<ffffffff80283c9c>] __sock_sendmsg_nosec+0x6c/0x90
> >  [<ffffffff80283e85>] sock_sendmsg+0xc5/0xf0
> >  [<ffffffff802847b6>] __sys_sendmsg+0x136/0x430
> >  [<ffffffff80284ec8>] sys_sendmsg+0x88/0x110
> >  [<ffffffff80711472>] system_call_fastpath+0x16/0x1b
> > Out of Memory: Kill process 2856 (bash) score 9999 or sacrifice child
> >
> > Signed-off-by: Eric Dumazet <edumazet@google.com>
> > Acked-by: David Rientjes <rientjes@google.com>
> Acked-by: "Eric W. Biederman" <ebiederm@xmission.com>
> 
> I have seen similiar order 3 allocation failures as well and reached the
> same conclusion that __GFP_NORETRY was the solution.

Thanks Eric !

Re: [PATCH] net: use __GFP_NORETRY for high order allocations

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 06 Feb 2014 10:42:42 -0800

> From: Eric Dumazet <edumazet@google.com>
> 
> sock_alloc_send_pskb() & sk_page_frag_refill()
> have a loop trying high order allocations to prepare
> skb with low number of fragments as this increases performance.
> 
> Problem is that under memory pressure/fragmentation, this can
> trigger OOM while the intent was only to try the high order
> allocations, then fallback to order-0 allocations.
> 
> We had various reports from unexpected regressions.
> 
> According to David, setting __GFP_NORETRY should be fine,
> as the asynchronous compaction is still enabled, and this
> will prevent OOM from kicking as in :
 ...
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Acked-by: David Rientjes <rientjes@google.com>

Applied, do we want this for -stable?

Re: [PATCH] net: use __GFP_NORETRY for high order allocations

[Eric W. Biederman]

David Miller <davem@davemloft.net> writes:

> From: Eric Dumazet <eric.dumazet@gmail.com>
> Date: Thu, 06 Feb 2014 10:42:42 -0800
>
>> From: Eric Dumazet <edumazet@google.com>
>> 
>> sock_alloc_send_pskb() & sk_page_frag_refill()
>> have a loop trying high order allocations to prepare
>> skb with low number of fragments as this increases performance.
>> 
>> Problem is that under memory pressure/fragmentation, this can
>> trigger OOM while the intent was only to try the high order
>> allocations, then fallback to order-0 allocations.
>> 
>> We had various reports from unexpected regressions.
>> 
>> According to David, setting __GFP_NORETRY should be fine,
>> as the asynchronous compaction is still enabled, and this
>> will prevent OOM from kicking as in :
>  ...
>> Signed-off-by: Eric Dumazet <edumazet@google.com>
>> Acked-by: David Rientjes <rientjes@google.com>
>
> Applied, do we want this for -stable?

The first hunk goes back to 3.12 and the second hunk goes back to 3.8.

I think so.    The change is safe and this class of problem can cause an
external attack to trigger an OOM on your box, by controlling the packet
flow.

Eric

Re: [PATCH] net: use __GFP_NORETRY for high order allocations

[David Miller]

From: ebiederm@xmission.com (Eric W. Biederman)
Date: Fri, 07 Feb 2014 16:22:56 -0800

> David Miller <davem@davemloft.net> writes:
> 
>> From: Eric Dumazet <eric.dumazet@gmail.com>
>> Date: Thu, 06 Feb 2014 10:42:42 -0800
>>
>>> From: Eric Dumazet <edumazet@google.com>
>>> 
>>> sock_alloc_send_pskb() & sk_page_frag_refill()
>>> have a loop trying high order allocations to prepare
>>> skb with low number of fragments as this increases performance.
>>> 
>>> Problem is that under memory pressure/fragmentation, this can
>>> trigger OOM while the intent was only to try the high order
>>> allocations, then fallback to order-0 allocations.
>>> 
>>> We had various reports from unexpected regressions.
>>> 
>>> According to David, setting __GFP_NORETRY should be fine,
>>> as the asynchronous compaction is still enabled, and this
>>> will prevent OOM from kicking as in :
>>  ...
>>> Signed-off-by: Eric Dumazet <edumazet@google.com>
>>> Acked-by: David Rientjes <rientjes@google.com>
>>
>> Applied, do we want this for -stable?
> 
> The first hunk goes back to 3.12 and the second hunk goes back to 3.8.
> 
> I think so.    The change is safe and this class of problem can cause an
> external attack to trigger an OOM on your box, by controlling the packet
> flow.

Great, I'm working integrating this into my -stable queue right now.