[PATCH] r8169: avoid OOM when allocating RX buffers

[PATCH] r8169: avoid OOM when allocating RX buffers

[Fabian Druschke]

From: Fabian Druschke <fdruschke@outlook.com>

r8169 allocates order-2 pages for RX buffers during rtl_open(). Under heavy
memory fragmentation this allocation may trigger the global OOM killer,
causing unrelated user processes to be killed.

Use a GFP mask that avoids OOM killer invocation so the allocation can fail
gracefully and rtl_open() returns -ENOMEM instead.

Cc: stable@vger.kernel.org
Signed-off-by: Fabian Druschke <fdruschke@outlook.com>
---
 drivers/net/ethernet/realtek/r8169_main.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
index 3507c2e28110..3525e889ec1c 100644
--- a/drivers/net/ethernet/realtek/r8169_main.c
+++ b/drivers/net/ethernet/realtek/r8169_main.c
@@ -3952,7 +3952,8 @@ static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
 	dma_addr_t mapping;
 	struct page *data;
 
-	data = alloc_pages_node(node, GFP_KERNEL, get_order(R8169_RX_BUF_SIZE));
+	gfp_t gfp = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN;
+	data = alloc_pages_node(node, gfp, get_order(R8169_RX_BUF_SIZE));
 	if (!data)
 		return NULL;
 
-- 
2.52.0

Re: [PATCH] r8169: avoid OOM when allocating RX buffers

[Eric Dumazet]

On Mon, Feb 16, 2026 at 7:53 PM Fabian Druschke <fabian@druschke.network> wrote:
>
> From: Fabian Druschke <fdruschke@outlook.com>
>
> r8169 allocates order-2 pages for RX buffers during rtl_open(). Under heavy
> memory fragmentation this allocation may trigger the global OOM killer,
> causing unrelated user processes to be killed.
>
> Use a GFP mask that avoids OOM killer invocation so the allocation can fail
> gracefully and rtl_open() returns -ENOMEM instead.
>
> Cc: stable@vger.kernel.org
> Signed-off-by: Fabian Druschke <fdruschke@outlook.com>
> ---
>  drivers/net/ethernet/realtek/r8169_main.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
> index 3507c2e28110..3525e889ec1c 100644
> --- a/drivers/net/ethernet/realtek/r8169_main.c
> +++ b/drivers/net/ethernet/realtek/r8169_main.c
> @@ -3952,7 +3952,8 @@ static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
>         dma_addr_t mapping;
>         struct page *data;
>
> -       data = alloc_pages_node(node, GFP_KERNEL, get_order(R8169_RX_BUF_SIZE));
> +       gfp_t gfp = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN;
> +       data = alloc_pages_node(node, gfp, get_order(R8169_RX_BUF_SIZE));
>         if (!data)
>                 return NULL;

Control path prefers to wait a bit so that the NIC can be setup, this
could be used for instance to enable swaping over the network.

Note that this is GFP_KERNEL here, not GFP_KERNEL_ACCOUNT , OOM seems
reasonable here.

Re: [PATCH] r8169: avoid OOM when allocating RX buffers

[Andrew Lunn]

On Mon, Feb 16, 2026 at 07:52:45PM +0100, Fabian Druschke wrote:
> From: Fabian Druschke <fdruschke@outlook.com>
> 
> r8169 allocates order-2 pages for RX buffers during rtl_open(). Under heavy
> memory fragmentation this allocation may trigger the global OOM killer,
> causing unrelated user processes to be killed.
> 
> Use a GFP mask that avoids OOM killer invocation so the allocation can fail
> gracefully and rtl_open() returns -ENOMEM instead.
> 
> Cc: stable@vger.kernel.org
> Signed-off-by: Fabian Druschke <fdruschke@outlook.com>
> ---
>  drivers/net/ethernet/realtek/r8169_main.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
> index 3507c2e28110..3525e889ec1c 100644
> --- a/drivers/net/ethernet/realtek/r8169_main.c
> +++ b/drivers/net/ethernet/realtek/r8169_main.c
> @@ -3952,7 +3952,8 @@ static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
>  	dma_addr_t mapping;
>  	struct page *data;
>  
> -	data = alloc_pages_node(node, GFP_KERNEL, get_order(R8169_RX_BUF_SIZE));
> +	gfp_t gfp = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN;
> +	data = alloc_pages_node(node, gfp, get_order(R8169_RX_BUF_SIZE));
>  	if (!data)
>  		return NULL;

~/linux/drivers/net$ grep -r alloc_pages_node
ethernet/chelsio/cxgb4/cxgb4_main.c:		newpage = alloc_pages_node(node, __GFP_NOWARN | GFP_KERNEL |
ethernet/chelsio/cxgb4/sge.c:		pg = alloc_pages_node(node, gfp | __GFP_COMP, s->fl_pg_order);
ethernet/chelsio/cxgb4/sge.c:		pg = alloc_pages_node(node, gfp, 0);
ethernet/amd/xgbe/xgbe-desc.c:		pages = alloc_pages_node(node, gfp, order);
ethernet/fungible/funcore/fun_queue.c:		rqinfo->page = alloc_pages_node(node, GFP_KERNEL, 0);
ethernet/fungible/funeth/funeth_rx.c:	p = __alloc_pages_node(node, gfp | __GFP_NOWARN, 0);
ethernet/mellanox/mlx5/core/pagealloc.c:	page = alloc_pages_node(nid, GFP_HIGHUSER, 0);
ethernet/mellanox/mlx5/core/en_main.c:		struct page *page = alloc_pages_node(node, GFP_KERNEL, 0);
ethernet/mellanox/mlx4/icm.c:	page = alloc_pages_node(node, gfp_mask, order);
ethernet/realtek/r8169_main.c:	data = alloc_pages_node(node, GFP_KERNEL, get_order(R8169_RX_BUF_SIZE));
ethernet/google/gve/gve_main.c:	*page = alloc_pages_node(priv->numa_node, gfp_flags, 0);
ethernet/google/gve/gve_rx.c:			struct page *page = alloc_pages_node(priv->numa_node,
ethernet/google/gve/gve_rx_dqo.c:	struct page *page = alloc_pages_node(rx->gve->numa_node, GFP_ATOMIC, 0);
ethernet/hisilicon/hns3/hns3_enet.c:	page = alloc_pages_node(dev_to_node(ring_to_dev(ring)),

:~/linux/drivers/net$ grep -r __GFP_RETRY_MAYFAIL
veth.c:			    GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL);

What makes the r8169 special?

     Andrew

Re: [PATCH] r8169: avoid OOM when allocating RX buffers

[Fabian Druschke]

Ahoy! Thanks for clarification! Didn't know it was intended behaviour.

We've encountered this issue specifically with this Realtek NIC on 
ShredOS due to lack of mlx5, mlx4 etc.

For NICs like ixgbe we didn't encounter this issue so i was thinking 
about a bug.

Thanks though!


BR,


On 16/02/2026 21:13, Andrew Lunn wrote:
> On Mon, Feb 16, 2026 at 07:52:45PM +0100, Fabian Druschke wrote:
>> From: Fabian Druschke <fdruschke@outlook.com>
>>
>> r8169 allocates order-2 pages for RX buffers during rtl_open(). Under heavy
>> memory fragmentation this allocation may trigger the global OOM killer,
>> causing unrelated user processes to be killed.
>>
>> Use a GFP mask that avoids OOM killer invocation so the allocation can fail
>> gracefully and rtl_open() returns -ENOMEM instead.
>>
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Fabian Druschke <fdruschke@outlook.com>
>> ---
>>   drivers/net/ethernet/realtek/r8169_main.c | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
>> index 3507c2e28110..3525e889ec1c 100644
>> --- a/drivers/net/ethernet/realtek/r8169_main.c
>> +++ b/drivers/net/ethernet/realtek/r8169_main.c
>> @@ -3952,7 +3952,8 @@ static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
>>   	dma_addr_t mapping;
>>   	struct page *data;
>>   
>> -	data = alloc_pages_node(node, GFP_KERNEL, get_order(R8169_RX_BUF_SIZE));
>> +	gfp_t gfp = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN;
>> +	data = alloc_pages_node(node, gfp, get_order(R8169_RX_BUF_SIZE));
>>   	if (!data)
>>   		return NULL;
> ~/linux/drivers/net$ grep -r alloc_pages_node
> ethernet/chelsio/cxgb4/cxgb4_main.c:		newpage = alloc_pages_node(node, __GFP_NOWARN | GFP_KERNEL |
> ethernet/chelsio/cxgb4/sge.c:		pg = alloc_pages_node(node, gfp | __GFP_COMP, s->fl_pg_order);
> ethernet/chelsio/cxgb4/sge.c:		pg = alloc_pages_node(node, gfp, 0);
> ethernet/amd/xgbe/xgbe-desc.c:		pages = alloc_pages_node(node, gfp, order);
> ethernet/fungible/funcore/fun_queue.c:		rqinfo->page = alloc_pages_node(node, GFP_KERNEL, 0);
> ethernet/fungible/funeth/funeth_rx.c:	p = __alloc_pages_node(node, gfp | __GFP_NOWARN, 0);
> ethernet/mellanox/mlx5/core/pagealloc.c:	page = alloc_pages_node(nid, GFP_HIGHUSER, 0);
> ethernet/mellanox/mlx5/core/en_main.c:		struct page *page = alloc_pages_node(node, GFP_KERNEL, 0);
> ethernet/mellanox/mlx4/icm.c:	page = alloc_pages_node(node, gfp_mask, order);
> ethernet/realtek/r8169_main.c:	data = alloc_pages_node(node, GFP_KERNEL, get_order(R8169_RX_BUF_SIZE));
> ethernet/google/gve/gve_main.c:	*page = alloc_pages_node(priv->numa_node, gfp_flags, 0);
> ethernet/google/gve/gve_rx.c:			struct page *page = alloc_pages_node(priv->numa_node,
> ethernet/google/gve/gve_rx_dqo.c:	struct page *page = alloc_pages_node(rx->gve->numa_node, GFP_ATOMIC, 0);
> ethernet/hisilicon/hns3/hns3_enet.c:	page = alloc_pages_node(dev_to_node(ring_to_dev(ring)),
>
> :~/linux/drivers/net$ grep -r __GFP_RETRY_MAYFAIL
> veth.c:			    GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL);
>
> What makes the r8169 special?
>
>       Andrew

Re: [PATCH] r8169: avoid OOM when allocating RX buffers

[Jakub Kicinski]

On Tue, 17 Feb 2026 22:50:30 +0100 Fabian Druschke wrote:
> Ahoy! Thanks for clarification! Didn't know it was intended behaviour.
> 
> We've encountered this issue specifically with this Realtek NIC on 
> ShredOS due to lack of mlx5, mlx4 etc.
> 
> For NICs like ixgbe we didn't encounter this issue so i was thinking 
> about a bug.

Most / all "professional grade"(??) NICs support scatter, where larger 
frames are written into multiple chunks, 4kB each. order-2 allocations
on the fast path are a bad idea. One way to alleviate the performance
implications would be to use page pool, but that doesn't help with the
initial fill, just the datapath :(

reminder: please avoid top posting when replying on the mailing list