[RFC]Page pool buffers stuck in App's socket queue

[RFC]Page pool buffers stuck in App's socket queue

[Ratheesh Kannoth]

Hi,

Recently customer faced a page pool leak issue And keeps on gettting following message in
console.
"page_pool_release_retry() stalled pool shutdown 1 inflight 60 sec"

Customer runs "ping" process in background and then does a interface down/up thru "ip" command.

Marvell octeotx2 driver does destroy all resources (including page pool allocated for each queue of
net device) during interface down event. This page pool destruction will wait for all page pool buffers
allocated by that instance to return to the pool, hence the above message (if some buffers
are stuck).

In the customer scenario, ping App opens both RAW and RAW6 sockets. Even though Customer ping
only ipv4 address, this RAW6 socket receives some IPV6 Router Advertisement messages which gets generated
in their network.

[   41.643448]  raw6_local_deliver+0xc0/0x1d8
[   41.647539]  ip6_protocol_deliver_rcu+0x60/0x490
[   41.652149]  ip6_input_finish+0x48/0x70
[   41.655976]  ip6_input+0x44/0xcc
[   41.659196]  ip6_sublist_rcv_finish+0x48/0x68
[   41.663546]  ip6_sublist_rcv+0x16c/0x22c
[   41.667460]  ipv6_list_rcv+0xf4/0x12c

Those packets will never gets processed. And if customer does a interface down/up, page pool
warnings will be shown in the console.

Customer was asking us for a mechanism to drain these sockets, as they dont want to kill their Apps.
The proposal is to have debugfs which shows "pid  last_processed_skb_time  number_of_packets  socket_fd/inode_number"
for each raw6/raw4 sockets created in the system. and
any write to the debugfs (any specific command) will drain the socket.

1. Could you please comment on the proposal ?
2. Could you suggest a better way ?

-Ratheesh

Re: [RFC]Page pool buffers stuck in App's socket queue

[Yunsheng Lin]

On 2025/6/16 16:05, Ratheesh Kannoth wrote:
> Hi,
> 
> Recently customer faced a page pool leak issue And keeps on gettting following message in
> console.
> "page_pool_release_retry() stalled pool shutdown 1 inflight 60 sec"
> 
> Customer runs "ping" process in background and then does a interface down/up thru "ip" command.
> 
> Marvell octeotx2 driver does destroy all resources (including page pool allocated for each queue of
> net device) during interface down event. This page pool destruction will wait for all page pool buffers
> allocated by that instance to return to the pool, hence the above message (if some buffers
> are stuck).
> 
> In the customer scenario, ping App opens both RAW and RAW6 sockets. Even though Customer ping
> only ipv4 address, this RAW6 socket receives some IPV6 Router Advertisement messages which gets generated
> in their network.
> 
> [   41.643448]  raw6_local_deliver+0xc0/0x1d8
> [   41.647539]  ip6_protocol_deliver_rcu+0x60/0x490
> [   41.652149]  ip6_input_finish+0x48/0x70
> [   41.655976]  ip6_input+0x44/0xcc
> [   41.659196]  ip6_sublist_rcv_finish+0x48/0x68
> [   41.663546]  ip6_sublist_rcv+0x16c/0x22c
> [   41.667460]  ipv6_list_rcv+0xf4/0x12c
> 
> Those packets will never gets processed. And if customer does a interface down/up, page pool
> warnings will be shown in the console.
> 
> Customer was asking us for a mechanism to drain these sockets, as they dont want to kill their Apps.
> The proposal is to have debugfs which shows "pid  last_processed_skb_time  number_of_packets  socket_fd/inode_number"
> for each raw6/raw4 sockets created in the system. and
> any write to the debugfs (any specific command) will drain the socket.
> 
> 1. Could you please comment on the proposal ?

I would say the above is kind of working around the problem.
It would be good to fix the Apps or fix the page_pool.

> 2. Could you suggest a better way ?

For fixing the page_pool part, I would be suggesting to keep track
of all the inflight pages and detach those pages from page_pool when
page_pool_destroy() is called, the tracking part was [1], unfortunately
the maintainers seemed to choose an easy way instead of a long term
direction, see [2].

1. https://lore.kernel.org/all/20250307092356.638242-1-linyunsheng@huawei.com/
2. https://lore.kernel.org/all/20250409-page-pool-track-dma-v9-0-6a9ef2e0cba8@redhat.com/

Re: [RFC]Page pool buffers stuck in App's socket queue

[Ratheesh Kannoth]

On 2025-06-17 at 12:03:41, Yunsheng Lin (linyunsheng@huawei.com) wrote:
> > Customer was asking us for a mechanism to drain these sockets, as they dont want to kill their Apps.
> > The proposal is to have debugfs which shows "pid  last_processed_skb_time  number_of_packets  socket_fd/inode_number"
> > for each raw6/raw4 sockets created in the system. and
> > any write to the debugfs (any specific command) will drain the socket.
> >
> > 1. Could you please comment on the proposal ?
>
> I would say the above is kind of working around the problem.
> It would be good to fix the Apps or fix the page_pool.
>
> > 2. Could you suggest a better way ?
>
> For fixing the page_pool part, I would be suggesting to keep track
> of all the inflight pages and detach those pages from page_pool when
> page_pool_destroy() is called, the tracking part was [1], unfortunately
> the maintainers seemed to choose an easy way instead of a long term
> direction, see [2].
>
> 1. https://lore.kernel.org/all/20250307092356.638242-1-linyunsheng@huawei.com/
> 2. https://lore.kernel.org/all/20250409-page-pool-track-dma-v9-0-6a9ef2e0cba8@redhat.com/

i think, both issues are different. Above links, fixes late DMA unmap issue. In my case,
buffers do not return at all. So some entity should purge those socket queues which are not getting
processed. App modification would be difficult as customer may use opensource Apps/third party apps.

Buffers are stuck in socket queue. And we need to free them. Page pool infra tracking socket queue and
process information would be an over kill ? And packets in socket queue of a process, being freed from page pool wont
work well. (That app could be killed later, causing skbs to be freed. or we dont know whether App processing socket queue
is slow).

I was thinking of extending /proc/net/raw and /proc/net/raw6 OR creating similar proc entries; which
can show "pid and timestamp of first skb in the queue" etc. And a .proc_write() function to purge a
specifc socket queue. Using these, customer can decide which queue is stuck and purge manualy (thru some script) after
netdevice down/up.

Re: [RFC]Page pool buffers stuck in App's socket queue

[Mina Almasry]

On Mon, Jun 16, 2025 at 11:34 PM Yunsheng Lin <linyunsheng@huawei.com> wrote:
>
> On 2025/6/16 16:05, Ratheesh Kannoth wrote:
> > Hi,
> >
> > Recently customer faced a page pool leak issue And keeps on gettting following message in
> > console.
> > "page_pool_release_retry() stalled pool shutdown 1 inflight 60 sec"
> >
> > Customer runs "ping" process in background and then does a interface down/up thru "ip" command.
> >
> > Marvell octeotx2 driver does destroy all resources (including page pool allocated for each queue of
> > net device) during interface down event. This page pool destruction will wait for all page pool buffers
> > allocated by that instance to return to the pool, hence the above message (if some buffers
> > are stuck).
> >
> > In the customer scenario, ping App opens both RAW and RAW6 sockets. Even though Customer ping
> > only ipv4 address, this RAW6 socket receives some IPV6 Router Advertisement messages which gets generated
> > in their network.
> >
> > [   41.643448]  raw6_local_deliver+0xc0/0x1d8
> > [   41.647539]  ip6_protocol_deliver_rcu+0x60/0x490
> > [   41.652149]  ip6_input_finish+0x48/0x70
> > [   41.655976]  ip6_input+0x44/0xcc
> > [   41.659196]  ip6_sublist_rcv_finish+0x48/0x68
> > [   41.663546]  ip6_sublist_rcv+0x16c/0x22c
> > [   41.667460]  ipv6_list_rcv+0xf4/0x12c
> >
> > Those packets will never gets processed. And if customer does a interface down/up, page pool
> > warnings will be shown in the console.
> >
> > Customer was asking us for a mechanism to drain these sockets, as they dont want to kill their Apps.
> > The proposal is to have debugfs which shows "pid  last_processed_skb_time  number_of_packets  socket_fd/inode_number"
> > for each raw6/raw4 sockets created in the system. and
> > any write to the debugfs (any specific command) will drain the socket.
> >
> > 1. Could you please comment on the proposal ?
>
> I would say the above is kind of working around the problem.
> It would be good to fix the Apps or fix the page_pool.
>
> > 2. Could you suggest a better way ?
>
> For fixing the page_pool part, I would be suggesting to keep track
> of all the inflight pages and detach those pages from page_pool when
> page_pool_destroy() is called, the tracking part was [1], unfortunately
> the maintainers seemed to choose an easy way instead of a long term
> direction, see [2].

This is not that accurate IMO. Your patch series and the merged patch
series from Toke does the same thing: both keep track of dma-mapped
pages, so that they can be unmapped at page_pool_destroy time. Toke
just did the tracking in a simpler way that people were willing to
review.

So, if you had a plan to detach pages on page_pool_destroy on top of
your tracking, the exact same plan should work on top of Toke's
tracking. It may be useful to code that and send an RFC if you have
time. It would indeed fix this periodic warning issue.

-- 
Thanks,
Mina

Re: [RFC]Page pool buffers stuck in App's socket queue

[Yunsheng Lin]

On 2025/6/18 5:02, Mina Almasry wrote:
> On Mon, Jun 16, 2025 at 11:34 PM Yunsheng Lin <linyunsheng@huawei.com> wrote:
>>
>> On 2025/6/16 16:05, Ratheesh Kannoth wrote:
>>> Hi,
>>>
>>> Recently customer faced a page pool leak issue And keeps on gettting following message in
>>> console.
>>> "page_pool_release_retry() stalled pool shutdown 1 inflight 60 sec"
>>>
>>> Customer runs "ping" process in background and then does a interface down/up thru "ip" command.
>>>
>>> Marvell octeotx2 driver does destroy all resources (including page pool allocated for each queue of
>>> net device) during interface down event. This page pool destruction will wait for all page pool buffers
>>> allocated by that instance to return to the pool, hence the above message (if some buffers
>>> are stuck).
>>>
>>> In the customer scenario, ping App opens both RAW and RAW6 sockets. Even though Customer ping
>>> only ipv4 address, this RAW6 socket receives some IPV6 Router Advertisement messages which gets generated
>>> in their network.
>>>
>>> [   41.643448]  raw6_local_deliver+0xc0/0x1d8
>>> [   41.647539]  ip6_protocol_deliver_rcu+0x60/0x490
>>> [   41.652149]  ip6_input_finish+0x48/0x70
>>> [   41.655976]  ip6_input+0x44/0xcc
>>> [   41.659196]  ip6_sublist_rcv_finish+0x48/0x68
>>> [   41.663546]  ip6_sublist_rcv+0x16c/0x22c
>>> [   41.667460]  ipv6_list_rcv+0xf4/0x12c
>>>
>>> Those packets will never gets processed. And if customer does a interface down/up, page pool
>>> warnings will be shown in the console.
>>>
>>> Customer was asking us for a mechanism to drain these sockets, as they dont want to kill their Apps.
>>> The proposal is to have debugfs which shows "pid  last_processed_skb_time  number_of_packets  socket_fd/inode_number"
>>> for each raw6/raw4 sockets created in the system. and
>>> any write to the debugfs (any specific command) will drain the socket.
>>>
>>> 1. Could you please comment on the proposal ?
>>
>> I would say the above is kind of working around the problem.
>> It would be good to fix the Apps or fix the page_pool.
>>
>>> 2. Could you suggest a better way ?
>>
>> For fixing the page_pool part, I would be suggesting to keep track
>> of all the inflight pages and detach those pages from page_pool when
>> page_pool_destroy() is called, the tracking part was [1], unfortunately
>> the maintainers seemed to choose an easy way instead of a long term
>> direction, see [2].
> 
> This is not that accurate IMO. Your patch series and the merged patch
> series from Toke does the same thing: both keep track of dma-mapped

The main difference is the above 'dma-mapped'.

> pages, so that they can be unmapped at page_pool_destroy time. Toke
> just did the tracking in a simpler way that people were willing to
> review.
> 
> So, if you had a plan to detach pages on page_pool_destroy on top of
> your tracking, the exact same plan should work on top of Toke's
> tracking. It may be useful to code that and send an RFC if you have
> time. It would indeed fix this periodic warning issue.

As above, when page_pool_create() is called without PP_FLAG_DMA_MAP, the
dma-mapped pages tracking is not enough.

Re: [RFC]Page pool buffers stuck in App's socket queue

[Mina Almasry]

On Mon, Jun 16, 2025 at 1:06 AM Ratheesh Kannoth <rkannoth@marvell.com> wrote:
>
> Hi,
>
> Recently customer faced a page pool leak issue And keeps on gettting following message in
> console.
> "page_pool_release_retry() stalled pool shutdown 1 inflight 60 sec"
>

This is not exactly a 'leak' per say. The page_pool doesn't allow
itself to exit until all the packets that came from it are freed. The
line just tells the user this is happening.

> Customer runs "ping" process in background and then does a interface down/up thru "ip" command.
>
> Marvell octeotx2 driver does destroy all resources (including page pool allocated for each queue of
> net device) during interface down event. This page pool destruction will wait for all page pool buffers
> allocated by that instance to return to the pool, hence the above message (if some buffers
> are stuck).
>
> In the customer scenario, ping App opens both RAW and RAW6 sockets. Even though Customer ping
> only ipv4 address, this RAW6 socket receives some IPV6 Router Advertisement messages which gets generated
> in their network.
>
> [   41.643448]  raw6_local_deliver+0xc0/0x1d8
> [   41.647539]  ip6_protocol_deliver_rcu+0x60/0x490
> [   41.652149]  ip6_input_finish+0x48/0x70
> [   41.655976]  ip6_input+0x44/0xcc
> [   41.659196]  ip6_sublist_rcv_finish+0x48/0x68
> [   41.663546]  ip6_sublist_rcv+0x16c/0x22c
> [   41.667460]  ipv6_list_rcv+0xf4/0x12c
>
> Those packets will never gets processed. And if customer does a interface down/up, page pool
> warnings will be shown in the console.
>

Right, I have a few recommendations here:

1. Check that commit be0096676e23 ("net: page_pool: mute the periodic
warning for visible page pools") is in your kernel. That mutes
warnings for visible page_pools.

2. Fix the application to not leave behind these RAW6 socket data.
Either processing the data incoming in the socket or closing the
socket itself would be sufficient.

> Customer was asking us for a mechanism to drain these sockets, as they dont want to kill their Apps.
> The proposal is to have debugfs which shows "pid  last_processed_skb_time  number_of_packets  socket_fd/inode_number"
> for each raw6/raw4 sockets created in the system. and
> any write to the debugfs (any specific command) will drain the socket.
>
> 1. Could you please comment on the proposal ?

Oh boy. I don't think this would fly at all. The userspace simply
closing the RAW6 socket would 'fix' the issue, unless I'm missing
something.

Having a roundabout debugfs entry that does the same thing that
`close(socket_fd);` would do is going to be a very hard sell upstream.

I think we could also mute the page_pool warning or make it less
visible. The kernel usually doesn't warn when the userspace is leaking
data.

We could also do what Yunsheng suggests and actually disconnect the
pages from the page_pool and let the page_pool clean up, but that may
be a complicated change.

Honsetly there are a lot of better solutions here than this debugfs file.

-- 
Thanks,
Mina

Re: [RFC]Page pool buffers stuck in App's socket queue

[Ratheesh Kannoth]

On 2025-06-18 at 02:30:04, Mina Almasry (almasrymina@google.com) wrote:
> >
> > Those packets will never gets processed. And if customer does a interface down/up, page pool
> > warnings will be shown in the console.
> >
>
> Right, I have a few recommendations here:
>
> 1. Check that commit be0096676e23 ("net: page_pool: mute the periodic
> warning for visible page pools") is in your kernel. That mutes
> warnings for visible page_pools.

Thanks. netdevice is not gettting unregistered in octeontx2 driver while interface
is brought down; but page pool is destroyed.

>
> 2. Fix the application to not leave behind these RAW6 socket data.
> Either processing the data incoming in the socket or closing the
> socket itself would be sufficient.

Customer is using opensource ping (from iptutils). They run a
ping to an ipv4 address (ping 192.x.x.x -A &). Here the app opens both
RAW4 and RAW6 sockets. IPv6 router advertisement messages from the network
lands up in this RAW6 socket qeueue. And ping App never dequeue from this RAW6
socket queue. They want to avoid killing and starting the ping APP.
Customer creates  RAW6 socket with other Apps also (third party) and page pool
issues pops up there as well.  Ping App reproduction steps are shared to us;
so quoting the same.

>
> > Customer was asking us for a mechanism to drain these sockets, as they dont want to kill their Apps.
> > The proposal is to have debugfs which shows "pid  last_processed_skb_time  number_of_packets  socket_fd/inode_number"
> > for each raw6/raw4 sockets created in the system. and
> > any write to the debugfs (any specific command) will drain the socket.
> >
> > 1. Could you please comment on the proposal ?
>
> Oh boy. I don't think this would fly at all. The userspace simply
> closing the RAW6 socket would 'fix' the issue, unless I'm missing
> something.
>
> Having a roundabout debugfs entry that does the same thing that
> `close(socket_fd);` would do is going to be a very hard sell upstream.

>
> I think we could also mute the page_pool warning or make it less
> visible. The kernel usually doesn't warn when the userspace is leaking
> data.
>
> We could also do what Yunsheng suggests and actually disconnect the
> pages from the page_pool and let the page_pool clean up, but that may
> be a complicated change.
>
> Honsetly there are a lot of better solutions here than this debugfs file.
Thanks a lot !. Could you suggest some solutions. I will try to work on those.

>
> --
> Thanks,
> Mina

Re: [RFC]Page pool buffers stuck in App's socket queue

[Lorenzo Bianconi]

[-- Attachment #1: Type: text/plain, Size: 2059 bytes --]

> Hi,
> 
> Recently customer faced a page pool leak issue And keeps on gettting following message in
> console.
> "page_pool_release_retry() stalled pool shutdown 1 inflight 60 sec"
> 
> Customer runs "ping" process in background and then does a interface down/up thru "ip" command.
> 
> Marvell octeotx2 driver does destroy all resources (including page pool allocated for each queue of
> net device) during interface down event. This page pool destruction will wait for all page pool buffers
> allocated by that instance to return to the pool, hence the above message (if some buffers
> are stuck).
> 
> In the customer scenario, ping App opens both RAW and RAW6 sockets. Even though Customer ping
> only ipv4 address, this RAW6 socket receives some IPV6 Router Advertisement messages which gets generated
> in their network.
> 
> [   41.643448]  raw6_local_deliver+0xc0/0x1d8
> [   41.647539]  ip6_protocol_deliver_rcu+0x60/0x490
> [   41.652149]  ip6_input_finish+0x48/0x70
> [   41.655976]  ip6_input+0x44/0xcc
> [   41.659196]  ip6_sublist_rcv_finish+0x48/0x68
> [   41.663546]  ip6_sublist_rcv+0x16c/0x22c
> [   41.667460]  ipv6_list_rcv+0xf4/0x12c
> 
> Those packets will never gets processed. And if customer does a interface down/up, page pool
> warnings will be shown in the console.
> 
> Customer was asking us for a mechanism to drain these sockets, as they dont want to kill their Apps.
> The proposal is to have debugfs which shows "pid  last_processed_skb_time  number_of_packets  socket_fd/inode_number"
> for each raw6/raw4 sockets created in the system. and
> any write to the debugfs (any specific command) will drain the socket.
> 
> 1. Could you please comment on the proposal ?
> 2. Could you suggest a better way ?
> 
> -Ratheesh

Hi,

this problem recall me an issue I had in the past with page_pool
and TCP traffic destroying the pool (not sure if it is still valid):

https://lore.kernel.org/netdev/ZD2HjZZSOjtsnQaf@lore-desk/

Do you have ongoing TCP flows?

Regards,
Lorenzo

> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [RFC]Page pool buffers stuck in App's socket queue

[Ratheesh Kannoth]

On 2025-06-18 at 03:26:53, Lorenzo Bianconi (lorenzo.bianconi@redhat.com) wrote:
>
> Hi,
>
> this problem recall me an issue I had in the past with page_pool
> and TCP traffic destroying the pool (not sure if it is still valid):
>
> https://lore.kernel.org/netdev/ZD2HjZZSOjtsnQaf@lore-desk/
>
> Do you have ongoing TCP flows?
No. it is just ping running. If we kill the ping process, Page pool prints stops.