Xen NIC driver have page_pool memory leaks

Xen NIC driver have page_pool memory leaks

[Jesper Dangaard Brouer]

Hi Arthur,

(Answer inlined below, which is custom on this mailing list)

On 23/03/2024 14.23, Arthur Borsboom wrote:
> Hi Jesper,
> 
> After a recent kernel upgrade 6.7.6 > 6.8.1 all my Xen guests on Arch
> Linux are dumping kernel traces.
> It seems to be indirectly caused by the page pool memory leak
> mechanism, which is probably a good thing.
> 
> I have created a bug report, but there is no response.
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=218618
> 
> I am uncertain where and to whom I need to report this page leak.
> Can you help me get this issue fixed?

I'm the page_pool maintainer, but as you say yourself in comment 2 then
since dba1b8a7ab68 ("mm/page_pool: catch page_pool memory leaks") this
indicated there is a problem in the xen_netfront driver, which was
previously not visible.

Cc'ing the "XEN NETWORK BACKEND DRIVER" maintainers, as this is a driver
bug.  What confuses me it that I cannot find any modules named
"xen_netfront" in the upstream tree.

IPSEC recently had a similar leak bug. That was fixed in commmit [1]
c3198822c6cb ("net: esp: fix bad handling of pages from page_pool").
This xen_netfront driver likely needs a similar fix.

--Jesper

[1] https://git.kernel.org/torvalds/c/c3198822c6cb


Text copied from bugzilla.kernel.org:
-------------------------------------
[88847.284348] Call Trace:
[88847.284354]  <IRQ>
[88847.284361]  dump_stack_lvl+0x47/0x60
[88847.284378]  bad_page+0x71/0x100
[88847.284393]  free_unref_page_prepare+0x236/0x390
[88847.284405]  free_unref_page+0x34/0x180
[88847.284416]  __pskb_pull_tail+0x3ff/0x4a0
[88847.284432]  xennet_poll+0x909/0xa40 [xen_netfront 
12c02fdcf84c692965d9cd6ca5a6ff0a530b4ce9]
[88847.284470]  __napi_poll+0x28/0x1b0
[88847.284483]  net_rx_action+0x2b5/0x370
[88847.284495]  ? handle_irq_desc+0x3e/0x60
[88847.284511]  __do_softirq+0xc9/0x2c8
[88847.284523]  __irq_exit_rcu+0xa3/0xc0
[88847.284536]  sysvec_xen_hvm_callback+0x72/0x90
[88847.284545]  </IRQ>
[88847.284549]  <TASK>
[88847.284552]  asm_sysvec_xen_hvm_callback+0x1a/0x20
[88847.284562] RIP: 0010:pv_native_safe_halt+0xf/0x20
[88847.284572] Code: 22 d7 c3 cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 
90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d e3 13 27 00 fb 
f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
[88847.284579] RSP: 0018:ffffb2a1800c3e58 EFLAGS: 00000246
[88847.284587] RAX: 0000000000004000 RBX: ffff91358033b864 RCX: 
000051404aebd79d
[88847.284594] RDX: ffff9136f9b00000 RSI: ffff91358033b800 RDI: 
0000000000000001
[88847.284599] RBP: ffff91358033b864 R08: ffffffff9b94dca0 R09: 
0000000000000001
[88847.284604] R10: 0000000000000018 R11: ffff9136f9b331a4 R12: 
ffffffff9b94dca0
[88847.284609] R13: ffffffff9b94dd20 R14: 0000000000000001 R15: 
0000000000000000
[88847.284623]  acpi_safe_halt+0x15/0x30
[88847.284634]  acpi_idle_do_entry+0x2f/0x50
[88847.284644]  acpi_idle_enter+0x7f/0xd0
[88847.284655]  cpuidle_enter_state+0x81/0x440
[88847.284667]  cpuidle_enter+0x2d/0x40
[88847.284678]  do_idle+0x1d8/0x230
[88847.284688]  cpu_startup_entry+0x2a/0x30
[88847.284695]  start_secondary+0x11e/0x140
[88847.284705]  secondary_startup_64_no_verify+0x184/0x18b
[88847.284725]  </TASK>

Re: Xen NIC driver have page_pool memory leaks

[Paul Durrant]

On 25/03/2024 12:21, Jesper Dangaard Brouer wrote:
> Hi Arthur,
> 
> (Answer inlined below, which is custom on this mailing list)
> 
> On 23/03/2024 14.23, Arthur Borsboom wrote:
>> Hi Jesper,
>>
>> After a recent kernel upgrade 6.7.6 > 6.8.1 all my Xen guests on Arch
>> Linux are dumping kernel traces.
>> It seems to be indirectly caused by the page pool memory leak
>> mechanism, which is probably a good thing.
>>
>> I have created a bug report, but there is no response.
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=218618
>>
>> I am uncertain where and to whom I need to report this page leak.
>> Can you help me get this issue fixed?
> 
> I'm the page_pool maintainer, but as you say yourself in comment 2 then
> since dba1b8a7ab68 ("mm/page_pool: catch page_pool memory leaks") this
> indicated there is a problem in the xen_netfront driver, which was
> previously not visible.
> 
> Cc'ing the "XEN NETWORK BACKEND DRIVER" maintainers, as this is a driver
> bug.  What confuses me it that I cannot find any modules named
> "xen_netfront" in the upstream tree.
> 

You should have tried '-' rather than '_' :-)

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/xen-netfront.c

Re: Xen NIC driver have page_pool memory leaks

[Jesper Dangaard Brouer]

On 25/03/2024 13.33, Paul Durrant wrote:
> On 25/03/2024 12:21, Jesper Dangaard Brouer wrote:
>> Hi Arthur,
>>
>> (Answer inlined below, which is custom on this mailing list)
>>
>> On 23/03/2024 14.23, Arthur Borsboom wrote:
>>> Hi Jesper,
>>>
>>> After a recent kernel upgrade 6.7.6 > 6.8.1 all my Xen guests on Arch
>>> Linux are dumping kernel traces.
>>> It seems to be indirectly caused by the page pool memory leak
>>> mechanism, which is probably a good thing.
>>>
>>> I have created a bug report, but there is no response.
>>>
>>> https://bugzilla.kernel.org/show_bug.cgi?id=218618
>>>
>>> I am uncertain where and to whom I need to report this page leak.
>>> Can you help me get this issue fixed?
>>
>> I'm the page_pool maintainer, but as you say yourself in comment 2 then
>> since dba1b8a7ab68 ("mm/page_pool: catch page_pool memory leaks") this
>> indicated there is a problem in the xen_netfront driver, which was
>> previously not visible.
>>
>> Cc'ing the "XEN NETWORK BACKEND DRIVER" maintainers, as this is a driver
>> bug.  What confuses me it that I cannot find any modules named
>> "xen_netfront" in the upstream tree.
>>
> 
> You should have tried '-' rather than '_' :-)
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/xen-netfront.c
> 

Looking at this driver, I think it is missing a call to 
skb_mark_for_recycle().

I'll will submit at patch for this, with details for stable maintainers.

As I think it dates back to v5.9 via commit 6c5aa6fc4def ("xen
networking: add basic XDP support for xen-netfront"). I think this
commit is missing a call to page_pool_release_page()
between v5.9 to v5.14, after which is should have used
skb_mark_for_recycle().

Since v6.6 the call page_pool_release_page() were removed (in
535b9c61bdef ("net: page_pool: hide page_pool_release_page()") and
remaining callers converted (in commit 6bfef2ec0172 ("Merge branch
'net-page_pool-remove-page_pool_release_page'")).

This leak became visible in v6.8 via commit dba1b8a7ab68 ("mm/page_pool:
catch page_pool memory leaks").

--Jesper

Re: Xen NIC driver have page_pool memory leaks

[Andrew Cooper]

On 27/03/2024 11:27 am, Jesper Dangaard Brouer wrote:
>
>
> On 25/03/2024 13.33, Paul Durrant wrote:
>> On 25/03/2024 12:21, Jesper Dangaard Brouer wrote:
>>> Hi Arthur,
>>>
>>> (Answer inlined below, which is custom on this mailing list)
>>>
>>> On 23/03/2024 14.23, Arthur Borsboom wrote:
>>>> Hi Jesper,
>>>>
>>>> After a recent kernel upgrade 6.7.6 > 6.8.1 all my Xen guests on Arch
>>>> Linux are dumping kernel traces.
>>>> It seems to be indirectly caused by the page pool memory leak
>>>> mechanism, which is probably a good thing.
>>>>
>>>> I have created a bug report, but there is no response.
>>>>
>>>> https://bugzilla.kernel.org/show_bug.cgi?id=218618
>>>>
>>>> I am uncertain where and to whom I need to report this page leak.
>>>> Can you help me get this issue fixed?
>>>
>>> I'm the page_pool maintainer, but as you say yourself in comment 2 then
>>> since dba1b8a7ab68 ("mm/page_pool: catch page_pool memory leaks") this
>>> indicated there is a problem in the xen_netfront driver, which was
>>> previously not visible.
>>>
>>> Cc'ing the "XEN NETWORK BACKEND DRIVER" maintainers, as this is a
>>> driver
>>> bug.  What confuses me it that I cannot find any modules named
>>> "xen_netfront" in the upstream tree.
>>>
>>
>> You should have tried '-' rather than '_' :-)
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/xen-netfront.c
>>
>>
>
> Looking at this driver, I think it is missing a call to
> skb_mark_for_recycle().
>
> I'll will submit at patch for this, with details for stable maintainers.
>
> As I think it dates back to v5.9 via commit 6c5aa6fc4def ("xen
> networking: add basic XDP support for xen-netfront"). I think this
> commit is missing a call to page_pool_release_page()
> between v5.9 to v5.14, after which is should have used
> skb_mark_for_recycle().
>
> Since v6.6 the call page_pool_release_page() were removed (in
> 535b9c61bdef ("net: page_pool: hide page_pool_release_page()") and
> remaining callers converted (in commit 6bfef2ec0172 ("Merge branch
> 'net-page_pool-remove-page_pool_release_page'")).
>
> This leak became visible in v6.8 via commit dba1b8a7ab68 ("mm/page_pool:
> catch page_pool memory leaks").

Thankyou very much for your help here.  Please CC
security@xenproject.org too, because we'll want to issue an XSA (Xen
Security Advisory) when this fix is ready.

~Andrew

Re: Xen NIC driver have page_pool memory leaks

[Denis Kirjanov]

On 3/27/24 14:27, Jesper Dangaard Brouer wrote:
> 
> 
> On 25/03/2024 13.33, Paul Durrant wrote:
>> On 25/03/2024 12:21, Jesper Dangaard Brouer wrote:
>>> Hi Arthur,
>>>
>>> (Answer inlined below, which is custom on this mailing list)
>>>
>>> On 23/03/2024 14.23, Arthur Borsboom wrote:
>>>> Hi Jesper,
>>>>
>>>> After a recent kernel upgrade 6.7.6 > 6.8.1 all my Xen guests on Arch
>>>> Linux are dumping kernel traces.
>>>> It seems to be indirectly caused by the page pool memory leak
>>>> mechanism, which is probably a good thing.
>>>>
>>>> I have created a bug report, but there is no response.
>>>>
>>>> https://bugzilla.kernel.org/show_bug.cgi?id=218618
>>>>
>>>> I am uncertain where and to whom I need to report this page leak.
>>>> Can you help me get this issue fixed?
>>>
>>> I'm the page_pool maintainer, but as you say yourself in comment 2 then
>>> since dba1b8a7ab68 ("mm/page_pool: catch page_pool memory leaks") this
>>> indicated there is a problem in the xen_netfront driver, which was
>>> previously not visible.
>>>
>>> Cc'ing the "XEN NETWORK BACKEND DRIVER" maintainers, as this is a driver
>>> bug.  What confuses me it that I cannot find any modules named
>>> "xen_netfront" in the upstream tree.
>>>
>>
>> You should have tried '-' rather than '_' :-)
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/xen-netfront.c
>>
> 
> Looking at this driver, I think it is missing a call to skb_mark_for_recycle().
> 
> I'll will submit at patch for this, with details for stable maintainers.

Please do

> 
> As I think it dates back to v5.9 via commit 6c5aa6fc4def ("xen
> networking: add basic XDP support for xen-netfront"). I think this
> commit is missing a call to page_pool_release_page()
> between v5.9 to v5.14, after which is should have used
> skb_mark_for_recycle().

Hmm, looks like I've missed it  
> 
> Since v6.6 the call page_pool_release_page() were removed (in
> 535b9c61bdef ("net: page_pool: hide page_pool_release_page()") and
> remaining callers converted (in commit 6bfef2ec0172 ("Merge branch
> 'net-page_pool-remove-page_pool_release_page'")).
> 
> This leak became visible in v6.8 via commit dba1b8a7ab68 ("mm/page_pool:
> catch page_pool memory leaks").
> 
> --Jesper
>