mlx5 XDP redirect leaking memory on kernel 6.3

mlx5 XDP redirect leaking memory on kernel 6.3

[Jesper Dangaard Brouer]

When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
works correctly. I tested both redirecting back out same mlx5 device and
cpumap redirect (with XDP_PASS), which both cause leaking.

After removing the XDP prog, which also cause the page_pool to be
released by mlx5, then the leaks are visible via the page_pool periodic
inflight reports. I have this bpftrace[1] tool that I also use to detect
the problem faster (not waiting 60 sec for a report).

  [1] 
https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt

I've been debugging and reading through the code for a couple of days,
but I've not found the root-cause, yet. I would appreciate new ideas
where to look and fresh eyes on the issue.

To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
suspicion is that mlx5 driver doesn't fully release the bias count (hint
see MLX5E_PAGECNT_BIAS_MAX).

--Jesper


Extra info about my device.  Providing these as mlx5 driver can have 
different allocation modes depending on HW and device priv-flags setup.

$ ethtool --show-priv-flags mlx5p1
Private flags for mlx5p1:
rx_cqe_moder       : on
tx_cqe_moder       : off
rx_cqe_compress    : off
rx_striding_rq     : on
rx_no_csum_complete: off
xdp_tx_mpwqe       : on
skb_tx_mpwqe       : on
tx_port_ts         : off

$ ethtool -i mlx5p1
driver: mlx5_core
version: 6.4.0-rc2-net-next-vm-lock-dbg+
firmware-version: 16.23.1020 (MT_0000000009)
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

$ lspci -v | grep 03:00.0
03:00.0 Ethernet controller: Mellanox Technologies MT28800 Family 
[ConnectX-5 Ex]

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Dragos Tatulea]

On Tue, 2023-05-23 at 17:55 +0200, Jesper Dangaard Brouer wrote:
> 
> When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
> is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
> works correctly. I tested both redirecting back out same mlx5 device and
> cpumap redirect (with XDP_PASS), which both cause leaking.
> 
> After removing the XDP prog, which also cause the page_pool to be
> released by mlx5, then the leaks are visible via the page_pool periodic
> inflight reports. I have this bpftrace[1] tool that I also use to detect
> the problem faster (not waiting 60 sec for a report).
> 
>   [1] 
> https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt
> 
> I've been debugging and reading through the code for a couple of days,
> but I've not found the root-cause, yet. I would appreciate new ideas
> where to look and fresh eyes on the issue.
> 
>
> To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
> suspicion is that mlx5 driver doesn't fully release the bias count (hint
> see MLX5E_PAGECNT_BIAS_MAX).
> 

Thanks for the report Jesper. Incidentally I've just picked up this issue today
as well.

On XDP redirect and tx, the page is set to skip the bias counter release with
the expectation that page_pool_put_defragged_page will be called from [1]. But,
as I found out now, during XDP redirect only one fragment of the page is
released in xdp core [2]. This is where the leak is coming from.

We'll provide a fix soon.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c#n665

[2]
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/core/xdp.c#n390

Thanks,
Dragos

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Yunsheng Lin]

On 2023/5/24 0:35, Dragos Tatulea wrote:
> 
> On Tue, 2023-05-23 at 17:55 +0200, Jesper Dangaard Brouer wrote:
>>
>> When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
>> is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
>> works correctly. I tested both redirecting back out same mlx5 device and
>> cpumap redirect (with XDP_PASS), which both cause leaking.
>>
>> After removing the XDP prog, which also cause the page_pool to be
>> released by mlx5, then the leaks are visible via the page_pool periodic
>> inflight reports. I have this bpftrace[1] tool that I also use to detect
>> the problem faster (not waiting 60 sec for a report).
>>
>>   [1] 
>> https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt
>>
>> I've been debugging and reading through the code for a couple of days,
>> but I've not found the root-cause, yet. I would appreciate new ideas
>> where to look and fresh eyes on the issue.
>>
>>
>> To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
>> suspicion is that mlx5 driver doesn't fully release the bias count (hint
>> see MLX5E_PAGECNT_BIAS_MAX).

It seems mlx5 is implementing it's own frag allocation scheme, it there a
reason why the native frag allocation scheme in page pool is not used? To
avoid the "((page->pp_magic & ~0x3UL) == PP_SIGNATURE)" checking?

>>
> 
> Thanks for the report Jesper. Incidentally I've just picked up this issue today
> as well.
> 
> On XDP redirect and tx, the page is set to skip the bias counter release with
> the expectation that page_pool_put_defragged_page will be called from [1]. But,

page_pool_put_defragged_page() can only be called when there is only user using
the page, I am not sure how it can ensure that yet.

> as I found out now, during XDP redirect only one fragment of the page is
> released in xdp core [2]. This is where the leak is coming from.
> 
> We'll provide a fix soon.
> 
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c#n665
> 
> [2]
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/core/xdp.c#n390
> 
> Thanks,
> Dragos
> 
> 

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Yunsheng Lin]

On 2023/5/24 19:26, Yunsheng Lin wrote:
>> On XDP redirect and tx, the page is set to skip the bias counter release with
>> the expectation that page_pool_put_defragged_page will be called from [1]. But,
> 
> page_pool_put_defragged_page() can only be called when there is only user using

only *one* user

> the page, I am not sure how it can ensure that yet.
> 

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Dragos Tatulea]

On Wed, 2023-05-24 at 19:26 +0800, Yunsheng Lin wrote:
> On 2023/5/24 0:35, Dragos Tatulea wrote:
> > 
> > On Tue, 2023-05-23 at 17:55 +0200, Jesper Dangaard Brouer wrote:
> > > 
> > > When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
> > > is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
> > > works correctly. I tested both redirecting back out same mlx5 device and
> > > cpumap redirect (with XDP_PASS), which both cause leaking.
> > > 
> > > After removing the XDP prog, which also cause the page_pool to be
> > > released by mlx5, then the leaks are visible via the page_pool periodic
> > > inflight reports. I have this bpftrace[1] tool that I also use to detect
> > > the problem faster (not waiting 60 sec for a report).
> > > 
> > >   [1] 
> > > https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt
> > > 
> > > I've been debugging and reading through the code for a couple of days,
> > > but I've not found the root-cause, yet. I would appreciate new ideas
> > > where to look and fresh eyes on the issue.
> > > 
> > > 
> > > To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
> > > suspicion is that mlx5 driver doesn't fully release the bias count (hint
> > > see MLX5E_PAGECNT_BIAS_MAX).
> 
> It seems mlx5 is implementing it's own frag allocation scheme, it there a
> reason why the native frag allocation scheme in page pool is not used? To
> avoid the "((page->pp_magic & ~0x3UL) == PP_SIGNATURE)" checking?

mlx5 uses fragmentation of the page from within the driver instead of the pre-
partitioning of the page using page_pool_alloc_frag(). As shown in commit
52cc6ffc0ab2 ("page_pool: Refactor page_pool to enable fragmenting after
allocation")

The exception is however the following optimization:
page_pool_put_defragged_page() can be called for XDP_TX directly to avoid the
overhead of fragment management. That's because mlx5 currently supports only one
packet per page for XDP.

> > > 
> > 
> > Thanks for the report Jesper. Incidentally I've just picked up this issue
> > today
> > as well.
> > 
> > On XDP redirect and tx, the page is set to skip the bias counter release
> > with
> > the expectation that page_pool_put_defragged_page will be called from [1].
> > But,
> 
> page_pool_put_defragged_page() can only be called when there is only user
> using
> the page, I am not sure how it can ensure that yet.
> 
See previous comment.

> > as I found out now, during XDP redirect only one fragment of the page is
> > released in xdp core [2]. This is where the leak is coming from.
> > 
> > We'll provide a fix soon.
> > 
> > [1]
> > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c#n665
> > 
> > [2]
> > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/core/xdp.c#n390
> > 
> > Thanks,
> > Dragos
> > 
> > 

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Yunsheng Lin]

On 2023/5/24 20:03, Dragos Tatulea wrote:
> On Wed, 2023-05-24 at 19:26 +0800, Yunsheng Lin wrote:
>> On 2023/5/24 0:35, Dragos Tatulea wrote:
>>>
>>> On Tue, 2023-05-23 at 17:55 +0200, Jesper Dangaard Brouer wrote:
>>>>
>>>> When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
>>>> is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
>>>> works correctly. I tested both redirecting back out same mlx5 device and
>>>> cpumap redirect (with XDP_PASS), which both cause leaking.
>>>>
>>>> After removing the XDP prog, which also cause the page_pool to be
>>>> released by mlx5, then the leaks are visible via the page_pool periodic
>>>> inflight reports. I have this bpftrace[1] tool that I also use to detect
>>>> the problem faster (not waiting 60 sec for a report).
>>>>
>>>>   [1] 
>>>> https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt
>>>>
>>>> I've been debugging and reading through the code for a couple of days,
>>>> but I've not found the root-cause, yet. I would appreciate new ideas
>>>> where to look and fresh eyes on the issue.
>>>>
>>>>
>>>> To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
>>>> suspicion is that mlx5 driver doesn't fully release the bias count (hint
>>>> see MLX5E_PAGECNT_BIAS_MAX).
>>
>> It seems mlx5 is implementing it's own frag allocation scheme, it there a
>> reason why the native frag allocation scheme in page pool is not used? To
>> avoid the "((page->pp_magic & ~0x3UL) == PP_SIGNATURE)" checking?
> 
> mlx5 uses fragmentation of the page from within the driver instead of the pre-
> partitioning of the page using page_pool_alloc_frag(). As shown in commit
> 52cc6ffc0ab2 ("page_pool: Refactor page_pool to enable fragmenting after
> allocation")

page_pool_alloc_frag() API does allow driver to allocate different number of
frag for the same page by specifying different 'size'.

> 
> The exception is however the following optimization:

The below rfc may be able to allow the following optimization using frag API
too.
https://patchwork.kernel.org/project/netdevbpf/cover/20230516124801.2465-1-linyunsheng@huawei.com/

> page_pool_put_defragged_page() can be called for XDP_TX directly to avoid the
> overhead of fragment management. That's because mlx5 currently supports only one
> packet per page for XDP.
> 

it seems almost everyone is doing the only one packet per page for XDP, but it is not
very memory saving for ususal case with 1.5K mtu with 4K page size if we reduce the xdp
headroom a little bit, and not to mention the 64K page size case.

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Jesper Dangaard Brouer]

On 23/05/2023 18.35, Dragos Tatulea wrote:
> 
> On Tue, 2023-05-23 at 17:55 +0200, Jesper Dangaard Brouer wrote:
>>
>> When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
>> is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
>> works correctly. I tested both redirecting back out same mlx5 device and
>> cpumap redirect (with XDP_PASS), which both cause leaking.
>>
>> After removing the XDP prog, which also cause the page_pool to be
>> released by mlx5, then the leaks are visible via the page_pool periodic
>> inflight reports. I have this bpftrace[1] tool that I also use to detect
>> the problem faster (not waiting 60 sec for a report).
>>
>>    [1]
>> https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt
>>
>> I've been debugging and reading through the code for a couple of days,
>> but I've not found the root-cause, yet. I would appreciate new ideas
>> where to look and fresh eyes on the issue.
>>
>>
>> To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
>> suspicion is that mlx5 driver doesn't fully release the bias count (hint
>> see MLX5E_PAGECNT_BIAS_MAX).
>>
> 
> Thanks for the report Jesper. Incidentally I've just picked up this issue today
> as well.
> 
> On XDP redirect and tx, the page is set to skip the bias counter release with
> the expectation that page_pool_put_defragged_page will be called from [1]. But,
> as I found out now, during XDP redirect only one fragment of the page is
> released in xdp core [2]. This is where the leak is coming from.
> 

Ohh, I guess I see the problem now. (As Lin also says indirectly) the
page_pool_put_defragged_page() call is not allowed or not intended to be
invoked directly.

In [1] the driver actually free a PP page that have been fragmented (via
page_pool_fragment_page), but not "defragged" yet.  Meaning
page->pp_frag_count will still be 64 (MLX5E_PAGECNT_BIAS_MAX).

I though about catching this invalid API usage in page_pool, but due to
an (atomic_read) optimization (in page_pool_defrag_page), we cannot
detect this reliably.

> We'll provide a fix soon.
> 
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c#n665
> 
> [2]
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/core/xdp.c#n390

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Jesper Dangaard Brouer]

Hi Dragos,

Below you promised to work on a fix for XDP redirect memory leak...
What is the status?

On 23/05/2023 18.35, Dragos Tatulea wrote:
> 
> On Tue, 2023-05-23 at 17:55 +0200, Jesper Dangaard Brouer wrote:
>>
>> When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
>> is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
>> works correctly. I tested both redirecting back out same mlx5 device and
>> cpumap redirect (with XDP_PASS), which both cause leaking.
>>
>> After removing the XDP prog, which also cause the page_pool to be
>> released by mlx5, then the leaks are visible via the page_pool periodic
>> inflight reports. I have this bpftrace[1] tool that I also use to detect
>> the problem faster (not waiting 60 sec for a report).
>>
>>    [1]
>> https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt
>>
>> I've been debugging and reading through the code for a couple of days,
>> but I've not found the root-cause, yet. I would appreciate new ideas
>> where to look and fresh eyes on the issue.
>>
>>
>> To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
>> suspicion is that mlx5 driver doesn't fully release the bias count (hint
>> see MLX5E_PAGECNT_BIAS_MAX).
>>
> 
> Thanks for the report Jesper. Incidentally I've just picked up this issue today
> as well.
> 
> On XDP redirect and tx, the page is set to skip the bias counter release with
> the expectation that page_pool_put_defragged_page will be called from [1]. But,
> as I found out now, during XDP redirect only one fragment of the page is
> released in xdp core [2]. This is where the leak is coming from.
> 
> We'll provide a fix soon.
> 
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c#n665
> 
> [2]
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/core/xdp.c#n390
> 
> Thanks,
> Dragos
> 
> 

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Dragos Tatulea]

Gi Jesper,
On Thu, 2023-07-13 at 11:20 +0200, Jesper Dangaard Brouer wrote:
> Hi Dragos,
> 
> Below you promised to work on a fix for XDP redirect memory leak...
> What is the status?
> 
The fix got merged into net a week ago:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/drivers/net/ethernet/mellanox/mlx5/core?id=7abd955a58fb0fcd4e756fa2065c03ae488fcfa7

Just forgot to follow up on this thread. Sorry about that...

Thanks,
Dragos

> On 23/05/2023 18.35, Dragos Tatulea wrote:
> > 
> > On Tue, 2023-05-23 at 17:55 +0200, Jesper Dangaard Brouer wrote:
> > > 
> > > When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
> > > is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
> > > works correctly. I tested both redirecting back out same mlx5 device and
> > > cpumap redirect (with XDP_PASS), which both cause leaking.
> > > 
> > > After removing the XDP prog, which also cause the page_pool to be
> > > released by mlx5, then the leaks are visible via the page_pool periodic
> > > inflight reports. I have this bpftrace[1] tool that I also use to detect
> > > the problem faster (not waiting 60 sec for a report).
> > > 
> > >    [1]
> > > https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt
> > > 
> > > I've been debugging and reading through the code for a couple of days,
> > > but I've not found the root-cause, yet. I would appreciate new ideas
> > > where to look and fresh eyes on the issue.
> > > 
> > > 
> > > To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
> > > suspicion is that mlx5 driver doesn't fully release the bias count (hint
> > > see MLX5E_PAGECNT_BIAS_MAX).
> > > 
> > 
> > Thanks for the report Jesper. Incidentally I've just picked up this issue
> > today
> > as well.
> > 
> > On XDP redirect and tx, the page is set to skip the bias counter release
> > with
> > the expectation that page_pool_put_defragged_page will be called from [1].
> > But,
> > as I found out now, during XDP redirect only one fragment of the page is
> > released in xdp core [2]. This is where the leak is coming from.
> > 
> > We'll provide a fix soon.
> > 
> > [1]
> > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c#n665
> > 
> > [2]
> > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/core/xdp.c#n390
> > 
> > Thanks,
> > Dragos
> > 
> > 
> 

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Jesper Dangaard Brouer]

On 13/07/2023 12.11, Dragos Tatulea wrote:
> Gi Jesper,
> On Thu, 2023-07-13 at 11:20 +0200, Jesper Dangaard Brouer wrote:
>> Hi Dragos,
>>
>> Below you promised to work on a fix for XDP redirect memory leak...
>> What is the status?
>>
> The fix got merged into net a week ago:
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/drivers/net/ethernet/mellanox/mlx5/core?id=7abd955a58fb0fcd4e756fa2065c03ae488fcfa7
> 
> Just forgot to follow up on this thread. Sorry about that...
> 

Good to see it being fixed in net.git commit:
  7abd955a58fb ("net/mlx5e: RX, Fix page_pool page fragment tracking for 
XDP")

This need to be backported into stable tree 6.3, but I can see 6.3.13 is 
marked EOL (End-of-Life).
Can we still get this fix applied? (Cc. GregKH)

--Jesper

> 
>> On 23/05/2023 18.35, Dragos Tatulea wrote:
>>>
>>> On Tue, 2023-05-23 at 17:55 +0200, Jesper Dangaard Brouer wrote:
>>>>
>>>> When the mlx5 driver runs an XDP program doing XDP_REDIRECT, then memory
>>>> is getting leaked. Other XDP actions, like XDP_DROP, XDP_PASS and XDP_TX
>>>> works correctly. I tested both redirecting back out same mlx5 device and
>>>> cpumap redirect (with XDP_PASS), which both cause leaking.
>>>>
>>>> After removing the XDP prog, which also cause the page_pool to be
>>>> released by mlx5, then the leaks are visible via the page_pool periodic
>>>> inflight reports. I have this bpftrace[1] tool that I also use to detect
>>>> the problem faster (not waiting 60 sec for a report).
>>>>
>>>>     [1]
>>>> https://github.com/xdp-project/xdp-project/blob/master/areas/mem/bpftrace/page_pool_track_shutdown01.bt
>>>>
>>>> I've been debugging and reading through the code for a couple of days,
>>>> but I've not found the root-cause, yet. I would appreciate new ideas
>>>> where to look and fresh eyes on the issue.
>>>>
>>>>
>>>> To Lin, it looks like mlx5 uses PP_FLAG_PAGE_FRAG, and my current
>>>> suspicion is that mlx5 driver doesn't fully release the bias count (hint
>>>> see MLX5E_PAGECNT_BIAS_MAX).
>>>>
>>>
>>> Thanks for the report Jesper. Incidentally I've just picked up this issue
>>> today
>>> as well.
>>>
>>> On XDP redirect and tx, the page is set to skip the bias counter release
>>> with
>>> the expectation that page_pool_put_defragged_page will be called from [1].
>>> But,
>>> as I found out now, during XDP redirect only one fragment of the page is
>>> released in xdp core [2]. This is where the leak is coming from.
>>>
>>> We'll provide a fix soon.
>>>
>>> [1]
>>> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c#n665
>>>
>>> [2]
>>> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/core/xdp.c#n390
>>>
>>> Thanks,
>>> Dragos
>>>
>>>
>>
> 

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Greg KH]

On Thu, Jul 13, 2023 at 04:58:04PM +0200, Jesper Dangaard Brouer wrote:
> 
> 
> On 13/07/2023 12.11, Dragos Tatulea wrote:
> > Gi Jesper,
> > On Thu, 2023-07-13 at 11:20 +0200, Jesper Dangaard Brouer wrote:
> > > Hi Dragos,
> > > 
> > > Below you promised to work on a fix for XDP redirect memory leak...
> > > What is the status?
> > > 
> > The fix got merged into net a week ago:
> > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/drivers/net/ethernet/mellanox/mlx5/core?id=7abd955a58fb0fcd4e756fa2065c03ae488fcfa7
> > 
> > Just forgot to follow up on this thread. Sorry about that...
> > 
> 
> Good to see it being fixed in net.git commit:
>  7abd955a58fb ("net/mlx5e: RX, Fix page_pool page fragment tracking for
> XDP")
> 
> This need to be backported into stable tree 6.3, but I can see 6.3.13 is
> marked EOL (End-of-Life).
> Can we still get this fix applied? (Cc. GregKH)

<formletter>

This is not the correct way to submit patches for inclusion in the
stable kernel tree.  Please read:
    https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
for how to do this properly.

</formletter>

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Dragos Tatulea]

On Thu, 2023-07-13 at 17:31 +0200, Greg KH wrote:
> On Thu, Jul 13, 2023 at 04:58:04PM +0200, Jesper Dangaard Brouer wrote:
> > 
> > 
> > On 13/07/2023 12.11, Dragos Tatulea wrote:
> > > Gi Jesper,
> > > On Thu, 2023-07-13 at 11:20 +0200, Jesper Dangaard Brouer wrote:
> > > > Hi Dragos,
> > > > 
> > > > Below you promised to work on a fix for XDP redirect memory leak...
> > > > What is the status?
> > > > 
> > > The fix got merged into net a week ago:
> > > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/drivers/net/ethernet/mellanox/mlx5/core?id=7abd955a58fb0fcd4e756fa2065c03ae488fcfa7
> > > 
> > > Just forgot to follow up on this thread. Sorry about that...
> > > 
> > 
> > Good to see it being fixed in net.git commit:
> >  7abd955a58fb ("net/mlx5e: RX, Fix page_pool page fragment tracking for
> > XDP")
> > 
> > This need to be backported into stable tree 6.3, but I can see 6.3.13 is
> > marked EOL (End-of-Life).
> > Can we still get this fix applied? (Cc. GregKH)
> 
> <formletter>
> 
> This is not the correct way to submit patches for inclusion in the
> stable kernel tree.  Please read:
>     https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
> for how to do this properly.
> 
> </formletter>
So...I am a bit confused: should I send the patch to stable for 6.13 according
to the stable submission rules or is it too late?

Thanks,
Dragos

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[gregkh]

On Mon, Jul 17, 2023 at 02:37:44PM +0000, Dragos Tatulea wrote:
> On Thu, 2023-07-13 at 17:31 +0200, Greg KH wrote:
> > On Thu, Jul 13, 2023 at 04:58:04PM +0200, Jesper Dangaard Brouer wrote:
> > > 
> > > 
> > > On 13/07/2023 12.11, Dragos Tatulea wrote:
> > > > Gi Jesper,
> > > > On Thu, 2023-07-13 at 11:20 +0200, Jesper Dangaard Brouer wrote:
> > > > > Hi Dragos,
> > > > > 
> > > > > Below you promised to work on a fix for XDP redirect memory leak...
> > > > > What is the status?
> > > > > 
> > > > The fix got merged into net a week ago:
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/drivers/net/ethernet/mellanox/mlx5/core?id=7abd955a58fb0fcd4e756fa2065c03ae488fcfa7
> > > > 
> > > > Just forgot to follow up on this thread. Sorry about that...
> > > > 
> > > 
> > > Good to see it being fixed in net.git commit:
> > >  7abd955a58fb ("net/mlx5e: RX, Fix page_pool page fragment tracking for
> > > XDP")
> > > 
> > > This need to be backported into stable tree 6.3, but I can see 6.3.13 is
> > > marked EOL (End-of-Life).
> > > Can we still get this fix applied? (Cc. GregKH)
> > 
> > <formletter>
> > 
> > This is not the correct way to submit patches for inclusion in the
> > stable kernel tree.  Please read:
> >     https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
> > for how to do this properly.
> > 
> > </formletter>
> So...I am a bit confused: should I send the patch to stable for 6.13 according
> to the stable submission rules or is it too late?

There is no "6.13" kernel version yet, that should not happen for
another year or so.

If you mean the "6.3.y" tree, yes, there is nothing to do here as that
tree is end-of-life and you should have moved to the 6.4.y kernel tree
at this point in time.

What is preventing you from moving?

thanks,

greg k-h

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Dragos Tatulea]

On Mon, 2023-07-17 at 16:42 +0200, gregkh@linuxfoundation.org wrote:
> On Mon, Jul 17, 2023 at 02:37:44PM +0000, Dragos Tatulea wrote:
> > On Thu, 2023-07-13 at 17:31 +0200, Greg KH wrote:
> > > On Thu, Jul 13, 2023 at 04:58:04PM +0200, Jesper Dangaard Brouer wrote:
> > > > 
> > > > 
> > > > On 13/07/2023 12.11, Dragos Tatulea wrote:
> > > > > Gi Jesper,
> > > > > On Thu, 2023-07-13 at 11:20 +0200, Jesper Dangaard Brouer wrote:
> > > > > > Hi Dragos,
> > > > > > 
> > > > > > Below you promised to work on a fix for XDP redirect memory leak...
> > > > > > What is the status?
> > > > > > 
> > > > > The fix got merged into net a week ago:
> > > > > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/drivers/net/ethernet/mellanox/mlx5/core?id=7abd955a58fb0fcd4e756fa2065c03ae488fcfa7
> > > > > 
> > > > > Just forgot to follow up on this thread. Sorry about that...
> > > > > 
> > > > 
> > > > Good to see it being fixed in net.git commit:
> > > >  7abd955a58fb ("net/mlx5e: RX, Fix page_pool page fragment tracking for
> > > > XDP")
> > > > 
> > > > This need to be backported into stable tree 6.3, but I can see 6.3.13 is
> > > > marked EOL (End-of-Life).
> > > > Can we still get this fix applied? (Cc. GregKH)
> > > 
> > > <formletter>
> > > 
> > > This is not the correct way to submit patches for inclusion in the
> > > stable kernel tree.  Please read:
> > >    
> > > https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
> > > for how to do this properly.
> > > 
> > > </formletter>
> > So...I am a bit confused: should I send the patch to stable for 6.13
> > according
> > to the stable submission rules or is it too late?
> 
> There is no "6.13" kernel version yet, that should not happen for
> another year or so.
> 
Sorry for the typo...

> If you mean the "6.3.y" tree, yes, there is nothing to do here as that
> tree is end-of-life and you should have moved to the 6.4.y kernel tree
> at this point in time.
> 
> What is preventing you from moving?
> 
I am fine with the state of things. But Jesper was asking. I suppose the answer
to his question is "it's too late".

Thanks,
Dragos

Re: mlx5 XDP redirect leaking memory on kernel 6.3

[Jesper Dangaard Brouer]

Correction: mlx5 XDP redirect leaking memory on kernel 6.4
  - Explained below

On 17/07/2023 17.15, Dragos Tatulea wrote:
> On Mon, 2023-07-17 at 16:42 +0200, gregkh@linuxfoundation.org wrote:
>> On Mon, Jul 17, 2023 at 02:37:44PM +0000, Dragos Tatulea wrote:
>>> On Thu, 2023-07-13 at 17:31 +0200, Greg KH wrote:
>>>> On Thu, Jul 13, 2023 at 04:58:04PM +0200, Jesper Dangaard Brouer wrote:
>>>>>
>>>>>
>>>>> On 13/07/2023 12.11, Dragos Tatulea wrote:
>>>>>> Gi Jesper,
>>>>>> On Thu, 2023-07-13 at 11:20 +0200, Jesper Dangaard Brouer wrote:
>>>>>>> Hi Dragos,
>>>>>>>
>>>>>>> Below you promised to work on a fix for XDP redirect memory leak...
>>>>>>> What is the status?
>>>>>>>
>>>>>> The fix got merged into net a week ago:
>>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/drivers/net/ethernet/mellanox/mlx5/core?id=7abd955a58fb0fcd4e756fa2065c03ae488fcfa7
>>>>>>
>>>>>> Just forgot to follow up on this thread. Sorry about that...
>>>>>>
>>>>>
>>>>> Good to see it being fixed in net.git commit:
>>>>>   7abd955a58fb ("net/mlx5e: RX, Fix page_pool page fragment tracking for XDP")
>>>>>
>>>>> This need to be backported into stable tree 6.3, but I can see 6.3.13 is
>>>>> marked EOL (End-of-Life).
>>>>> Can we still get this fix applied? (Cc. GregKH)
>>>>
>>>> <formletter>
>>>>
>>>> This is not the correct way to submit patches for inclusion in the
>>>> stable kernel tree.  Please read:
>>>>     
>>>> https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
>>>> for how to do this properly.
>>>>
>>>> </formletter>
[...]
>>> to the stable submission rules or is it too late?
>>
[...]
>> If you mean the "6.3.y" tree, yes, there is nothing to do here as that
>> tree is end-of-life and you should have moved to the 6.4.y kernel tree
>> at this point in time.
>>
>> What is preventing you from moving?
>>
> I am fine with the state of things. But Jesper was asking. I suppose
> the answer to his question is "it's too late".
I was looking for the answer "it is too late for 6.3.y".

This exercise is just to make Google searches and distro people (like
myself and cc) aware that mlx5 XDP *redirect* is (intermediately) broken
and we should remember to backport 7abd955a58fb ("net/mlx5e: RX, Fix
page_pool page fragment tracking for XDP") to fix this.

Looking at git details, I notice that I (and subject) were wrong. The
buggy feature first landed in v6.4 and not v6.3.
Thus, true subject "mlx5 XDP redirect leaking memory on kernel 6.4".
Sorry for the stable confusion.  Greg will "automatically" pickup
Dragos's fix commit for 6.4.y as it have correct fixes tag.  Guess, I
damaged the Google search parameter, but I trying to correct it with
this update.

--Jesper