Re: [BUG] mlx5_core memory management issue

[Chris Arges <carges@cloudflare.com>]

On 2025-07-04 12:37:36, Dragos Tatulea wrote:
> On Thu, Jul 03, 2025 at 10:49:20AM -0500, Chris Arges wrote:
> > When running iperf through a set of XDP programs we were able to crash
> > machines with NICs using the mlx5_core driver. We were able to confirm
> > that other NICs/drivers did not exhibit the same problem, and suspect
> > this could be a memory management issue in the driver code.
> > Specifically we found a WARNING at include/net/page_pool/helpers.h:277
> > mlx5e_page_release_fragmented.isra. We are able to demonstrate this
> > issue in production using hardware, but cannot easily bisect because
> > we don’t have a simple reproducer.
> >
> Thanks for the report! We will investigate.
> 
> > I wanted to share stack traces in
> > order to help us further debug and understand if anyone else has run
> > into this issue. We are currently working on getting more crashdumps
> > and doing further analysis.
> > 
> > 
> > The test setup looks like the following:
> >   ┌─────┐
> >   │mlx5 │
> >   │NIC  │
> >   └──┬──┘
> >      │xdp ebpf program (does encap and XDP_TX)
> >      │
> >      ▼
> >   ┌──────────────────────┐
> >   │xdp.frags             │
> >   │                      │
> >   └──┬───────────────────┘
> >      │tailcall
> >      │BPF_REDIRECT_MAP (using CPUMAP bpf type)
> >      ▼
> >   ┌──────────────────────┐
> >   │xdp.frags/cpumap      │
> >   │                      │
> >   └──┬───────────────────┘
> >      │BPF_REDIRECT to veth (*potential trigger for issue)
> >      │
> >      ▼
> >   ┌──────┐
> >   │veth  │
> >   │      │
> >   └──┬───┘
> >      │
> >      │
> >      ▼
> > 
> > Here an mlx5 NIC has an xdp.frags program attached which tailcalls via
> > BPF_REDIRECT_MAP into an xdp.frags/cpumap. For our reproducer we can
> > choose a random valid CPU to reproduce the issue. Once that packet
> > reaches the xdp.frags/cpumap program we then do another BPF_REDIRECT
> > to a veth device which has an XDP program which redirects to an
> > XSKMAP. It wasn’t until we added the additional BPF_REDIRECT to the
> > veth device that we noticed this issue.
> > 
> Would it be possible to try to use a single program that redirects to
> the XSKMAP and check that the issue reproduces?
> 
> > When running with 6.12.30 to 6.12.32 kernels we are able to see the
> > following KASAN use-after-free WARNINGs followed by a page fault which
> > crashes the machine. We have not been able to test earlier or later
> > kernels. I’ve tried to map symbols to lines of code for clarity.
> >
> Thanks for the KASAN reports, they are very useful. Keep us posted
> if you have other updates. A first quick look didn't reveal anything
> obvious from our side but we will keep looking.
> 
> Thanks,
> Dragos

Ok, we can reproduce this problem!

I tried to simplify this reproducer, but it seems like what's needed is:
- xdp program attached to mlx5 NIC
- cpumap redirect
- device redirect (map or just bpf_redirect)
- frame gets turned into an skb
Then from another machine send many flows of UDP traffic to trigger the problem.

I've put together a program that reproduces the issue here:
- https://github.com/arges/xdp-redirector

In general the failure manifests with many different WARNs such as:
include/net/page_pool/helpers.h:277 mlx5e_page_release_fragmented.isra.0+0xf7/0x150 [mlx5_core]
Then the machine crashes.

I was able to get a crashdump which shows:
```
PID: 0        TASK: ffff8c0910134380  CPU: 76   COMMAND: "swapper/76"
 #0 [fffffe10906d3ea8] crash_nmi_callback at ffffffffadc5c4fd
 #1 [fffffe10906d3eb0] default_do_nmi at ffffffffae9524f0
 #2 [fffffe10906d3ed0] exc_nmi at ffffffffae952733
 #3 [fffffe10906d3ef0] end_repeat_nmi at ffffffffaea01bfd
    [exception RIP: io_serial_in+25]
    RIP: ffffffffae4cd489  RSP: ffffb3c60d6049e8  RFLAGS: 00000002
    RAX: ffffffffae4cd400  RBX: 00000000000025d8  RCX: 0000000000000000
    RDX: 00000000000002fd  RSI: 0000000000000005  RDI: ffffffffb10a9cb0
    RBP: 0000000000000000   R8: 2d2d2d2d2d2d2d2d   R9: 656820747563205b
    R10: 000000002d2d2d2d  R11: 000000002d2d2d2d  R12: ffffffffb0fa5610
    R13: 0000000000000000  R14: 0000000000000000  R15: ffffffffb10a9cb0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
 #4 [ffffb3c60d6049e8] io_serial_in at ffffffffae4cd489
 #5 [ffffb3c60d6049e8] serial8250_console_write at ffffffffae4d2fcf
 #6 [ffffb3c60d604a80] console_flush_all at ffffffffadd1cf26
 #7 [ffffb3c60d604b00] console_unlock at ffffffffadd1d1df
 #8 [ffffb3c60d604b48] vprintk_emit at ffffffffadd1dda1
 #9 [ffffb3c60d604b98] _printk at ffffffffae90250c
#10 [ffffb3c60d604bf8] report_bug.cold at ffffffffae95001d
#11 [ffffb3c60d604c38] handle_bug at ffffffffae950e91
#12 [ffffb3c60d604c58] exc_invalid_op at ffffffffae9512b7
#13 [ffffb3c60d604c70] asm_exc_invalid_op at ffffffffaea0123a
    [exception RIP: mlx5e_page_release_fragmented+85]
    RIP: ffffffffc25f75c5  RSP: ffffb3c60d604d20  RFLAGS: 00010293
    RAX: 000000000000003f  RBX: ffff8bfa8f059fd0  RCX: ffffe3bf1992a180
    RDX: 000000000000003d  RSI: ffffe3bf1992a180  RDI: ffff8bf9b0784000
    RBP: 0000000000000040   R8: 00000000000001d2   R9: 0000000000000006
    R10: ffff8c06de22f380  R11: ffff8bfcfe6cd680  R12: 00000000000001d2
    R13: 000000000000002b  R14: ffff8bf9b0784000  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#14 [ffffb3c60d604d20] mlx5e_free_rx_wqes at ffffffffc25f7e2f [mlx5_core]
#15 [ffffb3c60d604d58] mlx5e_post_rx_wqes at ffffffffc25f877c [mlx5_core]
#16 [ffffb3c60d604dc0] mlx5e_napi_poll at ffffffffc25fdd27 [mlx5_core]
#17 [ffffb3c60d604e20] __napi_poll at ffffffffae6a8ddb
#18 [ffffb3c60d604e90] __napi_poll at ffffffffae6a8db5
#19 [ffffb3c60d604e98] net_rx_action at ffffffffae6a95f1
#20 [ffffb3c60d604f98] handle_softirqs at ffffffffadc9d4bf
#21 [ffffb3c60d604fe8] irq_exit_rcu at ffffffffadc9e057
#22 [ffffb3c60d604ff0] common_interrupt at ffffffffae952015
--- <IRQ stack> ---
#23 [ffffb3c60c837de8] asm_common_interrupt at ffffffffaea01466
    [exception RIP: cpuidle_enter_state+184]
    RIP: ffffffffae955c38  RSP: ffffb3c60c837e98  RFLAGS: 00000202
    RAX: ffff8c0cffc00000  RBX: ffff8c0911002400  RCX: 0000000000000000
    RDX: 00003c630b2d073a  RSI: ffffffe519600d10  RDI: 0000000000000000
    RBP: 0000000000000001   R8: 0000000000000002   R9: 0000000000000001
    R10: ffff8c0cffc330c4  R11: 071c71c71c71c71c  R12: ffffffffb05ff820
    R13: 00003c630b2d073a  R14: 0000000000000001  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#24 [ffffb3c60c837ed0] cpuidle_enter at ffffffffae64b4ad
#25 [ffffb3c60c837ef0] do_idle at ffffffffadcfa7c6
#26 [ffffb3c60c837f30] cpu_startup_entry at ffffffffadcfaa09
#27 [ffffb3c60c837f40] start_secondary at ffffffffadc5ec77
#28 [ffffb3c60c837f50] common_startup_64 at ffffffffadc24d5d
```

Assuming (this is x86_64):
RDI=ffff8bf9b0784000 (rq)
RSI=ffffe3bf1992a180 (frag_page)

```
static void mlx5e_page_release_fragmented(struct mlx5e_rq *rq,
                                          struct mlx5e_frag_page *frag_page)
{
        u16 drain_count = MLX5E_PAGECNT_BIAS_MAX - frag_page->frags;
        struct page *page = frag_page->page;

        if (page_pool_unref_page(page, drain_count) == 0)
                page_pool_put_unrefed_page(rq->page_pool, page, -1, true);
}
```

crash> struct mlx5e_frag_page ffffe3bf1992a180
struct mlx5e_frag_page {
  page = 0x26ffff800000000,
  frags = 49856
}

This means that drain_count could be an unexpected number (assuming that we
expect it to be less than MLX5E_PAGECNT_BIAS_MAX).

Let me know what additional experiments would be useful here.

--chris