mlx5: stale skb encapsulation metadata after native XDP decapsulation

mlx5: stale skb encapsulation metadata after native XDP decapsulation

[Vijay Nag]

Hi all, 

In a setup where native XDP performs tunnel decapsulation before TC ingress, I'm seeing an unexpected interaction with RX checksum offload.

XDP removes an outer IPv6 header using bpf_xdp_adjust_head() and returns XDP_PASS. TC ingress subsequently calls bpf_skb_change_tail() on the decapsulated packet.

On mlx5 with RX checksum offload enabled, bpf_skb_change_tail() returns -524 (-ENOTSUPP). Tracing shows skb->encapsulation == 1 at TC ingress, even though the outer header has already been removed. Disabling RX checksum offload makes the same processing path succeed.

The following Linux v6.6 code paths appear relevant.

In net/core/filter.c, __bpf_skb_change_tail():

if (skb->encapsulation)
    return -ENOTSUPP;

In drivers/net/ethernet/mellanox/mlx5/core/en_rx.c, mlx5e_handle_csum():

if (likely((cqe->hds_ip_ext & CQE_L3_OK) && (cqe->hds_ip_ext & CQE_L4_OK))) {
    skb->ip_summed = CHECKSUM_UNNECESSARY;
    if (cqe_is_tunneled(cqe)) {
        skb->csum_level = 1;
        skb->encapsulation = 1;
        stats->csum_unnecessary_inner++;
        return;
    }
}

My current understanding is that the CQE tunnel classification causes the subsequently constructed skb to retain encapsulation metadata corresponding to the original packet, even though XDP has already removed the outer header.

Is this expected behavior when native XDP decapsulates a packet before skb construction?

More generally, what is the intended handling of checksum/encapsulation metadata in this case? Is there an existing fix or recommended way to handle this interaction?

This was observed on a v6.6-based kernel with an mlx5 VF.

Source references:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/core/filter.c?h=v6.6
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c?h=v6.6

Thanks,
Vijay Nag

Re: mlx5: stale skb encapsulation metadata after native XDP decapsulation

[Stanislav Fomichev]

On 10/20, Vijay Nag wrote:
> Hi all, 
> 
> In a setup where native XDP performs tunnel decapsulation before TC ingress, I'm seeing an unexpected interaction with RX checksum offload.
> 
> XDP removes an outer IPv6 header using bpf_xdp_adjust_head() and returns XDP_PASS. TC ingress subsequently calls bpf_skb_change_tail() on the decapsulated packet.
> 
> On mlx5 with RX checksum offload enabled, bpf_skb_change_tail() returns -524 (-ENOTSUPP). Tracing shows skb->encapsulation == 1 at TC ingress, even though the outer header has already been removed. Disabling RX checksum offload makes the same processing path succeed.
> 
> The following Linux v6.6 code paths appear relevant.
> 
> In net/core/filter.c, __bpf_skb_change_tail():
> 
> if (skb->encapsulation)
>     return -ENOTSUPP;
> 
> In drivers/net/ethernet/mellanox/mlx5/core/en_rx.c, mlx5e_handle_csum():
> 
> if (likely((cqe->hds_ip_ext & CQE_L3_OK) && (cqe->hds_ip_ext & CQE_L4_OK))) {
>     skb->ip_summed = CHECKSUM_UNNECESSARY;
>     if (cqe_is_tunneled(cqe)) {
>         skb->csum_level = 1;
>         skb->encapsulation = 1;
>         stats->csum_unnecessary_inner++;
>         return;
>     }
> }
> 
> My current understanding is that the CQE tunnel classification causes the subsequently constructed skb to retain encapsulation metadata corresponding to the original packet, even though XDP has already removed the outer header.
> 
> Is this expected behavior when native XDP decapsulates a packet before skb construction?
> 
> More generally, what is the intended handling of checksum/encapsulation metadata in this case? Is there an existing fix or recommended way to handle this interaction?
> 
> This was observed on a v6.6-based kernel with an mlx5 VF.
> 
> Source references:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/core/filter.c?h=v6.6
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c?h=v6.6

I don't think we have a good answer for this, there has been some recent
discussion here: https://lore.kernel.org/netdev/amEojEGsIGkUjIWn@devvm7509.cco0.facebook.com/

In general, from my POW, we should not carry most of the rx metadata into
an skb when there is an xdp program (until we expose some helpers to the
xdp to change it). For mlx, we might want to skip that cqe_is_tunneled
branch in case of xdp, you can try to send a patch and see what happens :-)

For more recent kernels, and for tc, there is bpf_skb_adjust_room helper
with gives you more control over these fields (in case you can move your
decap fully to tc).

CC'd Willem who might know more about csum+encap specifics and Tariq for
mlx5.