[PATCH bpf] bpf: Fix metadata_dst leak __bpf_redirect_neigh_v{4,6}

[PATCH bpf] bpf: Fix metadata_dst leak __bpf_redirect_neigh_v{4,6}

[Daniel Borkmann]

Cilium has a BPF egress gateway feature which forces outgoing K8s Pod
traffic to pass through dedicated egress gateways which then SNAT the
traffic in order to interact with stable IPs outside the cluster.

The traffic is directed to the gateway via vxlan tunnel in collect md
mode. A recent BPF change utilized the bpf_redirect_neigh() helper to
forward packets after the arrival and decap on vxlan, which turned out
over time that the kmalloc-256 slab usage in kernel was ever-increasing.

The issue was that vxlan allocates the metadata_dst object and attaches
it through a fake dst entry to the skb. The latter was never released
though given bpf_redirect_neigh() was merely setting the new dst entry
via skb_dst_set() without dropping an existing one first.

Fixes: b4ab31414970 ("bpf: Add redirect_neigh helper as redirect drop-in")
Reported-by: Yusuke Suzuki <yusuke.suzuki@isovalent.com>
Reported-by: Julian Wiedmann <jwi@isovalent.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: Martin KaFai Lau <martin.lau@kernel.org>
Cc: Jakub Kicinski <kuba@kernel.org>
Cc: Jordan Rife <jrife@google.com>
---
 net/core/filter.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/net/core/filter.c b/net/core/filter.c
index b005363f482c..c3c0b5a37504 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -2281,6 +2281,7 @@ static int __bpf_redirect_neigh_v6(struct sk_buff *skb, struct net_device *dev,
 		if (IS_ERR(dst))
 			goto out_drop;
 
+		skb_dst_drop(skb);
 		skb_dst_set(skb, dst);
 	} else if (nh->nh_family != AF_INET6) {
 		goto out_drop;
@@ -2389,6 +2390,7 @@ static int __bpf_redirect_neigh_v4(struct sk_buff *skb, struct net_device *dev,
 			goto out_drop;
 		}
 
+		skb_dst_drop(skb);
 		skb_dst_set(skb, &rt->dst);
 	}
 
-- 
2.43.0

Re: [PATCH bpf] bpf: Fix metadata_dst leak __bpf_redirect_neigh_v{4,6}

[Simon Horman]

On Fri, Oct 03, 2025 at 09:34:18AM +0200, Daniel Borkmann wrote:
> Cilium has a BPF egress gateway feature which forces outgoing K8s Pod
> traffic to pass through dedicated egress gateways which then SNAT the
> traffic in order to interact with stable IPs outside the cluster.
> 
> The traffic is directed to the gateway via vxlan tunnel in collect md
> mode. A recent BPF change utilized the bpf_redirect_neigh() helper to
> forward packets after the arrival and decap on vxlan, which turned out
> over time that the kmalloc-256 slab usage in kernel was ever-increasing.
> 
> The issue was that vxlan allocates the metadata_dst object and attaches
> it through a fake dst entry to the skb. The latter was never released
> though given bpf_redirect_neigh() was merely setting the new dst entry
> via skb_dst_set() without dropping an existing one first.
> 
> Fixes: b4ab31414970 ("bpf: Add redirect_neigh helper as redirect drop-in")
> Reported-by: Yusuke Suzuki <yusuke.suzuki@isovalent.com>
> Reported-by: Julian Wiedmann <jwi@isovalent.com>
> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
> Cc: Martin KaFai Lau <martin.lau@kernel.org>
> Cc: Jakub Kicinski <kuba@kernel.org>
> Cc: Jordan Rife <jrife@google.com>

Reviewed-by: Simon Horman <horms@kernel.org>

Re: [PATCH bpf] bpf: Fix metadata_dst leak __bpf_redirect_neigh_v{4,6}

[Jordan Rife]

On Fri, Oct 03, 2025 at 09:34:18AM +0200, Daniel Borkmann wrote:
> Cilium has a BPF egress gateway feature which forces outgoing K8s Pod
> traffic to pass through dedicated egress gateways which then SNAT the
> traffic in order to interact with stable IPs outside the cluster.
> 
> The traffic is directed to the gateway via vxlan tunnel in collect md
> mode. A recent BPF change utilized the bpf_redirect_neigh() helper to
> forward packets after the arrival and decap on vxlan, which turned out
> over time that the kmalloc-256 slab usage in kernel was ever-increasing.
> 
> The issue was that vxlan allocates the metadata_dst object and attaches
> it through a fake dst entry to the skb. The latter was never released
> though given bpf_redirect_neigh() was merely setting the new dst entry
> via skb_dst_set() without dropping an existing one first.
> 
> Fixes: b4ab31414970 ("bpf: Add redirect_neigh helper as redirect drop-in")
> Reported-by: Yusuke Suzuki <yusuke.suzuki@isovalent.com>
> Reported-by: Julian Wiedmann <jwi@isovalent.com>
> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
> Cc: Martin KaFai Lau <martin.lau@kernel.org>
> Cc: Jakub Kicinski <kuba@kernel.org>
> Cc: Jordan Rife <jrife@google.com>
> ---
>  net/core/filter.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/net/core/filter.c b/net/core/filter.c
> index b005363f482c..c3c0b5a37504 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -2281,6 +2281,7 @@ static int __bpf_redirect_neigh_v6(struct sk_buff *skb, struct net_device *dev,
>  		if (IS_ERR(dst))
>  			goto out_drop;
>  
> +		skb_dst_drop(skb);
>  		skb_dst_set(skb, dst);
>  	} else if (nh->nh_family != AF_INET6) {
>  		goto out_drop;
> @@ -2389,6 +2390,7 @@ static int __bpf_redirect_neigh_v4(struct sk_buff *skb, struct net_device *dev,
>  			goto out_drop;
>  		}
>  
> +		skb_dst_drop(skb);
>  		skb_dst_set(skb, &rt->dst);
>  	}
>  
> -- 
> 2.43.0
>

Nice catch!

Reviewed-by: Jordan Rife <jrife@google.com>

Re: [PATCH bpf] bpf: Fix metadata_dst leak __bpf_redirect_neigh_v{4,6}

[Jakub Kicinski]

On Fri,  3 Oct 2025 09:34:18 +0200 Daniel Borkmann wrote:
> Cilium has a BPF egress gateway feature which forces outgoing K8s Pod
> traffic to pass through dedicated egress gateways which then SNAT the
> traffic in order to interact with stable IPs outside the cluster.

Nice! The warning Stan added at work?

Reviewed-by: Jakub Kicinski <kuba@kernel.org>

Re: [PATCH bpf] bpf: Fix metadata_dst leak __bpf_redirect_neigh_v{4,6}

[Martin KaFai Lau]

On 10/3/25 12:34 AM, Daniel Borkmann wrote:
> Cilium has a BPF egress gateway feature which forces outgoing K8s Pod
> traffic to pass through dedicated egress gateways which then SNAT the
> traffic in order to interact with stable IPs outside the cluster.
> 
> The traffic is directed to the gateway via vxlan tunnel in collect md
> mode. A recent BPF change utilized the bpf_redirect_neigh() helper to
> forward packets after the arrival and decap on vxlan, which turned out
> over time that the kmalloc-256 slab usage in kernel was ever-increasing.
> 
> The issue was that vxlan allocates the metadata_dst object and attaches
> it through a fake dst entry to the skb. The latter was never released
> though given bpf_redirect_neigh() was merely setting the new dst entry
> via skb_dst_set() without dropping an existing one first.

Reviewed-by: Martin KaFai Lau <martin.lau@kernel.org>

Re: [PATCH bpf] bpf: Fix metadata_dst leak __bpf_redirect_neigh_v{4,6}

[Daniel Borkmann]

On 10/3/25 6:02 PM, Jakub Kicinski wrote:
> On Fri,  3 Oct 2025 09:34:18 +0200 Daniel Borkmann wrote:
>> Cilium has a BPF egress gateway feature which forces outgoing K8s Pod
>> traffic to pass through dedicated egress gateways which then SNAT the
>> traffic in order to interact with stable IPs outside the cluster.
> 
> Nice! The warning Stan added at work?

We should add CONFIG_DEBUG_NET to our Cilium CI test kernels actually.
The memory leak was observed on AWS nodes with the above Cilium config,
so bpftrace came to the rescue in the end.

Thanks,
Daniel

Re: [PATCH bpf] bpf: Fix metadata_dst leak __bpf_redirect_neigh_v{4,6}

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to bpf/bpf.git (master)
by Alexei Starovoitov <ast@kernel.org>:

On Fri,  3 Oct 2025 09:34:18 +0200 you wrote:
> Cilium has a BPF egress gateway feature which forces outgoing K8s Pod
> traffic to pass through dedicated egress gateways which then SNAT the
> traffic in order to interact with stable IPs outside the cluster.
> 
> The traffic is directed to the gateway via vxlan tunnel in collect md
> mode. A recent BPF change utilized the bpf_redirect_neigh() helper to
> forward packets after the arrival and decap on vxlan, which turned out
> over time that the kmalloc-256 slab usage in kernel was ever-increasing.
> 
> [...]

Here is the summary with links:
  - [bpf] bpf: Fix metadata_dst leak __bpf_redirect_neigh_v{4,6}
    https://git.kernel.org/bpf/bpf/c/23f3770e1a53

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html