Questions about resilient next-hop groups (6.18.35)

Questions about resilient next-hop groups (6.18.35)

[Froehlich, Dominik]

Hello everyone,

I am running kernel 6.18.35 and I have the following multi path route setup:


10.181.0.0/16 nhid 400
      nexthop dev bond0-ip6tnl0 weight 1
      nexthop dev bond0-ip6tnl1 weight 1
100.80.0.0/12 nhid 400
      nexthop dev bond0-ip6tnl0 weight 1
      nexthop dev bond0-ip6tnl1 weight 1
100.112.0.0/13 nhid 400
      nexthop dev bond0-ip6tnl0 weight 1
      nexthop dev bond0-ip6tnl1 weight 1


The links are ip6tnl devices sending traffic over an IPv6 tunnel to the other end.

I am using L4 ECMP hash policy:

~ # sysctl net.ipv4.fib_multipath_hash_policy
net.ipv4.fib_multipath_hash_policy = 1
~ # sysctl net.ipv6.fib_multipath_hash_policy
net.ipv6.fib_multipath_hash_policy = 1

Here are the next hop definitions:

id 400 group 4000,128/4001,128 type resilient buckets 1024 idle_timer 60 unbalanced_timer 0 unbalanced_time 0
id 4000 dev bond0-ip6tnl0 scope host
id 4001 dev bond0-ip6tnl1 scope host


So I have two equal weighted next hops with 512 buckets each.

Now I am running a TCP connection in a shell on a host in the 10.181.0.0/16 network over one of the next hops that continuously produces traffic to keep its bucket busy:

while true; do 
  sleep 0.1
  date
done 


Thu Jul  9 13:27:03 UTC 2026
Thu Jul  9 13:27:03 UTC 2026
Thu Jul  9 13:27:03 UTC 2026
Thu Jul  9 13:27:04 UTC 2026
(…)

This roughly sends 10 pkts/s.
Now I want to drain next hop 4000 which is where my traffic is flowing:


ip nexthop replace id 400 group 4000,1/4001,255 type resilient buckets 1024 idle_timer 60 unbalanced_timer 0

Assigning it a weight of 1 should reduce the number of buckets assigned to next hop 4000 to 4 and move idle buckets to next hop 4001.

However, what I am seeing is that the connection is immediately terminated.


When I am changing the weights more slowly it kind of works:

~ # ip nexthop replace id 400 group 4000,128/4001,128 type resilient idle_timer 60 unbalanced_timer 0
~ # ip nexthop replace id 400 group 4000,64/4001,192 type resilient idle_timer 60 unbalanced_timer 0
~ # ip nexthop replace id 400 group 4000,5/4001,250 type resilient idle_timer 60 unbalanced_timer 0
~ # ip nexthop replace id 400 group 4000,1/4001,255 type resilient idle_timer 60 unbalanced_timer 0

I can then see the buckets reduced to just 4 over time as expected:

Every 1.0s: ip nexthop bucket list nhid 4000                                                                                                                               2026-07-09 13:44:22

id 400 index 447 idle_time 135.51 nhid 4000
id 400 index 631 idle_time 117.56 nhid 4000
id 400 index 720 idle_time 73.11 nhid 4000
id 400 index 742 idle_time 136.19 nhid 4000


My guess is that my connection is in bucket index 720 as it has the lowest idle_time reported. 

My questions:
- Why is the idle_time reported so high? There should be packets flowing through that bucket every 0.1 seconds, so I was expecting the idle_time to be near zero.
- Why does my connection get cut immediately when draining rapidly but kind of stays alive when changing weights more slowly? Shouldn't it be a busy (undrainable) bucket either way?
Bucket nr 720 already has idle_time higher than 60 so I think it would be migrated on any further weight changes - which would disrupt the existing connection again.
- When / how is the idle_time of a bucket updated? The docs mention that “Every time a packet is forwarded through a bucket, this timer is updated to current jiffies value.” I found that the bucket idle_time is refreshed more frequently when fever buckets are used. But why does the number of buckets even matter here? Fewer buckets mean more flows per bucket, I get that. But if there is at least one busy connection in a bucket, shouldn't that be enough to flag it busy?



Best regards,
Dom

Re: Questions about resilient next-hop groups (6.18.35)

[Ido Schimmel]

On Thu, Jul 09, 2026 at 02:46:47PM +0000, Froehlich, Dominik wrote:
> Hello everyone,
> 
> I am running kernel 6.18.35 and I have the following multi path route setup:
> 
> 
> 10.181.0.0/16 nhid 400
>       nexthop dev bond0-ip6tnl0 weight 1
>       nexthop dev bond0-ip6tnl1 weight 1
> 100.80.0.0/12 nhid 400
>       nexthop dev bond0-ip6tnl0 weight 1
>       nexthop dev bond0-ip6tnl1 weight 1
> 100.112.0.0/13 nhid 400
>       nexthop dev bond0-ip6tnl0 weight 1
>       nexthop dev bond0-ip6tnl1 weight 1
> 
> 
> The links are ip6tnl devices sending traffic over an IPv6 tunnel to the other end.
> 
> I am using L4 ECMP hash policy:
> 
> ~ # sysctl net.ipv4.fib_multipath_hash_policy
> net.ipv4.fib_multipath_hash_policy = 1
> ~ # sysctl net.ipv6.fib_multipath_hash_policy
> net.ipv6.fib_multipath_hash_policy = 1
> 
> Here are the next hop definitions:
> 
> id 400 group 4000,128/4001,128 type resilient buckets 1024 idle_timer 60 unbalanced_timer 0 unbalanced_time 0
> id 4000 dev bond0-ip6tnl0 scope host
> id 4001 dev bond0-ip6tnl1 scope host
> 
> 
> So I have two equal weighted next hops with 512 buckets each.
> 
> Now I am running a TCP connection in a shell on a host in the 10.181.0.0/16 network over one of the next hops that continuously produces traffic to keep its bucket busy:
> 
> while true; do 
>   sleep 0.1
>   date
> done 
> 
> 
> Thu Jul  9 13:27:03 UTC 2026
> Thu Jul  9 13:27:03 UTC 2026
> Thu Jul  9 13:27:03 UTC 2026
> Thu Jul  9 13:27:04 UTC 2026
> (…)
> 
> This roughly sends 10 pkts/s.
> Now I want to drain next hop 4000 which is where my traffic is flowing:
> 
> 
> ip nexthop replace id 400 group 4000,1/4001,255 type resilient buckets 1024 idle_timer 60 unbalanced_timer 0
> 
> Assigning it a weight of 1 should reduce the number of buckets assigned to next hop 4000 to 4 and move idle buckets to next hop 4001.
> 
> However, what I am seeing is that the connection is immediately terminated.
> 
> 
> When I am changing the weights more slowly it kind of works:
> 
> ~ # ip nexthop replace id 400 group 4000,128/4001,128 type resilient idle_timer 60 unbalanced_timer 0
> ~ # ip nexthop replace id 400 group 4000,64/4001,192 type resilient idle_timer 60 unbalanced_timer 0
> ~ # ip nexthop replace id 400 group 4000,5/4001,250 type resilient idle_timer 60 unbalanced_timer 0
> ~ # ip nexthop replace id 400 group 4000,1/4001,255 type resilient idle_timer 60 unbalanced_timer 0
> 
> I can then see the buckets reduced to just 4 over time as expected:
> 
> Every 1.0s: ip nexthop bucket list nhid 4000                                                                                                                               2026-07-09 13:44:22
> 
> id 400 index 447 idle_time 135.51 nhid 4000
> id 400 index 631 idle_time 117.56 nhid 4000
> id 400 index 720 idle_time 73.11 nhid 4000
> id 400 index 742 idle_time 136.19 nhid 4000
> 
> 
> My guess is that my connection is in bucket index 720 as it has the lowest idle_time reported. 
> 
> My questions:
> - Why is the idle_time reported so high? There should be packets flowing through that bucket every 0.1 seconds, so I was expecting the idle_time to be near zero.
> - Why does my connection get cut immediately when draining rapidly but kind of stays alive when changing weights more slowly? Shouldn't it be a busy (undrainable) bucket either way?
> Bucket nr 720 already has idle_time higher than 60 so I think it would be migrated on any further weight changes - which would disrupt the existing connection again.
> - When / how is the idle_time of a bucket updated? The docs mention that “Every time a packet is forwarded through a bucket, this timer is updated to current jiffies value.” I found that the bucket idle_time is refreshed more frequently when fever buckets are used. But why does the number of buckets even matter here? Fewer buckets mean more flows per bucket, I get that. But if there is at least one busy connection in a bucket, shouldn't that be enough to flag it busy?

IIUC you are using resilient nexthop groups with locally originated
traffic when these groups are meant to be used with forwarded traffic.
TCP caches the route lookup result in the socket and uses the cached dst
entry instead of performing a route lookup each time, so the routing
layer is bypassed and the nexthop bucket is not marked as used.

You can try to force a route lookup by flushing the cache, but it's a
hack and not something I recommend doing in production (it affects the
entire system, not only the flow that you care about):

echo 1 > /proc/sys/net/ipv4/route/flush