The Linux Kernel Mailing List
 help / color / mirror / Atom feed
* [PATCH net] selftests: bonding: fix failover race in bond_options.sh prio test
@ 2026-11-12  1:46 Pin-yen Lin
  2026-11-13  5:23 ` Hangbin Liu
  0 siblings, 1 reply; 7+ messages in thread
From: Pin-yen Lin @ 2026-11-12  1:46 UTC (permalink / raw)
  To: Jay Vosburgh, Andrew Lunn, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Shuah Khan
  Cc: netdev, Matthieu Baerts, linux-kernel, Jonathan Toppins,
	linux-kselftest, Hangbin Liu, Pin-yen Lin

The prio test intermittently fails in the arp_ip_target and
ns_ip6_target variants:

  TEST: prio (active-backup ns_ip6_target primary_reselect 1)   [FAIL]
  	Current active slave is eth0 but not eth2

The ARP/NS monitor marks a slave down when it has received nothing for
too long. With arp_interval 100 and the default arp_missed_max, that is
about 250ms for the active slave (eth1) and 350ms for a backup slave
(eth0, eth2). Every 100ms, eth1 sends a probe: eth1 receives the reply,
and eth0/eth2 receive a copy forwarded by the gateway bridge. Once eth1
is down, its probes reach no one and no slave receives anything, so
eth1 normally times out first and the new active slave is picked by
prio:

  t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
  t=10   test sets eth1 down
  t=300  monitor: eth1 idle 300ms -> down
                  eth0/eth2 idle 300ms -> up, eth2 picked by prio

But the test takes eth1 down right after bond_check_connection(), whose
ping replies only arrive on eth1:

  t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
  t=70   ping reply on eth1 only
  t=85   test sets eth1 down
  t=300  monitor: eth1 idle 230ms -> still up
  t=400  monitor: eth1 idle 330ms -> down
                  eth2 idle 400ms -> down too

eth2 can then no longer be picked. eth0 becomes the active slave if it
happened to receive some other packet (e.g. an MLD report), otherwise
whichever slave comes back first does. With primary_reselect 1 or 2,
the bond keeps eth0 even after eth2 comes back.

Before taking the active slave down, wait two monitor intervals
(200ms), so that at least one more probe is sent after the last ping
reply and all slaves last received that probe. The miimon variants do
not need the wait, but the extra delay is negligible.

Hit in our CI on GCE VMs (4 of 51 runs); the test only uses veth. On
the same VMs, 7 of 30 loops of the prio tests failed without this
change, and 20 of 20 loops passed with it.

Fixes: 481b56e0391e ("selftests: bonding: re-format bond option tests")
Signed-off-by: Pin-yen Lin <treapking@google.com>
---

 .../drivers/net/bonding/bond_options.sh       | 26 ++++++++++++++++---
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/drivers/net/bonding/bond_options.sh b/tools/testing/selftests/drivers/net/bonding/bond_options.sh
index 187b478d0ddf..16c1c2d97005 100755
--- a/tools/testing/selftests/drivers/net/bonding/bond_options.sh
+++ b/tools/testing/selftests/drivers/net/bonding/bond_options.sh
@@ -13,6 +13,24 @@ ALL_TESTS="
 
 lib_dir=$(dirname "$0")
 source ${lib_dir}/bond_topo_3d1c.sh
+
+# The ARP/NS monitor marks a slave down when it has received nothing for too
+# long: ~250ms for the active slave and ~350ms for a backup slave with
+# arp_interval 100. The backups only receive the copies of the active slave's
+# probes that the gateway bridge forwards, but the active slave also gets the
+# replies to the pings in bond_check_connection(). If the active slave goes
+# down right after such a reply, it times out in the same monitor run as the
+# backups, and the new active slave is then not picked by prio. Wait two
+# monitor intervals, so that at least one more probe is sent and all slaves
+# last received that probe.
+bond_link_down()
+{
+	local slave=$1
+
+	sleep 0.2
+	ip -n ${s_ns} link set "$slave" down
+}
+
 c_maddr="33:33:ff:00:00:10"
 g_maddr="33:33:ff:00:02:54"
 
@@ -97,12 +115,12 @@ prio_test()
 	check_active_slave eth1
 
 	# active slave should be the higher prio slave
-	ip -n ${s_ns} link set $active_slave down
+	bond_link_down $active_slave
 	check_active_slave eth2
 	bond_check_connection "fail over"
 
 	# when only 1 slave is up
-	ip -n ${s_ns} link set $active_slave down
+	bond_link_down $active_slave
 	check_active_slave eth0
 	bond_check_connection "only 1 slave up"
 
@@ -134,7 +152,7 @@ prio_test()
 			;;
 		"2")
 			check_active_slave "$pre_active_slave"
-			ip -n ${s_ns} link set $active_slave down
+			bond_link_down $active_slave
 			bond_check_connection "pre_active slave down"
 			check_active_slave "eth1"
 			;;
@@ -153,7 +171,7 @@ prio_test()
 		check_err $? "eth3 prio is not -50"
 		check_active_slave "eth1"
 
-		ip -n ${s_ns} link set $active_slave down
+		bond_link_down $active_slave
 		check_active_slave "eth0"
 		bond_check_connection "change slave prio"
 	fi
-- 
2.56.1.703.gf848ed807c-goog

base-commit: 665b4149e91b781a71bce8637c3a4d7d64dba05f

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH net] selftests: bonding: fix failover race in bond_options.sh prio test
  2026-11-12  1:46 [PATCH net] selftests: bonding: fix failover race in bond_options.sh prio test Pin-yen Lin
@ 2026-11-13  5:23 ` Hangbin Liu
  2026-11-13  5:59   ` Hangbin Liu
  0 siblings, 1 reply; 7+ messages in thread
From: Hangbin Liu @ 2026-11-13  5:23 UTC (permalink / raw)
  To: Pin-yen Lin
  Cc: Jay Vosburgh, Andrew Lunn, David S. Miller, Eric Dumazet,
	Bjoern Doebel, Jakub Kicinski, Paolo Abeni, Shuah Khan, netdev,
	Matthieu Baerts, linux-kernel, Jonathan Toppins, linux-kselftest,
	Hangbin Liu

Hi Pin-yen,
On Wed, Nov 11, 2026 at 05:46:18PM -0800, Pin-yen Lin wrote:
> The prio test intermittently fails in the arp_ip_target and
> ns_ip6_target variants:
> 
>   TEST: prio (active-backup ns_ip6_target primary_reselect 1)   [FAIL]
>   	Current active slave is eth0 but not eth2
> 
> The ARP/NS monitor marks a slave down when it has received nothing for
> too long. With arp_interval 100 and the default arp_missed_max, that is
> about 250ms for the active slave (eth1) and 350ms for a backup slave
> (eth0, eth2). Every 100ms, eth1 sends a probe: eth1 receives the reply,
> and eth0/eth2 receive a copy forwarded by the gateway bridge. Once eth1
> is down, its probes reach no one and no slave receives anything, so
> eth1 normally times out first and the new active slave is picked by
> prio:
> 
>   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
>   t=10   test sets eth1 down
>   t=300  monitor: eth1 idle 300ms -> down
>                   eth0/eth2 idle 300ms -> up, eth2 picked by prio
> 
> But the test takes eth1 down right after bond_check_connection(), whose
> ping replies only arrive on eth1:
> 
>   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
>   t=70   ping reply on eth1 only
>   t=85   test sets eth1 down
>   t=300  monitor: eth1 idle 230ms -> still up
>   t=400  monitor: eth1 idle 330ms -> down
>                   eth2 idle 400ms -> down too
> 
> eth2 can then no longer be picked. eth0 becomes the active slave if it
> happened to receive some other packet (e.g. an MLD report), otherwise
> whichever slave comes back first does. With primary_reselect 1 or 2,
> the bond keeps eth0 even after eth2 comes back.
> 
> Before taking the active slave down, wait two monitor intervals
> (200ms), so that at least one more probe is sent after the last ping
> reply and all slaves last received that probe. The miimon variants do
> not need the wait, but the extra delay is negligible.

I'm afraid the waiting time is too strict, especially with a debug kernel
since delays can happen.

Bjoern is also fixing[1] this issue by pinging a non-existent neighbor to
keep traffic flowing. Would you like to test whether that works for you?

[1] https://lore.kernel.org/netdev/20261104142018.552135-1-doebel@amazon.de/

Thanks
Hangbin

> 
> Hit in our CI on GCE VMs (4 of 51 runs); the test only uses veth. On
> the same VMs, 7 of 30 loops of the prio tests failed without this
> change, and 20 of 20 loops passed with it.
> 
> Fixes: 481b56e0391e ("selftests: bonding: re-format bond option tests")
> Signed-off-by: Pin-yen Lin <treapking@google.com>
> ---
> 
>  .../drivers/net/bonding/bond_options.sh       | 26 ++++++++++++++++---
>  1 file changed, 22 insertions(+), 4 deletions(-)
> 
> diff --git a/tools/testing/selftests/drivers/net/bonding/bond_options.sh b/tools/testing/selftests/drivers/net/bonding/bond_options.sh
> index 187b478d0ddf..16c1c2d97005 100755
> --- a/tools/testing/selftests/drivers/net/bonding/bond_options.sh
> +++ b/tools/testing/selftests/drivers/net/bonding/bond_options.sh
> @@ -13,6 +13,24 @@ ALL_TESTS="
>  
>  lib_dir=$(dirname "$0")
>  source ${lib_dir}/bond_topo_3d1c.sh
> +
> +# The ARP/NS monitor marks a slave down when it has received nothing for too
> +# long: ~250ms for the active slave and ~350ms for a backup slave with
> +# arp_interval 100. The backups only receive the copies of the active slave's
> +# probes that the gateway bridge forwards, but the active slave also gets the
> +# replies to the pings in bond_check_connection(). If the active slave goes
> +# down right after such a reply, it times out in the same monitor run as the
> +# backups, and the new active slave is then not picked by prio. Wait two
> +# monitor intervals, so that at least one more probe is sent and all slaves
> +# last received that probe.
> +bond_link_down()
> +{
> +	local slave=$1
> +
> +	sleep 0.2
> +	ip -n ${s_ns} link set "$slave" down
> +}
> +
>  c_maddr="33:33:ff:00:00:10"
>  g_maddr="33:33:ff:00:02:54"
>  
> @@ -97,12 +115,12 @@ prio_test()
>  	check_active_slave eth1
>  
>  	# active slave should be the higher prio slave
> -	ip -n ${s_ns} link set $active_slave down
> +	bond_link_down $active_slave
>  	check_active_slave eth2
>  	bond_check_connection "fail over"
>  
>  	# when only 1 slave is up
> -	ip -n ${s_ns} link set $active_slave down
> +	bond_link_down $active_slave
>  	check_active_slave eth0
>  	bond_check_connection "only 1 slave up"
>  
> @@ -134,7 +152,7 @@ prio_test()
>  			;;
>  		"2")
>  			check_active_slave "$pre_active_slave"
> -			ip -n ${s_ns} link set $active_slave down
> +			bond_link_down $active_slave
>  			bond_check_connection "pre_active slave down"
>  			check_active_slave "eth1"
>  			;;
> @@ -153,7 +171,7 @@ prio_test()
>  		check_err $? "eth3 prio is not -50"
>  		check_active_slave "eth1"
>  
> -		ip -n ${s_ns} link set $active_slave down
> +		bond_link_down $active_slave
>  		check_active_slave "eth0"
>  		bond_check_connection "change slave prio"
>  	fi
> -- 
> 2.56.1.703.gf848ed807c-goog
> 
> base-commit: 665b4149e91b781a71bce8637c3a4d7d64dba05f

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH net] selftests: bonding: fix failover race in bond_options.sh prio test
  2026-11-13  5:23 ` Hangbin Liu
@ 2026-11-13  5:59   ` Hangbin Liu
  2026-11-17  1:30     ` Pin-yen Lin
  0 siblings, 1 reply; 7+ messages in thread
From: Hangbin Liu @ 2026-11-13  5:59 UTC (permalink / raw)
  To: Pin-yen Lin
  Cc: Jay Vosburgh, Andrew Lunn, David S. Miller, Eric Dumazet,
	Bjoern Doebel, Jakub Kicinski, Paolo Abeni, Shuah Khan, netdev,
	Matthieu Baerts, linux-kernel, Jonathan Toppins, linux-kselftest,
	Hangbin Liu

On Fri, Nov 13, 2026 at 01:23:40PM +0800, Hangbin Liu wrote:
> Hi Pin-yen,
> On Wed, Nov 11, 2026 at 05:46:18PM -0800, Pin-yen Lin wrote:
> > The prio test intermittently fails in the arp_ip_target and
> > ns_ip6_target variants:
> > 
> >   TEST: prio (active-backup ns_ip6_target primary_reselect 1)   [FAIL]
> >   	Current active slave is eth0 but not eth2
> > 
> > The ARP/NS monitor marks a slave down when it has received nothing for
> > too long. With arp_interval 100 and the default arp_missed_max, that is
> > about 250ms for the active slave (eth1) and 350ms for a backup slave
> > (eth0, eth2). Every 100ms, eth1 sends a probe: eth1 receives the reply,
> > and eth0/eth2 receive a copy forwarded by the gateway bridge. Once eth1
> > is down, its probes reach no one and no slave receives anything, so
> > eth1 normally times out first and the new active slave is picked by
> > prio:
> > 
> >   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
> >   t=10   test sets eth1 down
> >   t=300  monitor: eth1 idle 300ms -> down
> >                   eth0/eth2 idle 300ms -> up, eth2 picked by prio
> > 
> > But the test takes eth1 down right after bond_check_connection(), whose
> > ping replies only arrive on eth1:
> > 
> >   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
> >   t=70   ping reply on eth1 only
> >   t=85   test sets eth1 down
> >   t=300  monitor: eth1 idle 230ms -> still up
> >   t=400  monitor: eth1 idle 330ms -> down
> >                   eth2 idle 400ms -> down too
> > 
> > eth2 can then no longer be picked. eth0 becomes the active slave if it
> > happened to receive some other packet (e.g. an MLD report), otherwise
> > whichever slave comes back first does. With primary_reselect 1 or 2,
> > the bond keeps eth0 even after eth2 comes back.
> > 
> > Before taking the active slave down, wait two monitor intervals
> > (200ms), so that at least one more probe is sent after the last ping
> > reply and all slaves last received that probe. The miimon variants do
> > not need the wait, but the extra delay is negligible.
> 
> I'm afraid the waiting time is too strict, especially with a debug kernel
> since delays can happen.
> 
> Bjoern is also fixing[1] this issue by pinging a non-existent neighbor to
> keep traffic flowing. Would you like to test whether that works for you?
> 
> [1] https://lore.kernel.org/netdev/20261104142018.552135-1-doebel@amazon.de/

BTW, using ping may not able to triger the arp request in time. You can try
arping instead.

Thanks
Hangbin

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH net] selftests: bonding: fix failover race in bond_options.sh prio test
  2026-11-13  5:59   ` Hangbin Liu
@ 2026-11-17  1:30     ` Pin-yen Lin
  2026-11-17  7:26       ` Hangbin Liu
  0 siblings, 1 reply; 7+ messages in thread
From: Pin-yen Lin @ 2026-11-17  1:30 UTC (permalink / raw)
  To: Hangbin Liu
  Cc: Jay Vosburgh, Andrew Lunn, David S. Miller, Eric Dumazet,
	Bjoern Doebel, Jakub Kicinski, Paolo Abeni, Shuah Khan, netdev,
	Matthieu Baerts, linux-kernel, Jonathan Toppins, linux-kselftest,
	Hangbin Liu

Hi Hangbin,

On Thu, Nov 12, 2026 at 10:00 PM Hangbin Liu <hangbin.liu@linux.dev> wrote:
>
> On Fri, Nov 13, 2026 at 01:23:40PM +0800, Hangbin Liu wrote:
> > Hi Pin-yen,
> > On Wed, Nov 11, 2026 at 05:46:18PM -0800, Pin-yen Lin wrote:
> > > The prio test intermittently fails in the arp_ip_target and
> > > ns_ip6_target variants:
> > >
> > >   TEST: prio (active-backup ns_ip6_target primary_reselect 1)   [FAIL]
> > >     Current active slave is eth0 but not eth2
> > >
> > > The ARP/NS monitor marks a slave down when it has received nothing for
> > > too long. With arp_interval 100 and the default arp_missed_max, that is
> > > about 250ms for the active slave (eth1) and 350ms for a backup slave
> > > (eth0, eth2). Every 100ms, eth1 sends a probe: eth1 receives the reply,
> > > and eth0/eth2 receive a copy forwarded by the gateway bridge. Once eth1
> > > is down, its probes reach no one and no slave receives anything, so
> > > eth1 normally times out first and the new active slave is picked by
> > > prio:
> > >
> > >   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
> > >   t=10   test sets eth1 down
> > >   t=300  monitor: eth1 idle 300ms -> down
> > >                   eth0/eth2 idle 300ms -> up, eth2 picked by prio
> > >
> > > But the test takes eth1 down right after bond_check_connection(), whose
> > > ping replies only arrive on eth1:
> > >
> > >   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
> > >   t=70   ping reply on eth1 only
> > >   t=85   test sets eth1 down
> > >   t=300  monitor: eth1 idle 230ms -> still up
> > >   t=400  monitor: eth1 idle 330ms -> down
> > >                   eth2 idle 400ms -> down too
> > >
> > > eth2 can then no longer be picked. eth0 becomes the active slave if it
> > > happened to receive some other packet (e.g. an MLD report), otherwise
> > > whichever slave comes back first does. With primary_reselect 1 or 2,
> > > the bond keeps eth0 even after eth2 comes back.
> > >
> > > Before taking the active slave down, wait two monitor intervals
> > > (200ms), so that at least one more probe is sent after the last ping
> > > reply and all slaves last received that probe. The miimon variants do
> > > not need the wait, but the extra delay is negligible.
> >
> > I'm afraid the waiting time is too strict, especially with a debug kernel
> > since delays can happen.
> >
> > Bjoern is also fixing[1] this issue by pinging a non-existent neighbor to
> > keep traffic flowing. Would you like to test whether that works for you?
> >
> > [1] https://lore.kernel.org/netdev/20261104142018.552135-1-doebel@amazon.de/
>
> BTW, using ping may not able to triger the arp request in time. You can try
> arping instead.

Thanks for the pointer, but I believe Bjoern's patch addresses a
different problem.

Bjoern's patch keeps the traffic flowing so that idle slaves are kept
alive between the test steps. This patch fixes the case where the ping
replies from bond_check_connection() only arrive on the active slave
right before it is set down. The active slave's failure is then
detected one monitor interval late, when the backups have also timed
out.

The ARP requests triggered by Bjoern's ping are sent only about once
per second, which isn't frequent enough for this failure. I tested
Bjoern's patch and confirmed that it didn't fix the issue in our
testing environment. We also can't simply use arping instead because
iputils arping doesn't support sub-second intervals.

If you're worried that the 200ms sleep is too short, would increasing
it to 0.5s or 1s work for you?
>
>
> Thanks
> Hangbin

Regards,
Pin-yen

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH net] selftests: bonding: fix failover race in bond_options.sh prio test
  2026-11-17  1:30     ` Pin-yen Lin
@ 2026-11-17  7:26       ` Hangbin Liu
  2026-11-18  2:32         ` Pin-yen Lin
  0 siblings, 1 reply; 7+ messages in thread
From: Hangbin Liu @ 2026-11-17  7:26 UTC (permalink / raw)
  To: Pin-yen Lin
  Cc: Jay Vosburgh, Andrew Lunn, David S. Miller, Eric Dumazet,
	Bjoern Doebel, Jakub Kicinski, Paolo Abeni, Shuah Khan, netdev,
	Matthieu Baerts, linux-kernel, Jonathan Toppins, linux-kselftest,
	Hangbin Liu

Hi Pin-yen,
On Mon, Nov 16, 2026 at 05:30:28PM -0800, Pin-yen Lin wrote:
> Hi Hangbin,
> 
> On Thu, Nov 12, 2026 at 10:00 PM Hangbin Liu <hangbin.liu@linux.dev> wrote:
> >
> > On Fri, Nov 13, 2026 at 01:23:40PM +0800, Hangbin Liu wrote:
> > > Hi Pin-yen,
> > > On Wed, Nov 11, 2026 at 05:46:18PM -0800, Pin-yen Lin wrote:
> > > > The prio test intermittently fails in the arp_ip_target and
> > > > ns_ip6_target variants:
> > > >
> > > >   TEST: prio (active-backup ns_ip6_target primary_reselect 1)   [FAIL]
> > > >     Current active slave is eth0 but not eth2
> > > >
> > > > The ARP/NS monitor marks a slave down when it has received nothing for
> > > > too long. With arp_interval 100 and the default arp_missed_max, that is
> > > > about 250ms for the active slave (eth1) and 350ms for a backup slave
> > > > (eth0, eth2). Every 100ms, eth1 sends a probe: eth1 receives the reply,
> > > > and eth0/eth2 receive a copy forwarded by the gateway bridge. Once eth1
> > > > is down, its probes reach no one and no slave receives anything, so
> > > > eth1 normally times out first and the new active slave is picked by
> > > > prio:
> > > >
> > > >   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
> > > >   t=10   test sets eth1 down
> > > >   t=300  monitor: eth1 idle 300ms -> down
> > > >                   eth0/eth2 idle 300ms -> up, eth2 picked by prio
> > > >
> > > > But the test takes eth1 down right after bond_check_connection(), whose
> > > > ping replies only arrive on eth1:
> > > >
> > > >   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
> > > >   t=70   ping reply on eth1 only
> > > >   t=85   test sets eth1 down
> > > >   t=300  monitor: eth1 idle 230ms -> still up
> > > >   t=400  monitor: eth1 idle 330ms -> down
> > > >                   eth2 idle 400ms -> down too
> > > >
> > > > eth2 can then no longer be picked. eth0 becomes the active slave if it
> > > > happened to receive some other packet (e.g. an MLD report), otherwise
> > > > whichever slave comes back first does. With primary_reselect 1 or 2,
> > > > the bond keeps eth0 even after eth2 comes back.
> > > >
> > > > Before taking the active slave down, wait two monitor intervals
> > > > (200ms), so that at least one more probe is sent after the last ping
> > > > reply and all slaves last received that probe. The miimon variants do
> > > > not need the wait, but the extra delay is negligible.
> > >
> > > I'm afraid the waiting time is too strict, especially with a debug kernel
> > > since delays can happen.
> > >
> > > Bjoern is also fixing[1] this issue by pinging a non-existent neighbor to
> > > keep traffic flowing. Would you like to test whether that works for you?
> > >
> > > [1] https://lore.kernel.org/netdev/20261104142018.552135-1-doebel@amazon.de/
> >
> > BTW, using ping may not able to triger the arp request in time. You can try
> > arping instead.
> 
> Thanks for the pointer, but I believe Bjoern's patch addresses a
> different problem.

I think they are the same issue, as you comment also said

The ARP/NS monitor marks a slave down when it has received nothing for
too long.

> 
> Bjoern's patch keeps the traffic flowing so that idle slaves are kept
> alive between the test steps. This patch fixes the case where the ping
> replies from bond_check_connection() only arrive on the active slave
> right before it is set down. The active slave's failure is then
> detected one monitor interval late, when the backups have also timed
> out.

The reason is we set arp interval to 100ms, and slowwait also check every
100ms. If the backup slave down at the same check point, e.g.

  t=300  monitor: eth1 idle 230ms -> still up
  t=400  monitor: eth1 idle 330ms -> down
                  eth2 idle 400ms -> down too

The test failed.

> 
> The ARP requests triggered by Bjoern's ping are sent only about once
> per second, which isn't frequent enough for this failure. I tested
> Bjoern's patch and confirmed that it didn't fix the issue in our
> testing environment. We also can't simply use arping instead because
> iputils arping doesn't support sub-second intervals.
> 
> If you're worried that the 200ms sleep is too short, would increasing
> it to 0.5s or 1s work for you?

It's not about the sleep time before setting active slave down. I'm afraid
about the slowwait checking time. Even we sleep 1s, with arp interval 100,
we still only have 100ms checking window. Before the window, the active slave
is up, after the window, the backup slave also down. With a debug kernel, this
time is a little tight.

So, either we have some background arp traffic, or let's extend the arp
interval to a larger one.

What do you think?

Thanks
Hangbin

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH net] selftests: bonding: fix failover race in bond_options.sh prio test
  2026-11-17  7:26       ` Hangbin Liu
@ 2026-11-18  2:32         ` Pin-yen Lin
  2026-11-18  8:27           ` Hangbin Liu
  0 siblings, 1 reply; 7+ messages in thread
From: Pin-yen Lin @ 2026-11-18  2:32 UTC (permalink / raw)
  To: Hangbin Liu
  Cc: Jay Vosburgh, Andrew Lunn, David S. Miller, Eric Dumazet,
	Bjoern Doebel, Jakub Kicinski, Paolo Abeni, Shuah Khan, netdev,
	Matthieu Baerts, linux-kernel, Jonathan Toppins, linux-kselftest,
	Hangbin Liu

Hi Hangbin,

On Mon, Nov 16, 2026 at 11:26 PM Hangbin Liu <hangbin.liu@linux.dev> wrote:
>
> Hi Pin-yen,
> On Mon, Nov 16, 2026 at 05:30:28PM -0800, Pin-yen Lin wrote:
> > Hi Hangbin,
> >
> > On Thu, Nov 12, 2026 at 10:00 PM Hangbin Liu <hangbin.liu@linux.dev> wrote:
> > >
> > > On Fri, Nov 13, 2026 at 01:23:40PM +0800, Hangbin Liu wrote:
> > > > Hi Pin-yen,
> > > > On Wed, Nov 11, 2026 at 05:46:18PM -0800, Pin-yen Lin wrote:
> > > > > The prio test intermittently fails in the arp_ip_target and
> > > > > ns_ip6_target variants:
> > > > >
> > > > >   TEST: prio (active-backup ns_ip6_target primary_reselect 1)   [FAIL]
> > > > >     Current active slave is eth0 but not eth2
> > > > >
> > > > > The ARP/NS monitor marks a slave down when it has received nothing for
> > > > > too long. With arp_interval 100 and the default arp_missed_max, that is
> > > > > about 250ms for the active slave (eth1) and 350ms for a backup slave
> > > > > (eth0, eth2). Every 100ms, eth1 sends a probe: eth1 receives the reply,
> > > > > and eth0/eth2 receive a copy forwarded by the gateway bridge. Once eth1
> > > > > is down, its probes reach no one and no slave receives anything, so
> > > > > eth1 normally times out first and the new active slave is picked by
> > > > > prio:
> > > > >
> > > > >   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
> > > > >   t=10   test sets eth1 down
> > > > >   t=300  monitor: eth1 idle 300ms -> down
> > > > >                   eth0/eth2 idle 300ms -> up, eth2 picked by prio
> > > > >
> > > > > But the test takes eth1 down right after bond_check_connection(), whose
> > > > > ping replies only arrive on eth1:
> > > > >
> > > > >   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
> > > > >   t=70   ping reply on eth1 only
> > > > >   t=85   test sets eth1 down
> > > > >   t=300  monitor: eth1 idle 230ms -> still up
> > > > >   t=400  monitor: eth1 idle 330ms -> down
> > > > >                   eth2 idle 400ms -> down too
> > > > >
> > > > > eth2 can then no longer be picked. eth0 becomes the active slave if it
> > > > > happened to receive some other packet (e.g. an MLD report), otherwise
> > > > > whichever slave comes back first does. With primary_reselect 1 or 2,
> > > > > the bond keeps eth0 even after eth2 comes back.
> > > > >
> > > > > Before taking the active slave down, wait two monitor intervals
> > > > > (200ms), so that at least one more probe is sent after the last ping
> > > > > reply and all slaves last received that probe. The miimon variants do
> > > > > not need the wait, but the extra delay is negligible.
> > > >
> > > > I'm afraid the waiting time is too strict, especially with a debug kernel
> > > > since delays can happen.
> > > >
> > > > Bjoern is also fixing[1] this issue by pinging a non-existent neighbor to
> > > > keep traffic flowing. Would you like to test whether that works for you?
> > > >
> > > > [1] https://lore.kernel.org/netdev/20261104142018.552135-1-doebel@amazon.de/
> > >
> > > BTW, using ping may not able to triger the arp request in time. You can try
> > > arping instead.
> >
> > Thanks for the pointer, but I believe Bjoern's patch addresses a
> > different problem.
>
> I think they are the same issue, as you comment also said
>
> The ARP/NS monitor marks a slave down when it has received nothing for
> too long.

This is the existing behavior of the bonding driver, and the prio test
uses it to check which backup slave takes over. Slaves being marked
down is not the root cause of the test failure. Instead, the root
cause is the ping replies described below.

>
> >
> > Bjoern's patch keeps the traffic flowing so that idle slaves are kept
> > alive between the test steps. This patch fixes the case where the ping
> > replies from bond_check_connection() only arrive on the active slave
> > right before it is set down. The active slave's failure is then
> > detected one monitor interval late, when the backups have also timed
> > out.
>
> The reason is we set arp interval to 100ms, and slowwait also check every
> 100ms. If the backup slave down at the same check point, e.g.
>
>   t=300  monitor: eth1 idle 230ms -> still up
>   t=400  monitor: eth1 idle 330ms -> down
>                   eth2 idle 400ms -> down too
>
> The test failed.

I don't think the slowwait interval makes a difference here.

The ARP monitor decides which slave becomes active, and slowwait only
reads the result afterwards. The test expects the active slave and the
backups to time out in different monitor runs, so that a backup can
take over. In the example above, eth2 is already down when the bond
picks the new active slave, no matter when slowwait polls.
>
> >
> > The ARP requests triggered by Bjoern's ping are sent only about once
> > per second, which isn't frequent enough for this failure. I tested
> > Bjoern's patch and confirmed that it didn't fix the issue in our
> > testing environment. We also can't simply use arping instead because
> > iputils arping doesn't support sub-second intervals.
> >
> > If you're worried that the 200ms sleep is too short, would increasing
> > it to 0.5s or 1s work for you?
>
> It's not about the sleep time before setting active slave down. I'm afraid
> about the slowwait checking time. Even we sleep 1s, with arp interval 100,
> we still only have 100ms checking window. Before the window, the active slave
> is up, after the window, the backup slave also down. With a debug kernel, this
> time is a little tight.
>
> So, either we have some background arp traffic, or let's extend the arp
> interval to a larger one.
>
> What do you think?

I agree that a 100ms ARP interval could be too short in environments
like a debug kernel.

Background ARP traffic doesn't widen that window, though: it is sent
through the active slave, so once the active slave is set down,
nothing reaches the backups. It could only help with the ping reply
race if it were very frequent; Bjoern's patch triggers about one ARP
request per second and didn't fix the failures in our testing.

Increasing the ARP interval alone won't address the ping reply race,
but it should help in environments like a debug kernel. I'd prefer to
keep this patch focused on the ping reply race and send the ARP
interval change as a follow-up, but I can fold it into v2 if you
prefer.
>
> Thanks
> Hangbin

Regards,
Pin-yen

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH net] selftests: bonding: fix failover race in bond_options.sh prio test
  2026-11-18  2:32         ` Pin-yen Lin
@ 2026-11-18  8:27           ` Hangbin Liu
  0 siblings, 0 replies; 7+ messages in thread
From: Hangbin Liu @ 2026-11-18  8:27 UTC (permalink / raw)
  To: Pin-yen Lin
  Cc: Jay Vosburgh, Andrew Lunn, David S. Miller, Eric Dumazet,
	Bjoern Doebel, Jakub Kicinski, Paolo Abeni, Shuah Khan, netdev,
	Matthieu Baerts, linux-kernel, Jonathan Toppins, linux-kselftest,
	Hangbin Liu

Hi Pin-yen,
On Tue, Nov 17, 2026 at 06:32:52PM -0800, Pin-yen Lin wrote:
> Hi Hangbin,
> 
> On Mon, Nov 16, 2026 at 11:26 PM Hangbin Liu <hangbin.liu@linux.dev> wrote:
> >
> > Hi Pin-yen,
> > On Mon, Nov 16, 2026 at 05:30:28PM -0800, Pin-yen Lin wrote:
> > > Thanks for the pointer, but I believe Bjoern's patch addresses a
> > > different problem.
> >
> > I think they are the same issue, as you comment also said
> >
> > The ARP/NS monitor marks a slave down when it has received nothing for
> > too long.
> 
> This is the existing behavior of the bonding driver, and the prio test
> uses it to check which backup slave takes over. Slaves being marked
> down is not the root cause of the test failure. Instead, the root
> cause is the ping replies described below.

If the backup slaves are not mark down, we can active the correct slave
based on the priority, right? I think that's the root cause. The ping testing
is just for checking connectivity.

> 
> >
> > >
> > > Bjoern's patch keeps the traffic flowing so that idle slaves are kept
> > > alive between the test steps. This patch fixes the case where the ping
> > > replies from bond_check_connection() only arrive on the active slave
> > > right before it is set down. The active slave's failure is then
> > > detected one monitor interval late, when the backups have also timed
> > > out.
> >
> > The reason is we set arp interval to 100ms, and slowwait also check every
> > 100ms. If the backup slave down at the same check point, e.g.
> >
> >   t=300  monitor: eth1 idle 230ms -> still up
> >   t=400  monitor: eth1 idle 330ms -> down
> >                   eth2 idle 400ms -> down too
> >
> > The test failed.
> 
> I don't think the slowwait interval makes a difference here.
> 
> The ARP monitor decides which slave becomes active, and slowwait only
> reads the result afterwards. The test expects the active slave and the
> backups to time out in different monitor runs, so that a backup can
> take over. In the example above, eth2 is already down when the bond
> picks the new active slave, no matter when slowwait polls.


My idea is, by extending the arp interval, there is a larger gap. e.g.

   t=300  monitor: eth1 still up
		   ^^ slowwait check, incorrect active slave
   t=400  monitor: eth1 down
                   eth2 still up, should become active slave
   t=500  monitor: eth1 down
                   eth2 still up, active slave
		   ^^ slowwait check, correct active

> >
> > >
> > > The ARP requests triggered by Bjoern's ping are sent only about once
> > > per second, which isn't frequent enough for this failure. I tested
> > > Bjoern's patch and confirmed that it didn't fix the issue in our
> > > testing environment. We also can't simply use arping instead because
> > > iputils arping doesn't support sub-second intervals.
> > >
> > > If you're worried that the 200ms sleep is too short, would increasing
> > > it to 0.5s or 1s work for you?
> >
> > It's not about the sleep time before setting active slave down. I'm afraid
> > about the slowwait checking time. Even we sleep 1s, with arp interval 100,
> > we still only have 100ms checking window. Before the window, the active slave
> > is up, after the window, the backup slave also down. With a debug kernel, this
> > time is a little tight.
> >
> > So, either we have some background arp traffic, or let's extend the arp
> > interval to a larger one.
> >
> > What do you think?
> 
> I agree that a 100ms ARP interval could be too short in environments
> like a debug kernel.
> 
> Background ARP traffic doesn't widen that window, though: it is sent
> through the active slave, so once the active slave is set down,
> nothing reaches the backups. It could only help with the ping reply
> race if it were very frequent; Bjoern's patch triggers about one ARP
> request per second and didn't fix the failures in our testing.

For background traffic, I mean let the client send the arp requests,
not from active slave of server. With arp_validate 0, the backup slaves
will update last_rx and won't be down with idle.

> 
> Increasing the ARP interval alone won't address the ping reply race,
> but it should help in environments like a debug kernel. I'd prefer to
> keep this patch focused on the ping reply race and send the ARP
> interval change as a follow-up, but I can fold it into v2 if you
> prefer.

Doesn't extending the arp_interval (e.g. to 500) resolve your issue?
Are you still able to reproduce the problem?

Thanks
Hangbin

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2026-11-18  8:27 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-12  1:46 [PATCH net] selftests: bonding: fix failover race in bond_options.sh prio test Pin-yen Lin
2026-11-13  5:23 ` Hangbin Liu
2026-11-13  5:59   ` Hangbin Liu
2026-11-17  1:30     ` Pin-yen Lin
2026-11-17  7:26       ` Hangbin Liu
2026-11-18  2:32         ` Pin-yen Lin
2026-11-18  8:27           ` Hangbin Liu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox