The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: Hangbin Liu <hangbin.liu@linux.dev>
To: Pin-yen Lin <treapking@google.com>
Cc: Jay Vosburgh <jv@jvosburgh.net>,
	Andrew Lunn <andrew+netdev@lunn.ch>,
	"David S. Miller" <davem@davemloft.net>,
	Eric Dumazet <edumazet@kernel.org>,
	Bjoern Doebel <doebel@amazon.de>,
	Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
	Shuah Khan <shuah@kernel.org>,
	netdev@vger.kernel.org, Matthieu Baerts <matttbe@kernel.org>,
	linux-kernel@vger.kernel.org,
	Jonathan Toppins <jon.toppins+linux@gmail.com>,
	linux-kselftest@vger.kernel.org,
	Hangbin Liu <liuhangbin@gmail.com>
Subject: Re: [PATCH net] selftests: bonding: fix failover race in bond_options.sh prio test
Date: Tue, 17 Nov 2026 15:26:37 +0800	[thread overview]
Message-ID: <avwBrbJfp4ET9ifT@fedora> (raw)
In-Reply-To: <CAHwYsio=qtCY10EPnh+qvWpdgtQUG5mw1b3WKMt9_Kytcc7h9A@mail.gmail.com>

Hi Pin-yen,
On Mon, Nov 16, 2026 at 05:30:28PM -0800, Pin-yen Lin wrote:
> Hi Hangbin,
> 
> On Thu, Nov 12, 2026 at 10:00 PM Hangbin Liu <hangbin.liu@linux.dev> wrote:
> >
> > On Fri, Nov 13, 2026 at 01:23:40PM +0800, Hangbin Liu wrote:
> > > Hi Pin-yen,
> > > On Wed, Nov 11, 2026 at 05:46:18PM -0800, Pin-yen Lin wrote:
> > > > The prio test intermittently fails in the arp_ip_target and
> > > > ns_ip6_target variants:
> > > >
> > > >   TEST: prio (active-backup ns_ip6_target primary_reselect 1)   [FAIL]
> > > >     Current active slave is eth0 but not eth2
> > > >
> > > > The ARP/NS monitor marks a slave down when it has received nothing for
> > > > too long. With arp_interval 100 and the default arp_missed_max, that is
> > > > about 250ms for the active slave (eth1) and 350ms for a backup slave
> > > > (eth0, eth2). Every 100ms, eth1 sends a probe: eth1 receives the reply,
> > > > and eth0/eth2 receive a copy forwarded by the gateway bridge. Once eth1
> > > > is down, its probes reach no one and no slave receives anything, so
> > > > eth1 normally times out first and the new active slave is picked by
> > > > prio:
> > > >
> > > >   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
> > > >   t=10   test sets eth1 down
> > > >   t=300  monitor: eth1 idle 300ms -> down
> > > >                   eth0/eth2 idle 300ms -> up, eth2 picked by prio
> > > >
> > > > But the test takes eth1 down right after bond_check_connection(), whose
> > > > ping replies only arrive on eth1:
> > > >
> > > >   t=0    probe: eth1 gets the reply, eth0/eth2 get a copy
> > > >   t=70   ping reply on eth1 only
> > > >   t=85   test sets eth1 down
> > > >   t=300  monitor: eth1 idle 230ms -> still up
> > > >   t=400  monitor: eth1 idle 330ms -> down
> > > >                   eth2 idle 400ms -> down too
> > > >
> > > > eth2 can then no longer be picked. eth0 becomes the active slave if it
> > > > happened to receive some other packet (e.g. an MLD report), otherwise
> > > > whichever slave comes back first does. With primary_reselect 1 or 2,
> > > > the bond keeps eth0 even after eth2 comes back.
> > > >
> > > > Before taking the active slave down, wait two monitor intervals
> > > > (200ms), so that at least one more probe is sent after the last ping
> > > > reply and all slaves last received that probe. The miimon variants do
> > > > not need the wait, but the extra delay is negligible.
> > >
> > > I'm afraid the waiting time is too strict, especially with a debug kernel
> > > since delays can happen.
> > >
> > > Bjoern is also fixing[1] this issue by pinging a non-existent neighbor to
> > > keep traffic flowing. Would you like to test whether that works for you?
> > >
> > > [1] https://lore.kernel.org/netdev/20261104142018.552135-1-doebel@amazon.de/
> >
> > BTW, using ping may not able to triger the arp request in time. You can try
> > arping instead.
> 
> Thanks for the pointer, but I believe Bjoern's patch addresses a
> different problem.

I think they are the same issue, as you comment also said

The ARP/NS monitor marks a slave down when it has received nothing for
too long.

> 
> Bjoern's patch keeps the traffic flowing so that idle slaves are kept
> alive between the test steps. This patch fixes the case where the ping
> replies from bond_check_connection() only arrive on the active slave
> right before it is set down. The active slave's failure is then
> detected one monitor interval late, when the backups have also timed
> out.

The reason is we set arp interval to 100ms, and slowwait also check every
100ms. If the backup slave down at the same check point, e.g.

  t=300  monitor: eth1 idle 230ms -> still up
  t=400  monitor: eth1 idle 330ms -> down
                  eth2 idle 400ms -> down too

The test failed.

> 
> The ARP requests triggered by Bjoern's ping are sent only about once
> per second, which isn't frequent enough for this failure. I tested
> Bjoern's patch and confirmed that it didn't fix the issue in our
> testing environment. We also can't simply use arping instead because
> iputils arping doesn't support sub-second intervals.
> 
> If you're worried that the 200ms sleep is too short, would increasing
> it to 0.5s or 1s work for you?

It's not about the sleep time before setting active slave down. I'm afraid
about the slowwait checking time. Even we sleep 1s, with arp interval 100,
we still only have 100ms checking window. Before the window, the active slave
is up, after the window, the backup slave also down. With a debug kernel, this
time is a little tight.

So, either we have some background arp traffic, or let's extend the arp
interval to a larger one.

What do you think?

Thanks
Hangbin

      reply	other threads:[~2026-11-17  7:26 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12  1:46 [PATCH net] selftests: bonding: fix failover race in bond_options.sh prio test Pin-yen Lin
2026-11-13  5:23 ` Hangbin Liu
2026-11-13  5:59   ` Hangbin Liu
2026-11-17  1:30     ` Pin-yen Lin
2026-11-17  7:26       ` Hangbin Liu [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=avwBrbJfp4ET9ifT@fedora \
    --to=hangbin.liu@linux.dev \
    --cc=andrew+netdev@lunn.ch \
    --cc=davem@davemloft.net \
    --cc=doebel@amazon.de \
    --cc=edumazet@kernel.org \
    --cc=jon.toppins+linux@gmail.com \
    --cc=jv@jvosburgh.net \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=liuhangbin@gmail.com \
    --cc=matttbe@kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=shuah@kernel.org \
    --cc=treapking@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox