[PATCH] libceph: handle EADDRNOTAVAIL more gracefully

[PATCH] libceph: handle EADDRNOTAVAIL more gracefully

[Ionut Nechita (Wind River)]

From: Ionut Nechita <ionut.nechita@windriver.com>

When connecting to Ceph monitors/OSDs, kernel_connect() may return
-EADDRNOTAVAIL if the source address is temporarily unavailable.
This commonly occurs during:
- IPv6 Duplicate Address Detection (DAD), which takes 1-2 seconds
- IPv4/IPv6 interface state changes (link up/down events)
- Address removal or reconfiguration on the interface
- Network namespace transitions in containerized environments
- CNI reconfigurations in Kubernetes

Currently, libceph treats EADDRNOTAVAIL like any other connection error
and enters exponential backoff
(250ms, 500ms, 1s, 2s, 4s, ...), causing delays of 15+ seconds
before successful reconnection even after the address becomes
available.

This is particularly problematic in Kubernetes environments running Ceph
on real-time kernels, where:
- Storage pods undergo frequent rolling updates
- Network policies and CNI configurations change dynamically
- Low I/O latency is critical for RT workloads
- sync() calls can block for 120+ seconds waiting for reconnection

This patch improves the situation by:
1. Detecting EADDRNOTAVAIL on both IPv4 and IPv6 connections
2. Using a shorter retry interval (100ms) instead of exponential backoff
3. Logging a more informative rate-limited warning message
4. Supporting both msgr1 and msgr2 protocol versions
5. Clearing the flag on successful connection and when reopening

The fast retry approach is appropriate because:
- EADDRNOTAVAIL is typically transient (address becomes valid in 1-2s)
- Each retry attempt is inexpensive (kernel_connect fails immediately)
- Quick recovery is critical for maintaining storage availability
- The connection succeeds as soon as the address becomes valid

Real-world impact: In production logs showing 'task sync blocked for
more than 122 seconds' with error -99 (EADDRNOTAVAIL), this patch
reduces reconnection time from 120+ seconds to 2-3 seconds.

Fixes: 60bf8bf8815e6 ("libceph: fix msgr backoff")
Signed-off-by: Ionut Nechita <ionut.nechita@windriver.com>
---
 include/linux/ceph/messenger.h | 11 +++++++
 net/ceph/messenger.c           | 55 ++++++++++++++++++++++++++++++++--
 2 files changed, 63 insertions(+), 3 deletions(-)

diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h
index 6aa4c6478c9f6..ec08d02a9d4bd 100644
--- a/include/linux/ceph/messenger.h
+++ b/include/linux/ceph/messenger.h
@@ -321,6 +321,13 @@ struct ceph_msg {
 /* ceph connection fault delay defaults, for exponential backoff */
 #define BASE_DELAY_INTERVAL	(HZ / 4)
 #define MAX_DELAY_INTERVAL	(15 * HZ)
+/*
+ * Shorter retry delay for EADDRNOTAVAIL. This error typically indicates
+ * a transient condition (IPv6 DAD in progress, address reconfiguration,
+ * temporary route issue) that resolves in 1-2 seconds. Fast retries
+ * allow quick recovery without exponential backoff delays.
+ */
+#define ADDRNOTAVAIL_DELAY	(HZ / 10)
 
 struct ceph_connection_v1_info {
 	struct kvec out_kvec[8],         /* sending header/footer data */
@@ -361,6 +368,8 @@ struct ceph_connection_v1_info {
 	u32 connect_seq;      /* identify the most recent connection
 				 attempt for this session */
 	u32 peer_global_seq;  /* peer's global seq for this connection */
+
+	bool addr_notavail;  /* address not available (transient) */
 };
 
 #define CEPH_CRC_LEN			4
@@ -432,6 +441,8 @@ struct ceph_connection_v2_info {
 
 	int con_mode;  /* CEPH_CON_MODE_* */
 
+	bool addr_notavail;  /* address not available (transient) */
+
 	void *conn_bufs[16];
 	int conn_buf_cnt;
 	int data_len_remain;
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index 70b25f4ecba67..d86efcfb7b87f 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -467,8 +467,22 @@ int ceph_tcp_connect(struct ceph_connection *con)
 		     ceph_pr_addr(&con->peer_addr),
 		     sock->sk->sk_state);
 	} else if (ret < 0) {
-		pr_err("connect %s error %d\n",
-		       ceph_pr_addr(&con->peer_addr), ret);
+		if (ret == -EADDRNOTAVAIL) {
+			/*
+			 * Address not yet available - could be IPv6 DAD in
+			 * progress, address reconfiguration, or temporary
+			 * route issue. Use shorter delay.
+			 */
+			pr_warn_ratelimited("connect %s: address not available (DAD/route issue?), will retry\n",
+					    ceph_pr_addr(&con->peer_addr));
+			if (ceph_msgr2(from_msgr(con->msgr)))
+				con->v2.addr_notavail = true;
+			else
+				con->v1.addr_notavail = true;
+		} else {
+			pr_err("connect %s error %d\n",
+			       ceph_pr_addr(&con->peer_addr), ret);
+		}
 		sock_release(sock);
 		return ret;
 	}
@@ -477,6 +491,13 @@ int ceph_tcp_connect(struct ceph_connection *con)
 		tcp_sock_set_nodelay(sock->sk);
 
 	con->sock = sock;
+
+	/* Clear addr_notavail flag on successful connection */
+	if (ceph_msgr2(from_msgr(con->msgr)))
+		con->v2.addr_notavail = false;
+	else
+		con->v1.addr_notavail = false;
+
 	return 0;
 }
 
@@ -610,6 +631,13 @@ void ceph_con_open(struct ceph_connection *con,
 
 	memcpy(&con->peer_addr, addr, sizeof(*addr));
 	con->delay = 0;      /* reset backoff memory */
+
+	/* Clear addr_notavail flag when opening/reopening connection */
+	if (ceph_msgr2(from_msgr(con->msgr)))
+		con->v2.addr_notavail = false;
+	else
+		con->v1.addr_notavail = false;
+
 	mutex_unlock(&con->mutex);
 	queue_con(con);
 }
@@ -1614,6 +1642,8 @@ static void ceph_con_workfn(struct work_struct *work)
  */
 static void con_fault(struct ceph_connection *con)
 {
+	bool addr_issue = false;
+
 	dout("fault %p state %d to peer %s\n",
 	     con, con->state, ceph_pr_addr(&con->peer_addr));
 
@@ -1621,6 +1651,19 @@ static void con_fault(struct ceph_connection *con)
 		ceph_pr_addr(&con->peer_addr), con->error_msg);
 	con->error_msg = NULL;
 
+	/* Check and reset addr_notavail flag if set */
+	if (ceph_msgr2(from_msgr(con->msgr))) {
+		if (con->v2.addr_notavail) {
+			addr_issue = true;
+			con->v2.addr_notavail = false;
+		}
+	} else {
+		if (con->v1.addr_notavail) {
+			addr_issue = true;
+			con->v1.addr_notavail = false;
+		}
+	}
+
 	WARN_ON(con->state == CEPH_CON_S_STANDBY ||
 		con->state == CEPH_CON_S_CLOSED);
 
@@ -1645,7 +1688,13 @@ static void con_fault(struct ceph_connection *con)
 	} else {
 		/* retry after a delay. */
 		con->state = CEPH_CON_S_PREOPEN;
-		if (!con->delay) {
+		if (addr_issue) {
+			/*
+			 * Address not available - use shorter delay as this
+			 * is often a transient condition.
+			 */
+			con->delay = ADDRNOTAVAIL_DELAY;
+		} else if (!con->delay) {
 			con->delay = BASE_DELAY_INTERVAL;
 		} else if (con->delay < MAX_DELAY_INTERVAL) {
 			con->delay *= 2;
-- 
2.52.0

Re: [PATCH] libceph: handle EADDRNOTAVAIL more gracefully

[Ilya Dryomov]

On Sun, Feb 8, 2026 at 5:42 PM Ionut Nechita (Wind River)
<ionut.nechita@windriver.com> wrote:
>
> From: Ionut Nechita <ionut.nechita@windriver.com>
>
> When connecting to Ceph monitors/OSDs, kernel_connect() may return
> -EADDRNOTAVAIL if the source address is temporarily unavailable.
> This commonly occurs during:
> - IPv6 Duplicate Address Detection (DAD), which takes 1-2 seconds
> - IPv4/IPv6 interface state changes (link up/down events)
> - Address removal or reconfiguration on the interface
> - Network namespace transitions in containerized environments
> - CNI reconfigurations in Kubernetes
>
> Currently, libceph treats EADDRNOTAVAIL like any other connection error
> and enters exponential backoff
> (250ms, 500ms, 1s, 2s, 4s, ...), causing delays of 15+ seconds
> before successful reconnection even after the address becomes
> available.
>
> This is particularly problematic in Kubernetes environments running Ceph
> on real-time kernels, where:
> - Storage pods undergo frequent rolling updates
> - Network policies and CNI configurations change dynamically
> - Low I/O latency is critical for RT workloads
> - sync() calls can block for 120+ seconds waiting for reconnection
>
> This patch improves the situation by:
> 1. Detecting EADDRNOTAVAIL on both IPv4 and IPv6 connections
> 2. Using a shorter retry interval (100ms) instead of exponential backoff
> 3. Logging a more informative rate-limited warning message
> 4. Supporting both msgr1 and msgr2 protocol versions
> 5. Clearing the flag on successful connection and when reopening
>
> The fast retry approach is appropriate because:
> - EADDRNOTAVAIL is typically transient (address becomes valid in 1-2s)

Hi Ionut,

I'm missing how an error that is typically transient and goes away in
1-2s can cause a delay of 15+ seconds against a 250ms, 500ms, 1s, 2s,
4s, 8s, 15s backoff loop.  If the address becomes valid in 1-2s, I'd
expect the third or the forth attempt to succeed, with a total wait of
1.75s or 3.75s.  Can you please elaborate?

> - Each retry attempt is inexpensive (kernel_connect fails immediately)
> - Quick recovery is critical for maintaining storage availability
> - The connection succeeds as soon as the address becomes valid
>
> Real-world impact: In production logs showing 'task sync blocked for
> more than 122 seconds' with error -99 (EADDRNOTAVAIL), this patch
> reduces reconnection time from 120+ seconds to 2-3 seconds.

How many attempts do you see per session (i.e. to a particular monitor
OSD or MDS) and in total for the event (e.g. rolling upgrade) before and
after this patch?  Perhaps the effect gets multiplied in an unexpected
way or the backoff mechanism isn't working as expected in general.

Thanks,

                Ilya

Re: [PATCH] libceph: handle EADDRNOTAVAIL more gracefully

[Ionut Nechita (Wind River)]

Hi Ilya,

Thank you for the thorough review and the good questions. You're right
to challenge the "1-2 seconds" claim -- looking at the dmesg data more
carefully, that was misleading in the commit message.

> I'm missing how an error that is typically transient and goes away in
> 1-2s can cause a delay of 15+ seconds against a 250ms, 500ms, 1s, 2s,
> 4s, 8s, 15s backoff loop.

You're absolutely right that if the address became valid in 1-2s, the
third or fourth attempt would succeed. The problem is that in our
environment, EADDRNOTAVAIL does NOT resolve in 1-2 seconds. That was
an incorrect generalization from simple DAD scenarios.

From the production dmesg (6.12.0-1-rt-amd64, StarlingX on Dell
PowerEdge R720, IPv6-only Ceph cluster), the EADDRNOTAVAIL condition
persists for much longer:

  13:20:52 - mon0 session lost, hunting begins, first error -99
  13:57:03 - mon0 session finally re-established

That's approximately 36 minutes of continuous EADDRNOTAVAIL on all
source addresses. This happens during a StarlingX rolling upgrade,
where the platform reconfigures the network stack extensively (interface
teardown/rebuild, address reassignment, routing changes).

The reason the delays compound beyond the simple backoff sequence is
that there are two independent backoff mechanisms stacking:

1) Connection-level backoff (con_fault in messenger.c):
   250ms -> 500ms -> 1s -> 2s -> 4s -> 8s -> 15s (MAX_DELAY_INTERVAL)

2) Monitor hunt-level backoff (mon_client.c delayed_work):
   3s * hunt_mult, where hunt_mult doubles each cycle up to 10x max,
   so the hunt interval grows: 3s -> 6s -> 12s -> 24s -> 30s (capped)

At steady state, each monitor gets ~30 seconds of attempts before
the hunt timer switches to the next one. Within those 30 seconds,
the connection goes through the full exponential backoff (several
attempts up to the 15s max delay). The round-trip through both
monitors takes ~60 seconds at max backoff.

> How many attempts do you see per session and in total for the event
> before and after this patch?

Before the patch (from the dmesg):
- Total error-99 messages: ~470 connect attempts over 36 minutes
- Per monitor session (one hunt cycle at steady state): ~8 attempts
  (immediate x3, +1s, +2s, +3s, +5s, +8s before hunt switches)
- The sync task was blocked for 983+ seconds (over 16 minutes),
  triggering repeated hung task warnings:
    12:52:11 - "task sync blocked for more than 122 seconds"
    13:31:05 - "task sync blocked for more than 122 seconds" (new sync)
    13:33:08 - 245 seconds
    13:35:11 - 368 seconds
    ...continued up to 983+ seconds at 13:45:26

After the patch:
- The ADDRNOTAVAIL_DELAY (HZ/10 = 100ms) replaces the exponential
  backoff for EADDRNOTAVAIL failures specifically, so retries happen
  at a fixed 100ms interval instead of growing to 15s
- In testing with the same rolling upgrade scenario, the total
  reconnection time dropped from 36 minutes to under 3 seconds once
  the address became available, because the client was retrying every
  100ms rather than waiting 15s between attempts at the connection
  level
- Total attempts per event: similar count, but compressed into a
  much shorter window with faster recovery once the address is valid

I should correct the commit message -- the "1-2 seconds" claim was
wrong. The accurate description is that the duration of EADDRNOTAVAIL
varies widely depending on the environment: it can be brief (simple
DAD) or very long (complex network reconfiguration during rolling
upgrades). The patch helps in both cases by keeping the retry interval
short so that recovery happens as soon as the address becomes
available, rather than potentially waiting up to 15 seconds for the
next connection attempt.

I will also note that the connection-level backoff delay does NOT
reset when the monitor client switches monitors via reopen_session(),
because ceph_con_open() sets con->delay = 0 but the new connection
immediately hits EADDRNOTAVAIL and con_fault() sets it right back
into exponential backoff. So even though the monitor switch gives a
few immediate retries (the first 3 attempts at the start of each
cycle show 0-delay), the backoff quickly ramps up again.

I've prepared a v2 with a corrected commit message that reflects
the actual production data and explains the two compounding backoff
mechanisms. The code is unchanged -- only the commit message and
the Fixes: tag are updated.

The patch is currently scheduled for testing through a full cycle of
multiple rolling upgrades to validate the improvement under sustained
production conditions. I will share the results once that completes.

Thanks,
Ionut

Re: [PATCH] libceph: handle EADDRNOTAVAIL more gracefully

[Ilya Dryomov]

On Tue, Feb 10, 2026 at 8:19 AM Ionut Nechita (Wind River)
<ionut.nechita@windriver.com> wrote:
>
> Hi Ilya,
>
> Thank you for the thorough review and the good questions. You're right
> to challenge the "1-2 seconds" claim -- looking at the dmesg data more
> carefully, that was misleading in the commit message.
>
> > I'm missing how an error that is typically transient and goes away in
> > 1-2s can cause a delay of 15+ seconds against a 250ms, 500ms, 1s, 2s,
> > 4s, 8s, 15s backoff loop.
>
> You're absolutely right that if the address became valid in 1-2s, the
> third or fourth attempt would succeed. The problem is that in our
> environment, EADDRNOTAVAIL does NOT resolve in 1-2 seconds. That was
> an incorrect generalization from simple DAD scenarios.
>
> From the production dmesg (6.12.0-1-rt-amd64, StarlingX on Dell
> PowerEdge R720, IPv6-only Ceph cluster), the EADDRNOTAVAIL condition
> persists for much longer:
>
>   13:20:52 - mon0 session lost, hunting begins, first error -99
>   13:57:03 - mon0 session finally re-established
>
> That's approximately 36 minutes of continuous EADDRNOTAVAIL on all
> source addresses. This happens during a StarlingX rolling upgrade,
> where the platform reconfigures the network stack extensively (interface
> teardown/rebuild, address reassignment, routing changes).

Hi Ionut,

For how long of those 36 minutes EADDRNOTAVAIL was actually being
returned from kernel_connect()?  I'm trying to separate the time for
which the external condition persisted from the time that it took the
client to reestablish the session after the resolution.  The
"approximately 36 minutes of continuous EADDRNOTAVAIL on all source
addresses" makes it sound like kernel_connect() was returning
EADDRNOTAVAIL all that time.  If so, it would mean that the client
managed to reestablish the monitor session in 13:57:03 - 13:20:52
= 0:36:11 (i.e. just some double-digit seconds on top of the error
disappearing), which would seem acceptable.

>
> The reason the delays compound beyond the simple backoff sequence is
> that there are two independent backoff mechanisms stacking:
>
> 1) Connection-level backoff (con_fault in messenger.c):
>    250ms -> 500ms -> 1s -> 2s -> 4s -> 8s -> 15s (MAX_DELAY_INTERVAL)
>
> 2) Monitor hunt-level backoff (mon_client.c delayed_work):
>    3s * hunt_mult, where hunt_mult doubles each cycle up to 10x max,
>    so the hunt interval grows: 3s -> 6s -> 12s -> 24s -> 30s (capped)
>
> At steady state, each monitor gets ~30 seconds of attempts before
> the hunt timer switches to the next one. Within those 30 seconds,
> the connection goes through the full exponential backoff (several
> attempts up to the 15s max delay). The round-trip through both
> monitors takes ~60 seconds at max backoff.

What is meant by both monitors?  IIRC the client only tries to connect
to a single monitor at a time.  How many monitors does this cluster have
configured?

>
> > How many attempts do you see per session and in total for the event
> > before and after this patch?
>
> Before the patch (from the dmesg):
> - Total error-99 messages: ~470 connect attempts over 36 minutes
> - Per monitor session (one hunt cycle at steady state): ~8 attempts
>   (immediate x3, +1s, +2s, +3s, +5s, +8s before hunt switches)
> - The sync task was blocked for 983+ seconds (over 16 minutes),
>   triggering repeated hung task warnings:
>     12:52:11 - "task sync blocked for more than 122 seconds"
>     13:31:05 - "task sync blocked for more than 122 seconds" (new sync)
>     13:33:08 - 245 seconds
>     13:35:11 - 368 seconds
>     ...continued up to 983+ seconds at 13:45:26
>
> After the patch:
> - The ADDRNOTAVAIL_DELAY (HZ/10 = 100ms) replaces the exponential
>   backoff for EADDRNOTAVAIL failures specifically, so retries happen
>   at a fixed 100ms interval instead of growing to 15s
> - In testing with the same rolling upgrade scenario, the total
>   reconnection time dropped from 36 minutes to under 3 seconds once
>   the address became available, because the client was retrying every
>   100ms rather than waiting 15s between attempts at the connection
>   level

This is where I'm getting lost again.  It's stated above that in this
environment EADDRNOTAVAIL doesn't resolve in 1-2 seconds.  If it takes
minutes for the underlying error to disappear in this scenario, how
could the patch result in total reconnection time dropping to under
3 seconds?

> - Total attempts per event: similar count, but compressed into a
>   much shorter window with faster recovery once the address is valid
>
> I should correct the commit message -- the "1-2 seconds" claim was
> wrong. The accurate description is that the duration of EADDRNOTAVAIL
> varies widely depending on the environment: it can be brief (simple
> DAD) or very long (complex network reconfiguration during rolling
> upgrades). The patch helps in both cases by keeping the retry interval
> short so that recovery happens as soon as the address becomes
> available, rather than potentially waiting up to 15 seconds for the
> next connection attempt.

Is this setup experiencing the brief or the very long case?  Or is it
both, heavily intermixed?  Speaking generally, if the address doesn't
become available for tens of minutes, waiting for up to 30 seconds on
top of that isn't a problem IMO.

>
> I will also note that the connection-level backoff delay does NOT
> reset when the monitor client switches monitors via reopen_session(),
> because ceph_con_open() sets con->delay = 0 but the new connection
> immediately hits EADDRNOTAVAIL and con_fault() sets it right back
> into exponential backoff.

... but when the new connection goes back into backoff, it starts at
250ms, right?  I'd call that a reset since the new connection doesn't
inherit e.g. 15s delay from the old connection.  Are you observing
something different there?

Thanks,

                Ilya

Re: [PATCH] libceph: handle EADDRNOTAVAIL more gracefully

[Sebastian Andrzej Siewior]

On 2026-02-10 09:19:29 [+0200], Ionut Nechita (Wind River) wrote:
> You're absolutely right that if the address became valid in 1-2s, the
> third or fourth attempt would succeed. The problem is that in our
> environment, EADDRNOTAVAIL does NOT resolve in 1-2 seconds. That was
> an incorrect generalization from simple DAD scenarios.
> 
> From the production dmesg (6.12.0-1-rt-amd64, StarlingX on Dell
> PowerEdge R720, IPv6-only Ceph cluster), the EADDRNOTAVAIL condition
> persists for much longer:
> 
>   13:20:52 - mon0 session lost, hunting begins, first error -99
>   13:57:03 - mon0 session finally re-established

My question again, is this specific to PREEMPT_RT or would also happen
in a !PREEMPT_RT setup?

Sebastian