All of lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Alexander Aring" <aahringo@redhat.com>, teigland@redhat.com
Cc: gfs2@lists.linux.dev
Subject: Re: [PATCHv2 RESEND v7.4-rc3 1/5] dlm: fix socket hang on send buffer limitation with socket error
Date: Fri, 20 Nov 2026 20:40:25 +0000	[thread overview]
Message-ID: <sashiko-outbox-226084@kernel.org> (raw)
In-Reply-To: <20261120200040.257152-2-aahringo@redhat.com>

Thank you for your contribution! Sashiko AI review found 4 potential
issues to consider:
- [High] dlm: missing CF_APP_LIMITED clear in lowcomms_error_report()
  hangs send queue
- [High] dlm: unconditional swork queue in lowcomms_error_report()
  breaks invariants
- [Medium] dlm: race in send_to_sock() sets CF_APP_LIMITED after socket
  error
- [Low] dlm: missing Fixes trailer in commit message
--

commit 796bee147607d968d752eb2e6e9611c521fe108f
Author: Alexander Aring <aahringo@redhat.com>

> dlm: fix socket hang on send buffer limitation with socket error
>
> As Sashiko AI bot mentioned [0], a socket connection can get stuck when
> the send buffer limitation is reached and a socket error simultaneously
> occurs.
>
> This issue can be reproduced with the following setup:
> 1. Set sk_sndbuf to SOCK_MIN_SNDBUF.
> 2. Slow down DLM connections using netem.
> 3. Use tcpkill to randomly send TCP resets to DLM connections.
>
> I instrumented debug printouts to confirm that the sk_write_space()
> notifier callbacks were being executed, using step 3 to trigger random
> socket errors.
>
> With these changes, I can no longer reproduce the hang.
>
> Changes included:
> - Removed sk_write_pending counting, as this should not be modified
>   at the socket application layer (or is at least unnecessary).
> - Moved clearing the CF_SEND_PENDING bit—which allows re-queuing
>   swork (send worker for sendmsg())—to the sk_write_space() callback,
>   since this callback notifies us that the underlying socket is no longer
>   constrained by its send buffer.
> - Handled the race condition between sendmsg() and evaluating
>   SOCK_NOSPACE after sendmsg(), where sk_write_space() could be called
>   in between, using CF_APP_LIMITED:
>   - In sk_write_space(), queue swork again if CF_APP_LIMITED is set.
>   - If CF_APP_LIMITED is not set, do nothing as send_to_sock() will handle
>     it, confirming the race occurred.
> - Introduced new handling in lowcomms_error_report() when a socket error
>   occurs during send buffer limitation. If CF_APP_LIMITED is set, swork
>   will be re-queued, which will fail and trigger a reconnect.
> - Added various comments explaining the interaction with CF_APP_LIMITED.
>
> [0] https://lore.kernel.org/netdev/179090395863.434549.3668493667259120759@kernel.org/
>
> Signed-off-by: Alexander Aring <aahringo@redhat.com>

[Severity: Low]
Should this bug fix include a Fixes: tag for the commit that gated
lowcomms_queue_swork() on CF_APP_LIMITED?

  Fixes: dbb751ffab0b ("fs: dlm: parallelize lowcomms socket handling")

> diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
> index 3929b164dc0d4..6c972f1b412ca 100644
> --- a/fs/dlm/lowcomms.c
> +++ b/fs/dlm/lowcomms.c

[ ... ]

> @@ -623,6 +631,20 @@ static void lowcomms_error_report(struct sock *sk)
>  		break;
>  	}
>  
> +	/* if waiting on sk_write_space() and an sk_err occurs, the callback
> +	 * won't fire. Clear CF_SEND_PENDING so resend tasks can re-queue
> +	 * swork, triggering a sendmsg() failure to initiate reconnection.
> +	 */
> +	spin_lock_bh(&con->writequeue_lock);
> +	clear_bit(CF_SEND_PENDING, &con->flags);
> +	/* dlm_midcomms_unack_msg_resend() does not always
> +	 * trigger lowcomms_queue_swork() as it tries to
> +	 * avoid to put pending messages into the lowcomms
> +	 * sending buffer. Force it here again.
> +	 */
> +	lowcomms_queue_swork(con);
> +	spin_unlock_bh(&con->writequeue_lock);
> +
>  	dlm_midcomms_unack_msg_resend(con->nodeid);
>  
>  	listen_sock.sk_error_report(sk);

[Severity: High]
Does lowcomms_error_report() also need to clear CF_APP_LIMITED before
calling lowcomms_queue_swork(con)?

lowcomms_queue_swork() refuses to schedule con->swork while CF_APP_LIMITED
is set:

fs/dlm/lowcomms.c:lowcomms_queue_swork() {
	if (!test_bit(CF_IO_STOP, &con->flags) &&
	    !test_bit(CF_APP_LIMITED, &con->flags) &&
	    !test_and_set_bit(CF_SEND_PENDING, &con->flags))
		queue_work(io_workqueue, &con->swork);
}

When a socket error arrives while CF_APP_LIMITED is set, sk_write_space()
will not fire, so leaving CF_APP_LIMITED set here causes every
lowcomms_queue_swork(con) call to be a no-op and leaves the connection
permanently hung.

[Severity: High]
Should lowcomms_error_report() only clear CF_SEND_PENDING and queue
con->swork when test_and_clear_bit(CF_APP_LIMITED, &con->flags) succeeds,
matching lowcomms_write_space()?

Accepting a crossover connection in accept_from_sock() attaches the socket
to othercon (which has CF_IS_OTHERCON set) via add_sock(), installing
lowcomms_error_report() as sk_error_report.  Because othercon is
receive-only, unconditionally queuing con->swork on a socket error
schedules process_send_sockets() on othercon, triggering
WARN_ON_ONCE(test_bit(CF_IS_OTHERCON, &con->flags)) and entering an
infinite reconnect loop calling dlm_connect(othercon) once othercon->sock
is closed.

On a primary connection where CF_APP_LIMITED is not set, unconditionally
clearing CF_SEND_PENDING also breaks swork serialization on the unbound
io_workqueue by allowing a second con->swork to be queued while
process_send_sockets() is already running.

> @@ -1391,22 +1413,19 @@ static int send_to_sock(struct connection *con)
>  		spin_lock_bh(&con->writequeue_lock);
>  		if (test_bit(SOCK_NOSPACE, &con->sock->flags) &&
>  		    !test_and_set_bit(CF_APP_LIMITED, &con->flags)) {
> -			/* Notify TCP that we're limited by the
> -			 * application window size.
> -			 */
> -			sk_set_nospace(con->sock->sk);
> -			con->sock->sk->sk_write_pending++;
> -
> -			clear_bit(CF_SEND_PENDING, &con->flags);
>  			spin_unlock_bh(&con->writequeue_lock);
>  			release_sock(con->sock->sk);
>  
> -			/* wait for write_space() event */
> +			/* wait for sk_write_space() event */
>  			return DLM_IO_END;
>  		}
>  		spin_unlock_bh(&con->writequeue_lock);
>  		release_sock(con->sock->sk);
>  
> +		/* the sk_write_space() came in between sock_sendmsg()
> +		 * and check on SOCK_NOSPACE and the socket became
> +		 * writeable again so just resched swork.
> +		 */
>  		return DLM_IO_RESCHED;
>  	} else if (ret < 0) {
>  		return ret;

[Severity: Medium]
Can send_to_sock() still set CF_APP_LIMITED and hang after a socket error
if the error arrives before lock_sock()?

kworker                                <softirq>
send_to_sock()
  sock_sendmsg()
    ...
      sk_set_nospace()
      return -EAGAIN
                                       tcp_v4_rcv() (TCP RST)
                                         tcp_reset()
                                           tcp_done_with_error()
                                             lowcomms_error_report()
  lock_sock(sk)
  spin_lock_bh(&con->writequeue_lock)
  test_bit(SOCK_NOSPACE, ...)
  !test_and_set_bit(CF_APP_LIMITED, ...)
  return DLM_IO_END  <-- waits forever

Unlike lowcomms_write_space(), neither tcp_done_with_error() nor
lowcomms_error_report() clears SOCK_NOSPACE.  When send_to_sock() then
checks SOCK_NOSPACE without checking sk->sk_err, it sets CF_APP_LIMITED
after lowcomms_error_report() has already finished and returns DLM_IO_END
on a dead socket that will never fire sk_write_space() or
sk_error_report() again.  Checking !con->sock->sk->sk_err under lock_sock()
(or clearing SOCK_NOSPACE in lowcomms_error_report()) would avoid entering
CF_APP_LIMITED on an errored socket.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261120200040.257152-1-aahringo@redhat.com?part=1

  reply	other threads:[~2026-11-20 20:40 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 20:00 [PATCHv2 RESEND v7.4-rc3 0/5] dlm: various fixes Alexander Aring
2026-11-20 20:00 ` [PATCHv2 RESEND v7.4-rc3 1/5] dlm: fix socket hang on send buffer limitation with socket error Alexander Aring
2026-11-20 20:40   ` sashiko-bot [this message]
2026-11-20 20:00 ` [PATCHv2 RESEND v7.4-rc3 2/5] dlm: reject wrapped messages with oversized inner lengths Alexander Aring
2026-11-20 20:00 ` [PATCHv2 RESEND v7.4-rc3 3/5] dlm: don't publish a lkb in ls_lkbxa before it has an rsb Alexander Aring
2026-11-20 20:12   ` sashiko-bot
2026-11-20 20:00 ` [PATCHv2 RESEND v7.4-rc3 4/5] dlm: don't set RSB_MASTER_UNCERTAIN on rsb with no master Alexander Aring
2026-11-20 20:00 ` [PATCHv2 RESEND v7.4-rc3 5/5] dlm: fix memory leak on active leaving member Alexander Aring

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-226084@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=aahringo@redhat.com \
    --cc=gfs2@lists.linux.dev \
    --cc=sashiko-reviews@lists.linux.dev \
    --cc=teigland@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.