From: sashiko-bot@kernel.org
To: "Alexander Aring" <aahringo@redhat.com>, teigland@redhat.com
Cc: gfs2@lists.linux.dev
Subject: Re: [PATCHv2 RESEND v7.4-rc3 1/5] dlm: fix socket hang on send buffer limitation with socket error
Date: Fri, 20 Nov 2026 20:40:25 +0000 [thread overview]
Message-ID: <sashiko-outbox-226084@kernel.org> (raw)
In-Reply-To: <20261120200040.257152-2-aahringo@redhat.com>
Thank you for your contribution! Sashiko AI review found 4 potential
issues to consider:
- [High] dlm: missing CF_APP_LIMITED clear in lowcomms_error_report()
hangs send queue
- [High] dlm: unconditional swork queue in lowcomms_error_report()
breaks invariants
- [Medium] dlm: race in send_to_sock() sets CF_APP_LIMITED after socket
error
- [Low] dlm: missing Fixes trailer in commit message
--
commit 796bee147607d968d752eb2e6e9611c521fe108f
Author: Alexander Aring <aahringo@redhat.com>
> dlm: fix socket hang on send buffer limitation with socket error
>
> As Sashiko AI bot mentioned [0], a socket connection can get stuck when
> the send buffer limitation is reached and a socket error simultaneously
> occurs.
>
> This issue can be reproduced with the following setup:
> 1. Set sk_sndbuf to SOCK_MIN_SNDBUF.
> 2. Slow down DLM connections using netem.
> 3. Use tcpkill to randomly send TCP resets to DLM connections.
>
> I instrumented debug printouts to confirm that the sk_write_space()
> notifier callbacks were being executed, using step 3 to trigger random
> socket errors.
>
> With these changes, I can no longer reproduce the hang.
>
> Changes included:
> - Removed sk_write_pending counting, as this should not be modified
> at the socket application layer (or is at least unnecessary).
> - Moved clearing the CF_SEND_PENDING bit—which allows re-queuing
> swork (send worker for sendmsg())—to the sk_write_space() callback,
> since this callback notifies us that the underlying socket is no longer
> constrained by its send buffer.
> - Handled the race condition between sendmsg() and evaluating
> SOCK_NOSPACE after sendmsg(), where sk_write_space() could be called
> in between, using CF_APP_LIMITED:
> - In sk_write_space(), queue swork again if CF_APP_LIMITED is set.
> - If CF_APP_LIMITED is not set, do nothing as send_to_sock() will handle
> it, confirming the race occurred.
> - Introduced new handling in lowcomms_error_report() when a socket error
> occurs during send buffer limitation. If CF_APP_LIMITED is set, swork
> will be re-queued, which will fail and trigger a reconnect.
> - Added various comments explaining the interaction with CF_APP_LIMITED.
>
> [0] https://lore.kernel.org/netdev/179090395863.434549.3668493667259120759@kernel.org/
>
> Signed-off-by: Alexander Aring <aahringo@redhat.com>
[Severity: Low]
Should this bug fix include a Fixes: tag for the commit that gated
lowcomms_queue_swork() on CF_APP_LIMITED?
Fixes: dbb751ffab0b ("fs: dlm: parallelize lowcomms socket handling")
> diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
> index 3929b164dc0d4..6c972f1b412ca 100644
> --- a/fs/dlm/lowcomms.c
> +++ b/fs/dlm/lowcomms.c
[ ... ]
> @@ -623,6 +631,20 @@ static void lowcomms_error_report(struct sock *sk)
> break;
> }
>
> + /* if waiting on sk_write_space() and an sk_err occurs, the callback
> + * won't fire. Clear CF_SEND_PENDING so resend tasks can re-queue
> + * swork, triggering a sendmsg() failure to initiate reconnection.
> + */
> + spin_lock_bh(&con->writequeue_lock);
> + clear_bit(CF_SEND_PENDING, &con->flags);
> + /* dlm_midcomms_unack_msg_resend() does not always
> + * trigger lowcomms_queue_swork() as it tries to
> + * avoid to put pending messages into the lowcomms
> + * sending buffer. Force it here again.
> + */
> + lowcomms_queue_swork(con);
> + spin_unlock_bh(&con->writequeue_lock);
> +
> dlm_midcomms_unack_msg_resend(con->nodeid);
>
> listen_sock.sk_error_report(sk);
[Severity: High]
Does lowcomms_error_report() also need to clear CF_APP_LIMITED before
calling lowcomms_queue_swork(con)?
lowcomms_queue_swork() refuses to schedule con->swork while CF_APP_LIMITED
is set:
fs/dlm/lowcomms.c:lowcomms_queue_swork() {
if (!test_bit(CF_IO_STOP, &con->flags) &&
!test_bit(CF_APP_LIMITED, &con->flags) &&
!test_and_set_bit(CF_SEND_PENDING, &con->flags))
queue_work(io_workqueue, &con->swork);
}
When a socket error arrives while CF_APP_LIMITED is set, sk_write_space()
will not fire, so leaving CF_APP_LIMITED set here causes every
lowcomms_queue_swork(con) call to be a no-op and leaves the connection
permanently hung.
[Severity: High]
Should lowcomms_error_report() only clear CF_SEND_PENDING and queue
con->swork when test_and_clear_bit(CF_APP_LIMITED, &con->flags) succeeds,
matching lowcomms_write_space()?
Accepting a crossover connection in accept_from_sock() attaches the socket
to othercon (which has CF_IS_OTHERCON set) via add_sock(), installing
lowcomms_error_report() as sk_error_report. Because othercon is
receive-only, unconditionally queuing con->swork on a socket error
schedules process_send_sockets() on othercon, triggering
WARN_ON_ONCE(test_bit(CF_IS_OTHERCON, &con->flags)) and entering an
infinite reconnect loop calling dlm_connect(othercon) once othercon->sock
is closed.
On a primary connection where CF_APP_LIMITED is not set, unconditionally
clearing CF_SEND_PENDING also breaks swork serialization on the unbound
io_workqueue by allowing a second con->swork to be queued while
process_send_sockets() is already running.
> @@ -1391,22 +1413,19 @@ static int send_to_sock(struct connection *con)
> spin_lock_bh(&con->writequeue_lock);
> if (test_bit(SOCK_NOSPACE, &con->sock->flags) &&
> !test_and_set_bit(CF_APP_LIMITED, &con->flags)) {
> - /* Notify TCP that we're limited by the
> - * application window size.
> - */
> - sk_set_nospace(con->sock->sk);
> - con->sock->sk->sk_write_pending++;
> -
> - clear_bit(CF_SEND_PENDING, &con->flags);
> spin_unlock_bh(&con->writequeue_lock);
> release_sock(con->sock->sk);
>
> - /* wait for write_space() event */
> + /* wait for sk_write_space() event */
> return DLM_IO_END;
> }
> spin_unlock_bh(&con->writequeue_lock);
> release_sock(con->sock->sk);
>
> + /* the sk_write_space() came in between sock_sendmsg()
> + * and check on SOCK_NOSPACE and the socket became
> + * writeable again so just resched swork.
> + */
> return DLM_IO_RESCHED;
> } else if (ret < 0) {
> return ret;
[Severity: Medium]
Can send_to_sock() still set CF_APP_LIMITED and hang after a socket error
if the error arrives before lock_sock()?
kworker <softirq>
send_to_sock()
sock_sendmsg()
...
sk_set_nospace()
return -EAGAIN
tcp_v4_rcv() (TCP RST)
tcp_reset()
tcp_done_with_error()
lowcomms_error_report()
lock_sock(sk)
spin_lock_bh(&con->writequeue_lock)
test_bit(SOCK_NOSPACE, ...)
!test_and_set_bit(CF_APP_LIMITED, ...)
return DLM_IO_END <-- waits forever
Unlike lowcomms_write_space(), neither tcp_done_with_error() nor
lowcomms_error_report() clears SOCK_NOSPACE. When send_to_sock() then
checks SOCK_NOSPACE without checking sk->sk_err, it sets CF_APP_LIMITED
after lowcomms_error_report() has already finished and returns DLM_IO_END
on a dead socket that will never fire sk_write_space() or
sk_error_report() again. Checking !con->sock->sk->sk_err under lock_sock()
(or clearing SOCK_NOSPACE in lowcomms_error_report()) would avoid entering
CF_APP_LIMITED on an errored socket.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261120200040.257152-1-aahringo@redhat.com?part=1
next prev parent reply other threads:[~2026-11-20 20:40 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 20:00 [PATCHv2 RESEND v7.4-rc3 0/5] dlm: various fixes Alexander Aring
2026-11-20 20:00 ` [PATCHv2 RESEND v7.4-rc3 1/5] dlm: fix socket hang on send buffer limitation with socket error Alexander Aring
2026-11-20 20:40 ` sashiko-bot [this message]
2026-11-20 20:00 ` [PATCHv2 RESEND v7.4-rc3 2/5] dlm: reject wrapped messages with oversized inner lengths Alexander Aring
2026-11-20 20:00 ` [PATCHv2 RESEND v7.4-rc3 3/5] dlm: don't publish a lkb in ls_lkbxa before it has an rsb Alexander Aring
2026-11-20 20:12 ` sashiko-bot
2026-11-20 20:00 ` [PATCHv2 RESEND v7.4-rc3 4/5] dlm: don't set RSB_MASTER_UNCERTAIN on rsb with no master Alexander Aring
2026-11-20 20:00 ` [PATCHv2 RESEND v7.4-rc3 5/5] dlm: fix memory leak on active leaving member Alexander Aring
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=sashiko-outbox-226084@kernel.org \
--to=sashiko-bot@kernel.org \
--cc=aahringo@redhat.com \
--cc=gfs2@lists.linux.dev \
--cc=sashiko-reviews@lists.linux.dev \
--cc=teigland@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox