From: Shakeel Butt <shakeel.butt@linux.dev>
To: Corey Minyard <corey@minyard.net>
Cc: openipmi-developer@lists.sourceforge.net,
Meta kernel team <kernel-team@meta.com>,
linux-kernel@vger.kernel.org
Subject: [PATCH] ipmi:msghandler: Free queued requests of a destroyed user
Date: Fri, 20 Nov 2026 19:15:49 -0800 [thread overview]
Message-ID: <20261121031549.3599823-1-shakeel.butt@linux.dev> (raw)
In the Meta fleet we found hosts whose BMC had stopped answering IPMI
requests while a power management daemon kept polling it:
- the daemon runs "timeout 2 ipmitool raw 0x30 0x7e" every two seconds;
- every run leaves one request and one closed IPMI user behind;
- one host held over a million closed users after 25 days, using about
33 GiB and growing by 1.3 GiB a day.
ipmitool opens /dev/ipmi0, sends a request and waits for the answer.
When timeout kills it, the file is closed and ipmi_destroy_user() runs.
A request that is still waiting on intf->xmit_msgs or intf->hp_xmit_msgs
stays there, and its receive message keeps a reference on the user.
With a BMC that does not answer, the lower layer is slow to fail each
request: about 15 seconds of retries for SSIF, and up to a minute for
KCS on kernels without the hosed BMC handling from
commit bc3a9d217755 ("ipmi:si: Gracefully handle if the BMC is non-functional").
Requests arrive faster than they leave, so the queue, the users and
their messages grow until the BMC answers again.
Nobody can receive these responses once the user is destroyed. Free the
user's requests that were not handed to the lower layer yet. Such a
request is then never sent to the BMC. This only matters for a program
that sends a request and closes the device without waiting for the
answer, and with a working BMC a request waits in these lists only
while another request is being handled.
The request the lower layer is working on (intf->curr_msg) is left
alone. IPMB and LAN commands do not point to their receive message, so
they are not touched here. Their receive messages are freed by the
sequence table loop just above, as before.
Tested in QEMU with a fake BMC that stops answering and four loops of
"timeout 2 ipmitool raw 0x30 0x7e", with KASAN and lockdep enabled:
- SSIF on linux-next: 465 closed users were held after 4 minutes,
growing by about 116 a minute. With this patch, 5 are held (the four
live callers and the request in flight), and the number stays flat.
- KCS on v6.16, which lacks the hosed BMC handling, with this change
adapted: 476 held after 4 minutes without it, 5 with it.
- KCS on linux-next: at most 5 held instead of up to 28.
- With a working BMC, 1600 requests from 8 parallel ipmitool loops all
succeed over KCS and SSIF, and nothing is left held afterwards.
QEMU needed small changes to behave like the failing BMC: ipmi-bmc-extern
makes up a timeout reply after 4 seconds, its KCS keeps IBF set while
the BMC has not answered, and smbus-ipmi returns the previous reply
instead of NAKing the read.
Fixes: 7ea0ed2b5be8 ("ipmi: Make the message handler easier to use for SMI interfaces")
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
drivers/char/ipmi/ipmi_msghandler.c | 41 +++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/drivers/char/ipmi/ipmi_msghandler.c b/drivers/char/ipmi/ipmi_msghandler.c
index 152dd6b5bb27..52e95e1949b9 100644
--- a/drivers/char/ipmi/ipmi_msghandler.c
+++ b/drivers/char/ipmi/ipmi_msghandler.c
@@ -1339,6 +1339,45 @@ int ipmi_get_smi_info(int if_num, struct ipmi_smi_info *data)
}
EXPORT_SYMBOL(ipmi_get_smi_info);
+static void move_user_xmit_msgs(struct list_head *from, struct list_head *to,
+ struct ipmi_user *user)
+{
+ struct ipmi_smi_msg *msg, *msg2;
+
+ list_for_each_entry_safe(msg, msg2, from, link) {
+ if (msg->recv_msg && msg->recv_msg->user == user)
+ list_move_tail(&msg->link, to);
+ }
+}
+
+/*
+ * Free the requests of a user that are still waiting to be handed to the
+ * lower layer. Once the user is destroyed, nobody can receive their
+ * responses. If the BMC is slow or does not answer, they could otherwise
+ * stay queued for a long time, each one keeping the user and its own
+ * messages allocated. A request that was already handed to the lower
+ * layer (intf->curr_msg) is not on these lists and is left alone.
+ */
+static void remove_user_xmit_msgs(struct ipmi_smi *intf,
+ struct ipmi_user *user)
+{
+ int run_to_completion = READ_ONCE(intf->run_to_completion);
+ struct ipmi_smi_msg *msg, *msg2;
+ unsigned long flags = 0;
+ LIST_HEAD(msgs);
+
+ ipmi_lock_xmit_msgs(intf, run_to_completion, &flags);
+ move_user_xmit_msgs(&intf->hp_xmit_msgs, &msgs, user);
+ move_user_xmit_msgs(&intf->xmit_msgs, &msgs, user);
+ ipmi_unlock_xmit_msgs(intf, run_to_completion, &flags);
+
+ list_for_each_entry_safe(msg, msg2, &msgs, link) {
+ list_del(&msg->link);
+ ipmi_free_recv_msg(msg->recv_msg);
+ ipmi_free_smi_msg(msg);
+ }
+}
+
/* Must be called with intf->users_mutex held. */
static void _ipmi_destroy_user(struct ipmi_user *user)
{
@@ -1375,6 +1414,8 @@ static void _ipmi_destroy_user(struct ipmi_user *user)
}
mutex_unlock(&intf->seq_lock);
+ remove_user_xmit_msgs(intf, user);
+
/*
* Remove the user from the command receiver's table. First
* we build a list of everything (not using the standard link,
--
2.53.1-Meta
reply other threads:[~2026-11-21 3:15 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261121031549.3599823-1-shakeel.butt@linux.dev \
--to=shakeel.butt@linux.dev \
--cc=corey@minyard.net \
--cc=kernel-team@meta.com \
--cc=linux-kernel@vger.kernel.org \
--cc=openipmi-developer@lists.sourceforge.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.