From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3AC83BFE34 for ; Mon, 10 Aug 2026 11:31:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786361477; cv=none; b=W/nHRx9AtMtSEJj77QWC0EGgAXV5v7HvxE48uk2hJ5kDKPUqMxd/X7gcskQwV/AmHJCM9hSsiY5TLTT9HcMKl92ooSOzO01TT+ZmJQKwvoC/h5rJ8AnC/8vwt/7szsGvC1pEBJVltEjvlKRVBjmvInJEmgAEcrlIlvx4vo3J2Yc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786361477; c=relaxed/simple; bh=WrvGD1S/8CGoBEjut1hb4sF6WNKFjkQG4pMFNG5BpTY=; h=From:Subject:Date:Message-Id:MIME-Version:Content-Type:To:Cc; b=UHfvhcGC4mcOf59IeBP9FdM7ez6Luptlfoqbw5FKhR3NFj21cIA3YNy5cDOc8db0LOQaTL3Qw/hdWxxgYx1+fkxQQF+0KatJ3A1OkPhFsjnotcvpzHGwiPyMxdsZ6T4XvgQOXkKKVVx5DR6ypKzFjaXXtgseJmTr1kRZaHTpiUc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=wRj/cvFO; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="wRj/cvFO" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:Content-Transfer-Encoding: Content-Type:MIME-Version:Message-Id:Date:Subject:From:Reply-To:Content-ID: Content-Description:In-Reply-To:References; bh=h+4pHdev69tFgH9ZU2rtrXHQZ5yG7J/PSCVHmMETH6Y=; b=wRj/cvFOe1dLuBlptHlTbNf2Nn GbL3CsDIpwEJ6reas6GjFcOjxiREZx7zV/BCdxBUZ4nFKnqCCLX3D5VLwm08Mc5JgZCVuV+jPuuHV DoPAY2Oao1L55RFtc8h+UrteB2XCQVAcDNMcjzwAIZze/Buaw8pjz62BI4wKgSkW22j+vg837qPk5 HtX9KkgwnVrW80L9WQVbEAAS55qTCRY8TYinBUPhPszf2hf5mgf6b4VNYgm6Y86gjOk91NsciIzTh XmV9AJnbxBWABMlrUu5MiBRE/LbNoSkZwI7CROleodDChZ+CFkt44eEeqUCGmeSthAk24/JsBRDZS UR7jPZbg==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wtODh-002iep-1d; Mon, 10 Aug 2026 11:31:07 +0000 From: Breno Leitao Subject: [PATCH v2 0/3] locking/csd-lock: Report how long a CSD stall lasted Date: Mon, 10 Aug 2026 04:29:23 -0700 Message-Id: <20260810-csd-stall-duration-v2-0-795083bf04a4@debian.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-B4-Tracking: v=1; b=H4sIABO2eWoC/33NQQ6DIBBA0auQWTMNiFTDqvdoXFAZdRIDDVjSx nj3ph6g2794f4dCmamAEztkqlw4RXCikQLGxceZkAM4AY1qrqpXFscSsGx+XTG8st84RTSmt6Y 1ofOqByngmWni94neBylg4bKl/DkfVf/qX65q1NhpbdrJT6StugV6sI+XlGcYjuP4AhVKi1e1A AAA X-Change-ID: 20260805-csd-stall-duration-3385343d7a08 To: paulmck@kernel.org, Andrew Morton , d@ilvokhin.com Cc: linux-kernel@vger.kernel.org, Peter Zijlstra , Ingo Molnar , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, kernel-team@meta.com, Thomas Gleixner , Breno Leitao X-Mailer: b4 0.16-dev-f8e9d X-Developer-Signature: v=1; a=openpgp-sha256; l=1766; i=leitao@debian.org; h=from:subject:message-id; bh=WrvGD1S/8CGoBEjut1hb4sF6WNKFjkQG4pMFNG5BpTY=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqebZxIMSEhZBkQxzptVehuEHg5AMLLt0nG5PeG DTKuy8lw8iJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCanm2cQAKCRA1o5Of/Hh3 bVYYEACbbngppKbCy4ZAEzj8tyQLsGQU2fnWwo01EDRMhZstGHQ2GR5td3cLTEoQNuTnmwR+1F6 ltBdjpOFCGbvIYYuZYSy+delHrQAz/R2zLxWXijSBg/pgQq85NN9y/Nm/LC4mMw8JX15Uf/kcjC QYv6GnBXtUsuk9lWt4qF6Y4xDeGbJ3ImD/Imh0RMxwV0va+vV7D62ySFxMPj6uSos2jR1c73dXK JqHJDcZIBgpXA572sinICTm1CyzMOUcEJTJ348srMh/tFGiN/oMwIrmVQFRSrsl0m5q3uLLQL5O T9yXW+7YDIGyGxBIbrbOwX0ysmWQhLg5b6AJacfWVXda6O8wrgAANACF3ppNfBJKMwbxit41hQb vaT7C7B3wzDDghOFB0VI6NdDyX0uvBOauv+0fowQUEkB6itccmF+NGwvyJofW/VLovWus1cRBUe AeH7tcj7bNhVMXQco6s8STGocp8W5EbYitOLVoJceAQ5t1wEK6L+QqMrVFJlinxB9wkwHXKu2uQ 7hk12TxVM/rnZjI4GJs9R6C1mnWJ/mViDq8gdmskNJufK5a/ZWYIsRRpMStSm+7eh3xjYmxlkcQ Anw2XLojmyzdlmilhXFGtzwj1k5Mj9b8VB3tEEG+zeuXA44Z8RzU1sH/SZCvCJZr7OTbokhM7yl kFDymQLWX+iLiIQ== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao The CSD lock debug code says when a CPU stops answering an IPI and when it finally answers, but the recovery line leaves out how long the stall lasted, even though csd_lock_wait_toolong() has the timestamp the wait started from. On Meta's fleet that line fired 211K times in the last 24 hours, none of them with a duration. Patch 1 packs the state that csd_lock_wait_toolong() carries across the polling loop into a struct, as suggested on v1, which gives the timestamps names and leaves room for another one. Patch 2 prints how long the lock was stuck and, when the IPI was re-sent, how long after the last re-send the target released it. I've created a simple selftest to test both patches, and I am including it here, given it might be helpful for whoever is hacking this code. --- Changes in v2: - Add patch 1, packing csd_lock_wait_toolong() state into a struct, as suggested by Dmitry Ilvokhin. - Add patch 3, a module that stalls a CPU so the reporting can be exercised. - Link to v1: https://patch.msgid.link/20260805-csd-stall-duration-v1-1-71134fafe150@debian.org --- Breno Leitao (3): locking/csd-lock: Pack csd_lock_wait_toolong() state into a struct locking/csd-lock: Report how long a stuck CSD lock took to recover lib/test_csd_lock: Add a module to stall a CPU on a CSD lock kernel/smp.c | 77 +++++++++++++++-------- lib/Kconfig.debug | 12 ++++ lib/Makefile | 1 + lib/test_csd_lock.c | 173 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 237 insertions(+), 26 deletions(-) --- base-commit: f9a2394a23482bfd330911e9c8295b71724feacd change-id: 20260805-csd-stall-duration-3385343d7a08 Best regards, -- Breno Leitao