* [PATCH v4 1/2] tpm: Disable a TPM that stops responding
2026-11-20 20:47 [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Breno Leitao
@ 2026-11-20 20:47 ` Breno Leitao
2026-11-20 20:47 ` [PATCH v4 2/2] tpm: Disable TPM on TPM_RC_INITIALIZE Breno Leitao
2026-11-20 21:29 ` [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Jarkko Sakkinen
2 siblings, 0 replies; 5+ messages in thread
From: Breno Leitao @ 2026-11-20 20:47 UTC (permalink / raw)
To: Peter Huewe, Jarkko Sakkinen, Jason Gunthorpe
Cc: linux-integrity, linux-kernel, surenkj, noodles, David Laight,
Stefan Mahnke-Hartmann, Breno Leitao, kernel-team
Some TPMs stop responding at runtime, hours to months after they passed
probe. The first sign is usually "invalid TPM_STS.x 0xff", and from then
on every command fails only after the transport timeout. A TPM that is
already dead at boot fails to probe and never registers, which this
patch does not change.
The callers keep sending more commands: the hwrng thread every 10
seconds forever, and userspace through /dev/tpmrm0. Each attempt holds
tpm_mutex for the whole timeout and logs an error:
tpm tpm0: invalid TPM_STS.x 0xff, dumping stack for forensics
tpm tpm0: tpm_try_transmit: send(): error -62
tpm tpm0: tpm_try_transmit: send(): error -62
...
The core already stops using a TPM it cannot trust: tpm_try_get_ops()
fails with -EIO once TPM_CHIP_FLAG_DISABLE is set, which
tpm2_load_null() does when it cannot restore the null key. Set the same
flag when two commands in a row time out.
The first timeout sets TPM_CHIP_FLAG_MISSED_RESPONSE, and a command that
gets an answer, whatever its TPM return code, clears it.
With interrupts, the wait helpers in tpm_tis_core, st33zp24 and
xen-tpmfront return -ETIME as soon as a signal is pending, so an -ETIME
with a signal pending is not counted. Otherwise any user could disable
the TPM by interrupting two reads of a PCR file in sysfs.
Some TPM 1.2 chips do not get ready while their self test runs, and
tpm1_do_selftest() keeps polling through those timeouts, see
commit 4643826a3da1 ("TPM: Work around buggy TPMs that block during
continue self test"). It now clears the flag after each of them.
A disabled TPM stays disabled until the driver is rebound or the machine
reboots.
Suggested-by: Jarkko Sakkinen <jarkko@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
drivers/char/tpm/tpm-interface.c | 15 +++++++++++++++
drivers/char/tpm/tpm1-cmd.c | 1 +
include/linux/tpm.h | 1 +
3 files changed, 17 insertions(+)
diff --git a/drivers/char/tpm/tpm-interface.c b/drivers/char/tpm/tpm-interface.c
index e0385b16616b46..cb794cfc5f63e8 100644
--- a/drivers/char/tpm/tpm-interface.c
+++ b/drivers/char/tpm/tpm-interface.c
@@ -26,6 +26,7 @@
#include <linux/spinlock.h>
#include <linux/suspend.h>
#include <linux/freezer.h>
+#include <linux/sched/signal.h>
#include <linux/tpm_eventlog.h>
#include "tpm.h"
@@ -268,6 +269,20 @@ ssize_t tpm_transmit(struct tpm_chip *chip, u8 *buf, size_t bufsiz)
delay_msec *= 2;
memcpy(buf, save, save_size);
}
+
+ if (ret >= 0)
+ chip->flags &= ~TPM_CHIP_FLAG_MISSED_RESPONSE;
+
+ /* Some drivers also return -ETIME when a signal interrupts the wait. */
+ if (ret == -ETIME && !signal_pending(current)) {
+ if (!(chip->flags & TPM_CHIP_FLAG_MISSED_RESPONSE)) {
+ chip->flags |= TPM_CHIP_FLAG_MISSED_RESPONSE;
+ } else if (!(chip->flags & TPM_CHIP_FLAG_DISABLE)) {
+ dev_err(&chip->dev,
+ "two commands in a row timed out, disabling the TPM\n");
+ chip->flags |= TPM_CHIP_FLAG_DISABLE;
+ }
+ }
return ret;
}
diff --git a/drivers/char/tpm/tpm1-cmd.c b/drivers/char/tpm/tpm1-cmd.c
index 7a8606e8c7117b..e4e1f0799da196 100644
--- a/drivers/char/tpm/tpm1-cmd.c
+++ b/drivers/char/tpm/tpm1-cmd.c
@@ -647,6 +647,7 @@ int tpm1_do_selftest(struct tpm_chip *chip)
* until the self test duration expires.
*/
if (rc == -ETIME) {
+ chip->flags &= ~TPM_CHIP_FLAG_MISSED_RESPONSE;
dev_info(&chip->dev, HW_ERR "TPM command timed out during continue self test");
tpm_msleep(delay_msec);
continue;
diff --git a/include/linux/tpm.h b/include/linux/tpm.h
index 0db277af45c393..0f97abc82846ae 100644
--- a/include/linux/tpm.h
+++ b/include/linux/tpm.h
@@ -197,6 +197,7 @@ enum tpm_chip_flags {
TPM_CHIP_FLAG_HWRNG_DISABLED = BIT(9),
TPM_CHIP_FLAG_DISABLE = BIT(10),
TPM_CHIP_FLAG_SYNC = BIT(11),
+ TPM_CHIP_FLAG_MISSED_RESPONSE = BIT(12),
};
#define to_tpm_chip(d) container_of(d, struct tpm_chip, dev)
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH v4 2/2] tpm: Disable TPM on TPM_RC_INITIALIZE
2026-11-20 20:47 [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Breno Leitao
2026-11-20 20:47 ` [PATCH v4 1/2] tpm: Disable a TPM that stops responding Breno Leitao
@ 2026-11-20 20:47 ` Breno Leitao
2026-11-20 21:29 ` [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Jarkko Sakkinen
2 siblings, 0 replies; 5+ messages in thread
From: Breno Leitao @ 2026-11-20 20:47 UTC (permalink / raw)
To: Peter Huewe, Jarkko Sakkinen, Jason Gunthorpe
Cc: linux-integrity, linux-kernel, surenkj, noodles, David Laight,
Stefan Mahnke-Hartmann, Breno Leitao, kernel-team
From: Jarkko Sakkinen <jarkko@kernel.org>
If the TPM has been spuriously disabled, the game is over. Let's reflect
this truth with TPM_CHIP_FLAG_DISABLE.
Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
[leitao@debian.org: check rc before the loop breaks out on anything but
RETRY/TESTING, skip TPM2_Startup, which a TPM that is already started
fails with TPM2_RC_INITIALIZE, and fix the TPM2_RC_INITIALIZE and
chip->flags typos]
Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
drivers/char/tpm/tpm-interface.c | 5 +++++
drivers/char/tpm/tpm2-cmd.c | 2 ++
2 files changed, 7 insertions(+)
diff --git a/drivers/char/tpm/tpm-interface.c b/drivers/char/tpm/tpm-interface.c
index cb794cfc5f63e8..2ea10fd89a70f1 100644
--- a/drivers/char/tpm/tpm-interface.c
+++ b/drivers/char/tpm/tpm-interface.c
@@ -248,6 +248,11 @@ ssize_t tpm_transmit(struct tpm_chip *chip, u8 *buf, size_t bufsiz)
if (ret < 0)
break;
rc = be32_to_cpu(header->return_code);
+ /* A TPM that is already started fails TPM2_Startup this way. */
+ if (rc == TPM2_RC_INITIALIZE && cc != TPM2_CC_STARTUP) {
+ chip->flags |= TPM_CHIP_FLAG_DISABLE;
+ break;
+ }
if (rc != TPM2_RC_RETRY && rc != TPM2_RC_TESTING)
break;
/*
diff --git a/drivers/char/tpm/tpm2-cmd.c b/drivers/char/tpm/tpm2-cmd.c
index 4b2bb2ddc321a9..65928ae85b496c 100644
--- a/drivers/char/tpm/tpm2-cmd.c
+++ b/drivers/char/tpm/tpm2-cmd.c
@@ -724,6 +724,8 @@ int tpm2_auto_startup(struct tpm_chip *chip)
goto out;
if (rc == TPM2_RC_INITIALIZE) {
+ chip->flags &= ~TPM_CHIP_FLAG_DISABLE;
+
rc = tpm2_startup(chip);
if (rc)
goto out;
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working
2026-11-20 20:47 [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Breno Leitao
2026-11-20 20:47 ` [PATCH v4 1/2] tpm: Disable a TPM that stops responding Breno Leitao
2026-11-20 20:47 ` [PATCH v4 2/2] tpm: Disable TPM on TPM_RC_INITIALIZE Breno Leitao
@ 2026-11-20 21:29 ` Jarkko Sakkinen
2026-11-20 21:30 ` Jarkko Sakkinen
2 siblings, 1 reply; 5+ messages in thread
From: Jarkko Sakkinen @ 2026-11-20 21:29 UTC (permalink / raw)
To: Breno Leitao
Cc: Peter Huewe, Jason Gunthorpe, linux-integrity, linux-kernel,
surenkj, noodles, David Laight, Stefan Mahnke-Hartmann,
kernel-team
On Fri, Nov 20, 2026 at 12:47:06PM -0800, Breno Leitao wrote:
> On Meta's datacenter, given the size, TPMs that stop working write about
> 54 million lines a day to the kernel log, because nothing in the kernel
> gives up on them.
>
> Two cases make up nearly all of it.
>
> Some TPMs stop responding. Every command then fails only after the
> transport timeout, while the hwrng thread and userspace keep sending
> more.
>
> Other TPMs get reset under the running OS, and answer every command with
> TPM_RC_INITIALIZE (256) from then on, while the hwrng thread retries
> every 10 seconds:
>
> tpm tpm0: A TPM error (256) occurred attempting get random
>
> The goal is to retire TPMs that are not properly working, similarly to
> what was done for EFI in commit 4b2c033b5bc971 ("efi/runtime-wrappers:
> retire the worker if a wedged call ever returns")
>
> Patch 1 disables the chip once two commands in a row have timed out,
> instead of keeping it in this desolate state. Patch 2, from Jarkko,
> disables it once it answers TPM_RC_INITIALIZE to anything but
> TPM2_Startup.
>
> Disabling the chip makes the failure explicit: every later command
> fails at once instead of timing out.
>
> A health-check daemon, such as machine checker, can then detect
> the broken hardware and send the machine to repair.
>
> Signed-off-by: Breno Leitao <leitao@debian.org>
> ---
> Changes in v4:
> - Replace the count of timeouts and the tpm.max_missed_responses
> parameter with TPM_CHIP_FLAG_MISSED_RESPONSE: one timeout is
> tolerated, and a second one in a row disables the TPM (Jarkko)
> - Say in patch 1 when TPMs stop responding (Jarkko), and drop the
> claim that some of them recover
> - Fix sashiko findings:
> - A timeout while a signal is pending no longer counts, since some
> drivers return -ETIME when a signal interrupts the wait
> - tpm1_do_selftest() clears the flag on the timeouts it expects from
> TPM 1.2 chips that block during the self test
> - Link to v3: https://patch.msgid.link/20261113-tpm_unresponsive-v3-0-8138c8e283b8@debian.org
>
> Changes in v3:
> - Rename tpm.max_unanswered to tpm.max_missed_responses and
> nr_unanswered to nr_missed_responses, and drop a redundant comment
> (Jarkko)
> - Document tpm.max_missed_responses in kernel-parameters.txt (Jarkko)
> - Check the limit with a modulo (Jarkko), guarded so that 0 still
> means never
> - Link to v2: https://patch.msgid.link/20261112-tpm_unresponsive-v2-0-7454d2c899ef@debian.org
>
> Changes in v2:
> - Disable the TPM after 60 commands in a row time out, instead of after
> ten minutes of timeouts, and make the limit the tpm.max_unanswered
> boot parameter (Jarkko)
> - Open-code the check in tpm_transmit() and drop the "timeout" field
> names (Jarkko)
> - Replace the hwrng failure counter with Jarkko's patch, which disables
> the TPM on TPM_RC_INITIALIZE
> - In that patch, move the check ahead of the RETRY/TESTING break, where
> it never ran, and skip TPM2_Startup, which a TPM that is already
> started fails with TPM_RC_INITIALIZE
> - Fix sashiko findings:
> - Two timeouts ten minutes apart no longer disable an idle TPM
> - Link to v1: https://patch.msgid.link/20261111-tpm_unresponsive-v1-0-42ac1b965eb9@debian.org
>
> ---
> Breno Leitao (1):
> tpm: Disable a TPM that stops responding
>
> Jarkko Sakkinen (1):
> tpm: Disable TPM on TPM_RC_INITIALIZE
>
> drivers/char/tpm/tpm-interface.c | 20 ++++++++++++++++++++
> drivers/char/tpm/tpm1-cmd.c | 1 +
> drivers/char/tpm/tpm2-cmd.c | 2 ++
> include/linux/tpm.h | 1 +
> 4 files changed, 24 insertions(+)
> ---
> base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47
> change-id: 20261110-tpm_unresponsive-a5f71c1db026
>
> Best regards,
> --
> Breno Leitao <leitao@debian.org>
>
Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>
Br, Jarkko
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working
2026-11-20 21:29 ` [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Jarkko Sakkinen
@ 2026-11-20 21:30 ` Jarkko Sakkinen
0 siblings, 0 replies; 5+ messages in thread
From: Jarkko Sakkinen @ 2026-11-20 21:30 UTC (permalink / raw)
To: Breno Leitao
Cc: Peter Huewe, Jason Gunthorpe, linux-integrity, linux-kernel,
surenkj, noodles, David Laight, Stefan Mahnke-Hartmann,
kernel-team
On Fri, Nov 20, 2026 at 11:29:56PM +0200, Jarkko Sakkinen wrote:
> On Fri, Nov 20, 2026 at 12:47:06PM -0800, Breno Leitao wrote:
> > On Meta's datacenter, given the size, TPMs that stop working write about
> > 54 million lines a day to the kernel log, because nothing in the kernel
> > gives up on them.
> >
> > Two cases make up nearly all of it.
> >
> > Some TPMs stop responding. Every command then fails only after the
> > transport timeout, while the hwrng thread and userspace keep sending
> > more.
> >
> > Other TPMs get reset under the running OS, and answer every command with
> > TPM_RC_INITIALIZE (256) from then on, while the hwrng thread retries
> > every 10 seconds:
> >
> > tpm tpm0: A TPM error (256) occurred attempting get random
> >
> > The goal is to retire TPMs that are not properly working, similarly to
> > what was done for EFI in commit 4b2c033b5bc971 ("efi/runtime-wrappers:
> > retire the worker if a wedged call ever returns")
> >
> > Patch 1 disables the chip once two commands in a row have timed out,
> > instead of keeping it in this desolate state. Patch 2, from Jarkko,
> > disables it once it answers TPM_RC_INITIALIZE to anything but
> > TPM2_Startup.
> >
> > Disabling the chip makes the failure explicit: every later command
> > fails at once instead of timing out.
> >
> > A health-check daemon, such as machine checker, can then detect
> > the broken hardware and send the machine to repair.
> >
> > Signed-off-by: Breno Leitao <leitao@debian.org>
> > ---
> > Changes in v4:
> > - Replace the count of timeouts and the tpm.max_missed_responses
> > parameter with TPM_CHIP_FLAG_MISSED_RESPONSE: one timeout is
> > tolerated, and a second one in a row disables the TPM (Jarkko)
> > - Say in patch 1 when TPMs stop responding (Jarkko), and drop the
> > claim that some of them recover
> > - Fix sashiko findings:
> > - A timeout while a signal is pending no longer counts, since some
> > drivers return -ETIME when a signal interrupts the wait
> > - tpm1_do_selftest() clears the flag on the timeouts it expects from
> > TPM 1.2 chips that block during the self test
> > - Link to v3: https://patch.msgid.link/20261113-tpm_unresponsive-v3-0-8138c8e283b8@debian.org
> >
> > Changes in v3:
> > - Rename tpm.max_unanswered to tpm.max_missed_responses and
> > nr_unanswered to nr_missed_responses, and drop a redundant comment
> > (Jarkko)
> > - Document tpm.max_missed_responses in kernel-parameters.txt (Jarkko)
> > - Check the limit with a modulo (Jarkko), guarded so that 0 still
> > means never
> > - Link to v2: https://patch.msgid.link/20261112-tpm_unresponsive-v2-0-7454d2c899ef@debian.org
> >
> > Changes in v2:
> > - Disable the TPM after 60 commands in a row time out, instead of after
> > ten minutes of timeouts, and make the limit the tpm.max_unanswered
> > boot parameter (Jarkko)
> > - Open-code the check in tpm_transmit() and drop the "timeout" field
> > names (Jarkko)
> > - Replace the hwrng failure counter with Jarkko's patch, which disables
> > the TPM on TPM_RC_INITIALIZE
> > - In that patch, move the check ahead of the RETRY/TESTING break, where
> > it never ran, and skip TPM2_Startup, which a TPM that is already
> > started fails with TPM_RC_INITIALIZE
> > - Fix sashiko findings:
> > - Two timeouts ten minutes apart no longer disable an idle TPM
> > - Link to v1: https://patch.msgid.link/20261111-tpm_unresponsive-v1-0-42ac1b965eb9@debian.org
> >
> > ---
> > Breno Leitao (1):
> > tpm: Disable a TPM that stops responding
> >
> > Jarkko Sakkinen (1):
> > tpm: Disable TPM on TPM_RC_INITIALIZE
> >
> > drivers/char/tpm/tpm-interface.c | 20 ++++++++++++++++++++
> > drivers/char/tpm/tpm1-cmd.c | 1 +
> > drivers/char/tpm/tpm2-cmd.c | 2 ++
> > include/linux/tpm.h | 1 +
> > 4 files changed, 24 insertions(+)
> > ---
> > base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47
> > change-id: 20261110-tpm_unresponsive-a5f71c1db026
> >
> > Best regards,
> > --
> > Breno Leitao <leitao@debian.org>
> >
>
> Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>
It's considered enough. If a tree falls somewhere because of this, we
deal with that separately (and it's just life).
Br, Jarkko
^ permalink raw reply [flat|nested] 5+ messages in thread