[PATCH v3 0/2] tpm: Stop retrying TPMs that have stopped working

[PATCH v3 0/2] tpm: Stop retrying TPMs that have stopped working

[Breno Leitao]

On Meta's datacenter, given the size, TPMs that stop working write about
54 million lines a day to the kernel log, because nothing in the kernel
gives up on them.

Two cases make up nearly all of it.

Some TPMs stop responding. Every command then fails only after the
transport timeout, while the hwrng thread and userspace keep sending
more.

Other TPMs get reset under the running OS, and answer every command with
TPM_RC_INITIALIZE (256) from then on, while the hwrng thread retries
every 10 seconds:

  tpm tpm0: A TPM error (256) occurred attempting get random

The goal is to retire TPMs that are not properly working, similarly to
what was done for EFI in commit 4b2c033b5bc971 ("efi/runtime-wrappers:
retire the worker if a wedged call ever returns")

Patch 1 disables the chip once 60 commands in a row have timed out,
instead of keeping it in this desolate state. Patch 2, from Jarkko,
disables it once it answers TPM_RC_INITIALIZE to anything but
TPM2_Startup.

Disabling the chip makes the failure explicit: the log gets one last
line about it, and every later command fails at once instead of timing
out.

A health-check daemon, such as machine checker, can then detect
the broken hardware and send the machine to repair.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Changes in v3:
- Rename tpm.max_unanswered to tpm.max_missed_responses and
  nr_unanswered to nr_missed_responses, and drop a redundant comment
  (Jarkko)
- Document tpm.max_missed_responses in kernel-parameters.txt (Jarkko)
- Check the limit with a modulo (Jarkko), guarded so that 0 still
  means never
- Link to v2: https://patch.msgid.link/20261112-tpm_unresponsive-v2-0-7454d2c899ef@debian.org

Changes in v2:
- Disable the TPM after 60 commands in a row time out, instead of after
  ten minutes of timeouts, and make the limit the tpm.max_unanswered
  boot parameter (Jarkko)
- Open-code the check in tpm_transmit() and drop the "timeout" field
  names (Jarkko)
- Replace the hwrng failure counter with Jarkko's patch, which disables
  the TPM on TPM_RC_INITIALIZE
- In that patch, move the check ahead of the RETRY/TESTING break, where
  it never ran, and skip TPM2_Startup, which a TPM that is already
  started fails with TPM_RC_INITIALIZE
- Fix sashiko findings:
  - Two timeouts ten minutes apart no longer disable an idle TPM
- Link to v1: https://patch.msgid.link/20261111-tpm_unresponsive-v1-0-42ac1b965eb9@debian.org

---
Breno Leitao (1):
      tpm: Disable a TPM that stops responding

Jarkko Sakkinen (1):
      tpm: Disable TPM on TPM_RC_INITIALIZE

 Documentation/admin-guide/kernel-parameters.txt |  8 ++++++++
 drivers/char/tpm/tpm-interface.c                | 19 +++++++++++++++++++
 drivers/char/tpm/tpm2-cmd.c                     |  2 ++
 include/linux/tpm.h                             |  1 +
 4 files changed, 30 insertions(+)
---
base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47
change-id: 20261110-tpm_unresponsive-a5f71c1db026

Best regards,
--  
Breno Leitao <leitao@debian.org>

[PATCH v3 1/2] tpm: Disable a TPM that stops responding

[Breno Leitao]

A TPM that stops responding fails each command only after the transport
timeout, and its callers keep sending more: the hwrng thread every 10
seconds forever, and userspace through /dev/tpmrm0. Each attempt holds
tpm_mutex for the whole timeout and logs an error:

  tpm tpm0: invalid TPM_STS.x 0xff, dumping stack for forensics
  tpm tpm0: tpm_try_transmit: send(): error -62
  tpm tpm0: tpm_try_transmit: send(): error -62
  ...

The core already stops using a TPM it cannot trust: tpm_try_get_ops()
fails with -EIO once TPM_CHIP_FLAG_DISABLE is set, which
tpm2_load_null() does when it cannot restore the null key. Set the same
flag once 60 commands in a row have timed out.

A completed command, whatever its TPM return code, resets the count.
Some TPMs stall for half a minute, time out a handful of commands, and
then recover, so a short streak must not disable them.

With only the hwrng retrying every 10 seconds, 60 timeouts take more
than ten minutes. A userspace poller gets there sooner, but each of its
commands still waits out the transport timeout.

The limit is the tpm.max_missed_responses boot parameter, so that slow
pre-production parts, such as FPGA prototypes, can raise it, or set it
to 0 to never disable the TPM. A disabled TPM stays disabled until the
driver is rebound or the machine reboots.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 Documentation/admin-guide/kernel-parameters.txt |  8 ++++++++
 drivers/char/tpm/tpm-interface.c                | 14 ++++++++++++++
 include/linux/tpm.h                             |  1 +
 3 files changed, 23 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 1e8317483844d9..cedea8bd468aba 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -7682,6 +7682,14 @@ Kernel parameters
 			causing a major performance hit, and the space where
 			machines are deployed is by other means guarded.
 
+	tpm.max_missed_responses= [HW,TPM]
+			Number of commands in a row that may time out before
+			the TPM is disabled. It stays disabled until the
+			driver is rebound or the machine reboots. Set to 0
+			to never disable the TPM.
+			Format: <unsigned int>
+			Default: 60
+
 	tpm_crb_ffa.busy_timeout_ms= [ARM64,TPM]
 			Maximum time in milliseconds to retry sending a message
 			to the TPM service before giving up. This parameter controls
diff --git a/drivers/char/tpm/tpm-interface.c b/drivers/char/tpm/tpm-interface.c
index e0385b16616b46..977e205807033d 100644
--- a/drivers/char/tpm/tpm-interface.c
+++ b/drivers/char/tpm/tpm-interface.c
@@ -40,6 +40,11 @@ module_param_named(suspend_pcr, tpm_suspend_pcr, uint, 0644);
 MODULE_PARM_DESC(suspend_pcr,
 		 "PCR to use for dummy writes to facilitate flush on suspend.");
 
+static unsigned int tpm_max_missed_responses = 60;
+module_param_named(max_missed_responses, tpm_max_missed_responses, uint, 0444);
+MODULE_PARM_DESC(max_missed_responses,
+		 "Disable the TPM after this many commands in a row time out (0 = never).");
+
 /**
  * tpm_calc_ordinal_duration() - calculate the maximum command duration
  * @chip:    TPM chip to use.
@@ -268,6 +273,15 @@ ssize_t tpm_transmit(struct tpm_chip *chip, u8 *buf, size_t bufsiz)
 		delay_msec *= 2;
 		memcpy(buf, save, save_size);
 	}
+
+	if (ret >= 0) {
+		chip->nr_missed_responses = 0;
+	} else if (ret == -ETIME && tpm_max_missed_responses &&
+		   !(++chip->nr_missed_responses % tpm_max_missed_responses)) {
+		dev_err(&chip->dev, "%u commands in a row timed out, disabling the TPM\n",
+			chip->nr_missed_responses);
+		chip->flags |= TPM_CHIP_FLAG_DISABLE;
+	}
 	return ret;
 }
 
diff --git a/include/linux/tpm.h b/include/linux/tpm.h
index 0db277af45c393..6ad1e254831f5f 100644
--- a/include/linux/tpm.h
+++ b/include/linux/tpm.h
@@ -131,6 +131,7 @@ struct tpm_chip {
 	struct hwrng hwrng;
 
 	struct mutex tpm_mutex;	/* tpm is processing */
+	unsigned int nr_missed_responses; /* commands in a row that timed out */
 
 	unsigned long timeout_a; /* jiffies */
 	unsigned long timeout_b; /* jiffies */

-- 
2.53.1-Meta

[PATCH v3 2/2] tpm: Disable TPM on TPM_RC_INITIALIZE

[Breno Leitao]

From: Jarkko Sakkinen <jarkko@kernel.org>

If the TPM has been spuriously disabled, the game is over. Let's reflect
this truth with TPM_CHIP_FLAG_DISABLE.

Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
[leitao@debian.org: check rc before the loop breaks out on anything but
 RETRY/TESTING, skip TPM2_Startup, which a TPM that is already started
 fails with TPM2_RC_INITIALIZE, and fix the TPM2_RC_INITIALIZE and
 chip->flags typos]
Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
 drivers/char/tpm/tpm-interface.c | 5 +++++
 drivers/char/tpm/tpm2-cmd.c      | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/drivers/char/tpm/tpm-interface.c b/drivers/char/tpm/tpm-interface.c
index 977e205807033d..dd6bb518ab7d9c 100644
--- a/drivers/char/tpm/tpm-interface.c
+++ b/drivers/char/tpm/tpm-interface.c
@@ -252,6 +252,11 @@ ssize_t tpm_transmit(struct tpm_chip *chip, u8 *buf, size_t bufsiz)
 		if (ret < 0)
 			break;
 		rc = be32_to_cpu(header->return_code);
+		/* A TPM that is already started fails TPM2_Startup this way. */
+		if (rc == TPM2_RC_INITIALIZE && cc != TPM2_CC_STARTUP) {
+			chip->flags |= TPM_CHIP_FLAG_DISABLE;
+			break;
+		}
 		if (rc != TPM2_RC_RETRY && rc != TPM2_RC_TESTING)
 			break;
 		/*
diff --git a/drivers/char/tpm/tpm2-cmd.c b/drivers/char/tpm/tpm2-cmd.c
index 4b2bb2ddc321a9..65928ae85b496c 100644
--- a/drivers/char/tpm/tpm2-cmd.c
+++ b/drivers/char/tpm/tpm2-cmd.c
@@ -724,6 +724,8 @@ int tpm2_auto_startup(struct tpm_chip *chip)
 		goto out;
 
 	if (rc == TPM2_RC_INITIALIZE) {
+		chip->flags &= ~TPM_CHIP_FLAG_DISABLE;
+
 		rc = tpm2_startup(chip);
 		if (rc)
 			goto out;

-- 
2.53.1-Meta

Re: [PATCH v3 1/2] tpm: Disable a TPM that stops responding

[Jarkko Sakkinen]

On Fri, Nov 13, 2026 at 02:38:58AM -0800, Breno Leitao wrote:
> A TPM that stops responding fails each command only after the transport
> timeout, and its callers keep sending more: the hwrng thread every 10
> seconds forever, and userspace through /dev/tpmrm0. Each attempt holds
> tpm_mutex for the whole timeout and logs an error:
> 
>   tpm tpm0: invalid TPM_STS.x 0xff, dumping stack for forensics
>   tpm tpm0: tpm_try_transmit: send(): error -62
>   tpm tpm0: tpm_try_transmit: send(): error -62
>   ...
> 
> The core already stops using a TPM it cannot trust: tpm_try_get_ops()
> fails with -EIO once TPM_CHIP_FLAG_DISABLE is set, which
> tpm2_load_null() does when it cannot restore the null key. Set the same
> flag once 60 commands in a row have timed out.
> 
> A completed command, whatever its TPM return code, resets the count.
> Some TPMs stall for half a minute, time out a handful of commands, and
> then recover, so a short streak must not disable them.
> 
> With only the hwrng retrying every 10 seconds, 60 timeouts take more
> than ten minutes. A userspace poller gets there sooner, but each of its
> commands still waits out the transport timeout.
> 
> The limit is the tpm.max_missed_responses boot parameter, so that slow
> pre-production parts, such as FPGA prototypes, can raise it, or set it
> to 0 to never disable the TPM. A disabled TPM stays disabled until the
> driver is rebound or the machine reboots.
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>
> ---
>  Documentation/admin-guide/kernel-parameters.txt |  8 ++++++++
>  drivers/char/tpm/tpm-interface.c                | 14 ++++++++++++++
>  include/linux/tpm.h                             |  1 +
>  3 files changed, 23 insertions(+)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 1e8317483844d9..cedea8bd468aba 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -7682,6 +7682,14 @@ Kernel parameters
>  			causing a major performance hit, and the space where
>  			machines are deployed is by other means guarded.
>  
> +	tpm.max_missed_responses= [HW,TPM]
> +			Number of commands in a row that may time out before
> +			the TPM is disabled. It stays disabled until the
> +			driver is rebound or the machine reboots. Set to 0
> +			to never disable the TPM.
> +			Format: <unsigned int>
> +			Default: 60
> +
>  	tpm_crb_ffa.busy_timeout_ms= [ARM64,TPM]
>  			Maximum time in milliseconds to retry sending a message
>  			to the TPM service before giving up. This parameter controls
> diff --git a/drivers/char/tpm/tpm-interface.c b/drivers/char/tpm/tpm-interface.c
> index e0385b16616b46..977e205807033d 100644
> --- a/drivers/char/tpm/tpm-interface.c
> +++ b/drivers/char/tpm/tpm-interface.c
> @@ -40,6 +40,11 @@ module_param_named(suspend_pcr, tpm_suspend_pcr, uint, 0644);
>  MODULE_PARM_DESC(suspend_pcr,
>  		 "PCR to use for dummy writes to facilitate flush on suspend.");
>  
> +static unsigned int tpm_max_missed_responses = 60;

This looks good but I started to think this number. I guess it is high
because there's never reset-to-zero for chip->nr_missed_responses.

I think three or four trials at most should be enough, and when there 
is a successful transmission chip->nr_missed_responses should reset.

One thing that bothers me in the commit message is a piece of
information that is missing: when does it start to return error?
From the get go?

I mean based on provided information one could presume that this
could be also fixed at driver init by doing a few smoke tests
and don't let it fully ever initialize.

"A TPM that stops responding fails" is not very locked in scenario.

Br, Jarkko

Re: [PATCH v3 1/2] tpm: Disable a TPM that stops responding

[Breno Leitao]

On Fri, Nov 13, 2026 at 04:42:20PM +0200, Jarkko Sakkinen wrote:
> On Fri, Nov 13, 2026 at 02:38:58AM -0800, Breno Leitao wrote:
> > @@ -40,6 +40,11 @@ module_param_named(suspend_pcr, tpm_suspend_pcr, uint, 0644);
> >  MODULE_PARM_DESC(suspend_pcr,
> >  		 "PCR to use for dummy writes to facilitate flush on suspend.");
> >  
> > +static unsigned int tpm_max_missed_responses = 60;
> 
> This looks good but I started to think this number. I guess it is high
> because there's never reset-to-zero for chip->nr_missed_responses.
>
> I think three or four trials at most should be enough, and when there 
> is a successful transmission chip->nr_missed_responses should reset.

It does reset. Any command that gets an answer, whatever its return
code, sets the count back to zero:

	if (ret >= 0) {
		chip->nr_missed_responses = 0;

so the TPM is only disabled after 60 timeouts with no answer in between.

> One thing that bothers me in the commit message is a piece of
> information that is missing: when does it start to return error?
> From the get go?
> 
> I mean based on provided information one could presume that this
> could be also fixed at driver init by doing a few smoke tests
> and don't let it fully ever initialize.
> 
> "A TPM that stops responding fails" is not very locked in scenario.

I cam update the commit. What about something like:

    Some TPMs stop responding at runtime, hours to months after they
    passed probe. The first sign is usually "invalid TPM_STS.x 0xff",
    and from then on every command fails only after the transport
    timeout. A TPM that is already dead at boot fails tpm2_probe() and
    never registers, which is not affected by this patchset/feature.

Thanks,
--breno

Re: [PATCH v3 1/2] tpm: Disable a TPM that stops responding

[Jarkko Sakkinen]

On Fri, Nov 13, 2026 at 07:14:39AM -0800, Breno Leitao wrote:
> On Fri, Nov 13, 2026 at 04:42:20PM +0200, Jarkko Sakkinen wrote:
> > On Fri, Nov 13, 2026 at 02:38:58AM -0800, Breno Leitao wrote:
> > > @@ -40,6 +40,11 @@ module_param_named(suspend_pcr, tpm_suspend_pcr, uint, 0644);
> > >  MODULE_PARM_DESC(suspend_pcr,
> > >  		 "PCR to use for dummy writes to facilitate flush on suspend.");
> > >  
> > > +static unsigned int tpm_max_missed_responses = 60;
> > 
> > This looks good but I started to think this number. I guess it is high
> > because there's never reset-to-zero for chip->nr_missed_responses.
> >
> > I think three or four trials at most should be enough, and when there 
> > is a successful transmission chip->nr_missed_responses should reset.
> 
> It does reset. Any command that gets an answer, whatever its return
> code, sets the count back to zero:
> 
> 	if (ret >= 0) {
> 		chip->nr_missed_responses = 0;
> 
> so the TPM is only disabled after 60 timeouts with no answer in between.

Oops, sorry had a blind spot.

This made me thinking that it is unlikely that there would be a sequence
of -ETIME's and then things started suddenly working. It is not a likely
world scenario.

So really TPM_CHIP_FLAG_MISSED_RESPONSE could be the counter: it is set
when -ETIME happens, allowing a single -ETIME failure but not a sequence
of them.

Sorry, this simply did not cross mind earlier.

Second open question: do we want to disable or move into firmware
upgrade mode?

Stefan, could you review this patch set from this angle? I don't feel
I have enough expertise on this topic.


>     Some TPMs stop responding at runtime, hours to months after they
>     passed probe. The first sign is usually "invalid TPM_STS.x 0xff",
>     and from then on every command fails only after the transport
>     timeout. A TPM that is already dead at boot fails tpm2_probe() and
>     never registers, which is not affected by this patchset/feature.

This is much better. It is as accurate as possible, which is good
enough.

> 
> Thanks,
> --breno

Br, Jarkko