All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jarkko Sakkinen <jarkko@kernel.org>
To: Breno Leitao <leitao@debian.org>
Cc: Peter Huewe <peterhuewe@gmx.de>, Jason Gunthorpe <jgg@ziepe.ca>,
	linux-integrity@vger.kernel.org, linux-kernel@vger.kernel.org,
	surenkj@amazon.com, noodles@meta.com,
	David Laight <david.laight.linux@gmail.com>,
	Stefan Mahnke-Hartmann <stefan.mahnke-hartmann@infineon.com>,
	kernel-team@meta.com
Subject: Re: [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working
Date: Fri, 20 Nov 2026 23:30:55 +0200	[thread overview]
Message-ID: <awC8D-17ShOMXkp2@kernel.org> (raw)
In-Reply-To: <awC70MnZ4eXVQrdS@kernel.org>

On Fri, Nov 20, 2026 at 11:29:56PM +0200, Jarkko Sakkinen wrote:
> On Fri, Nov 20, 2026 at 12:47:06PM -0800, Breno Leitao wrote:
> > On Meta's datacenter, given the size, TPMs that stop working write about
> > 54 million lines a day to the kernel log, because nothing in the kernel
> > gives up on them.
> > 
> > Two cases make up nearly all of it.
> > 
> > Some TPMs stop responding. Every command then fails only after the
> > transport timeout, while the hwrng thread and userspace keep sending
> > more.
> > 
> > Other TPMs get reset under the running OS, and answer every command with
> > TPM_RC_INITIALIZE (256) from then on, while the hwrng thread retries
> > every 10 seconds:
> > 
> >   tpm tpm0: A TPM error (256) occurred attempting get random
> > 
> > The goal is to retire TPMs that are not properly working, similarly to
> > what was done for EFI in commit 4b2c033b5bc971 ("efi/runtime-wrappers:
> > retire the worker if a wedged call ever returns")
> > 
> > Patch 1 disables the chip once two commands in a row have timed out,
> > instead of keeping it in this desolate state. Patch 2, from Jarkko,
> > disables it once it answers TPM_RC_INITIALIZE to anything but
> > TPM2_Startup.
> > 
> > Disabling the chip makes the failure explicit: every later command
> > fails at once instead of timing out.
> > 
> > A health-check daemon, such as machine checker, can then detect
> > the broken hardware and send the machine to repair.
> > 
> > Signed-off-by: Breno Leitao <leitao@debian.org>
> > ---
> > Changes in v4:
> > - Replace the count of timeouts and the tpm.max_missed_responses
> >   parameter with TPM_CHIP_FLAG_MISSED_RESPONSE: one timeout is
> >   tolerated, and a second one in a row disables the TPM (Jarkko)
> > - Say in patch 1 when TPMs stop responding (Jarkko), and drop the
> >   claim that some of them recover
> > - Fix sashiko findings:
> >   - A timeout while a signal is pending no longer counts, since some
> >     drivers return -ETIME when a signal interrupts the wait
> >   - tpm1_do_selftest() clears the flag on the timeouts it expects from
> >     TPM 1.2 chips that block during the self test
> > - Link to v3: https://patch.msgid.link/20261113-tpm_unresponsive-v3-0-8138c8e283b8@debian.org
> > 
> > Changes in v3:
> > - Rename tpm.max_unanswered to tpm.max_missed_responses and
> >   nr_unanswered to nr_missed_responses, and drop a redundant comment
> >   (Jarkko)
> > - Document tpm.max_missed_responses in kernel-parameters.txt (Jarkko)
> > - Check the limit with a modulo (Jarkko), guarded so that 0 still
> >   means never
> > - Link to v2: https://patch.msgid.link/20261112-tpm_unresponsive-v2-0-7454d2c899ef@debian.org
> > 
> > Changes in v2:
> > - Disable the TPM after 60 commands in a row time out, instead of after
> >   ten minutes of timeouts, and make the limit the tpm.max_unanswered
> >   boot parameter (Jarkko)
> > - Open-code the check in tpm_transmit() and drop the "timeout" field
> >   names (Jarkko)
> > - Replace the hwrng failure counter with Jarkko's patch, which disables
> >   the TPM on TPM_RC_INITIALIZE
> > - In that patch, move the check ahead of the RETRY/TESTING break, where
> >   it never ran, and skip TPM2_Startup, which a TPM that is already
> >   started fails with TPM_RC_INITIALIZE
> > - Fix sashiko findings:
> >   - Two timeouts ten minutes apart no longer disable an idle TPM
> > - Link to v1: https://patch.msgid.link/20261111-tpm_unresponsive-v1-0-42ac1b965eb9@debian.org
> > 
> > ---
> > Breno Leitao (1):
> >       tpm: Disable a TPM that stops responding
> > 
> > Jarkko Sakkinen (1):
> >       tpm: Disable TPM on TPM_RC_INITIALIZE
> > 
> >  drivers/char/tpm/tpm-interface.c | 20 ++++++++++++++++++++
> >  drivers/char/tpm/tpm1-cmd.c      |  1 +
> >  drivers/char/tpm/tpm2-cmd.c      |  2 ++
> >  include/linux/tpm.h              |  1 +
> >  4 files changed, 24 insertions(+)
> > ---
> > base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47
> > change-id: 20261110-tpm_unresponsive-a5f71c1db026
> > 
> > Best regards,
> > --  
> > Breno Leitao <leitao@debian.org>
> > 
> 
> Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>

It's considered enough. If a tree falls somewhere because of this, we
deal with that separately (and it's just life).

Br, Jarkko

      reply	other threads:[~2026-11-20 21:30 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 20:47 [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Breno Leitao
2026-11-20 20:47 ` [PATCH v4 1/2] tpm: Disable a TPM that stops responding Breno Leitao
2026-11-20 20:47 ` [PATCH v4 2/2] tpm: Disable TPM on TPM_RC_INITIALIZE Breno Leitao
2026-11-20 21:29 ` [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Jarkko Sakkinen
2026-11-20 21:30   ` Jarkko Sakkinen [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=awC8D-17ShOMXkp2@kernel.org \
    --to=jarkko@kernel.org \
    --cc=david.laight.linux@gmail.com \
    --cc=jgg@ziepe.ca \
    --cc=kernel-team@meta.com \
    --cc=leitao@debian.org \
    --cc=linux-integrity@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=noodles@meta.com \
    --cc=peterhuewe@gmx.de \
    --cc=stefan.mahnke-hartmann@infineon.com \
    --cc=surenkj@amazon.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.