* [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working
@ 2026-11-20 20:47 Breno Leitao
2026-11-20 20:47 ` [PATCH v4 1/2] tpm: Disable a TPM that stops responding Breno Leitao
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Breno Leitao @ 2026-11-20 20:47 UTC (permalink / raw)
To: Peter Huewe, Jarkko Sakkinen, Jason Gunthorpe
Cc: linux-integrity, linux-kernel, surenkj, noodles, David Laight,
Stefan Mahnke-Hartmann, Breno Leitao, kernel-team
On Meta's datacenter, given the size, TPMs that stop working write about
54 million lines a day to the kernel log, because nothing in the kernel
gives up on them.
Two cases make up nearly all of it.
Some TPMs stop responding. Every command then fails only after the
transport timeout, while the hwrng thread and userspace keep sending
more.
Other TPMs get reset under the running OS, and answer every command with
TPM_RC_INITIALIZE (256) from then on, while the hwrng thread retries
every 10 seconds:
tpm tpm0: A TPM error (256) occurred attempting get random
The goal is to retire TPMs that are not properly working, similarly to
what was done for EFI in commit 4b2c033b5bc971 ("efi/runtime-wrappers:
retire the worker if a wedged call ever returns")
Patch 1 disables the chip once two commands in a row have timed out,
instead of keeping it in this desolate state. Patch 2, from Jarkko,
disables it once it answers TPM_RC_INITIALIZE to anything but
TPM2_Startup.
Disabling the chip makes the failure explicit: every later command
fails at once instead of timing out.
A health-check daemon, such as machine checker, can then detect
the broken hardware and send the machine to repair.
Signed-off-by: Breno Leitao <leitao@debian.org>
---
Changes in v4:
- Replace the count of timeouts and the tpm.max_missed_responses
parameter with TPM_CHIP_FLAG_MISSED_RESPONSE: one timeout is
tolerated, and a second one in a row disables the TPM (Jarkko)
- Say in patch 1 when TPMs stop responding (Jarkko), and drop the
claim that some of them recover
- Fix sashiko findings:
- A timeout while a signal is pending no longer counts, since some
drivers return -ETIME when a signal interrupts the wait
- tpm1_do_selftest() clears the flag on the timeouts it expects from
TPM 1.2 chips that block during the self test
- Link to v3: https://patch.msgid.link/20261113-tpm_unresponsive-v3-0-8138c8e283b8@debian.org
Changes in v3:
- Rename tpm.max_unanswered to tpm.max_missed_responses and
nr_unanswered to nr_missed_responses, and drop a redundant comment
(Jarkko)
- Document tpm.max_missed_responses in kernel-parameters.txt (Jarkko)
- Check the limit with a modulo (Jarkko), guarded so that 0 still
means never
- Link to v2: https://patch.msgid.link/20261112-tpm_unresponsive-v2-0-7454d2c899ef@debian.org
Changes in v2:
- Disable the TPM after 60 commands in a row time out, instead of after
ten minutes of timeouts, and make the limit the tpm.max_unanswered
boot parameter (Jarkko)
- Open-code the check in tpm_transmit() and drop the "timeout" field
names (Jarkko)
- Replace the hwrng failure counter with Jarkko's patch, which disables
the TPM on TPM_RC_INITIALIZE
- In that patch, move the check ahead of the RETRY/TESTING break, where
it never ran, and skip TPM2_Startup, which a TPM that is already
started fails with TPM_RC_INITIALIZE
- Fix sashiko findings:
- Two timeouts ten minutes apart no longer disable an idle TPM
- Link to v1: https://patch.msgid.link/20261111-tpm_unresponsive-v1-0-42ac1b965eb9@debian.org
---
Breno Leitao (1):
tpm: Disable a TPM that stops responding
Jarkko Sakkinen (1):
tpm: Disable TPM on TPM_RC_INITIALIZE
drivers/char/tpm/tpm-interface.c | 20 ++++++++++++++++++++
drivers/char/tpm/tpm1-cmd.c | 1 +
drivers/char/tpm/tpm2-cmd.c | 2 ++
include/linux/tpm.h | 1 +
4 files changed, 24 insertions(+)
---
base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47
change-id: 20261110-tpm_unresponsive-a5f71c1db026
Best regards,
--
Breno Leitao <leitao@debian.org>
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v4 1/2] tpm: Disable a TPM that stops responding
2026-11-20 20:47 [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Breno Leitao
@ 2026-11-20 20:47 ` Breno Leitao
2026-11-20 20:47 ` [PATCH v4 2/2] tpm: Disable TPM on TPM_RC_INITIALIZE Breno Leitao
2026-11-20 21:29 ` [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Jarkko Sakkinen
2 siblings, 0 replies; 5+ messages in thread
From: Breno Leitao @ 2026-11-20 20:47 UTC (permalink / raw)
To: Peter Huewe, Jarkko Sakkinen, Jason Gunthorpe
Cc: linux-integrity, linux-kernel, surenkj, noodles, David Laight,
Stefan Mahnke-Hartmann, Breno Leitao, kernel-team
Some TPMs stop responding at runtime, hours to months after they passed
probe. The first sign is usually "invalid TPM_STS.x 0xff", and from then
on every command fails only after the transport timeout. A TPM that is
already dead at boot fails to probe and never registers, which this
patch does not change.
The callers keep sending more commands: the hwrng thread every 10
seconds forever, and userspace through /dev/tpmrm0. Each attempt holds
tpm_mutex for the whole timeout and logs an error:
tpm tpm0: invalid TPM_STS.x 0xff, dumping stack for forensics
tpm tpm0: tpm_try_transmit: send(): error -62
tpm tpm0: tpm_try_transmit: send(): error -62
...
The core already stops using a TPM it cannot trust: tpm_try_get_ops()
fails with -EIO once TPM_CHIP_FLAG_DISABLE is set, which
tpm2_load_null() does when it cannot restore the null key. Set the same
flag when two commands in a row time out.
The first timeout sets TPM_CHIP_FLAG_MISSED_RESPONSE, and a command that
gets an answer, whatever its TPM return code, clears it.
With interrupts, the wait helpers in tpm_tis_core, st33zp24 and
xen-tpmfront return -ETIME as soon as a signal is pending, so an -ETIME
with a signal pending is not counted. Otherwise any user could disable
the TPM by interrupting two reads of a PCR file in sysfs.
Some TPM 1.2 chips do not get ready while their self test runs, and
tpm1_do_selftest() keeps polling through those timeouts, see
commit 4643826a3da1 ("TPM: Work around buggy TPMs that block during
continue self test"). It now clears the flag after each of them.
A disabled TPM stays disabled until the driver is rebound or the machine
reboots.
Suggested-by: Jarkko Sakkinen <jarkko@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
drivers/char/tpm/tpm-interface.c | 15 +++++++++++++++
drivers/char/tpm/tpm1-cmd.c | 1 +
include/linux/tpm.h | 1 +
3 files changed, 17 insertions(+)
diff --git a/drivers/char/tpm/tpm-interface.c b/drivers/char/tpm/tpm-interface.c
index e0385b16616b46..cb794cfc5f63e8 100644
--- a/drivers/char/tpm/tpm-interface.c
+++ b/drivers/char/tpm/tpm-interface.c
@@ -26,6 +26,7 @@
#include <linux/spinlock.h>
#include <linux/suspend.h>
#include <linux/freezer.h>
+#include <linux/sched/signal.h>
#include <linux/tpm_eventlog.h>
#include "tpm.h"
@@ -268,6 +269,20 @@ ssize_t tpm_transmit(struct tpm_chip *chip, u8 *buf, size_t bufsiz)
delay_msec *= 2;
memcpy(buf, save, save_size);
}
+
+ if (ret >= 0)
+ chip->flags &= ~TPM_CHIP_FLAG_MISSED_RESPONSE;
+
+ /* Some drivers also return -ETIME when a signal interrupts the wait. */
+ if (ret == -ETIME && !signal_pending(current)) {
+ if (!(chip->flags & TPM_CHIP_FLAG_MISSED_RESPONSE)) {
+ chip->flags |= TPM_CHIP_FLAG_MISSED_RESPONSE;
+ } else if (!(chip->flags & TPM_CHIP_FLAG_DISABLE)) {
+ dev_err(&chip->dev,
+ "two commands in a row timed out, disabling the TPM\n");
+ chip->flags |= TPM_CHIP_FLAG_DISABLE;
+ }
+ }
return ret;
}
diff --git a/drivers/char/tpm/tpm1-cmd.c b/drivers/char/tpm/tpm1-cmd.c
index 7a8606e8c7117b..e4e1f0799da196 100644
--- a/drivers/char/tpm/tpm1-cmd.c
+++ b/drivers/char/tpm/tpm1-cmd.c
@@ -647,6 +647,7 @@ int tpm1_do_selftest(struct tpm_chip *chip)
* until the self test duration expires.
*/
if (rc == -ETIME) {
+ chip->flags &= ~TPM_CHIP_FLAG_MISSED_RESPONSE;
dev_info(&chip->dev, HW_ERR "TPM command timed out during continue self test");
tpm_msleep(delay_msec);
continue;
diff --git a/include/linux/tpm.h b/include/linux/tpm.h
index 0db277af45c393..0f97abc82846ae 100644
--- a/include/linux/tpm.h
+++ b/include/linux/tpm.h
@@ -197,6 +197,7 @@ enum tpm_chip_flags {
TPM_CHIP_FLAG_HWRNG_DISABLED = BIT(9),
TPM_CHIP_FLAG_DISABLE = BIT(10),
TPM_CHIP_FLAG_SYNC = BIT(11),
+ TPM_CHIP_FLAG_MISSED_RESPONSE = BIT(12),
};
#define to_tpm_chip(d) container_of(d, struct tpm_chip, dev)
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v4 2/2] tpm: Disable TPM on TPM_RC_INITIALIZE
2026-11-20 20:47 [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Breno Leitao
2026-11-20 20:47 ` [PATCH v4 1/2] tpm: Disable a TPM that stops responding Breno Leitao
@ 2026-11-20 20:47 ` Breno Leitao
2026-11-20 21:29 ` [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Jarkko Sakkinen
2 siblings, 0 replies; 5+ messages in thread
From: Breno Leitao @ 2026-11-20 20:47 UTC (permalink / raw)
To: Peter Huewe, Jarkko Sakkinen, Jason Gunthorpe
Cc: linux-integrity, linux-kernel, surenkj, noodles, David Laight,
Stefan Mahnke-Hartmann, Breno Leitao, kernel-team
From: Jarkko Sakkinen <jarkko@kernel.org>
If the TPM has been spuriously disabled, the game is over. Let's reflect
this truth with TPM_CHIP_FLAG_DISABLE.
Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
[leitao@debian.org: check rc before the loop breaks out on anything but
RETRY/TESTING, skip TPM2_Startup, which a TPM that is already started
fails with TPM2_RC_INITIALIZE, and fix the TPM2_RC_INITIALIZE and
chip->flags typos]
Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
drivers/char/tpm/tpm-interface.c | 5 +++++
drivers/char/tpm/tpm2-cmd.c | 2 ++
2 files changed, 7 insertions(+)
diff --git a/drivers/char/tpm/tpm-interface.c b/drivers/char/tpm/tpm-interface.c
index cb794cfc5f63e8..2ea10fd89a70f1 100644
--- a/drivers/char/tpm/tpm-interface.c
+++ b/drivers/char/tpm/tpm-interface.c
@@ -248,6 +248,11 @@ ssize_t tpm_transmit(struct tpm_chip *chip, u8 *buf, size_t bufsiz)
if (ret < 0)
break;
rc = be32_to_cpu(header->return_code);
+ /* A TPM that is already started fails TPM2_Startup this way. */
+ if (rc == TPM2_RC_INITIALIZE && cc != TPM2_CC_STARTUP) {
+ chip->flags |= TPM_CHIP_FLAG_DISABLE;
+ break;
+ }
if (rc != TPM2_RC_RETRY && rc != TPM2_RC_TESTING)
break;
/*
diff --git a/drivers/char/tpm/tpm2-cmd.c b/drivers/char/tpm/tpm2-cmd.c
index 4b2bb2ddc321a9..65928ae85b496c 100644
--- a/drivers/char/tpm/tpm2-cmd.c
+++ b/drivers/char/tpm/tpm2-cmd.c
@@ -724,6 +724,8 @@ int tpm2_auto_startup(struct tpm_chip *chip)
goto out;
if (rc == TPM2_RC_INITIALIZE) {
+ chip->flags &= ~TPM_CHIP_FLAG_DISABLE;
+
rc = tpm2_startup(chip);
if (rc)
goto out;
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working
2026-11-20 20:47 [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Breno Leitao
2026-11-20 20:47 ` [PATCH v4 1/2] tpm: Disable a TPM that stops responding Breno Leitao
2026-11-20 20:47 ` [PATCH v4 2/2] tpm: Disable TPM on TPM_RC_INITIALIZE Breno Leitao
@ 2026-11-20 21:29 ` Jarkko Sakkinen
2026-11-20 21:30 ` Jarkko Sakkinen
2 siblings, 1 reply; 5+ messages in thread
From: Jarkko Sakkinen @ 2026-11-20 21:29 UTC (permalink / raw)
To: Breno Leitao
Cc: Peter Huewe, Jason Gunthorpe, linux-integrity, linux-kernel,
surenkj, noodles, David Laight, Stefan Mahnke-Hartmann,
kernel-team
On Fri, Nov 20, 2026 at 12:47:06PM -0800, Breno Leitao wrote:
> On Meta's datacenter, given the size, TPMs that stop working write about
> 54 million lines a day to the kernel log, because nothing in the kernel
> gives up on them.
>
> Two cases make up nearly all of it.
>
> Some TPMs stop responding. Every command then fails only after the
> transport timeout, while the hwrng thread and userspace keep sending
> more.
>
> Other TPMs get reset under the running OS, and answer every command with
> TPM_RC_INITIALIZE (256) from then on, while the hwrng thread retries
> every 10 seconds:
>
> tpm tpm0: A TPM error (256) occurred attempting get random
>
> The goal is to retire TPMs that are not properly working, similarly to
> what was done for EFI in commit 4b2c033b5bc971 ("efi/runtime-wrappers:
> retire the worker if a wedged call ever returns")
>
> Patch 1 disables the chip once two commands in a row have timed out,
> instead of keeping it in this desolate state. Patch 2, from Jarkko,
> disables it once it answers TPM_RC_INITIALIZE to anything but
> TPM2_Startup.
>
> Disabling the chip makes the failure explicit: every later command
> fails at once instead of timing out.
>
> A health-check daemon, such as machine checker, can then detect
> the broken hardware and send the machine to repair.
>
> Signed-off-by: Breno Leitao <leitao@debian.org>
> ---
> Changes in v4:
> - Replace the count of timeouts and the tpm.max_missed_responses
> parameter with TPM_CHIP_FLAG_MISSED_RESPONSE: one timeout is
> tolerated, and a second one in a row disables the TPM (Jarkko)
> - Say in patch 1 when TPMs stop responding (Jarkko), and drop the
> claim that some of them recover
> - Fix sashiko findings:
> - A timeout while a signal is pending no longer counts, since some
> drivers return -ETIME when a signal interrupts the wait
> - tpm1_do_selftest() clears the flag on the timeouts it expects from
> TPM 1.2 chips that block during the self test
> - Link to v3: https://patch.msgid.link/20261113-tpm_unresponsive-v3-0-8138c8e283b8@debian.org
>
> Changes in v3:
> - Rename tpm.max_unanswered to tpm.max_missed_responses and
> nr_unanswered to nr_missed_responses, and drop a redundant comment
> (Jarkko)
> - Document tpm.max_missed_responses in kernel-parameters.txt (Jarkko)
> - Check the limit with a modulo (Jarkko), guarded so that 0 still
> means never
> - Link to v2: https://patch.msgid.link/20261112-tpm_unresponsive-v2-0-7454d2c899ef@debian.org
>
> Changes in v2:
> - Disable the TPM after 60 commands in a row time out, instead of after
> ten minutes of timeouts, and make the limit the tpm.max_unanswered
> boot parameter (Jarkko)
> - Open-code the check in tpm_transmit() and drop the "timeout" field
> names (Jarkko)
> - Replace the hwrng failure counter with Jarkko's patch, which disables
> the TPM on TPM_RC_INITIALIZE
> - In that patch, move the check ahead of the RETRY/TESTING break, where
> it never ran, and skip TPM2_Startup, which a TPM that is already
> started fails with TPM_RC_INITIALIZE
> - Fix sashiko findings:
> - Two timeouts ten minutes apart no longer disable an idle TPM
> - Link to v1: https://patch.msgid.link/20261111-tpm_unresponsive-v1-0-42ac1b965eb9@debian.org
>
> ---
> Breno Leitao (1):
> tpm: Disable a TPM that stops responding
>
> Jarkko Sakkinen (1):
> tpm: Disable TPM on TPM_RC_INITIALIZE
>
> drivers/char/tpm/tpm-interface.c | 20 ++++++++++++++++++++
> drivers/char/tpm/tpm1-cmd.c | 1 +
> drivers/char/tpm/tpm2-cmd.c | 2 ++
> include/linux/tpm.h | 1 +
> 4 files changed, 24 insertions(+)
> ---
> base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47
> change-id: 20261110-tpm_unresponsive-a5f71c1db026
>
> Best regards,
> --
> Breno Leitao <leitao@debian.org>
>
Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>
Br, Jarkko
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working
2026-11-20 21:29 ` [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Jarkko Sakkinen
@ 2026-11-20 21:30 ` Jarkko Sakkinen
0 siblings, 0 replies; 5+ messages in thread
From: Jarkko Sakkinen @ 2026-11-20 21:30 UTC (permalink / raw)
To: Breno Leitao
Cc: Peter Huewe, Jason Gunthorpe, linux-integrity, linux-kernel,
surenkj, noodles, David Laight, Stefan Mahnke-Hartmann,
kernel-team
On Fri, Nov 20, 2026 at 11:29:56PM +0200, Jarkko Sakkinen wrote:
> On Fri, Nov 20, 2026 at 12:47:06PM -0800, Breno Leitao wrote:
> > On Meta's datacenter, given the size, TPMs that stop working write about
> > 54 million lines a day to the kernel log, because nothing in the kernel
> > gives up on them.
> >
> > Two cases make up nearly all of it.
> >
> > Some TPMs stop responding. Every command then fails only after the
> > transport timeout, while the hwrng thread and userspace keep sending
> > more.
> >
> > Other TPMs get reset under the running OS, and answer every command with
> > TPM_RC_INITIALIZE (256) from then on, while the hwrng thread retries
> > every 10 seconds:
> >
> > tpm tpm0: A TPM error (256) occurred attempting get random
> >
> > The goal is to retire TPMs that are not properly working, similarly to
> > what was done for EFI in commit 4b2c033b5bc971 ("efi/runtime-wrappers:
> > retire the worker if a wedged call ever returns")
> >
> > Patch 1 disables the chip once two commands in a row have timed out,
> > instead of keeping it in this desolate state. Patch 2, from Jarkko,
> > disables it once it answers TPM_RC_INITIALIZE to anything but
> > TPM2_Startup.
> >
> > Disabling the chip makes the failure explicit: every later command
> > fails at once instead of timing out.
> >
> > A health-check daemon, such as machine checker, can then detect
> > the broken hardware and send the machine to repair.
> >
> > Signed-off-by: Breno Leitao <leitao@debian.org>
> > ---
> > Changes in v4:
> > - Replace the count of timeouts and the tpm.max_missed_responses
> > parameter with TPM_CHIP_FLAG_MISSED_RESPONSE: one timeout is
> > tolerated, and a second one in a row disables the TPM (Jarkko)
> > - Say in patch 1 when TPMs stop responding (Jarkko), and drop the
> > claim that some of them recover
> > - Fix sashiko findings:
> > - A timeout while a signal is pending no longer counts, since some
> > drivers return -ETIME when a signal interrupts the wait
> > - tpm1_do_selftest() clears the flag on the timeouts it expects from
> > TPM 1.2 chips that block during the self test
> > - Link to v3: https://patch.msgid.link/20261113-tpm_unresponsive-v3-0-8138c8e283b8@debian.org
> >
> > Changes in v3:
> > - Rename tpm.max_unanswered to tpm.max_missed_responses and
> > nr_unanswered to nr_missed_responses, and drop a redundant comment
> > (Jarkko)
> > - Document tpm.max_missed_responses in kernel-parameters.txt (Jarkko)
> > - Check the limit with a modulo (Jarkko), guarded so that 0 still
> > means never
> > - Link to v2: https://patch.msgid.link/20261112-tpm_unresponsive-v2-0-7454d2c899ef@debian.org
> >
> > Changes in v2:
> > - Disable the TPM after 60 commands in a row time out, instead of after
> > ten minutes of timeouts, and make the limit the tpm.max_unanswered
> > boot parameter (Jarkko)
> > - Open-code the check in tpm_transmit() and drop the "timeout" field
> > names (Jarkko)
> > - Replace the hwrng failure counter with Jarkko's patch, which disables
> > the TPM on TPM_RC_INITIALIZE
> > - In that patch, move the check ahead of the RETRY/TESTING break, where
> > it never ran, and skip TPM2_Startup, which a TPM that is already
> > started fails with TPM_RC_INITIALIZE
> > - Fix sashiko findings:
> > - Two timeouts ten minutes apart no longer disable an idle TPM
> > - Link to v1: https://patch.msgid.link/20261111-tpm_unresponsive-v1-0-42ac1b965eb9@debian.org
> >
> > ---
> > Breno Leitao (1):
> > tpm: Disable a TPM that stops responding
> >
> > Jarkko Sakkinen (1):
> > tpm: Disable TPM on TPM_RC_INITIALIZE
> >
> > drivers/char/tpm/tpm-interface.c | 20 ++++++++++++++++++++
> > drivers/char/tpm/tpm1-cmd.c | 1 +
> > drivers/char/tpm/tpm2-cmd.c | 2 ++
> > include/linux/tpm.h | 1 +
> > 4 files changed, 24 insertions(+)
> > ---
> > base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47
> > change-id: 20261110-tpm_unresponsive-a5f71c1db026
> >
> > Best regards,
> > --
> > Breno Leitao <leitao@debian.org>
> >
>
> Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>
It's considered enough. If a tree falls somewhere because of this, we
deal with that separately (and it's just life).
Br, Jarkko
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-20 21:30 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 20:47 [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Breno Leitao
2026-11-20 20:47 ` [PATCH v4 1/2] tpm: Disable a TPM that stops responding Breno Leitao
2026-11-20 20:47 ` [PATCH v4 2/2] tpm: Disable TPM on TPM_RC_INITIALIZE Breno Leitao
2026-11-20 21:29 ` [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Jarkko Sakkinen
2026-11-20 21:30 ` Jarkko Sakkinen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.