Re: [PATCH] KVM: arm64: Disable TRBE Trace Buffer Unit when running in guest context

[Marc Zyngier <maz@kernel.org>]

On Fri, 20 Feb 2026 11:42:11 +0000,
James Clark <james.clark@linaro.org> wrote:
> 
> 
> 
> On 16/02/2026 4:49 pm, Marc Zyngier wrote:
> > On Mon, 16 Feb 2026 16:10:14 +0000,
> > James Clark <james.clark@linaro.org> wrote:
> >> 
> >> 
> >> 
> >> On 16/02/2026 3:51 pm, Marc Zyngier wrote:
> >>> On Mon, 16 Feb 2026 15:05:10 +0000,
> >>> James Clark <james.clark@linaro.org> wrote:
> >>>> 
> >>>> 
> >>>> 
> >>>> On 16/02/2026 2:29 pm, Marc Zyngier wrote:
> >>>>> On Mon, 16 Feb 2026 13:09:59 +0000,
> >>>>> Will Deacon <will@kernel.org> wrote:
> >>>>>> 
> >>>>>> The nVHE world-switch code relies on zeroing TRFCR_EL1 to disable trace
> >>>>>> generation in guest context when self-hosted TRBE is in use by the host.
> >>>>>> 
> >>>>>> Per D3.2.1 ("Controls to prohibit trace at Exception levels"), clearing
> >>>>>> TRFCR_EL1 means that trace generation is prohibited at EL1 and EL0 but
> >>>>>> per R_YCHKJ the Trace Buffer Unit will still be enabled if
> >>>>>> TRBLIMITR_EL1.E is set. R_SJFRQ goes on to state that, when enabled, the
> >>>>>> Trace Buffer Unit can perform address translation for the "owning
> >>>>>> exception level" even when it is out of context.
> >>>>> 
> >>>>> Great. So TRBE violates all the principles that we hold true in the
> >>>>> architecture. Does SPE suffer from the same level of brokenness?
> >>>>> 
> >>>>>> Consequently, we can end up in a state where TRBE performs speculative
> >>>>>> page-table walks for a host VA/IPA in guest/hypervisor context depending
> >>>>>> on the value of MDCR_EL2.E2TB, which changes over world-switch. The
> >>>>>> result appears to be a heady mixture of data corruption and hardware
> >>>>>> lockups.
> >>>>>> 
> >>>>>> Extend the TRBE world-switch code to clear TRBLIMITR_EL1.E after
> >>>>>> draining the buffer, restoring the register on return to the host.
> >>>>>> 
> >>>>>> Cc: Marc Zyngier <maz@kernel.org>
> >>>>>> Cc: Oliver Upton <oupton@kernel.org>
> >>>>>> Cc: James Clark <james.clark@linaro.org>
> >>>>>> Cc: Leo Yan <leo.yan@arm.com>
> >>>>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
> >>>>>> Cc: Fuad Tabba <tabba@google.com>
> >>>>>> Fixes: a1319260bf62 ("arm64: KVM: Enable access to TRBE support for host")
> >>>>>> Signed-off-by: Will Deacon <will@kernel.org>
> >>>>>> ---
> >>>>>> 
> >>>>>> NOTE: This is *untested* as I don't have a TRBE-capable device that can
> >>>>>> run upstream but I noticed this by inspection when triaging occasional
> >>>>>> hardware lockups on systems using a 6.12-based kernel with TRBE running
> >>>>>> at the same time as a vCPU is loaded. This code has changed quite a bit
> >>>>>> over time, so stable backports are not entirely straightforward.
> >>>>>> Hopefully James/Leo/Suzuki can help us test if folks agree with the
> >>>>>> general approach taken here.
> >>>>>> 
> >>>>>>     arch/arm64/include/asm/kvm_host.h  |  1 +
> >>>>>>     arch/arm64/kvm/hyp/nvhe/debug-sr.c | 36 ++++++++++++++++++++++--------
> >>>>>>     2 files changed, 28 insertions(+), 9 deletions(-)
> >>>>>> 
> >>>>>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> >>>>>> index ac7f970c7883..a932cf043b83 100644
> >>>>>> --- a/arch/arm64/include/asm/kvm_host.h
> >>>>>> +++ b/arch/arm64/include/asm/kvm_host.h
> >>>>>> @@ -746,6 +746,7 @@ struct kvm_host_data {
> >>>>>>     		u64 pmscr_el1;
> >>>>>>     		/* Self-hosted trace */
> >>>>>>     		u64 trfcr_el1;
> >>>>>> +		u64 trblimitr_el1;
> >>>>>>     		/* Values of trap registers for the host before guest entry. */
> >>>>>>     		u64 mdcr_el2;
> >>>>>>     		u64 brbcr_el1;
> >>>>>> diff --git a/arch/arm64/kvm/hyp/nvhe/debug-sr.c b/arch/arm64/kvm/hyp/nvhe/debug-sr.c
> >>>>>> index 2a1c0f49792b..fd389a26bc59 100644
> >>>>>> --- a/arch/arm64/kvm/hyp/nvhe/debug-sr.c
> >>>>>> +++ b/arch/arm64/kvm/hyp/nvhe/debug-sr.c
> >>>>>> @@ -57,12 +57,27 @@ static void __trace_do_switch(u64 *saved_trfcr, u64 new_trfcr)
> >>>>>>     	write_sysreg_el1(new_trfcr, SYS_TRFCR);
> >>>>>>     }
> >>>>>>     -static bool __trace_needs_drain(void)
> >>>>>> +static void __trace_drain_and_disable(void)
> >>>>>>     {
> >>>>>> -	if (is_protected_kvm_enabled() && host_data_test_flag(HAS_TRBE))
> >>>>>> -		return read_sysreg_s(SYS_TRBLIMITR_EL1) & TRBLIMITR_EL1_E;
> >>>>>> +	u64 *trblimitr_el1 = host_data_ptr(host_debug_state.trblimitr_el1);
> >>>>>>     -	return host_data_test_flag(TRBE_ENABLED);
> >>>>>> +	*trblimitr_el1 = 0;
> >>>>>> +
> >>>>>> +	if (is_protected_kvm_enabled()) {
> >>>>>> +		if (!host_data_test_flag(HAS_TRBE))
> >>>>>> +			return;
> >>>>>> +	} else {
> >>>>>> +		if (!host_data_test_flag(TRBE_ENABLED))
> >>>>>> +			return;
> >>>>>> +	}
> >>>>>> +
> >>>>>> +	*trblimitr_el1 = read_sysreg_s(SYS_TRBLIMITR_EL1);
> >>>>>> +	if (*trblimitr_el1 & TRBLIMITR_EL1_E) {
> >>>>>> +		isb();
> >>>>>> +		tsb_csync();
> >>>>>> +		write_sysreg_s(0, SYS_TRBLIMITR_EL1);
> >>>>>> +		isb();
> >>>> 
> >>>> The TRBE driver might do an extra drain here as a workaround. Hard to
> >>>> tell if it's actually required in this case (seems like probably not)
> >>>> but it might be worth doing it anyway to avoid hitting the
> >>>> issue. Especially if we add guest support later where some of the
> >>>> affected registers might start being used.
> >>> 
> >>> Just to set the expectations: guest TRBE support is not happening
> >>> until the architecture is fixed. It cannot reliably give a trace that
> >>> includes emulated exceptions, and until then, no TRBE for you.
> >>> 
> >>>> See:
> >>>> 
> >>>>       if (trbe_needs_drain_after_disable(cpudata))
> >>>>           trbe_drain_buffer();
> >>>> 
> >>>> 
> >>>>>> +	}
> >>>>> 
> >>>>> Doesn't this mean we should be able to get rid of most of the TRFCR
> >>>>> messing about that litters the entry/exit code and leave that to VHE
> >>>> 
> >>>> Technically you could have ETMs that and are connected to sinks other
> >>>> than TRBE. Unless you somehow switch off those sinks you still need to
> >>>> do the TRFCR switching stuff.
> >>>> 
> >>>>> only? And even then, I'm tempted to simply get rid of any sort of
> >>>>> guest-only tracing, given that TRBE is not capable of representing
> >>>>> exceptions that are synthesised by the host, making it the resulting
> >>>>> traces useless.
> >>>> 
> >>>> I haven't heard of anyone tracing a guest from the host, but until we
> >>>> add support for guests to be able to trace themselves it's the only
> >>>> way of doing it, so it could be useful.
> >>> 
> >>> But that's *not* working. If you trace EL1 only, even with a VHE host,
> >>> the result is not usable.
> >>> 
> >> 
> >> Do you mean not working because of the missing exceptions? I did a bit
> >> of testing before and the trace did seem somewhat usable to me. It had
> >> EL1 and EL0 atoms in there.
> > 
> > Sure. Now try to look at what that means for NV, where all the
> > EL1->EL2 exceptions are emulated, where all the EL2->EL1 exception
> > returns are emulated.
> > 
> > What does it give you? A bag of nonsense.
> > 
> > Same thing for EL2->EL0, by the way, so you can't even correctly
> > profile an EL0 program that performs a syscall, or that gets
> > interrupted. And while without NV, these exceptions are rare, having a
> > trace that is unreliable has the potential of being worse than no
> > trace at all.
> 
> If there are issues with NV perhaps we can skip it for the initial
> trace virtualisation implementation?

No. This is broken for *any* hypervisor-generated exception.

> I'm not familiar with it but isn't NV still an experimental feature
> anyway?

Let me give you a clue: if I have to choose between TRBE and NV, it's
not TRBE I'm going to pick.

> I can't imagine actual users who want to do tracing in guests would
> accept that they can't do tracing on a non-NV guest because there is
> something that doesn't work in NV.

But that's the thing: they are not getting a trace. They are getting
nonsense.

> Also do you have an example of these exceptions that you mean without
> NV so I can have a look?

Anything that ends up in arch/arm64/kvm/hyp/exception.c, where the
exception is emulated by changing PC.

	M.

-- 
Without deviation from the norm, progress is not possible.