Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

[Tangnianyao]

Hi, all

I'm trying to understand the TLB and I-cache invalidation in
`kvm_arch_vcpu_load()` that is intended to "guarantee that both TLBs and
I-cache are private to each vCPU".

As I understand it, when `VTTBR_EL2.CnP == 1`, `__kvm_flush_cpu_context()`
only performs a local TLB and I-cache invalidation, which does not seem
sufficient to guarantee that property.

In fact, even if the invalidation were extended to the Inner Shareable
domain, it still seems difficult to guarantee “TLBs and I-cache are
private to each vCPU”, when `VTTBR_EL2.CnP == 1`, as long as multiple
vCPUs from the same VM may be running concurrently on different PEs.

So I have two questions:

1. What is the rationale behind the comment that "guarantee that both TLBs
and I-cache are private to each vCPU"?
2. Should the effect of `VTTBR_EL2.CnP` be taken into account when
reasoning about this guarantee?

Thanks
Nianyao Tang

Re: Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

[Wei-Lin Chang]

Hi,

Let me try to answer this:

On Sat, Jul 04, 2026 at 03:45:56PM +0800, Tangnianyao wrote:
> Hi, all
> 
> I'm trying to understand the TLB and I-cache invalidation in
> `kvm_arch_vcpu_load()` that is intended to "guarantee that both TLBs and
> I-cache are private to each vCPU".
> 
> As I understand it, when `VTTBR_EL2.CnP == 1`, `__kvm_flush_cpu_context()`
> only performs a local TLB and I-cache invalidation, which does not seem
> sufficient to guarantee that property.
> 
> In fact, even if the invalidation were extended to the Inner Shareable
> domain, it still seems difficult to guarantee “TLBs and I-cache are
> private to each vCPU”, when `VTTBR_EL2.CnP == 1`, as long as multiple
> vCPUs from the same VM may be running concurrently on different PEs.

I think you have missed that when 2 stages are involved, both stages
have to set CnP == 1 in order to share TLB entries (Arm ARM R_ZVRZW).
So if TLB entry sharing happens, the guest kernel must have allowed it
in the first place (by setting TTBR0/1_EL1.CnP == 1), hence accidental
sharing that you are worried about won't happen.

__kvm_flush_cpu_context() is solving problems that occur when multiple
vCPUs of a VM are multiplexed on a single physical CPU.

> 
> So I have two questions:
> 
> 1. What is the rationale behind the comment that "guarantee that both TLBs
> and I-cache are private to each vCPU"?

I assume you are asking why keeping both TLBs and I-cache private per
each vCPU is required. The fundamental answer is that each physical CPU
is expected to have its own TLB and I-cache, so we must uphold that
property for vCPUs as well. vCPUs can be scheduled on the same physical
CPU, and use the same physical TLB/I-cache, obviously, so extra
invalidations need to be done.

As for how this can go wrong without __kvm_flush_cpu_context(), you can
look at commit 94d0e5980d67 ("arm/arm64: KVM: Perform local TLB
invalidation when multiplexing vcpus on a single CPU") and commit
01dc9262ff57 ("VM: arm64: Ensure I-cache isolation between vcpus of a
same VM"). The commit messages and the linked thread explained the
problems pretty well for me.

> 2. Should the effect of `VTTBR_EL2.CnP` be taken into account when
> reasoning about this guarantee?

Please see the first part of the answer.

Hope this helps!

Thanks,
Wei-Lin Chang

> 
> Thanks
> Nianyao Tang
> 

Re: Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

[Tangnianyao]

On 7/6/2026 1:28, Wei-Lin Chang wrote:
> Hi,
>
> Let me try to answer this:
>
> On Sat, Jul 04, 2026 at 03:45:56PM +0800, Tangnianyao wrote:
>> Hi, all
>>
>> I'm trying to understand the TLB and I-cache invalidation in
>> `kvm_arch_vcpu_load()` that is intended to "guarantee that both TLBs and
>> I-cache are private to each vCPU".
>>
>> As I understand it, when `VTTBR_EL2.CnP == 1`, `__kvm_flush_cpu_context()`
>> only performs a local TLB and I-cache invalidation, which does not seem
>> sufficient to guarantee that property.
>>
>> In fact, even if the invalidation were extended to the Inner Shareable
>> domain, it still seems difficult to guarantee “TLBs and I-cache are
>> private to each vCPU”, when `VTTBR_EL2.CnP == 1`, as long as multiple
>> vCPUs from the same VM may be running concurrently on different PEs.
> I think you have missed that when 2 stages are involved, both stages
> have to set CnP == 1 in order to share TLB entries (Arm ARM R_ZVRZW).
> So if TLB entry sharing happens, the guest kernel must have allowed it
> in the first place (by setting TTBR0/1_EL1.CnP == 1), hence accidental
> sharing that you are worried about won't happen.
>
> __kvm_flush_cpu_context() is solving problems that occur when multiple
> vCPUs of a VM are multiplexed on a single physical CPU.

Thanks for you answer.

If guest kernel allow TLB shared across CPUs by setting TTBR0/1_EL1.CnP == 1,
does kvm still need to guarantee that TLBs are private to each vCPU?
 

>> So I have two questions:
>>
>> 1. What is the rationale behind the comment that "guarantee that both TLBs
>> and I-cache are private to each vCPU"?
> I assume you are asking why keeping both TLBs and I-cache private per
> each vCPU is required. The fundamental answer is that each physical CPU
> is expected to have its own TLB and I-cache, so we must uphold that
> property for vCPUs as well. vCPUs can be scheduled on the same physical
> CPU, and use the same physical TLB/I-cache, obviously, so extra
> invalidations need to be done.
Let's assume that both Stage-1 CnP and Stage-2 CnP are enabled.

As I understand it, the architecture permits TLB to be shared by multiple
PEs within an Inner Shareable domain. Right?

If an implementation allows TLB entries to be shared in this way, it seems
that the current invalidation performed by kvm would no longer be sufficient
to guarantee that TLBs are private to each vCPU.

>
> As for how this can go wrong without __kvm_flush_cpu_context(), you can
> look at commit 94d0e5980d67 ("arm/arm64: KVM: Perform local TLB
> invalidation when multiplexing vcpus on a single CPU") and commit
> 01dc9262ff57 ("VM: arm64: Ensure I-cache isolation between vcpus of a
> same VM"). The commit messages and the linked thread explained the
> problems pretty well for me.
>
>> 2. Should the effect of `VTTBR_EL2.CnP` be taken into account when
>> reasoning about this guarantee?
> Please see the first part of the answer.
>
> Hope this helps!
>
> Thanks,
> Wei-Lin Chang
>
>> Thanks
>> Nianyao Tang
>>
> .
>
Thanks
Nianyao Tang

Re: Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

[Marc Zyngier]

On Mon, 06 Jul 2026 04:30:30 +0100,
Tangnianyao <tangnianyao@huawei.com> wrote:
> 
> 
> 
> On 7/6/2026 1:28, Wei-Lin Chang wrote:
> > Hi,
> >
> > Let me try to answer this:
> >
> > On Sat, Jul 04, 2026 at 03:45:56PM +0800, Tangnianyao wrote:
> >> Hi, all
> >>
> >> I'm trying to understand the TLB and I-cache invalidation in
> >> `kvm_arch_vcpu_load()` that is intended to "guarantee that both TLBs and
> >> I-cache are private to each vCPU".
> >>
> >> As I understand it, when `VTTBR_EL2.CnP == 1`, `__kvm_flush_cpu_context()`
> >> only performs a local TLB and I-cache invalidation, which does not seem
> >> sufficient to guarantee that property.
> >>
> >> In fact, even if the invalidation were extended to the Inner Shareable
> >> domain, it still seems difficult to guarantee “TLBs and I-cache are
> >> private to each vCPU”, when `VTTBR_EL2.CnP == 1`, as long as multiple
> >> vCPUs from the same VM may be running concurrently on different PEs.
> > I think you have missed that when 2 stages are involved, both stages
> > have to set CnP == 1 in order to share TLB entries (Arm ARM R_ZVRZW).
> > So if TLB entry sharing happens, the guest kernel must have allowed it
> > in the first place (by setting TTBR0/1_EL1.CnP == 1), hence accidental
> > sharing that you are worried about won't happen.
> >
> > __kvm_flush_cpu_context() is solving problems that occur when multiple
> > vCPUs of a VM are multiplexed on a single physical CPU.
> 
> Thanks for you answer.
> 
> If guest kernel allow TLB shared across CPUs by setting TTBR0/1_EL1.CnP == 1,
> does kvm still need to guarantee that TLBs are private to each vCPU?

Yes, because there is nothing that describes which physical CPUs
actually share TLBs. So the only possible course of action is to
ignore what the guest says and fallback to something that is safe.

>
> >> So I have two questions:
> >>
> >> 1. What is the rationale behind the comment that "guarantee that both TLBs
> >> and I-cache are private to each vCPU"?
> > I assume you are asking why keeping both TLBs and I-cache private per
> > each vCPU is required. The fundamental answer is that each physical CPU
> > is expected to have its own TLB and I-cache, so we must uphold that
> > property for vCPUs as well. vCPUs can be scheduled on the same physical
> > CPU, and use the same physical TLB/I-cache, obviously, so extra
> > invalidations need to be done.
> Let's assume that both Stage-1 CnP and Stage-2 CnP are enabled.
> 
> As I understand it, the architecture permits TLB to be shared by multiple
> PEs within an Inner Shareable domain. Right?
> 
> If an implementation allows TLB entries to be shared in this way, it seems
> that the current invalidation performed by kvm would no longer be sufficient
> to guarantee that TLBs are private to each vCPU.

Care to explain why?

The core assumption is that a TLBI take effect on all the PEs the TLB
is shared with. If this doesn't work, then CnP is unusable, because it
is then impossible to guarantee that a translation will be refetched
(you could always hit in another PEs TLBs). Such an implementation
would be terminally broken.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

[Tangnianyao]

On 7/6/2026 15:25, Marc Zyngier wrote:
> On Mon, 06 Jul 2026 04:30:30 +0100,
> Tangnianyao <tangnianyao@huawei.com> wrote:
>>
>>
>> On 7/6/2026 1:28, Wei-Lin Chang wrote:
>>> Hi,
>>>
>>> Let me try to answer this:
>>>
>>> On Sat, Jul 04, 2026 at 03:45:56PM +0800, Tangnianyao wrote:
>>>> Hi, all
>>>>
>>>> I'm trying to understand the TLB and I-cache invalidation in
>>>> `kvm_arch_vcpu_load()` that is intended to "guarantee that both TLBs and
>>>> I-cache are private to each vCPU".
>>>>
>>>> As I understand it, when `VTTBR_EL2.CnP == 1`, `__kvm_flush_cpu_context()`
>>>> only performs a local TLB and I-cache invalidation, which does not seem
>>>> sufficient to guarantee that property.
>>>>
>>>> In fact, even if the invalidation were extended to the Inner Shareable
>>>> domain, it still seems difficult to guarantee “TLBs and I-cache are
>>>> private to each vCPU”, when `VTTBR_EL2.CnP == 1`, as long as multiple
>>>> vCPUs from the same VM may be running concurrently on different PEs.
>>> I think you have missed that when 2 stages are involved, both stages
>>> have to set CnP == 1 in order to share TLB entries (Arm ARM R_ZVRZW).
>>> So if TLB entry sharing happens, the guest kernel must have allowed it
>>> in the first place (by setting TTBR0/1_EL1.CnP == 1), hence accidental
>>> sharing that you are worried about won't happen.
>>>
>>> __kvm_flush_cpu_context() is solving problems that occur when multiple
>>> vCPUs of a VM are multiplexed on a single physical CPU.
>> Thanks for you answer.
>>
>> If guest kernel allow TLB shared across CPUs by setting TTBR0/1_EL1.CnP == 1,
>> does kvm still need to guarantee that TLBs are private to each vCPU?
> Yes, because there is nothing that describes which physical CPUs
> actually share TLBs. So the only possible course of action is to
> ignore what the guest says and fallback to something that is safe.
>
>>>> So I have two questions:
>>>>
>>>> 1. What is the rationale behind the comment that "guarantee that both TLBs
>>>> and I-cache are private to each vCPU"?
>>> I assume you are asking why keeping both TLBs and I-cache private per
>>> each vCPU is required. The fundamental answer is that each physical CPU
>>> is expected to have its own TLB and I-cache, so we must uphold that
>>> property for vCPUs as well. vCPUs can be scheduled on the same physical
>>> CPU, and use the same physical TLB/I-cache, obviously, so extra
>>> invalidations need to be done.
>> Let's assume that both Stage-1 CnP and Stage-2 CnP are enabled.
>>
>> As I understand it, the architecture permits TLB to be shared by multiple
>> PEs within an Inner Shareable domain. Right?
>>
>> If an implementation allows TLB entries to be shared in this way, it seems
>> that the current invalidation performed by kvm would no longer be sufficient
>> to guarantee that TLBs are private to each vCPU.
> Care to explain why?
>
> The core assumption is that a TLBI take effect on all the PEs the TLB
> is shared with. If this doesn't work, then CnP is unusable, because it
> is then impossible to guarantee that a translation will be refetched
> (you could always hit in another PEs TLBs). Such an implementation
> would be terminally broken.
>
> 	M.
>
For example:
Sharing the TLB between the two SMT threads of the same physical core can
reduce hardware cost while increasing the effective TLB coverage.
A local TLBI take effect on the whole TLB shared by the two SMT threads,
with the sharing enabled by CnP.

In this scenario, enabling CnP in KVM appears to break the guarantee that
TLBs are private to each vCPU when multiple vCPUs of the same VM run
concurrently on different SMT threads of the same PE.

Thanks
Nianyao Tang

Re: Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

[Marc Zyngier]

On Mon, 06 Jul 2026 09:25:46 +0100,
Tangnianyao <tangnianyao@huawei.com> wrote:
> 
> 
> 
> On 7/6/2026 15:25, Marc Zyngier wrote:
> > On Mon, 06 Jul 2026 04:30:30 +0100,
> > Tangnianyao <tangnianyao@huawei.com> wrote:
> >>
> >>
> >> On 7/6/2026 1:28, Wei-Lin Chang wrote:
> >>> Hi,
> >>>
> >>> Let me try to answer this:
> >>>
> >>> On Sat, Jul 04, 2026 at 03:45:56PM +0800, Tangnianyao wrote:
> >>>> Hi, all
> >>>>
> >>>> I'm trying to understand the TLB and I-cache invalidation in
> >>>> `kvm_arch_vcpu_load()` that is intended to "guarantee that both TLBs and
> >>>> I-cache are private to each vCPU".
> >>>>
> >>>> As I understand it, when `VTTBR_EL2.CnP == 1`, `__kvm_flush_cpu_context()`
> >>>> only performs a local TLB and I-cache invalidation, which does not seem
> >>>> sufficient to guarantee that property.
> >>>>
> >>>> In fact, even if the invalidation were extended to the Inner Shareable
> >>>> domain, it still seems difficult to guarantee “TLBs and I-cache are
> >>>> private to each vCPU”, when `VTTBR_EL2.CnP == 1`, as long as multiple
> >>>> vCPUs from the same VM may be running concurrently on different PEs.
> >>> I think you have missed that when 2 stages are involved, both stages
> >>> have to set CnP == 1 in order to share TLB entries (Arm ARM R_ZVRZW).
> >>> So if TLB entry sharing happens, the guest kernel must have allowed it
> >>> in the first place (by setting TTBR0/1_EL1.CnP == 1), hence accidental
> >>> sharing that you are worried about won't happen.
> >>>
> >>> __kvm_flush_cpu_context() is solving problems that occur when multiple
> >>> vCPUs of a VM are multiplexed on a single physical CPU.
> >> Thanks for you answer.
> >>
> >> If guest kernel allow TLB shared across CPUs by setting TTBR0/1_EL1.CnP == 1,
> >> does kvm still need to guarantee that TLBs are private to each vCPU?
> > Yes, because there is nothing that describes which physical CPUs
> > actually share TLBs. So the only possible course of action is to
> > ignore what the guest says and fallback to something that is safe.
> >
> >>>> So I have two questions:
> >>>>
> >>>> 1. What is the rationale behind the comment that "guarantee that both TLBs
> >>>> and I-cache are private to each vCPU"?
> >>> I assume you are asking why keeping both TLBs and I-cache private per
> >>> each vCPU is required. The fundamental answer is that each physical CPU
> >>> is expected to have its own TLB and I-cache, so we must uphold that
> >>> property for vCPUs as well. vCPUs can be scheduled on the same physical
> >>> CPU, and use the same physical TLB/I-cache, obviously, so extra
> >>> invalidations need to be done.
> >> Let's assume that both Stage-1 CnP and Stage-2 CnP are enabled.
> >>
> >> As I understand it, the architecture permits TLB to be shared by multiple
> >> PEs within an Inner Shareable domain. Right?
> >>
> >> If an implementation allows TLB entries to be shared in this way, it seems
> >> that the current invalidation performed by kvm would no longer be sufficient
> >> to guarantee that TLBs are private to each vCPU.
> > Care to explain why?
> >
> > The core assumption is that a TLBI take effect on all the PEs the TLB
> > is shared with. If this doesn't work, then CnP is unusable, because it
> > is then impossible to guarantee that a translation will be refetched
> > (you could always hit in another PEs TLBs). Such an implementation
> > would be terminally broken.
> >
> > 	M.
> >
> For example:
> Sharing the TLB between the two SMT threads of the same physical core can
> reduce hardware cost while increasing the effective TLB coverage.

I have a precise idea of what TLB sharing can achieve.

> A local TLBI take effect on the whole TLB shared by the two SMT threads,
> with the sharing enabled by CnP.
> 
> In this scenario, enabling CnP in KVM appears to break the guarantee that
> TLBs are private to each vCPU when multiple vCPUs of the same VM run
> concurrently on different SMT threads of the same PE.

Please stop inventing your own terminology. A PE *is* a thread in the
architecture. There is no such thing as "threads of the same PE".

But more to the point: if TLBI invalidates the relevant TLBs for all
the PEs that share them, *why* isn't the current KVM behaviour not
enough to ensure that the vcpu will not hit old TLBs that are there as
a result of a vcpu having run there previously?

Please explain.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

[Tangnianyao]

On 7/6/2026 16:44, Marc Zyngier wrote:
> On Mon, 06 Jul 2026 09:25:46 +0100,
> Tangnianyao <tangnianyao@huawei.com> wrote:
>>
>>
>> On 7/6/2026 15:25, Marc Zyngier wrote:
>>> On Mon, 06 Jul 2026 04:30:30 +0100,
>>> Tangnianyao <tangnianyao@huawei.com> wrote:
>>>>
>>>> On 7/6/2026 1:28, Wei-Lin Chang wrote:
>>>>> Hi,
>>>>>
>>>>> Let me try to answer this:
>>>>>
>>>>> On Sat, Jul 04, 2026 at 03:45:56PM +0800, Tangnianyao wrote:
>>>>>> Hi, all
>>>>>>
>>>>>> I'm trying to understand the TLB and I-cache invalidation in
>>>>>> `kvm_arch_vcpu_load()` that is intended to "guarantee that both TLBs and
>>>>>> I-cache are private to each vCPU".
>>>>>>
>>>>>> As I understand it, when `VTTBR_EL2.CnP == 1`, `__kvm_flush_cpu_context()`
>>>>>> only performs a local TLB and I-cache invalidation, which does not seem
>>>>>> sufficient to guarantee that property.
>>>>>>
>>>>>> In fact, even if the invalidation were extended to the Inner Shareable
>>>>>> domain, it still seems difficult to guarantee “TLBs and I-cache are
>>>>>> private to each vCPU”, when `VTTBR_EL2.CnP == 1`, as long as multiple
>>>>>> vCPUs from the same VM may be running concurrently on different PEs.
>>>>> I think you have missed that when 2 stages are involved, both stages
>>>>> have to set CnP == 1 in order to share TLB entries (Arm ARM R_ZVRZW).
>>>>> So if TLB entry sharing happens, the guest kernel must have allowed it
>>>>> in the first place (by setting TTBR0/1_EL1.CnP == 1), hence accidental
>>>>> sharing that you are worried about won't happen.
>>>>>
>>>>> __kvm_flush_cpu_context() is solving problems that occur when multiple
>>>>> vCPUs of a VM are multiplexed on a single physical CPU.
>>>> Thanks for you answer.
>>>>
>>>> If guest kernel allow TLB shared across CPUs by setting TTBR0/1_EL1.CnP == 1,
>>>> does kvm still need to guarantee that TLBs are private to each vCPU?
>>> Yes, because there is nothing that describes which physical CPUs
>>> actually share TLBs. So the only possible course of action is to
>>> ignore what the guest says and fallback to something that is safe.
>>>
>>>>>> So I have two questions:
>>>>>>
>>>>>> 1. What is the rationale behind the comment that "guarantee that both TLBs
>>>>>> and I-cache are private to each vCPU"?
>>>>> I assume you are asking why keeping both TLBs and I-cache private per
>>>>> each vCPU is required. The fundamental answer is that each physical CPU
>>>>> is expected to have its own TLB and I-cache, so we must uphold that
>>>>> property for vCPUs as well. vCPUs can be scheduled on the same physical
>>>>> CPU, and use the same physical TLB/I-cache, obviously, so extra
>>>>> invalidations need to be done.
>>>> Let's assume that both Stage-1 CnP and Stage-2 CnP are enabled.
>>>>
>>>> As I understand it, the architecture permits TLB to be shared by multiple
>>>> PEs within an Inner Shareable domain. Right?
>>>>
>>>> If an implementation allows TLB entries to be shared in this way, it seems
>>>> that the current invalidation performed by kvm would no longer be sufficient
>>>> to guarantee that TLBs are private to each vCPU.
>>> Care to explain why?
>>>
>>> The core assumption is that a TLBI take effect on all the PEs the TLB
>>> is shared with. If this doesn't work, then CnP is unusable, because it
>>> is then impossible to guarantee that a translation will be refetched
>>> (you could always hit in another PEs TLBs). Such an implementation
>>> would be terminally broken.
>>>
>>> 	M.
>>>
>> For example:
>> Sharing the TLB between the two SMT threads of the same physical core can
>> reduce hardware cost while increasing the effective TLB coverage.
> I have a precise idea of what TLB sharing can achieve.
>
>> A local TLBI take effect on the whole TLB shared by the two SMT threads,
>> with the sharing enabled by CnP.
>>
>> In this scenario, enabling CnP in KVM appears to break the guarantee that
>> TLBs are private to each vCPU when multiple vCPUs of the same VM run
>> concurrently on different SMT threads of the same PE.
> Please stop inventing your own terminology. A PE *is* a thread in the
> architecture. There is no such thing as "threads of the same PE".
>
> But more to the point: if TLBI invalidates the relevant TLBs for all
> the PEs that share them, *why* isn't the current KVM behaviour not
> enough to ensure that the vcpu will not hit old TLBs that are there as
> a result of a vcpu having run there previously?
>
> Please explain.
>
> 	M.
>

Two SMT threads(PE0,PE1) on the same physical core share TLB.

VM0 has 2 vcpus, vcpu0 and vcpu1 that share all architectural context
except the address translation context.

Vcpu0 may observe TLB entries that are supposed to be private to vcpu1
in the following case:

    PE0(core0,smt0)         PE1(core0,smt1)
vcpu0 load           
vcpu0 va->pa0        
vcpu0 put            
                                       vcpu1 load
                                       vcpu1 flush local tlb
                                       vcpu1 modify desc to va->pa1
vcpu0 load
vcpu0 hit *va->pa1*


Thanks
Nianyao Tang

Re: Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

[Will Deacon]

On Mon, Jul 06, 2026 at 10:15:04PM +0800, Tangnianyao wrote:
> Two SMT threads(PE0,PE1) on the same physical core share TLB.

For that to be observable, I think both PEs must be configured with CnP
enabled.

> VM0 has 2 vcpus, vcpu0 and vcpu1 that share all architectural context
> except the address translation context.
> 
> Vcpu0 may observe TLB entries that are supposed to be private to vcpu1
> in the following case:
> 
>     PE0(core0,smt0)         PE1(core0,smt1)
> vcpu0 load           
> vcpu0 va->pa0        
> vcpu0 put            
>                                        vcpu1 load
>                                        vcpu1 flush local tlb
>                                        vcpu1 modify desc to va->pa1
> vcpu0 load
> vcpu0 hit *va->pa1*

If they're "supposed to be private", why have the vCPUs configured CnP
at stage-1?

Will

Re: Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

[Mark Rutland]

On Mon, Jul 06, 2026 at 10:15:04PM +0800, Tangnianyao wrote:
> Two SMT threads(PE0,PE1) on the same physical core share TLB.

Critially those are *NOT* allowed to share entries allocated with
CnP==0, and are only allowed to share entries where CnP was enabled at
stage 1 (and stage 2 if applicable).

Please see the ARM ARM:

  https://developer.arm.com/documentation/ddi0487/mc/

Specifically, section D8.12.3.4 "Common not private translations":

  https://developer.arm.com/documentation/ddi0487/mc/-Part-D-The-AArch64-System-Level-Architecture/-Chapter-D8-The-AArch64-Virtual-Memory-System-Architecture/-D8-16-Translation-Lookaside-Buffers/-D8-16-3-Use-of-ASIDs-and-VMIDs-to-reduce-TLB-maintenance-requirements

> VM0 has 2 vcpus, vcpu0 and vcpu1 that share all architectural context
> except the address translation context.
> 
> Vcpu0 may observe TLB entries that are supposed to be private to vcpu1
> in the following case:
> 
>     PE0(core0,smt0)         PE1(core0,smt1)
> vcpu0 load           
> vcpu0 va->pa0        
> vcpu0 put            
>                                        vcpu1 load
>                                        vcpu1 flush local tlb
>                                        vcpu1 modify desc to va->pa1
> vcpu0 load
> vcpu0 hit *va->pa1*

How is CnP managed in this example?

If *either* of the vCPUs don't set TTBRn_EL1.CnP, that is not permitted
to happen.

If *both* of the vCPUs set TTBRn_EL1.CnP, then surely that is
indistinguishable from physical CPUs:

    PE0(core0,smt0)         PE1(core0,smt1)
    cpu0 va->pa0        
                            cpu1 flush local tlb
                            cpu1 modify desc to va->pa1
    cpu0 hit *va->pa1*

Mark. 

Re: Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

[Tangnianyao]

On 7/6/2026 23:33, Mark Rutland wrote:
> On Mon, Jul 06, 2026 at 10:15:04PM +0800, Tangnianyao wrote:
>> Two SMT threads(PE0,PE1) on the same physical core share TLB.
> Critially those are *NOT* allowed to share entries allocated with
> CnP==0, and are only allowed to share entries where CnP was enabled at
> stage 1 (and stage 2 if applicable).
>
> Please see the ARM ARM:
>
>   https://developer.arm.com/documentation/ddi0487/mc/
>
> Specifically, section D8.12.3.4 "Common not private translations":
>
>   https://developer.arm.com/documentation/ddi0487/mc/-Part-D-The-AArch64-System-Level-Architecture/-Chapter-D8-The-AArch64-Virtual-Memory-System-Architecture/-D8-16-Translation-Lookaside-Buffers/-D8-16-3-Use-of-ASIDs-and-VMIDs-to-reduce-TLB-maintenance-requirements
>
>> VM0 has 2 vcpus, vcpu0 and vcpu1 that share all architectural context
>> except the address translation context.
>>
>> Vcpu0 may observe TLB entries that are supposed to be private to vcpu1
>> in the following case:
>>
>>     PE0(core0,smt0)         PE1(core0,smt1)
>> vcpu0 load           
>> vcpu0 va->pa0        
>> vcpu0 put            
>>                                        vcpu1 load
>>                                        vcpu1 flush local tlb
>>                                        vcpu1 modify desc to va->pa1
>> vcpu0 load
>> vcpu0 hit *va->pa1*
> How is CnP managed in this example?
>
> If *either* of the vCPUs don't set TTBRn_EL1.CnP, that is not permitted
> to happen.
>
> If *both* of the vCPUs set TTBRn_EL1.CnP, then surely that is
> indistinguishable from physical CPUs:
>
>     PE0(core0,smt0)         PE1(core0,smt1)
>     cpu0 va->pa0        
>                             cpu1 flush local tlb
>                             cpu1 modify desc to va->pa1
>     cpu0 hit *va->pa1*
>
> Mark. 
>
> .
>
Thanks for the clarification.
Stage-1 CnP determines whether Stage-1 translation entries may be shared
across vCPUs, and the hardware is responsible for enforcing the
architectural semantics.

Given that, why does KVM still need to guarantee that TLBs are private to
each vCPU?

Assuming VTTBR_EL2.CnP == 1:
If TTBRx_EL1.CnP == 1, the guest is responsible for ensuring that the
translations referenced by TTBRx_EL1 are shareable, as required by the
architecture.
If TTBRx_EL1.CnP == 0, the hardware must ensure that Stage-1 translations
are not shared, again according to the architectural definition.

The reason I'm asking is the potential performance impact. In a scenario
where multiple vCPUs of the same VM are scheduled onto a single PE, this
TLB flush may prevent a vCPU from reusing its previously populated
translation entries, potentially increasing TLB misses.

Thanks,
Nianyao Tang

Re: Question about the "TLBs and I-cache are private to each vCPU" guarantee with VTTBR_EL2.CnP

[Oliver Upton]

On Tue, Jul 07, 2026 at 10:41:35AM +0800, Tangnianyao wrote:
> On 7/6/2026 23:33, Mark Rutland wrote:
> > If *both* of the vCPUs set TTBRn_EL1.CnP, then surely that is
> > indistinguishable from physical CPUs:
> >
> >     PE0(core0,smt0)         PE1(core0,smt1)
> >     cpu0 va->pa0        
> >                             cpu1 flush local tlb
> >                             cpu1 modify desc to va->pa1
> >     cpu0 hit *va->pa1*
> >
> > Mark. 
> >
> > .
> >
> Thanks for the clarification.
> Stage-1 CnP determines whether Stage-1 translation entries may be shared
> across vCPUs, and the hardware is responsible for enforcing the
> architectural semantics.
> 
> Given that, why does KVM still need to guarantee that TLBs are private to
> each vCPU?
> 
> Assuming VTTBR_EL2.CnP == 1:
> If TTBRx_EL1.CnP == 1, the guest is responsible for ensuring that the
> translations referenced by TTBRx_EL1 are shareable, as required by the
> architecture.
> If TTBRx_EL1.CnP == 0, the hardware must ensure that Stage-1 translations
> are not shared, again according to the architectural definition.
> 
> The reason I'm asking is the potential performance impact. In a scenario
> where multiple vCPUs of the same VM are scheduled onto a single PE, this
> TLB flush may prevent a vCPU from reusing its previously populated
> translation entries, potentially increasing TLB misses.

The problem is you can't infer the state of the TLB based on the current
value of CnP at stage-1. CnP only applies to the current TTBR;
nothing stops the guest from using a mix of CnP=0/1 for different TTBRs.

KVM still needs to invalidate in the case that the PE retained a CnP=0
TLB entry for a different stage-1 context than the one configured at
the time of vcpu_load().

Thanks,
Oliver