HCR_EL2 GET_ONE_REG value difference

HCR_EL2 GET_ONE_REG value difference

[Jan Kotas]

Hello,

During NV debugging I noticed a difference in HCR_EL2 value.

The value I get from accessing this register, using KVM_GET_ONE_REG:
0x30480000000

However, X1 value after executing mrs x1, hcr_el2 in guest:
0x100030080000000

I access both of these at the same place.
I get the list of available registers from KVM_GET_REG_LIST.

I double checked my encoding, and it seems to be correct.
HCR_EL2 3, 4, 1, 1, 0

I can check other registers, but I first wanted to check, if this expected?
Is there something I’m missing, when accessing emulated registers?

Regards,
Jan

Re: HCR_EL2 GET_ONE_REG value difference

[Marc Zyngier]

On Fri, 10 Oct 2025 09:30:12 +0100,
Jan Kotas <jank@cadence.com> wrote:
> 
> Hello,
> 
> During NV debugging I noticed a difference in HCR_EL2 value.
> 
> The value I get from accessing this register, using KVM_GET_ONE_REG:
> 0x30480000000

These bits are API, APK, E2H, RW.

When did you access this? After the vcpu has run? Before it has run?
What NV configuration did you use?

>
> However, X1 value after executing mrs x1, hcr_el2 in guest:
> 0x100030080000000

You have ATA, API, APK, RW. Is that the *only* thing that has run in
your guest?

> I access both of these at the same place.

"at the same place"? What do you mean? One is accessed from host
userspace, and the other from the guest. How can that be the same
place?

> I get the list of available registers from KVM_GET_REG_LIST.
> 
> I double checked my encoding, and it seems to be correct.
> HCR_EL2 3, 4, 1, 1, 0
> 
> I can check other registers, but I first wanted to check, if this expected?
> Is there something I’m missing, when accessing emulated registers?

Could you please start by clarifying what you are doing?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: HCR_EL2 GET_ONE_REG value difference

[Jan Kotas]

> On 10 Oct 2025, at 11:37, Marc Zyngier <maz@kernel.org> wrote:
> 
> EXTERNAL MAIL
> 
> 
> On Fri, 10 Oct 2025 09:30:12 +0100,
> Jan Kotas <jank@cadence.com> wrote:
>> 
>> Hello,
>> 
>> During NV debugging I noticed a difference in HCR_EL2 value.
>> 
>> The value I get from accessing this register, using KVM_GET_ONE_REG:
>> 0x30480000000
> 
> These bits are API, APK, E2H, RW.
> 
> When did you access this? After the vcpu has run? Before it has run?
> What NV configuration did you use?

Hi Marc,

I wanted to reproduce the conditions when I observed it.

I run the guest without the improved EL2 detection,
which was posted yesterday, in VHE mode. 
I’m investigating the place at which E2H detection happens,
just before an exception on ZCR_EL2.

> 
>> 
>> However, X1 value after executing mrs x1, hcr_el2 in guest:
>> 0x100030080000000
> 
> You have ATA, API, APK, RW. Is that the *only* thing that has run in
> your guest?

I just booted the kernel directly without E2H0.

> 
>> I access both of these at the same place.
> 
> "at the same place"? What do you mean? One is accessed from host
> userspace, and the other from the guest. How can that be the same
> place?

By the same place I mean the same KVM_EXIT.
Without resuming vCPUs, at the same PC.

> 
>> I get the list of available registers from KVM_GET_REG_LIST.
>> 
>> I double checked my encoding, and it seems to be correct.
>> HCR_EL2 3, 4, 1, 1, 0
>> 
>> I can check other registers, but I first wanted to check, if this expected?
>> Is there something I’m missing, when accessing emulated registers?
> 
> Could you please start by clarifying what you are doing?

I placed a HW breakpoint at the MRS instruction.
I did a single step, and then I checked X1 and HCR_EL2 values.
By calling GET_ONE_REG on both, I see different values.

Regards,
Jan

> 
> Thanks,
> 
> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.
> 

Re: HCR_EL2 GET_ONE_REG value difference

[Marc Zyngier]

On Fri, 10 Oct 2025 10:51:52 +0100,
Jan Kotas <jank@cadence.com> wrote:
> 
> 
> > On 10 Oct 2025, at 11:37, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > EXTERNAL MAIL
> > 
> > 
> > On Fri, 10 Oct 2025 09:30:12 +0100,
> > Jan Kotas <jank@cadence.com> wrote:
> >> 
> >> Hello,
> >> 
> >> During NV debugging I noticed a difference in HCR_EL2 value.
> >> 
> >> The value I get from accessing this register, using KVM_GET_ONE_REG:
> >> 0x30480000000
> > 
> > These bits are API, APK, E2H, RW.
> > 
> > When did you access this? After the vcpu has run? Before it has run?
> > What NV configuration did you use?
> 
> Hi Marc,
> 
> I wanted to reproduce the conditions when I observed it.
> 
> I run the guest without the improved EL2 detection,
> which was posted yesterday, in VHE mode. 
> I’m investigating the place at which E2H detection happens,
> just before an exception on ZCR_EL2.
> 
> > 
> >> 
> >> However, X1 value after executing mrs x1, hcr_el2 in guest:
> >> 0x100030080000000
> > 
> > You have ATA, API, APK, RW. Is that the *only* thing that has run in
> > your guest?
> 
> I just booted the kernel directly without E2H0.
> 
> > 
> >> I access both of these at the same place.
> > 
> > "at the same place"? What do you mean? One is accessed from host
> > userspace, and the other from the guest. How can that be the same
> > place?
> 
> By the same place I mean the same KVM_EXIT.
> Without resuming vCPUs, at the same PC.
> 
> > 
> >> I get the list of available registers from KVM_GET_REG_LIST.
> >> 
> >> I double checked my encoding, and it seems to be correct.
> >> HCR_EL2 3, 4, 1, 1, 0
> >> 
> >> I can check other registers, but I first wanted to check, if this expected?
> >> Is there something I’m missing, when accessing emulated registers?
> > 
> > Could you please start by clarifying what you are doing?
> 
> I placed a HW breakpoint at the MRS instruction.
> I did a single step, and then I checked X1 and HCR_EL2 values.
> By calling GET_ONE_REG on both, I see different values.

I'm very surprised that debug actually works, but hey, maybe I should
admit that miracles can happen... ;-)

The guest can write whatever it wants to the HCR_EL2 register. That's
just a 64bit chunk of memory, and if it wants to write Pi to it, it
can. Nothing the guest writes there will have any effect as long as it
doesn't ERET to EL1 anyway.

What GET_ONE_REG returns is the *sanitised* view of that register,
after having applied RES0 and RES1 masks, as well as the constraints
that are specific to your exact configuration.

Since you are running without the patch that makes the two views
consistent, your kernel is probably taking the wrong path, and
configuring bits that have no meaning in that configuration and
resulting in, amongst other things, E2H appearing to be 0 while
GET_ONE_REG clearly tells you it is 1.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: HCR_EL2 GET_ONE_REG value difference

[Jan Kotas]

> On 10 Oct 2025, at 12:31, Marc Zyngier <maz@kernel.org> wrote:
> 
> EXTERNAL MAIL
> 
> 
> On Fri, 10 Oct 2025 10:51:52 +0100,
> Jan Kotas <jank@cadence.com> wrote:
>> 
>> 
>>> On 10 Oct 2025, at 11:37, Marc Zyngier <maz@kernel.org> wrote:
>>> 
>>> EXTERNAL MAIL
>>> 
>>> 
>>> On Fri, 10 Oct 2025 09:30:12 +0100,
>>> Jan Kotas <jank@cadence.com> wrote:
>>>> 
>>>> Hello,
>>>> 
>>>> During NV debugging I noticed a difference in HCR_EL2 value.
>>>> 
>>>> The value I get from accessing this register, using KVM_GET_ONE_REG:
>>>> 0x30480000000
>>> 
>>> These bits are API, APK, E2H, RW.
>>> 
>>> When did you access this? After the vcpu has run? Before it has run?
>>> What NV configuration did you use?
>> 
>> Hi Marc,
>> 
>> I wanted to reproduce the conditions when I observed it.
>> 
>> I run the guest without the improved EL2 detection,
>> which was posted yesterday, in VHE mode. 
>> I’m investigating the place at which E2H detection happens,
>> just before an exception on ZCR_EL2.
>> 
>>> 
>>>> 
>>>> However, X1 value after executing mrs x1, hcr_el2 in guest:
>>>> 0x100030080000000
>>> 
>>> You have ATA, API, APK, RW. Is that the *only* thing that has run in
>>> your guest?
>> 
>> I just booted the kernel directly without E2H0.
>> 
>>> 
>>>> I access both of these at the same place.
>>> 
>>> "at the same place"? What do you mean? One is accessed from host
>>> userspace, and the other from the guest. How can that be the same
>>> place?
>> 
>> By the same place I mean the same KVM_EXIT.
>> Without resuming vCPUs, at the same PC.
>> 
>>> 
>>>> I get the list of available registers from KVM_GET_REG_LIST.
>>>> 
>>>> I double checked my encoding, and it seems to be correct.
>>>> HCR_EL2 3, 4, 1, 1, 0
>>>> 
>>>> I can check other registers, but I first wanted to check, if this expected?
>>>> Is there something I’m missing, when accessing emulated registers?
>>> 
>>> Could you please start by clarifying what you are doing?
>> 
>> I placed a HW breakpoint at the MRS instruction.
>> I did a single step, and then I checked X1 and HCR_EL2 values.
>> By calling GET_ONE_REG on both, I see different values.
> 
> I'm very surprised that debug actually works, but hey, maybe I should
> admit that miracles can happen... ;-)

Great, I gave some positive feedback for once. :)

> The guest can write whatever it wants to the HCR_EL2 register. That's
> just a 64bit chunk of memory, and if it wants to write Pi to it, it
> can. Nothing the guest writes there will have any effect as long as it
> doesn't ERET to EL1 anyway.
> 
> What GET_ONE_REG returns is the *sanitised* view of that register,
> after having applied RES0 and RES1 masks, as well as the constraints
> that are specific to your exact configuration.
> 
> Since you are running without the patch that makes the two views
> consistent, your kernel is probably taking the wrong path, and
> configuring bits that have no meaning in that configuration and
> resulting in, amongst other things, E2H appearing to be 0 while
> GET_ONE_REG clearly tells you it is 1.

I got a feeling it will be something like this.

Maybe it would be useful to have a capability/flag
for GET_ONE_REG to get non-sanitized values?
Or a way for a VMM to access the raw NV2 memory?

I’m just wondering, if Guest does something unexpected,
debugging it, may be harder, if a user sees different stuff than SW.

Regards,
Jan

> 
> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: HCR_EL2 GET_ONE_REG value difference

[Marc Zyngier]

On Fri, 10 Oct 2025 11:59:45 +0100,
Jan Kotas <jank@cadence.com> wrote:
> 
> 
> > On 10 Oct 2025, at 12:31, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > EXTERNAL MAIL
> > 
> > 
> > On Fri, 10 Oct 2025 10:51:52 +0100,
> > Jan Kotas <jank@cadence.com> wrote:
> >> 
> >> 
> >>> On 10 Oct 2025, at 11:37, Marc Zyngier <maz@kernel.org> wrote:
> >>> 
> >>> EXTERNAL MAIL
> >>> 
> >>> 
> >>> On Fri, 10 Oct 2025 09:30:12 +0100,
> >>> Jan Kotas <jank@cadence.com> wrote:
> >>>> 
> >>>> Hello,
> >>>> 
> >>>> During NV debugging I noticed a difference in HCR_EL2 value.
> >>>> 
> >>>> The value I get from accessing this register, using KVM_GET_ONE_REG:
> >>>> 0x30480000000
> >>> 
> >>> These bits are API, APK, E2H, RW.
> >>> 
> >>> When did you access this? After the vcpu has run? Before it has run?
> >>> What NV configuration did you use?
> >> 
> >> Hi Marc,
> >> 
> >> I wanted to reproduce the conditions when I observed it.
> >> 
> >> I run the guest without the improved EL2 detection,
> >> which was posted yesterday, in VHE mode. 
> >> I’m investigating the place at which E2H detection happens,
> >> just before an exception on ZCR_EL2.
> >> 
> >>> 
> >>>> 
> >>>> However, X1 value after executing mrs x1, hcr_el2 in guest:
> >>>> 0x100030080000000
> >>> 
> >>> You have ATA, API, APK, RW. Is that the *only* thing that has run in
> >>> your guest?
> >> 
> >> I just booted the kernel directly without E2H0.
> >> 
> >>> 
> >>>> I access both of these at the same place.
> >>> 
> >>> "at the same place"? What do you mean? One is accessed from host
> >>> userspace, and the other from the guest. How can that be the same
> >>> place?
> >> 
> >> By the same place I mean the same KVM_EXIT.
> >> Without resuming vCPUs, at the same PC.
> >> 
> >>> 
> >>>> I get the list of available registers from KVM_GET_REG_LIST.
> >>>> 
> >>>> I double checked my encoding, and it seems to be correct.
> >>>> HCR_EL2 3, 4, 1, 1, 0
> >>>> 
> >>>> I can check other registers, but I first wanted to check, if this expected?
> >>>> Is there something I’m missing, when accessing emulated registers?
> >>> 
> >>> Could you please start by clarifying what you are doing?
> >> 
> >> I placed a HW breakpoint at the MRS instruction.
> >> I did a single step, and then I checked X1 and HCR_EL2 values.
> >> By calling GET_ONE_REG on both, I see different values.
> > 
> > I'm very surprised that debug actually works, but hey, maybe I should
> > admit that miracles can happen... ;-)
> 
> Great, I gave some positive feedback for once. :)

Hey, there's no bad feedback, and your contribution is greatly
appreciated.

> 
> > The guest can write whatever it wants to the HCR_EL2 register. That's
> > just a 64bit chunk of memory, and if it wants to write Pi to it, it
> > can. Nothing the guest writes there will have any effect as long as it
> > doesn't ERET to EL1 anyway.
> > 
> > What GET_ONE_REG returns is the *sanitised* view of that register,
> > after having applied RES0 and RES1 masks, as well as the constraints
> > that are specific to your exact configuration.
> > 
> > Since you are running without the patch that makes the two views
> > consistent, your kernel is probably taking the wrong path, and
> > configuring bits that have no meaning in that configuration and
> > resulting in, amongst other things, E2H appearing to be 0 while
> > GET_ONE_REG clearly tells you it is 1.
> 
> I got a feeling it will be something like this.
> 
> Maybe it would be useful to have a capability/flag
> for GET_ONE_REG to get non-sanitized values?

GET_ONE_REG is the save/restore interface. It is there to present
architecturally correct information that can be restored to another
machine. Having non-sanitised information would potentially prevent
that.

> Or a way for a VMM to access the raw NV2 memory?

The whole point of this sanitisation is that there should be no way to
obtain bad data from within KVM, even if the guest messes things up.
Also, see below.

> I’m just wondering, if Guest does something unexpected,
> debugging it, may be harder, if a user sees different stuff than SW.

This sort of problems exists on real HW already. If a bit is RES1 in
the architecture, it doesn't mean that this bit is always 1. It means
it is interpreted as 1 by the HW, but the bit could be set to any
value and read as such.

And that's exactly what we have here. SW writes 0 to E2H, and that's
what you read from the register in the context of the guest itself.
But GET_ONE_REG shows you how the "HW" interprets this bit, and that
is where having the two views is actually valuable: different values
are likely to result in different behaviours. KVM gives you the tool
to see what both the guest programs (debug or access to the GPR using
GET_ONE_REG), and how KVM interpret the programmed value (access to
HCR_EL2 using GET_ONE_REG).

Given that, my take is that what we have today is not only correct,
but actually the desirable behaviour.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: HCR_EL2 GET_ONE_REG value difference

[Jan Kotas]

> On 10 Oct 2025, at 13:59, Marc Zyngier <maz@kernel.org> wrote:
> 
> EXTERNAL MAIL
> 
> 
> On Fri, 10 Oct 2025 11:59:45 +0100,
> Jan Kotas <jank@cadence.com> wrote:
>> 
>> 
>>> On 10 Oct 2025, at 12:31, Marc Zyngier <maz@kernel.org> wrote:
>>> 
>>> EXTERNAL MAIL
>>> 
>>> 
>>> On Fri, 10 Oct 2025 10:51:52 +0100,
>>> Jan Kotas <jank@cadence.com> wrote:
>>>> 
>>>> 
>>>>> On 10 Oct 2025, at 11:37, Marc Zyngier <maz@kernel.org> wrote:
>>>>> 
>>>>> EXTERNAL MAIL
>>>>> 
>>>>> 
>>>>> On Fri, 10 Oct 2025 09:30:12 +0100,
>>>>> Jan Kotas <jank@cadence.com> wrote:
>>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> During NV debugging I noticed a difference in HCR_EL2 value.
>>>>>> 
>>>>>> The value I get from accessing this register, using KVM_GET_ONE_REG:
>>>>>> 0x30480000000
>>>>> 
>>>>> These bits are API, APK, E2H, RW.
>>>>> 
>>>>> When did you access this? After the vcpu has run? Before it has run?
>>>>> What NV configuration did you use?
>>>> 
>>>> Hi Marc,
>>>> 
>>>> I wanted to reproduce the conditions when I observed it.
>>>> 
>>>> I run the guest without the improved EL2 detection,
>>>> which was posted yesterday, in VHE mode. 
>>>> I’m investigating the place at which E2H detection happens,
>>>> just before an exception on ZCR_EL2.
>>>> 
>>>>> 
>>>>>> 
>>>>>> However, X1 value after executing mrs x1, hcr_el2 in guest:
>>>>>> 0x100030080000000
>>>>> 
>>>>> You have ATA, API, APK, RW. Is that the *only* thing that has run in
>>>>> your guest?
>>>> 
>>>> I just booted the kernel directly without E2H0.
>>>> 
>>>>> 
>>>>>> I access both of these at the same place.
>>>>> 
>>>>> "at the same place"? What do you mean? One is accessed from host
>>>>> userspace, and the other from the guest. How can that be the same
>>>>> place?
>>>> 
>>>> By the same place I mean the same KVM_EXIT.
>>>> Without resuming vCPUs, at the same PC.
>>>> 
>>>>> 
>>>>>> I get the list of available registers from KVM_GET_REG_LIST.
>>>>>> 
>>>>>> I double checked my encoding, and it seems to be correct.
>>>>>> HCR_EL2 3, 4, 1, 1, 0
>>>>>> 
>>>>>> I can check other registers, but I first wanted to check, if this expected?
>>>>>> Is there something I’m missing, when accessing emulated registers?
>>>>> 
>>>>> Could you please start by clarifying what you are doing?
>>>> 
>>>> I placed a HW breakpoint at the MRS instruction.
>>>> I did a single step, and then I checked X1 and HCR_EL2 values.
>>>> By calling GET_ONE_REG on both, I see different values.
>>> 
>>> I'm very surprised that debug actually works, but hey, maybe I should
>>> admit that miracles can happen... ;-)
>> 
>> Great, I gave some positive feedback for once. :)
> 
> Hey, there's no bad feedback, and your contribution is greatly
> appreciated.
> 
>> 
>>> The guest can write whatever it wants to the HCR_EL2 register. That's
>>> just a 64bit chunk of memory, and if it wants to write Pi to it, it
>>> can. Nothing the guest writes there will have any effect as long as it
>>> doesn't ERET to EL1 anyway.
>>> 
>>> What GET_ONE_REG returns is the *sanitised* view of that register,
>>> after having applied RES0 and RES1 masks, as well as the constraints
>>> that are specific to your exact configuration.
>>> 
>>> Since you are running without the patch that makes the two views
>>> consistent, your kernel is probably taking the wrong path, and
>>> configuring bits that have no meaning in that configuration and
>>> resulting in, amongst other things, E2H appearing to be 0 while
>>> GET_ONE_REG clearly tells you it is 1.
>> 
>> I got a feeling it will be something like this.
>> 
>> Maybe it would be useful to have a capability/flag
>> for GET_ONE_REG to get non-sanitized values?
> 
> GET_ONE_REG is the save/restore interface. It is there to present
> architecturally correct information that can be restored to another
> machine. Having non-sanitised information would potentially prevent
> that.
> 
>> Or a way for a VMM to access the raw NV2 memory?
> 
> The whole point of this sanitisation is that there should be no way to
> obtain bad data from within KVM, even if the guest messes things up.
> Also, see below.
> 
>> I’m just wondering, if Guest does something unexpected,
>> debugging it, may be harder, if a user sees different stuff than SW.
> 
> This sort of problems exists on real HW already. If a bit is RES1 in
> the architecture, it doesn't mean that this bit is always 1. It means
> it is interpreted as 1 by the HW, but the bit could be set to any
> value and read as such.
> 
> And that's exactly what we have here. SW writes 0 to E2H, and that's
> what you read from the register in the context of the guest itself.
> But GET_ONE_REG shows you how the "HW" interprets this bit, and that
> is where having the two views is actually valuable: different values
> are likely to result in different behaviours. KVM gives you the tool
> to see what both the guest programs (debug or access to the GPR using
> GET_ONE_REG), and how KVM interpret the programmed value (access to
> HCR_EL2 using GET_ONE_REG).
> 
> Given that, my take is that what we have today is not only correct,
> but actually the desirable behaviour.

From this point of view, I agree as well.

And as you said, it allows us to compare what the HW actually does,
and what the Guest may try to do. It could be a neat debug feature.
Maybe it would be worth adding a note about sanitization to kernel docs?

Thanks for taking the time to respond.
It also helped me to rule out bugs in our VMM implementation.

Regards,
Jan

> Thanks,
> 
> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: HCR_EL2 GET_ONE_REG value difference

[Marc Zyngier]

On Fri, 10 Oct 2025 13:49:43 +0100,
Jan Kotas <jank@cadence.com> wrote:
> 
> > On 10 Oct 2025, at 13:59, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > Given that, my take is that what we have today is not only correct,
> > but actually the desirable behaviour.
> 
> From this point of view, I agree as well.
> 
> And as you said, it allows us to compare what the HW actually does,
> and what the Guest may try to do. It could be a neat debug feature.
> Maybe it would be worth adding a note about sanitization to kernel docs?

We take patches! ;-)

> Thanks for taking the time to respond.
> It also helped me to rule out bugs in our VMM implementation.

Oh gawd... good luck with that! ;-)

	M.

-- 
Without deviation from the norm, progress is not possible.