Re: [PATCH v2 07/15] arm64: hyperv: Add support for mshv_vtl_return_call

[Mark Rutland <mark.rutland@arm.com>]

On Wed, Apr 29, 2026 at 03:26:11PM +0530, Naman Jain wrote:
> On 4/23/2026 7:26 PM, Mark Rutland wrote:
> > On Thu, Apr 23, 2026 at 12:41:57PM +0000, Naman Jain wrote:

[ non-SMMC hypercall code omitted for brevity ]

> > NAK to this.
> > 
> > * This is a non-SMCCC hypercall, which we have NAK'd in general in the
> >    past for various reasons that I am not going to rehash here.
> > 
> > * It's not clear how this is going to be extended with necessary
> >    architecture state in future (e.g. SVE, SME). This is not
> >    future-proof, and I don't believe this is maintainable.
> > 
> > * This breaks general requirements for reliable stacktracing by
> >    clobbering state (e.g. x29) that we depend upon being valid AT ALL
> >    TIMES outside of entry code.
> > 
> > * IMO, if this needs to be saved/restored, that should happen in
> >    whatever you are calling.
> > 
> > Mark.
> 
> Merging threads for addressing comments from Mark Rutland and Marc Zyngier
> on this patch.
> 
> Thanks for reviewing the changes. Please allow me to briefly explain the use
> case here and then address your comments.
> 
> Hyper-V's Virtual Trust Levels (VTLs) provide hardware-enforced isolation
> within a single VM, analogous to ARM TrustZone. The kernel runs in VTL2
> (higher privilege) as a "paravisor", a security monitor that handles
> intercepts for the primary OS in VTL0 (lower privilege). The VTL switch
> (mshv_vtl_return_call) is functionally equivalent to KVM's guest enter/exit.

It's worth noting that for KVM, the KVM hyp code is *tightly* coupled
with the host kernel (they are one single binary object), and the
calling convention between the two is an implementation detail that can
change at any time without any ABI concerns.

While I appreciate this might be trying to do the same thing from a
*functional* perspective, it's certainly different from a
maintainability perspective, and can't be treated in the same way.

> It saves VTL2 state, loads VTL0's GPRs other registers from a shared context
> structure, issues hvc #3 to let VTL0 run, and on return saves VTL0's updated
> state back.
> 
> Coming to the problems with the code, I have identified a few ways to
> address them.
> 
> I can put the assembly code in a separate .S file with
> SYM_FUNC_START/SYM_FUNC_END and marked as noinstr, to prevent ftrace/kprobes
> from instrumenting between the GPR load and the hvc, which could have
> corrupted VTL0 register state. This should solve x29 clobbering, stack
> tracing problems.

My point was that you must not clobber those registers.

Looking at the TLFS document you linked below, it says:

| Note: X29 (FP/frame pointer), X30 (LR/link register), and SP are private
| per-VTL

... so clobbering those doesn't seem to be necessary anyway. Clearly
having an arbitrary calling convention is confusing for everyone.

> I should use kernel_neon_begin()/kernel_neon_end() to save/restore the full
> extended FP state of the current task in VTL2. VTL0's Q0-Q31 can be
> loaded/saved separately via fpsimd_load_state()/fpsimd_save_state(). This
> way, the assembly touches none of the SIMD registers. This is SVE/SME-safe
> for VTL2's task state. VTL0 still only carries Q0-Q31 in the context struct,
> and extending to SVE, SME is a future context struct change, which will need
> Hyper-V arm64 ABI support.
> This way, VTL2's callee-saved regs (x19-x28, x29, x30) are explicitly saved
> to the stack frame at the top and restored at the bottom of assembly code.
> The C caller (in hv_vtl.c) is a clean function call.

That doesn't really address my concerns here.

I do not think that Linux should have to save/restore anything here;
that should be the job of the real hypervisor. The arbitrary separation
of PE state into private and shared (with shred state being directly
exposed to Linux) is a problem for maintainability and forward
compatibility.

Looking at the TLFS document you linked below, I see:

| Note: SVE state (Z0-Z31, P0-P15, FFR) and SME state are VTL-private.
| The lower 128-bit portion (Q registers) is shared, but the upper bits
| of Z registers may be corrupted on VTL transitions. Software should
| not rely on Z register contents being preserved across VTL switches.

... which is certainly going to be a pain to manage.

Note in particular "SME state" is not an architectural term. I don't
know which state in particular that is intended to cover (e.g. ZA, ZT0,
SVCR, all streaming mode state)?

There's no mention of SVCR, so I don't know how this is going to
interact with management of ZA state (ZA and ZT0, which are dependent
upon SVCR.ZA) or streaming mode (dependent upon SVCR.SM). That state has
been *incredibly* painful for us to manage generally. Regardless of the
SMCCC concerns, that needs to be specified better.

> Regarding Non-SMCCC "hvc #3" call, I have a limitation here owing to the ABI
> that is defined by the Hyper-V hypervisor. Fixing this requires a
> hypervisor-side change to support SMCCC-style dispatch for VTL return. Until
> then, hvc #3 is the only working interface. Moreover there would be backward
> compatibility issues with this new ABI interface, if at all it is added.

To be clear, that's Microsoft's problem, not the Linux kernel
community's problem. My NAK still stands.

Multiple years ago now, we made it clear that we would not accept a
non-SMCCC calling convention. Ignoring the substance of that feedback,
and inventing a new calling convention after that point is a
self-inflicted problem.

[...]

> Link to TLFS: https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/vsm#on-arm64-platforms-3

For shared state, aside fomr GPRs and FPSIMD/SVE/SME state, that says:

| * System Information Registers (read-only or non-security-critical):
|   * System identification and feature registers
|   * Cache and TLB type information

It's *implied* that some of those registers might be writable, but as
the specific set of registers is not described I cannot tell. Are there
any writable system registers which are shared?

I don't see how we can know which registers we might need to
save/restore without that being explicitly documented.

I also see:

| Note: SPE (Statistical Profiling Extension) state is shared across VTLs,
| except for PMBSR_EL1 which is VTL-private.

If "SPE state" includes PMBPTR or PMBLIMITR (which is the obvious
reading), this would be a security problem, as a lower-privileged VTL
could clobber those and cause SPE to write to arbitrary memory
immediately upon return to the higher-privileged VTL. Having PMBSR be
private on its own isn't sufficient to prevent that (e.g. since the
higher-privileged VTL could have its own active SPE profiling session).

I'm not keen on requiring hyper-v specific hooks in the SPE driver to
achieve that, and I'm also not keen on having hyper-v support code poke
SPE registers behind the SPE driver's back.

This does not give me confidence that any future PE state (e.g. things
like TRBE) will be managed in a safe way either.

Mark.