[PATCH v2 0/3] TDX attestation support and GHCI fixup

[PATCH v2 0/3] TDX attestation support and GHCI fixup

[Paolo Bonzini]

This is a refresh of Binbin's patches with a change to the userspace
API.  I am consolidating everything into a single KVM_EXIT_TDX and
adding to the contract that userspace is free to ignore it *except*
for having to reenter the guest with KVM_RUN.

If in the future this does not work, it should be possible to introduce
an opt-in interface.  Hopefully that will not be necessary.

Paolo

Binbin Wu (3):
  KVM: TDX: Add new TDVMCALL status code for unsupported subfuncs
  KVM: TDX: Handle TDG.VP.VMCALL<GetQuote>
  KVM: TDX: Exit to userspace for GetTdVmCallInfo

 Documentation/virt/kvm/api.rst    | 62 ++++++++++++++++++++++++-
 arch/x86/include/asm/shared/tdx.h |  1 +
 arch/x86/kvm/vmx/tdx.c            | 77 ++++++++++++++++++++++++++++---
 include/uapi/linux/kvm.h          | 22 +++++++++
 4 files changed, 154 insertions(+), 8 deletions(-)

-- 
2.43.5

[PATCH 1/3] KVM: TDX: Add new TDVMCALL status code for unsupported subfuncs

[Paolo Bonzini]

From: Binbin Wu <binbin.wu@linux.intel.com>

Add the new TDVMCALL status code TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED and
return it for unimplemented TDVMCALL subfunctions.

Returning TDVMCALL_STATUS_INVALID_OPERAND when a subfunction is not
implemented is vague because TDX guests can't tell the error is due to
the subfunction is not supported or an invalid input of the subfunction.
New GHCI spec adds TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED to avoid the
ambiguity. Use it instead of TDVMCALL_STATUS_INVALID_OPERAND.

Before the change, for common guest implementations, when a TDX guest
receives TDVMCALL_STATUS_INVALID_OPERAND, it has two cases:
1. Some operand is invalid. It could change the operand to another value
   retry.
2. The subfunction is not supported.

For case 1, an invalid operand usually means the guest implementation bug.
Since the TDX guest can't tell which case is, the best practice for
handling TDVMCALL_STATUS_INVALID_OPERAND is stopping calling such leaf,
treating the failure as fatal if the TDVMCALL is essential or ignoring
it if the TDVMCALL is optional.

With this change, TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED could be sent to
old TDX guest that do not know about it, but it is expected that the
guest will make the same action as TDVMCALL_STATUS_INVALID_OPERAND.
Currently, no known TDX guest checks TDVMCALL_STATUS_INVALID_OPERAND
specifically; for example Linux just checks for success.

Signed-off-by: Binbin Wu <binbin.wu@linux.intel.com>
[Return it for untrapped KVM_HC_MAP_GPA_RANGE. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/include/asm/shared/tdx.h |  1 +
 arch/x86/kvm/vmx/tdx.c            | 10 ++++++----
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/shared/tdx.h b/arch/x86/include/asm/shared/tdx.h
index 2f3820342598..d8525e6ef50a 100644
--- a/arch/x86/include/asm/shared/tdx.h
+++ b/arch/x86/include/asm/shared/tdx.h
@@ -80,6 +80,7 @@
 #define TDVMCALL_STATUS_RETRY		0x0000000000000001ULL
 #define TDVMCALL_STATUS_INVALID_OPERAND	0x8000000000000000ULL
 #define TDVMCALL_STATUS_ALIGN_ERROR	0x8000000000000002ULL
+#define TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED	0x8000000000000003ULL
 
 /*
  * Bitmasks of exposed registers (with VMM).
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index b952bc673271..5d100c240ab3 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1212,11 +1212,13 @@ static int tdx_map_gpa(struct kvm_vcpu *vcpu)
 	/*
 	 * Converting TDVMCALL_MAP_GPA to KVM_HC_MAP_GPA_RANGE requires
 	 * userspace to enable KVM_CAP_EXIT_HYPERCALL with KVM_HC_MAP_GPA_RANGE
-	 * bit set.  If not, the error code is not defined in GHCI for TDX, use
-	 * TDVMCALL_STATUS_INVALID_OPERAND for this case.
+	 * bit set.  This is a base call so it should always be supported, but
+	 * KVM has no way to ensure that userspace implements the GHCI correctly.
+	 * So if KVM_HC_MAP_GPA_RANGE does not cause a VMEXIT, return an error
+	 * to the guest.
 	 */
 	if (!user_exit_on_hypercall(vcpu->kvm, KVM_HC_MAP_GPA_RANGE)) {
-		ret = TDVMCALL_STATUS_INVALID_OPERAND;
+		ret = TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED;
 		goto error;
 	}
 
@@ -1476,7 +1478,7 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)
 		break;
 	}
 
-	tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_INVALID_OPERAND);
+	tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED);
 	return 1;
 }
 
-- 
2.43.5

[PATCH 2/3] KVM: TDX: Handle TDG.VP.VMCALL<GetQuote>

[Paolo Bonzini]

From: Binbin Wu <binbin.wu@linux.intel.com>

Handle TDVMCALL for GetQuote to generate a TD-Quote.

GetQuote is a doorbell-like interface used by TDX guests to request VMM
to generate a TD-Quote signed by a service hosting TD-Quoting Enclave
operating on the host.  A TDX guest passes a TD Report (TDREPORT_STRUCT) in
a shared-memory area as parameter.  Host VMM can access it and queue the
operation for a service hosting TD-Quoting enclave.  When completed, the
Quote is returned via the same shared-memory area.

KVM only checks the GPA from the TDX guest has the shared-bit set and drops
the shared-bit before exiting to userspace to avoid bleeding the shared-bit
into KVM's exit ABI.  KVM forwards the request to userspace VMM (e.g. QEMU)
and userspace VMM queues the operation asynchronously.  KVM sets the return
code according to the 'ret' field set by userspace to notify the TDX guest
whether the request has been queued successfully or not.  When the request
has been queued successfully, the TDX guest can poll the status field in
the shared-memory area to check whether the Quote generation is completed
or not.  When completed, the generated Quote is returned via the same
buffer.

Add KVM_EXIT_TDX as a new exit reason to userspace. Userspace is
required to handle the KVM exit reason as the initial support for TDX,
by reentering KVM to ensure that the TDVMCALL is complete.  While at it,
add a note that KVM_EXIT_HYPERCALL also requires reentry with KVM_RUN.

Signed-off-by: Binbin Wu <binbin.wu@linux.intel.com>
Tested-by: Mikko Ylinen <mikko.ylinen@linux.intel.com>
Acked-by: Kai Huang <kai.huang@intel.com>
[Adjust userspace API. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 Documentation/virt/kvm/api.rst | 49 +++++++++++++++++++++++++++++++++-
 arch/x86/kvm/vmx/tdx.c         | 31 +++++++++++++++++++++
 include/uapi/linux/kvm.h       | 17 ++++++++++++
 3 files changed, 96 insertions(+), 1 deletion(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 1bd2d42e6424..3643d853a634 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6645,7 +6645,8 @@ to the byte array.
 .. note::
 
       For KVM_EXIT_IO, KVM_EXIT_MMIO, KVM_EXIT_OSI, KVM_EXIT_PAPR, KVM_EXIT_XEN,
-      KVM_EXIT_EPR, KVM_EXIT_X86_RDMSR and KVM_EXIT_X86_WRMSR the corresponding
+      KVM_EXIT_EPR, KVM_EXIT_HYPERCALL, KVM_EXIT_TDX,
+      KVM_EXIT_X86_RDMSR and KVM_EXIT_X86_WRMSR the corresponding
       operations are complete (and guest state is consistent) only after userspace
       has re-entered the kernel with KVM_RUN.  The kernel side will first finish
       incomplete operations and then check for pending signals.
@@ -7174,6 +7175,52 @@ The valid value for 'flags' is:
   - KVM_NOTIFY_CONTEXT_INVALID -- the VM context is corrupted and not valid
     in VMCS. It would run into unknown result if resume the target VM.
 
+::
+
+		/* KVM_EXIT_TDX */
+		struct {
+			__u64 flags;
+			__u64 nr;
+			union {
+				struct {
+					u64 ret;
+					u64 data[5];
+				} unknown;
+				struct {
+					u64 ret;
+					u64 gpa;
+					u64 size;
+				} get_quote;
+			};
+		} tdx;
+
+Process a TDVMCALL from the guest.  KVM forwards select TDVMCALL based
+on the Guest-Hypervisor Communication Interface (GHCI) specification;
+KVM bridges these requests to the userspace VMM with minimal changes,
+placing the inputs in the union and copying them back to the guest
+on re-entry.
+
+Flags are currently always zero, whereas ``nr`` contains the TDVMCALL
+number from register R11.  The remaining field of the union provide the
+inputs and outputs of the TDVMCALL.  Currently the following values of
+``nr`` are defined:
+
+* ``TDVMCALL_GET_QUOTE``: the guest has requested to generate a TD-Quote
+signed by a service hosting TD-Quoting Enclave operating on the host.
+Parameters and return value are in the ``get_quote`` field of the union.
+The ``gpa`` field and ``size`` specify the guest physical address
+(without the shared bit set) and the size of a shared-memory buffer, in
+which the TDX guest passes a TD Report.  The ``ret`` field represents
+the return value of the GetQuote request.  When the request has been
+queued successfully, the TDX guest can poll the status field in the
+shared-memory area to check whether the Quote generation is completed or
+not. When completed, the generated Quote is returned via the same buffer.
+
+KVM may add support for more values in the future that may cause a userspace
+exit, even without calls to ``KVM_ENABLE_CAP`` or similar.  In this case,
+it will enter with output fields already valid; in the common case, the
+``unknown.ret`` field of the union will be ``TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED``.
+Userspace need not do anything if it does not wish to support a TDVMCALL.
 ::
 
 		/* Fix the size of the union. */
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index 5d100c240ab3..6878a76069f8 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1465,6 +1465,36 @@ static int tdx_get_td_vm_call_info(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+static int tdx_complete_simple(struct kvm_vcpu *vcpu)
+{
+	tdvmcall_set_return_code(vcpu, vcpu->run->tdx.unknown.ret);
+	return 1;
+}
+
+static int tdx_get_quote(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	u64 gpa = tdx->vp_enter_args.r12;
+	u64 size = tdx->vp_enter_args.r13;
+
+	/* The gpa of buffer must have shared bit set. */
+	if (vt_is_tdx_private_gpa(vcpu->kvm, gpa)) {
+		tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_INVALID_OPERAND);
+		return 1;
+	}
+
+	vcpu->run->exit_reason = KVM_EXIT_TDX;
+	vcpu->run->tdx.flags = 0;
+	vcpu->run->tdx.nr = TDVMCALL_GET_QUOTE;
+	vcpu->run->tdx.get_quote.ret = TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED;
+	vcpu->run->tdx.get_quote.gpa = gpa & ~gfn_to_gpa(kvm_gfn_direct_bits(tdx->vcpu.kvm));
+	vcpu->run->tdx.get_quote.size = size;
+
+	vcpu->arch.complete_userspace_io = tdx_complete_simple;
+
+	return 0;
+}
+
 static int handle_tdvmcall(struct kvm_vcpu *vcpu)
 {
 	switch (tdvmcall_leaf(vcpu)) {
@@ -1474,6 +1503,8 @@ static int handle_tdvmcall(struct kvm_vcpu *vcpu)
 		return tdx_report_fatal_error(vcpu);
 	case TDVMCALL_GET_TD_VM_CALL_INFO:
 		return tdx_get_td_vm_call_info(vcpu);
+	case TDVMCALL_GET_QUOTE:
+		return tdx_get_quote(vcpu);
 	default:
 		break;
 	}
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index d00b85cb168c..6708bc88ae69 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -178,6 +178,7 @@ struct kvm_xen_exit {
 #define KVM_EXIT_NOTIFY           37
 #define KVM_EXIT_LOONGARCH_IOCSR  38
 #define KVM_EXIT_MEMORY_FAULT     39
+#define KVM_EXIT_TDX              40
 
 /* For KVM_EXIT_INTERNAL_ERROR */
 /* Emulate instruction failed. */
@@ -447,6 +448,22 @@ struct kvm_run {
 			__u64 gpa;
 			__u64 size;
 		} memory_fault;
+		/* KVM_EXIT_TDX */
+		struct {
+			__u64 flags;
+			__u64 nr;
+			union {
+				struct {
+					__u64 ret;
+					__u64 data[5];
+				} unknown;
+				struct {
+					__u64 ret;
+					__u64 gpa;
+					__u64 size;
+				} get_quote;
+			};
+		} tdx;
 		/* Fix the size of the union. */
 		char padding[256];
 	};
-- 
2.43.5

[PATCH 3/3] KVM: TDX: Exit to userspace for GetTdVmCallInfo

[Paolo Bonzini]

From: Binbin Wu <binbin.wu@linux.intel.com>

Exit to userspace for TDG.VP.VMCALL<GetTdVmCallInfo> via KVM_EXIT_TDX,
to allow userspace to provide information about the support of
TDVMCALLs when r12 is 1 for the TDVMCALLs beyond the GHCI base API.

GHCI spec defines the GHCI base TDVMCALLs: <GetTdVmCallInfo>, <MapGPA>,
<ReportFatalError>, <Instruction.CPUID>, <#VE.RequestMMIO>,
<Instruction.HLT>, <Instruction.IO>, <Instruction.RDMSR> and
<Instruction.WRMSR>. They must be supported by VMM to support TDX guests.

For GetTdVmCallInfo
- When leaf (r12) to enumerate TDVMCALL functionality is set to 0,
  successful execution indicates all GHCI base TDVMCALLs listed above are
  supported.

  Update the KVM TDX document with the set of the GHCI base APIs.

- When leaf (r12) to enumerate TDVMCALL functionality is set to 1, it
  indicates the TDX guest is querying the supported TDVMCALLs beyond
  the GHCI base TDVMCALLs.
  Exit to userspace to let userspace set the TDVMCALL sub-function bit(s)
  accordingly to the leaf outputs.  KVM could set the TDVMCALL bit(s)
  supported by itself when the TDVMCALLs don't need support from userspace
  after returning from userspace and before entering guest. Currently, no
  such TDVMCALLs implemented, KVM just sets the values returned from
  userspace.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Binbin Wu <binbin.wu@linux.intel.com>
[Adjust userspace API. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 Documentation/virt/kvm/api.rst | 15 +++++++++++++-
 arch/x86/kvm/vmx/tdx.c         | 38 ++++++++++++++++++++++++++++++----
 include/uapi/linux/kvm.h       |  5 +++++
 3 files changed, 53 insertions(+), 5 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 3643d853a634..2b1656907356 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -7190,6 +7190,11 @@ The valid value for 'flags' is:
 					u64 gpa;
 					u64 size;
 				} get_quote;
+				struct {
+					u64 ret;
+					u64 leaf;
+					u64 r11, r12, r13, r14;
+				} get_tdvmcall_info;
 			};
 		} tdx;

@@ -7216,9 +7221,17 @@ queued successfully, the TDX guest can poll the status field in the
 shared-memory area to check whether the Quote generation is completed or
 not. When completed, the generated Quote is returned via the same buffer.
 
+* ``TDVMCALL_GET_TD_VM_CALL_INFO``: the guest has requested the support
+status of TDVMCALLs.  The output values for the given leaf should be
+placed in fields from ``r11`` to ``r14`` of the ``get_tdvmcall_info``
+field of the union.  This TDVMCALL must succeed, therefore KVM leaves
+``ret`` equal to ``TDVMCALL_STATUS_SUCCESS`` and ``r11`` to ``r14``
+equal to zero on entry.
+
 KVM may add support for more values in the future that may cause a userspace
 exit, even without calls to ``KVM_ENABLE_CAP`` or similar.  In this case,
-it will enter with output fields already valid; in the common case, the
+it will enter with output fields already valid as mentioned for
+``TDVMCALL_GET_TD_VM_CALL_INFO`` above; in the common case, the
 ``unknown.ret`` field of the union will be ``TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED``.
 Userspace need not do anything if it does not wish to support a TDVMCALL.
 ::
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index 6878a76069f8..5804d1b1ea0e 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1451,18 +1451,49 @@ static int tdx_emulate_mmio(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+static int tdx_complete_get_td_vm_call_info(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+
+	tdvmcall_set_return_code(vcpu, vcpu->run->tdx.get_tdvmcall_info.ret);
+
+	/*
+	 * For now, there is no TDVMCALL beyond GHCI base API supported by KVM
+	 * directly without the support from userspace, just set the value
+	 * returned from userspace.
+	 */
+	tdx->vp_enter_args.r11 = vcpu->run->tdx.get_tdvmcall_info.r11;
+	tdx->vp_enter_args.r12 = vcpu->run->tdx.get_tdvmcall_info.r12;
+	tdx->vp_enter_args.r13 = vcpu->run->tdx.get_tdvmcall_info.r13;
+	tdx->vp_enter_args.r14 = vcpu->run->tdx.get_tdvmcall_info.r14;
+
+	return 1;
+}
+
 static int tdx_get_td_vm_call_info(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_tdx *tdx = to_tdx(vcpu);
 
-	if (tdx->vp_enter_args.r12)
-		tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_INVALID_OPERAND);
-	else {
+	switch (tdx->vp_enter_args.r12) {
+	case 1:
+		vcpu->run->tdx.get_tdvmcall_info.leaf = tdx->vp_enter_args.r12;
+		vcpu->run->exit_reason = KVM_EXIT_TDX;
+		vcpu->run->tdx.flags = 0;
+		vcpu->run->tdx.nr = TDVMCALL_GET_TD_VM_CALL_INFO;
+		vcpu->run->tdx.get_tdvmcall_info.ret = TDVMCALL_STATUS_SUCCESS;
+		vcpu->run->tdx.get_tdvmcall_info.r11 = 0;
+		vcpu->run->tdx.get_tdvmcall_info.r12 = 0;
+		vcpu->run->tdx.get_tdvmcall_info.r13 = 0;
+		vcpu->run->tdx.get_tdvmcall_info.r14 = 0;
+		vcpu->arch.complete_userspace_io = tdx_complete_get_td_vm_call_info;
+		return 0;
+	default:
 		tdx->vp_enter_args.r11 = 0;
+		tdx->vp_enter_args.r12 = 0;
 		tdx->vp_enter_args.r13 = 0;
 		tdx->vp_enter_args.r14 = 0;
+		return 1;
 	}
-	return 1;
 }
 
 static int tdx_complete_simple(struct kvm_vcpu *vcpu)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 6708bc88ae69..fb3b4cd8d662 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -461,6 +461,11 @@ struct kvm_run {
 					__u64 gpa;
 					__u64 size;
 				} get_quote;
+				struct {
+					__u64 ret;
+					__u64 leaf;
+					__u64 r11, r12, r13, r14;
+				} get_tdvmcall_info;
 			};
 		} tdx;
 		/* Fix the size of the union. */
-- 
2.43.5

Re: [PATCH 3/3] KVM: TDX: Exit to userspace for GetTdVmCallInfo

[Xiaoyao Li]

On 6/20/2025 2:01 AM, Paolo Bonzini wrote:
> From: Binbin Wu <binbin.wu@linux.intel.com>
> 
> Exit to userspace for TDG.VP.VMCALL<GetTdVmCallInfo> via KVM_EXIT_TDX,
> to allow userspace to provide information about the support of
> TDVMCALLs when r12 is 1 for the TDVMCALLs beyond the GHCI base API.
> 
> GHCI spec defines the GHCI base TDVMCALLs: <GetTdVmCallInfo>, <MapGPA>,
> <ReportFatalError>, <Instruction.CPUID>, <#VE.RequestMMIO>,
> <Instruction.HLT>, <Instruction.IO>, <Instruction.RDMSR> and
> <Instruction.WRMSR>. They must be supported by VMM to support TDX guests.
> 
> For GetTdVmCallInfo
> - When leaf (r12) to enumerate TDVMCALL functionality is set to 0,
>    successful execution indicates all GHCI base TDVMCALLs listed above are
>    supported.
> 
>    Update the KVM TDX document with the set of the GHCI base APIs.
> 
> - When leaf (r12) to enumerate TDVMCALL functionality is set to 1, it
>    indicates the TDX guest is querying the supported TDVMCALLs beyond
>    the GHCI base TDVMCALLs.
>    Exit to userspace to let userspace set the TDVMCALL sub-function bit(s)
>    accordingly to the leaf outputs.  KVM could set the TDVMCALL bit(s)
>    supported by itself when the TDVMCALLs don't need support from userspace
>    after returning from userspace and before entering guest. Currently, no
>    such TDVMCALLs implemented, KVM just sets the values returned from
>    userspace.
> 
> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Binbin Wu <binbin.wu@linux.intel.com>
> [Adjust userspace API. - Paolo]
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>   Documentation/virt/kvm/api.rst | 15 +++++++++++++-
>   arch/x86/kvm/vmx/tdx.c         | 38 ++++++++++++++++++++++++++++++----
>   include/uapi/linux/kvm.h       |  5 +++++
>   3 files changed, 53 insertions(+), 5 deletions(-)
> 
> diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
> index 3643d853a634..2b1656907356 100644
> --- a/Documentation/virt/kvm/api.rst
> +++ b/Documentation/virt/kvm/api.rst
> @@ -7190,6 +7190,11 @@ The valid value for 'flags' is:
>   					u64 gpa;
>   					u64 size;
>   				} get_quote;
> +				struct {
> +					u64 ret;
> +					u64 leaf;
> +					u64 r11, r12, r13, r14;
> +				} get_tdvmcall_info;
>   			};
>   		} tdx;
> 
> @@ -7216,9 +7221,17 @@ queued successfully, the TDX guest can poll the status field in the
>   shared-memory area to check whether the Quote generation is completed or
>   not. When completed, the generated Quote is returned via the same buffer.
>   
> +* ``TDVMCALL_GET_TD_VM_CALL_INFO``: the guest has requested the support
> +status of TDVMCALLs.  The output values for the given leaf should be
> +placed in fields from ``r11`` to ``r14`` of the ``get_tdvmcall_info``
> +field of the union.  This TDVMCALL must succeed, therefore KVM leaves
> +``ret`` equal to ``TDVMCALL_STATUS_SUCCESS`` and ``r11`` to ``r14``
> +equal to zero on entry.
> +
>   KVM may add support for more values in the future that may cause a userspace
>   exit, even without calls to ``KVM_ENABLE_CAP`` or similar.  In this case,
> -it will enter with output fields already valid; in the common case, the
> +it will enter with output fields already valid as mentioned for
> +``TDVMCALL_GET_TD_VM_CALL_INFO`` above; in the common case, the
>   ``unknown.ret`` field of the union will be ``TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED``.
>   Userspace need not do anything if it does not wish to support a TDVMCALL.
>   ::
> diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
> index 6878a76069f8..5804d1b1ea0e 100644
> --- a/arch/x86/kvm/vmx/tdx.c
> +++ b/arch/x86/kvm/vmx/tdx.c
> @@ -1451,18 +1451,49 @@ static int tdx_emulate_mmio(struct kvm_vcpu *vcpu)
>   	return 1;
>   }
>   
> +static int tdx_complete_get_td_vm_call_info(struct kvm_vcpu *vcpu)
> +{
> +	struct vcpu_tdx *tdx = to_tdx(vcpu);
> +
> +	tdvmcall_set_return_code(vcpu, vcpu->run->tdx.get_tdvmcall_info.ret);
> +
> +	/*
> +	 * For now, there is no TDVMCALL beyond GHCI base API supported by KVM
> +	 * directly without the support from userspace, just set the value
> +	 * returned from userspace.
> +	 */
> +	tdx->vp_enter_args.r11 = vcpu->run->tdx.get_tdvmcall_info.r11;
> +	tdx->vp_enter_args.r12 = vcpu->run->tdx.get_tdvmcall_info.r12;
> +	tdx->vp_enter_args.r13 = vcpu->run->tdx.get_tdvmcall_info.r13;
> +	tdx->vp_enter_args.r14 = vcpu->run->tdx.get_tdvmcall_info.r14;
> +
> +	return 1;
> +}
> +
>   static int tdx_get_td_vm_call_info(struct kvm_vcpu *vcpu)
>   {
>   	struct vcpu_tdx *tdx = to_tdx(vcpu);
>   
> -	if (tdx->vp_enter_args.r12)
> -		tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_INVALID_OPERAND);
> -	else {
> +	switch (tdx->vp_enter_args.r12) {
> +	case 1:
> +		vcpu->run->tdx.get_tdvmcall_info.leaf = tdx->vp_enter_args.r12;
> +		vcpu->run->exit_reason = KVM_EXIT_TDX;
> +		vcpu->run->tdx.flags = 0;
> +		vcpu->run->tdx.nr = TDVMCALL_GET_TD_VM_CALL_INFO;
> +		vcpu->run->tdx.get_tdvmcall_info.ret = TDVMCALL_STATUS_SUCCESS;
> +		vcpu->run->tdx.get_tdvmcall_info.r11 = 0;
> +		vcpu->run->tdx.get_tdvmcall_info.r12 = 0;
> +		vcpu->run->tdx.get_tdvmcall_info.r13 = 0;
> +		vcpu->run->tdx.get_tdvmcall_info.r14 = 0;
> +		vcpu->arch.complete_userspace_io = tdx_complete_get_td_vm_call_info;
> +		return 0;
> +	default:
>   		tdx->vp_enter_args.r11 = 0;
> +		tdx->vp_enter_args.r12 = 0;
>   		tdx->vp_enter_args.r13 = 0;
>   		tdx->vp_enter_args.r14 = 0;
> +		return 1;

Though it looks OK to return all-0 for r12 == 0 and undefined case of 
r12 > 1, I prefer returning TDVMCALL_STATUS_INVALID_OPERAND for 
undefined case.

So please make above "case 0:", and make the "default:" return 
TDVMCALL_STATUS_INVALID_OPERAND

>   	}
> -	return 1;
>   }
>   
>   static int tdx_complete_simple(struct kvm_vcpu *vcpu)
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 6708bc88ae69..fb3b4cd8d662 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -461,6 +461,11 @@ struct kvm_run {
>   					__u64 gpa;
>   					__u64 size;
>   				} get_quote;
> +				struct {
> +					__u64 ret;
> +					__u64 leaf;
> +					__u64 r11, r12, r13, r14;
> +				} get_tdvmcall_info;
>   			};
>   		} tdx;
>   		/* Fix the size of the union. */

Re: [PATCH v2 0/3] TDX attestation support and GHCI fixup

[Xiaoyao Li]

On 6/20/2025 2:01 AM, Paolo Bonzini wrote:
> This is a refresh of Binbin's patches with a change to the userspace
> API.  I am consolidating everything into a single KVM_EXIT_TDX and
> adding to the contract that userspace is free to ignore it *except*
> for having to reenter the guest with KVM_RUN.
> 
> If in the future this does not work, it should be possible to introduce
> an opt-in interface.  Hopefully that will not be necessary.

For <GetTdVmCallInfo> exit, I think KVM still needs to report which 
TDVMCALL leaf will exit to userspace, to differentiate between different 
KVMs.

But it's not a must for current <GetQuote> since it exits to userspace 
from day 0. So that we can leave the report interface until KVM needs to 
support user exit of another TDVMCALL leaf.

> Paolo
> 
> Binbin Wu (3):
>    KVM: TDX: Add new TDVMCALL status code for unsupported subfuncs
>    KVM: TDX: Handle TDG.VP.VMCALL<GetQuote>
>    KVM: TDX: Exit to userspace for GetTdVmCallInfo
> 
>   Documentation/virt/kvm/api.rst    | 62 ++++++++++++++++++++++++-
>   arch/x86/include/asm/shared/tdx.h |  1 +
>   arch/x86/kvm/vmx/tdx.c            | 77 ++++++++++++++++++++++++++++---
>   include/uapi/linux/kvm.h          | 22 +++++++++
>   4 files changed, 154 insertions(+), 8 deletions(-)
> 

Re: [PATCH v2 0/3] TDX attestation support and GHCI fixup

[Binbin Wu]

On 6/20/2025 9:30 AM, Xiaoyao Li wrote:
> On 6/20/2025 2:01 AM, Paolo Bonzini wrote:
>> This is a refresh of Binbin's patches with a change to the userspace
>> API.  I am consolidating everything into a single KVM_EXIT_TDX and
>> adding to the contract that userspace is free to ignore it *except*
>> for having to reenter the guest with KVM_RUN.
>>
>> If in the future this does not work, it should be possible to introduce
>> an opt-in interface.  Hopefully that will not be necessary.
>
> For <GetTdVmCallInfo> exit, I think KVM still needs to report which TDVMCALL leaf will exit to userspace, to differentiate between different KVMs.

Yes, I planned a v2 to expose the bitmap of TDVMCALLs that KVM will exit to
userspace VMM for handling via KVM_TDX_CAPABILITIES.

>
> But it's not a must for current <GetQuote> since it exits to userspace from day 0. So that we can leave the report interface until KVM needs to support user exit of another TDVMCALL leaf.

Agree. This report interface can be added later when needed.

About the compatibility:
Since <GetQuote> is the only optional TDVMCALL for now and KVM always exit to
userspace for <GetQuote>, a userspace VMM can always set the bit for <GetQuote>
if it's supported in userspace.
Then
- First KVM release + new userspace VMM release with report interface.
   Userspace will see nothing reported by the interface, and it always sets
   <GetQuote> , which is expected.
- New KVM release with report interface + first userspace VMM release
   Userspace doesn't know the report interface and it only sets <GetQuote>, which
   is expected.

>
>> Paolo
>>
>> Binbin Wu (3):
>>    KVM: TDX: Add new TDVMCALL status code for unsupported subfuncs
>>    KVM: TDX: Handle TDG.VP.VMCALL<GetQuote>
>>    KVM: TDX: Exit to userspace for GetTdVmCallInfo
>>
>>   Documentation/virt/kvm/api.rst    | 62 ++++++++++++++++++++++++-
>>   arch/x86/include/asm/shared/tdx.h |  1 +
>>   arch/x86/kvm/vmx/tdx.c            | 77 ++++++++++++++++++++++++++++---
>>   include/uapi/linux/kvm.h          | 22 +++++++++
>>   4 files changed, 154 insertions(+), 8 deletions(-)
>>
>

Re: [PATCH 2/3] KVM: TDX: Handle TDG.VP.VMCALL<GetQuote>

[Binbin Wu]

On 6/20/2025 2:01 AM, Paolo Bonzini wrote:
[...]
> @@ -7174,6 +7175,52 @@ The valid value for 'flags' is:
>     - KVM_NOTIFY_CONTEXT_INVALID -- the VM context is corrupted and not valid
>       in VMCS. It would run into unknown result if resume the target VM.
>   
> +::
> +
> +		/* KVM_EXIT_TDX */
> +		struct {
> +			__u64 flags;
> +			__u64 nr;
> +			union {
> +				struct {
> +					u64 ret;
> +					u64 data[5];
Should the interface reserve more elements?

Without considering XMM registers, the possible registers according to GHCI spec
are RBX, RDX, RBP, RDI, RSI, R8, R9, R12-R15. Since RBP is not suggested to be
used to pass information, how about make the array 10 elements?


> +				} unknown;
> +				struct {
> +					u64 ret;
> +					u64 gpa;
> +					u64 size;
> +				} get_quote;
> +			};
> +		} tdx;
> +

Re: [PATCH 2/3] KVM: TDX: Handle TDG.VP.VMCALL<GetQuote>

[Binbin Wu]

On 6/20/2025 10:57 AM, Binbin Wu wrote:
>
>
> On 6/20/2025 2:01 AM, Paolo Bonzini wrote:
> [...]
>> @@ -7174,6 +7175,52 @@ The valid value for 'flags' is:
>>     - KVM_NOTIFY_CONTEXT_INVALID -- the VM context is corrupted and not valid
>>       in VMCS. It would run into unknown result if resume the target VM.
>>   +::
>> +
>> +        /* KVM_EXIT_TDX */
>> +        struct {
>> +            __u64 flags;
>> +            __u64 nr;
>> +            union {
>> +                struct {
>> +                    u64 ret;
>> +                    u64 data[5];
> Should the interface reserve more elements?
>
> Without considering XMM registers, the possible registers according to GHCI spec
> are RBX, RDX, RBP, RDI, RSI, R8, R9, R12-R15. Since RBP is not suggested to be
> used to pass information, how about make the array 10 elements?

Please Ignore it, there is no need to do this.

>
>
>> +                } unknown;
>> +                struct {
>> +                    u64 ret;
>> +                    u64 gpa;
>> +                    u64 size;
>> +                } get_quote;
>> +            };
>> +        } tdx;
>> +
>

Re: [PATCH v2 0/3] TDX attestation support and GHCI fixup

[Binbin Wu]

On 6/20/2025 2:01 AM, Paolo Bonzini wrote:
> This is a refresh of Binbin's patches with a change to the userspace
> API.  I am consolidating everything into a single KVM_EXIT_TDX and
> adding to the contract that userspace is free to ignore it *except*
> for having to reenter the guest with KVM_RUN.
>
> If in the future this does not work, it should be possible to introduce
> an opt-in interface.  Hopefully that will not be necessary.
>
> Paolo
>
> Binbin Wu (3):
>    KVM: TDX: Add new TDVMCALL status code for unsupported subfuncs
>    KVM: TDX: Handle TDG.VP.VMCALL<GetQuote>
>    KVM: TDX: Exit to userspace for GetTdVmCallInfo
>
>   Documentation/virt/kvm/api.rst    | 62 ++++++++++++++++++++++++-
>   arch/x86/include/asm/shared/tdx.h |  1 +
>   arch/x86/kvm/vmx/tdx.c            | 77 ++++++++++++++++++++++++++++---
>   include/uapi/linux/kvm.h          | 22 +++++++++
>   4 files changed, 154 insertions(+), 8 deletions(-)
>
Tested the patch set with the TDX kvm-unit-tests, TDX enhanced KVM selftests,
booting a Linux TD, and TDX related test cases defined in the LKVS test suite
as described in:
https://github.com/intel/lkvs/blob/main/KVM/docs/lkvs_on_avocado.md

Xiaoyao has tested the flow for GetQuote and had some comments for small issues
on qemu patch:
https://mail.gnu.org/archive/html/qemu-devel/2025-06/msg03154.html

Re: [PATCH v2 0/3] TDX attestation support and GHCI fixup

[Paolo Bonzini]

Il ven 20 giu 2025, 03:30 Xiaoyao Li <xiaoyao.li@intel.com> ha scritto:
>
> On 6/20/2025 2:01 AM, Paolo Bonzini wrote:
> > This is a refresh of Binbin's patches with a change to the userspace
> > API.  I am consolidating everything into a single KVM_EXIT_TDX and
> > adding to the contract that userspace is free to ignore it *except*
> > for having to reenter the guest with KVM_RUN.
> >
> > If in the future this does not work, it should be possible to introduce
> > an opt-in interface.  Hopefully that will not be necessary.
>
> For <GetTdVmCallInfo> exit, I think KVM still needs to report which
> TDVMCALL leaf will exit to userspace, to differentiate between different
> KVMs.


The interface I chose is that KVM always exits, but it initializes the
output values such that userspace can leave them untouched for unknown
TDVMCALLs or unknown leaves. So there is no need for this.

Querying kernel support of other services can be added later, but
unless the GHCI adds more input or output fields to TdVmCallInfo there
is no need to limit the userspace exit to leaf 1.


Paolo

>
> But it's not a must for current <GetQuote> since it exits to userspace
> from day 0. So that we can leave the report interface until KVM needs to
> support user exit of another TDVMCALL leaf.
>
> > Paolo
> >
> > Binbin Wu (3):
> >    KVM: TDX: Add new TDVMCALL status code for unsupported subfuncs
> >    KVM: TDX: Handle TDG.VP.VMCALL<GetQuote>
> >    KVM: TDX: Exit to userspace for GetTdVmCallInfo
> >
> >   Documentation/virt/kvm/api.rst    | 62 ++++++++++++++++++++++++-
> >   arch/x86/include/asm/shared/tdx.h |  1 +
> >   arch/x86/kvm/vmx/tdx.c            | 77 ++++++++++++++++++++++++++++---
> >   include/uapi/linux/kvm.h          | 22 +++++++++
> >   4 files changed, 154 insertions(+), 8 deletions(-)
> >
>

Re: [PATCH 3/3] KVM: TDX: Exit to userspace for GetTdVmCallInfo

[Paolo Bonzini]

Il ven 20 giu 2025, 03:21 Xiaoyao Li <xiaoyao.li@intel.com> ha scritto:
>
> >               tdx->vp_enter_args.r11 = 0;
> > +             tdx->vp_enter_args.r12 = 0;
> >               tdx->vp_enter_args.r13 = 0;
> >               tdx->vp_enter_args.r14 = 0;
> > +             return 1;
>
> Though it looks OK to return all-0 for r12 == 0 and undefined case of
> r12 > 1, I prefer returning TDVMCALL_STATUS_INVALID_OPERAND for
> undefined case.


From the GHCI I wasn't sure that TDVMCALL_STATUS_INVALID_OPERAND is a
valid result at all.

Paolo

>
> So please make above "case 0:", and make the "default:" return
> TDVMCALL_STATUS_INVALID_OPERAND
>
> >       }
> > -     return 1;
> >   }
> >
> >   static int tdx_complete_simple(struct kvm_vcpu *vcpu)
> > diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> > index 6708bc88ae69..fb3b4cd8d662 100644
> > --- a/include/uapi/linux/kvm.h
> > +++ b/include/uapi/linux/kvm.h
> > @@ -461,6 +461,11 @@ struct kvm_run {
> >                                       __u64 gpa;
> >                                       __u64 size;
> >                               } get_quote;
> > +                             struct {
> > +                                     __u64 ret;
> > +                                     __u64 leaf;
> > +                                     __u64 r11, r12, r13, r14;
> > +                             } get_tdvmcall_info;
> >                       };
> >               } tdx;
> >               /* Fix the size of the union. */
>

Re: [PATCH 3/3] KVM: TDX: Exit to userspace for GetTdVmCallInfo

[Xiaoyao Li]

On 6/20/2025 8:03 PM, Paolo Bonzini wrote:
> Il ven 20 giu 2025, 03:21 Xiaoyao Li <xiaoyao.li@intel.com> ha scritto:
>>
>>>                tdx->vp_enter_args.r11 = 0;
>>> +             tdx->vp_enter_args.r12 = 0;
>>>                tdx->vp_enter_args.r13 = 0;
>>>                tdx->vp_enter_args.r14 = 0;
>>> +             return 1;
>>
>> Though it looks OK to return all-0 for r12 == 0 and undefined case of
>> r12 > 1, I prefer returning TDVMCALL_STATUS_INVALID_OPERAND for
>> undefined case.
> 
> 
>  From the GHCI I wasn't sure that TDVMCALL_STATUS_INVALID_OPERAND is a
> valid result at all.

It's part of the new GHCI change, which currently is still in draft 
state. (Sorry for not informing you)

The proposed GHCI update defines VMCALL_OPERAND_INVALID for the case of 
input R12 value is not supported. So for VMM that doesn't implement the 
enumeration for the optional leafs when r12 = 1 can return this status 
code. As well, VMM can return this status code for the case of input R12 
 >= 2, to avoid the VMM introduces its own defined behavior.

> Paolo
> 
>>
>> So please make above "case 0:", and make the "default:" return
>> TDVMCALL_STATUS_INVALID_OPERAND
>>
>>>        }
>>> -     return 1;
>>>    }
>>>
>>>    static int tdx_complete_simple(struct kvm_vcpu *vcpu)
>>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>>> index 6708bc88ae69..fb3b4cd8d662 100644
>>> --- a/include/uapi/linux/kvm.h
>>> +++ b/include/uapi/linux/kvm.h
>>> @@ -461,6 +461,11 @@ struct kvm_run {
>>>                                        __u64 gpa;
>>>                                        __u64 size;
>>>                                } get_quote;
>>> +                             struct {
>>> +                                     __u64 ret;
>>> +                                     __u64 leaf;
>>> +                                     __u64 r11, r12, r13, r14;
>>> +                             } get_tdvmcall_info;
>>>                        };
>>>                } tdx;
>>>                /* Fix the size of the union. */
>>
> 
> 

Re: [PATCH v2 0/3] TDX attestation support and GHCI fixup

[Xiaoyao Li]

On 6/20/2025 8:03 PM, Paolo Bonzini wrote:
> Il ven 20 giu 2025, 03:30 Xiaoyao Li <xiaoyao.li@intel.com> ha scritto:
>>
>> On 6/20/2025 2:01 AM, Paolo Bonzini wrote:
>>> This is a refresh of Binbin's patches with a change to the userspace
>>> API.  I am consolidating everything into a single KVM_EXIT_TDX and
>>> adding to the contract that userspace is free to ignore it *except*
>>> for having to reenter the guest with KVM_RUN.
>>>
>>> If in the future this does not work, it should be possible to introduce
>>> an opt-in interface.  Hopefully that will not be necessary.
>>
>> For <GetTdVmCallInfo> exit, I think KVM still needs to report which
>> TDVMCALL leaf will exit to userspace, to differentiate between different
>> KVMs.
> 
> 
> The interface I chose is that KVM always exits, but it initializes the
> output values such that userspace can leave them untouched for unknown
> TDVMCALLs or unknown leaves. So there is no need for this.
> 
> Querying kernel support of other services can be added later, but
> unless the GHCI adds more input or output fields to TdVmCallInfo there
> is no need to limit the userspace exit to leaf 1.

I meant the case where KVM is going to support another optional TDVMCALL 
leaf in the future, e.g., SetEventNotifyInterrupt. At that time, 
userspace needs to differentiate between old KVM which only supports 
<GetQuote> and new KVM which supports both <GetQuote> and 
<SetEventNotifyInterrupt>.

- If it's old KVM, userspace should only set <GetQuote> bit in 
GetTdVmCallInfo exit. If userspace sets <SetEventNotifyInterrupt> in 
GetTdVmCallInfo exit and enumerate to TD guest, but it's wrong info 
since the KVM doesn't support <SetEventNotifyInterrupt> and userspace 
won't get any chance to handle the guest call of <SetEventNotifyInterrupt>

- But if it's new KVM, userspace can <SetEventNotifyInterrupt> bit in 
GetTdVmCallInfo exit and enumerate to TD guest.

Anyway, its the future problem, there should be various options to 
handle it in the future. This series works for the current need.

> 
> Paolo
> 
>>
>> But it's not a must for current <GetQuote> since it exits to userspace
>> from day 0. So that we can leave the report interface until KVM needs to
>> support user exit of another TDVMCALL leaf.
>>
>>> Paolo
>>>
>>> Binbin Wu (3):
>>>     KVM: TDX: Add new TDVMCALL status code for unsupported subfuncs
>>>     KVM: TDX: Handle TDG.VP.VMCALL<GetQuote>
>>>     KVM: TDX: Exit to userspace for GetTdVmCallInfo
>>>
>>>    Documentation/virt/kvm/api.rst    | 62 ++++++++++++++++++++++++-
>>>    arch/x86/include/asm/shared/tdx.h |  1 +
>>>    arch/x86/kvm/vmx/tdx.c            | 77 ++++++++++++++++++++++++++++---
>>>    include/uapi/linux/kvm.h          | 22 +++++++++
>>>    4 files changed, 154 insertions(+), 8 deletions(-)
>>>
>>
> 

Re: [PATCH v2 0/3] TDX attestation support and GHCI fixup

[Paolo Bonzini]

On Fri, Jun 20, 2025 at 2:48 PM Xiaoyao Li <xiaoyao.li@intel.com> wrote:
> > The interface I chose is that KVM always exits, but it initializes the
> > output values such that userspace can leave them untouched for unknown
> > TDVMCALLs or unknown leaves. So there is no need for this.
> >
> > Querying kernel support of other services can be added later, but
> > unless the GHCI adds more input or output fields to TdVmCallInfo there
> > is no need to limit the userspace exit to leaf 1.
>
> I meant the case where KVM is going to support another optional TDVMCALL
> leaf in the future, e.g., SetEventNotifyInterrupt. At that time,
> userspace needs to differentiate between old KVM which only supports
> <GetQuote> and new KVM which supports both <GetQuote> and
> <SetEventNotifyInterrupt>.

Yeah, I see what you mean now. Userspace cannot know which TDVMCALL
will exit, other than GET_QUOTE which we know is in the first part.

By the way I'm tempted to implement SetupEventNotifyInterrupt as well,
it's just a handful of lines of code.

Paolo

Re: [PATCH v2 0/3] TDX attestation support and GHCI fixup

[Edgecombe, Rick P]

On Fri, 2025-06-20 at 19:24 +0200, Paolo Bonzini wrote:
> On Fri, Jun 20, 2025 at 2:48 PM Xiaoyao Li <xiaoyao.li@intel.com> wrote:
> > > The interface I chose is that KVM always exits, but it initializes the
> > > output values such that userspace can leave them untouched for unknown
> > > TDVMCALLs or unknown leaves. So there is no need for this.
> > > 
> > > Querying kernel support of other services can be added later, but
> > > unless the GHCI adds more input or output fields to TdVmCallInfo there
> > > is no need to limit the userspace exit to leaf 1.
> > 
> > I meant the case where KVM is going to support another optional TDVMCALL
> > leaf in the future, e.g., SetEventNotifyInterrupt. At that time,
> > userspace needs to differentiate between old KVM which only supports
> > <GetQuote> and new KVM which supports both <GetQuote> and
> > <SetEventNotifyInterrupt>.
> 
> Yeah, I see what you mean now. Userspace cannot know which TDVMCALL
> will exit, other than GET_QUOTE which we know is in the first part.

How about we expose the KVM supported GHCI exits in KVM_TDX_CAPABILITIES? We had
been discussing this as an option. It is not needed for the initial fixup series
I think.

It can include GHCI calls handled within KVM, and ones that are supported via
exits.

> 
> By the way I'm tempted to implement SetupEventNotifyInterrupt as well,
> it's just a handful of lines of code.

Seems ok to me. In general I think we should lean towards implementing the
minimum. It seems we are still in the learning period and have already had some
TDX arch course corrections. If a GetQuote2, SetEventNotifyInterrupt2, etc shows
up, it all starts to add up. Having a KVM_EXIT_TDX will help there, from the KVM
side at least.

[PATCH 3/3] KVM: TDX: Exit to userspace for GetTdVmCallInfo

[Paolo Bonzini]

From: Binbin Wu <binbin.wu@linux.intel.com>

Exit to userspace for TDG.VP.VMCALL<GetTdVmCallInfo> via KVM_EXIT_TDX,
to allow userspace to provide information about the support of
TDVMCALLs when r12 is 1 for the TDVMCALLs beyond the GHCI base API.

GHCI spec defines the GHCI base TDVMCALLs: <GetTdVmCallInfo>, <MapGPA>,
<ReportFatalError>, <Instruction.CPUID>, <#VE.RequestMMIO>,
<Instruction.HLT>, <Instruction.IO>, <Instruction.RDMSR> and
<Instruction.WRMSR>. They must be supported by VMM to support TDX guests.

For GetTdVmCallInfo
- When leaf (r12) to enumerate TDVMCALL functionality is set to 0,
  successful execution indicates all GHCI base TDVMCALLs listed above are
  supported.

  Update the KVM TDX document with the set of the GHCI base APIs.

- When leaf (r12) to enumerate TDVMCALL functionality is set to 1, it
  indicates the TDX guest is querying the supported TDVMCALLs beyond
  the GHCI base TDVMCALLs.
  Exit to userspace to let userspace set the TDVMCALL sub-function bit(s)
  accordingly to the leaf outputs.  KVM could set the TDVMCALL bit(s)
  supported by itself when the TDVMCALLs don't need support from userspace
  after returning from userspace and before entering guest. Currently, no
  such TDVMCALLs implemented, KVM just sets the values returned from
  userspace.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Binbin Wu <binbin.wu@linux.intel.com>
[Adjust userspace API. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 Documentation/virt/kvm/api.rst | 10 ++++++++
 arch/x86/kvm/vmx/tdx.c         | 43 ++++++++++++++++++++++++++++++----
 include/uapi/linux/kvm.h       |  5 ++++
 3 files changed, 54 insertions(+), 4 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 115ec3c2b641..9abf93ee5f65 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -7191,6 +7191,11 @@ The valid value for 'flags' is:
 					u64 gpa;
 					u64 size;
 				} get_quote;
+				struct {
+					u64 ret;
+					u64 leaf;
+					u64 r11, r12, r13, r14;
+				} get_tdvmcall_info;
 			};
 		} tdx;
 
@@ -7216,6 +7221,11 @@ queued successfully, the TDX guest can poll the status field in the
 shared-memory area to check whether the Quote generation is completed or
 not. When completed, the generated Quote is returned via the same buffer.
 
+* ``TDVMCALL_GET_TD_VM_CALL_INFO``: the guest has requested the support
+status of TDVMCALLs.  The output values for the given leaf should be
+placed in fields from ``r11`` to ``r14`` of the ``get_tdvmcall_info``
+field of the union.
+
 KVM may add support for more values in the future that may cause a userspace
 exit, even without calls to ``KVM_ENABLE_CAP`` or similar.  In this case,
 it will enter with output fields already valid; in the common case, the
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index b619a3478983..1ad20c273f3b 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1451,18 +1451,53 @@ static int tdx_emulate_mmio(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+static int tdx_complete_get_td_vm_call_info(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+
+	tdvmcall_set_return_code(vcpu, vcpu->run->tdx.get_tdvmcall_info.ret);
+
+	/*
+	 * For now, there is no TDVMCALL beyond GHCI base API supported by KVM
+	 * directly without the support from userspace, just set the value
+	 * returned from userspace.
+	 */
+	tdx->vp_enter_args.r11 = vcpu->run->tdx.get_tdvmcall_info.r11;
+	tdx->vp_enter_args.r12 = vcpu->run->tdx.get_tdvmcall_info.r12;
+	tdx->vp_enter_args.r13 = vcpu->run->tdx.get_tdvmcall_info.r13;
+	tdx->vp_enter_args.r14 = vcpu->run->tdx.get_tdvmcall_info.r14;
+
+	return 1;
+}
+
 static int tdx_get_td_vm_call_info(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_tdx *tdx = to_tdx(vcpu);
 
-	if (tdx->vp_enter_args.r12)
-		tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_INVALID_OPERAND);
-	else {
+	switch (tdx->vp_enter_args.r12) {
+	case 0:
 		tdx->vp_enter_args.r11 = 0;
+		tdx->vp_enter_args.r12 = 0;
 		tdx->vp_enter_args.r13 = 0;
 		tdx->vp_enter_args.r14 = 0;
+		tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_SUCCESS);
+		return 1;
+	case 1:
+		vcpu->run->tdx.get_tdvmcall_info.leaf = tdx->vp_enter_args.r12;
+		vcpu->run->exit_reason = KVM_EXIT_TDX;
+		vcpu->run->tdx.flags = 0;
+		vcpu->run->tdx.nr = TDVMCALL_GET_TD_VM_CALL_INFO;
+		vcpu->run->tdx.get_tdvmcall_info.ret = TDVMCALL_STATUS_SUCCESS;
+		vcpu->run->tdx.get_tdvmcall_info.r11 = 0;
+		vcpu->run->tdx.get_tdvmcall_info.r12 = 0;
+		vcpu->run->tdx.get_tdvmcall_info.r13 = 0;
+		vcpu->run->tdx.get_tdvmcall_info.r14 = 0;
+		vcpu->arch.complete_userspace_io = tdx_complete_get_td_vm_call_info;
+		return 0;
+	default:
+		tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_INVALID_OPERAND);
+		return 1;
 	}
-	return 1;
 }
 
 static int tdx_complete_simple(struct kvm_vcpu *vcpu)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index e23e7286ad1a..37891580d05d 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -462,6 +462,11 @@ struct kvm_run {
 					__u64 gpa;
 					__u64 size;
 				} get_quote;
+				struct {
+					__u64 ret;
+					__u64 leaf;
+					__u64 r11, r12, r13, r14;
+				} get_tdvmcall_info;
 			};
 		} tdx;
 		/* Fix the size of the union. */
-- 
2.43.5

Re: [PATCH 1/3] KVM: TDX: Add new TDVMCALL status code for unsupported subfuncs

[Huang, Kai]

On Thu, 2025-06-19 at 14:01 -0400, Paolo Bonzini wrote:
> With this change, TDVMCALL_STATUS_SUBFUNC_UNSUPPORTED could be sent to
> old TDX guest that do not know about it,

Nit:

... old TDX guests that do not ...