[RFC PATCH 0/4] Physical address hypercall ABI ("HVMv2")

[RFC PATCH 0/4] Physical address hypercall ABI ("HVMv2")

[Teddy Astie]

In current HVM mode, when a hypercall references a structure in guest memory,
it is passed to the hypervisor as its "linear address" (e.g virtual address for
the x86 long mode).
One of the caveats is that this linear address (GVA) is generally not directly
usable by the Xen and needs to be translated from GVA to GPA then HPA. This
implies a complex and potentially expensive lookup of the pagetables inside the
guest. This can be significant, especially if the P2M cannot use efficiently
superpages (or with e.g XSA-304).

This proposal introduce a new mode where all addresses used for hypercalls are
GPADDR instead of GVADDR, therefore, looking up the HPA related to this GPA
only needs a P2M lookup and not looking through the inside-guest pagetables.

This mode is opt-in and must be enabled explicitely by the toolstack.

This is also mandatory for confidential-computing guests (e.g SEV) where the
guest pagetable are not visible to the hypervisor.

In a synthetic xtf-based hypercall benchmark (VCPUOP_get_runstate_info loop),
it gives broadly a 30% overhead reduction when tested on a AMD EPYC 9124.

This serie only implement support for x86 yet this ABI is also meaningful for
other architectures as well.
A separate patch for adding support in Linux is planned.

Teddy Astie (4):
  xen: Introduce physaddr_abi CDF flag
  x86/hvm: Consider phyaddr_abi when copying from/to guest memory
  x86/public: Expose physaddr_abi through Xen HVM CPUID leaf
  libxl: Add support for enabling physaddr_abi

 tools/include/libxl.h               |  5 +++++
 tools/libs/light/libxl_create.c     |  4 ++++
 tools/libs/light/libxl_types.idl    |  1 +
 tools/xl/xl_parse.c                 |  1 +
 xen/arch/x86/cpuid.c                |  3 +++
 xen/arch/x86/hvm/hvm.c              | 17 ++++++++++++++---
 xen/common/domain.c                 | 10 +++++++++-
 xen/include/public/arch-x86/cpuid.h |  2 ++
 xen/include/public/domctl.h         |  4 +++-
 xen/include/xen/sched.h             |  6 ++++++
 10 files changed, 48 insertions(+), 5 deletions(-)

-- 
2.47.2



Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[RFC PATCH 1/4] xen: Introduce physaddr_abi CDF flag

[Teddy Astie]

Introduce the physaddr_abi flag to indicate that the guest uses
physical addresses (gpaddr) instead of virtual addresses when
performing hypercalls.

Signed-off-by: Teddy Astie <teddy.astie@vates.tech>
---
 xen/common/domain.c         | 10 +++++++++-
 xen/include/public/domctl.h |  4 +++-
 xen/include/xen/sched.h     |  6 ++++++
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/xen/common/domain.c b/xen/common/domain.c
index 585fd726a9..1a1e51c32d 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -716,12 +716,14 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
     bool hap = config->flags & XEN_DOMCTL_CDF_hap;
     bool iommu = config->flags & XEN_DOMCTL_CDF_iommu;
     bool vpmu = config->flags & XEN_DOMCTL_CDF_vpmu;
+    bool physaddr_abi = config->flags & XEN_DOMCTL_CDF_physaddr_abi;
 
     if ( config->flags &
          ~(XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
            XEN_DOMCTL_CDF_s3_integrity | XEN_DOMCTL_CDF_oos_off |
            XEN_DOMCTL_CDF_xs_domain | XEN_DOMCTL_CDF_iommu |
-           XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu) )
+           XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu |
+           XEN_DOMCTL_CDF_physaddr_abi) )
     {
         dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
         return -EINVAL;
@@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( physaddr_abi && !hvm )
+    {
+        dprintk(XENLOG_INFO, "Physical address ABI requested for non-HVM guest");
+        return -EINVAL;
+    }
+
     if ( iommu )
     {
         if ( config->iommu_opts & ~XEN_DOMCTL_IOMMU_no_sharept )
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 5b2063eed9..3a77efe673 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -66,9 +66,11 @@ struct xen_domctl_createdomain {
 #define XEN_DOMCTL_CDF_nested_virt    (1U << _XEN_DOMCTL_CDF_nested_virt)
 /* Should we expose the vPMU to the guest? */
 #define XEN_DOMCTL_CDF_vpmu           (1U << 7)
+/* Do the guest use physical addresses for hypercalls ? */
+#define XEN_DOMCTL_CDF_physaddr_abi   (1U << 8)
 
 /* Max XEN_DOMCTL_CDF_* constant.  Used for ABI checking. */
-#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_vpmu
+#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_physaddr_abi
 
     uint32_t flags;
 
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 559d201e0c..a29c63c737 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -1201,6 +1201,12 @@ static always_inline bool hap_enabled(const struct domain *d)
         evaluate_nospec(d->options & XEN_DOMCTL_CDF_hap);
 }
 
+static always_inline bool is_hvm_physaddr_abi(const struct domain *d)
+{
+    return IS_ENABLED(CONFIG_HVM) &&
+        evaluate_nospec(d->options & XEN_DOMCTL_CDF_physaddr_abi);
+}
+
 static inline bool is_hwdom_pinned_vcpu(const struct vcpu *v)
 {
     return (is_hardware_domain(v->domain) &&
-- 
2.47.2



Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [RFC PATCH 1/4] xen: Introduce physaddr_abi CDF flag

[Jan Beulich]

On 18.04.2025 16:18, Teddy Astie wrote:
> @@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
>          return -EINVAL;
>      }
>  
> +    if ( physaddr_abi && !hvm )
> +    {
> +        dprintk(XENLOG_INFO, "Physical address ABI requested for non-HVM guest");
> +        return -EINVAL;
> +    }

Why this restriction?

Jan

Re: [RFC PATCH 1/4] xen: Introduce physaddr_abi CDF flag

[Teddy Astie]

Le 30/04/2025 à 17:59, Jan Beulich a écrit :
> On 18.04.2025 16:18, Teddy Astie wrote:
>> @@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
>>           return -EINVAL;
>>       }
>>
>> +    if ( physaddr_abi && !hvm )
>> +    {
>> +        dprintk(XENLOG_INFO, "Physical address ABI requested for non-HVM guest");
>> +        return -EINVAL;
>> +    }

>
> Why this restriction?
>

physaddr_abi changes how copy_from/to_guest works to make it use GPA
instead of GVA. As non-HVM probably means PV guest, it would mean
something like PV guest hypercalls uses physical addresses (derived from
MFN?) instead of virtual addresses, which would not really be practical
for both the guest and the hypervisor.

> Jan
>

Teddy


Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [RFC PATCH 1/4] xen: Introduce physaddr_abi CDF flag

[Jan Beulich]

On 02.05.2025 13:55, Teddy Astie wrote:
> Le 30/04/2025 à 17:59, Jan Beulich a écrit :
>> On 18.04.2025 16:18, Teddy Astie wrote:
>>> @@ -745,6 +747,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
>>>           return -EINVAL;
>>>       }
>>>   
>>> +    if ( physaddr_abi && !hvm )
>>> +    {
>>> +        dprintk(XENLOG_INFO, "Physical address ABI requested for non-HVM guest");
>>> +        return -EINVAL;
>>> +    }
> 
>>
>> Why this restriction?
>>
> 
> physaddr_abi changes how copy_from/to_guest works to make it use GPA 
> instead of GVA. As non-HVM probably means PV guest, it would mean 
> something like PV guest hypercalls uses physical addresses (derived from 
> MFN?)

Machine addresses, yes (hence MFN). If it was PFNs / (pseudo-)physical
addresses, ...

> instead of virtual addresses, which would not really be practical 
> for both the guest and the hypervisor.

... I'd maybe agree here.

Jan

[RFC PATCH 3/4] x86/public: Expose physaddr_abi through Xen HVM CPUID leaf

[Teddy Astie]

The guest needs to know this information before attempting to make a hypercall
(especially ones to fetch XENFEAT_*).

Signed-off-by: Teddy Astie <teddy.astie@vates.tech>
---
 xen/arch/x86/cpuid.c                | 3 +++
 xen/include/public/arch-x86/cpuid.h | 2 ++
 2 files changed, 5 insertions(+)

diff --git a/xen/arch/x86/cpuid.c b/xen/arch/x86/cpuid.c
index 8dc68945f7..a721499920 100644
--- a/xen/arch/x86/cpuid.c
+++ b/xen/arch/x86/cpuid.c
@@ -153,6 +153,9 @@ static void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,
          */
         res->a |= XEN_HVM_CPUID_UPCALL_VECTOR;
 
+        /* Indicate if the guest uses the physical addresses hypercall ABI. */
+        if ( is_hvm_physaddr_abi(v->domain) )
+            res->a |= XEN_HVM_CPUID_PHYS_ADDR_ABI;
         break;
 
     case 5: /* PV-specific parameters */
diff --git a/xen/include/public/arch-x86/cpuid.h b/xen/include/public/arch-x86/cpuid.h
index 3bb0dd249f..b8ceb833d7 100644
--- a/xen/include/public/arch-x86/cpuid.h
+++ b/xen/include/public/arch-x86/cpuid.h
@@ -106,6 +106,8 @@
  * bound to event channels.
  */
 #define XEN_HVM_CPUID_UPCALL_VECTOR    (1u << 6)
+/* Hypercalls are using physical addresses instead of linear ones. */
+#define XEN_HVM_CPUID_PHYS_ADDR_ABI    (1u << 7)
 
 /*
  * Leaf 6 (0x40000x05)
-- 
2.47.2



Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[RFC PATCH 2/4] x86/hvm: Consider phyaddr_abi when copying from/to guest memory

[Teddy Astie]

Signed-off-by: Teddy Astie <teddy.astie@vates.tech>
---
 xen/arch/x86/hvm/hvm.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 6f1174c512..5923166b9b 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -3499,7 +3499,11 @@ unsigned int copy_to_user_hvm(void *to, const void *from, unsigned int len)
         return 0;
     }
 
-    rc = hvm_copy_to_guest_linear((unsigned long)to, from, len, 0, NULL);
+    if ( is_hvm_physaddr_abi(current->domain) )    
+        rc = hvm_copy_to_guest_phys((unsigned long)to, from, len, current);
+    else
+        rc = hvm_copy_to_guest_linear((unsigned long)to, from, len, 0, NULL);
+
     return rc ? len : 0; /* fake a copy_to_user() return code */
 }
 
@@ -3513,7 +3517,10 @@ unsigned int clear_user_hvm(void *to, unsigned int len)
         return 0;
     }
 
-    rc = hvm_copy_to_guest_linear((unsigned long)to, NULL, len, 0, NULL);
+    if ( is_hvm_physaddr_abi(current->domain) )
+        rc = hvm_copy_to_guest_phys((unsigned long)to, NULL, len, current);
+    else
+        rc = hvm_copy_to_guest_linear((unsigned long)to, NULL, len, 0, NULL);
 
     return rc ? len : 0; /* fake a clear_user() return code */
 }
@@ -3528,7 +3535,11 @@ unsigned int copy_from_user_hvm(void *to, const void *from, unsigned int len)
         return 0;
     }
 
-    rc = hvm_copy_from_guest_linear(to, (unsigned long)from, len, 0, NULL);
+    if ( is_hvm_physaddr_abi(current->domain) )
+        rc = hvm_copy_from_guest_phys(to, (unsigned long)from, len);
+    else
+        rc = hvm_copy_from_guest_linear(to, (unsigned long)from, len, 0, NULL);
+
     return rc ? len : 0; /* fake a copy_from_user() return code */
 }
 
-- 
2.47.2



Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[RFC PATCH 4/4] libxl: Add support for enabling physaddr_abi

[Teddy Astie]

Signed-off-by: Teddy Astie <teddy.astie@vates.tech>
---
 tools/include/libxl.h            | 5 +++++
 tools/libs/light/libxl_create.c  | 4 ++++
 tools/libs/light/libxl_types.idl | 1 +
 tools/xl/xl_parse.c              | 1 +
 4 files changed, 11 insertions(+)

diff --git a/tools/include/libxl.h b/tools/include/libxl.h
index b7ad7735ca..6af9a658c9 100644
--- a/tools/include/libxl.h
+++ b/tools/include/libxl.h
@@ -178,6 +178,11 @@
  */
 #define LIBXL_HAVE_BUILDINFO_EVENT_CHANNELS 1
 
+/*
+ * The libxl_domain_build_info has physaddr_abi field.
+ */
+#define LIBXL_HAVE_BUILDINFO_PHYS_ADDR_ABI 1
+
 /*
  * libxl_domain_build_info has the u.hvm.ms_vm_genid field.
  */
diff --git a/tools/libs/light/libxl_create.c b/tools/libs/light/libxl_create.c
index e03599ea99..43e8c18768 100644
--- a/tools/libs/light/libxl_create.c
+++ b/tools/libs/light/libxl_create.c
@@ -466,6 +466,7 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
 
         libxl_defbool_setdefault(&b_info->apic,                     true);
         libxl_defbool_setdefault(&b_info->nested_hvm,               false);
+        libxl_defbool_setdefault(&b_info->physaddr_abi,             false);
     }
 
     if (b_info->max_grant_version == LIBXL_MAX_GRANT_DEFAULT) {
@@ -662,6 +663,9 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
 
             if ( libxl_defbool_val(b_info->nested_hvm) )
                 create.flags |= XEN_DOMCTL_CDF_nested_virt;
+
+            if (libxl_defbool_val(b_info->physaddr_abi))
+                create.flags |= XEN_DOMCTL_CDF_physaddr_abi;
         }
 
         if (libxl_defbool_val(b_info->vpmu))
diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
index 9bb2969931..d02e84471b 100644
--- a/tools/libs/light/libxl_types.idl
+++ b/tools/libs/light/libxl_types.idl
@@ -637,6 +637,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
     ("nested_hvm",       libxl_defbool),
     ("apic",             libxl_defbool),
     ("dm_restrict",      libxl_defbool),
+    ("physaddr_abi",     libxl_defbool),
     ("tee",              libxl_tee_type),
     ("u", KeyedUnion(None, libxl_domain_type, "type",
                 [("hvm", Struct(None, [("firmware",         string),
diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
index 089a88935a..bd674684fa 100644
--- a/tools/xl/xl_parse.c
+++ b/tools/xl/xl_parse.c
@@ -1803,6 +1803,7 @@ void parse_config_data(const char *config_source,
     }
 
     xlu_cfg_get_defbool(config, "nestedhvm", &b_info->nested_hvm, 0);
+    xlu_cfg_get_defbool(config, "physaddr_abi", &b_info->physaddr_abi, 0);
 
     switch(b_info->type) {
     case LIBXL_DOMAIN_TYPE_HVM:
-- 
2.47.2



Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [RFC PATCH 0/4] Physical address hypercall ABI ("HVMv2")

[Jan Beulich]

On 18.04.2025 16:18, Teddy Astie wrote:
> In current HVM mode, when a hypercall references a structure in guest memory,
> it is passed to the hypervisor as its "linear address" (e.g virtual address for
> the x86 long mode).
> One of the caveats is that this linear address (GVA) is generally not directly
> usable by the Xen and needs to be translated from GVA to GPA then HPA. This
> implies a complex and potentially expensive lookup of the pagetables inside the
> guest. This can be significant, especially if the P2M cannot use efficiently
> superpages (or with e.g XSA-304).
> 
> This proposal introduce a new mode where all addresses used for hypercalls are
> GPADDR instead of GVADDR, therefore, looking up the HPA related to this GPA
> only needs a P2M lookup and not looking through the inside-guest pagetables.
> 
> This mode is opt-in and must be enabled explicitely by the toolstack.

Which I view as a severe downside (leaving aside the PVH Dom0 aspect): This way
a guest needs to be converted all in one go. While doable, it'll be increasingly
risky with the size of the guest kernel code base.

A prior proposal of mine was to add an indicator to hypercall numbers (e.g. to
set the top bit there), to indicate which of the two models a particular
hypercall invocation uses.

Aiui Andrew had yet different (albeit also never spelled out) plans.

Jan

Re: [RFC PATCH 0/4] Physical address hypercall ABI ("HVMv2")

[Julien Grall]

Hi,

On 22/04/2025 08:45, Jan Beulich wrote:
> On 18.04.2025 16:18, Teddy Astie wrote:
>> In current HVM mode, when a hypercall references a structure in guest memory,
>> it is passed to the hypervisor as its "linear address" (e.g virtual address for
>> the x86 long mode).
>> One of the caveats is that this linear address (GVA) is generally not directly
>> usable by the Xen and needs to be translated from GVA to GPA then HPA. This
>> implies a complex and potentially expensive lookup of the pagetables inside the
>> guest. This can be significant, especially if the P2M cannot use efficiently
>> superpages (or with e.g XSA-304).
>>
>> This proposal introduce a new mode where all addresses used for hypercalls are
>> GPADDR instead of GVADDR, therefore, looking up the HPA related to this GPA
>> only needs a P2M lookup and not looking through the inside-guest pagetables.
>>
>> This mode is opt-in and must be enabled explicitely by the toolstack.
> 
> Which I view as a severe downside (leaving aside the PVH Dom0 aspect): This way
> a guest needs to be converted all in one go. While doable, it'll be increasingly
> risky with the size of the guest kernel code base.

+1. It is not only the guest kernel, but also the firmware (UEFI, U-boot).

Furthermore, I don't think this can be easily adopted in public cloud 
where the admin for Xen and the guest will be different. So any 
indication of the ABI would have to come from the guest itself rather 
than the configuration.

Cheers,

-- 
Julien Grall

Re: [RFC PATCH 0/4] Physical address hypercall ABI ("HVMv2")

[Teddy Astie]

Le 01/05/2025 à 13:14, Julien Grall a écrit :
> Hi,
>
> On 22/04/2025 08:45, Jan Beulich wrote:
>> On 18.04.2025 16:18, Teddy Astie wrote:
>>> In current HVM mode, when a hypercall references a structure in guest
>>> memory,
>>> it is passed to the hypervisor as its "linear address" (e.g virtual
>>> address for
>>> the x86 long mode).
>>> One of the caveats is that this linear address (GVA) is generally not
>>> directly
>>> usable by the Xen and needs to be translated from GVA to GPA then
>>> HPA. This
>>> implies a complex and potentially expensive lookup of the pagetables
>>> inside the
>>> guest. This can be significant, especially if the P2M cannot use
>>> efficiently
>>> superpages (or with e.g XSA-304).
>>>
>>> This proposal introduce a new mode where all addresses used for
>>> hypercalls are
>>> GPADDR instead of GVADDR, therefore, looking up the HPA related to
>>> this GPA
>>> only needs a P2M lookup and not looking through the inside-guest
>>> pagetables.
>>>
>>> This mode is opt-in and must be enabled explicitely by the toolstack.
>>
>> Which I view as a severe downside (leaving aside the PVH Dom0 aspect):
>> This way
>> a guest needs to be converted all in one go. While doable, it'll be
>> increasingly
>> risky with the size of the guest kernel code base.
>
> +1. It is not only the guest kernel, but also the firmware (UEFI, U-boot).
>
> Furthermore, I don't think this can be easily adopted in public cloud
> where the admin for Xen and the guest will be different. So any
> indication of the ABI would have to come from the guest itself rather
> than the configuration.
>

Makes sense, I am experimenting with a alternative design which requires
setting the 30th bit (31th is already used for viridian) of the eax
register to indicate the use of this new ABI. Also keeping a CPUID to
indicate that the feature is supported by the hypervisor (thus no need
to enable it in advance for a guest).

> Cheers,
>

Teddy


Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech