[PATCH 0/4] XSM changes for split hardware / control domain

[PATCH 0/4] XSM changes for split hardware / control domain

[Jason Andryuk]

Theses are the broad changes needed for a split hardware / control
domain.

An earlier posting gave device_model privileges to hardware domain.  For
this posting, it was split out into a new capability.  This way the
operator can choose where to run the device models without making the
hardware domain have the permissions.

The first patch add XSM_HW_PRIV for the hardware hypercalls.  Unlike the
first posting, the control domain can call these hypercalls even though
it doesn't really make sense.  The idea was to keep the control domain
all powerful from an XSM perspective.

SILO is changed to allow control, hardwware or xenstore to service
domUs.  Xenstore and hardware will use grants for PV interfaces.
Control wouldn't typically provide PV interfaces to domUs, but it is
given the permision to do so.  Again, to keep control all powerful.

xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo this is not strictly
needed.  xenconsoled could read Xen's dmesg.  If it's in hwdom, then
that permission would be required.  SYSCTL_physinfo is mainly to silence
xl messages, which are mostly cosmetic.

Jason Andryuk (4):
  xen/xsm: Add XSM_HW_PRIV
  xsm/silo: Support hwdom/control domains
  xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model
  xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

 tools/ocaml/libs/xc/xenctrl.ml          |  1 +
 tools/ocaml/libs/xc/xenctrl.mli         |  1 +
 xen/arch/arm/domain.c                   |  3 ++-
 xen/arch/arm/platform_hypercall.c       |  2 +-
 xen/arch/x86/msi.c                      |  2 +-
 xen/arch/x86/physdev.c                  | 12 ++++-----
 xen/arch/x86/platform_hypercall.c       |  2 +-
 xen/common/device-tree/dom0less-build.c |  3 +++
 xen/common/domain.c                     |  3 ++-
 xen/common/sysctl.c                     |  2 +-
 xen/drivers/passthrough/pci.c           |  5 ++--
 xen/drivers/pci/physdev.c               |  2 +-
 xen/include/public/bootfdt.h            | 12 +++++++--
 xen/include/public/domctl.h             |  4 ++-
 xen/include/xen/sched.h                 |  9 +++++++
 xen/include/xsm/dummy.h                 | 36 ++++++++++++++++++-------
 xen/include/xsm/xsm.h                   |  1 +
 xen/xsm/silo.c                          | 10 +++++--
 18 files changed, 80 insertions(+), 30 deletions(-)

-- 
2.49.0

[PATCH 1/4] xen/xsm: Add XSM_HW_PRIV

[Jason Andryuk]

Xen includes disctinct concepts of a control domain (privileged) and a
hardware domain, but there is only a single XSM_PRIV check.  For dom0
this is not an issue as they are one and the same.

With hyperlaunch and its build capabilities, a non-privileged hwdom and a
privileged control domain should be possible.  Today the hwdom fails the
XSM_PRIV checks for hardware-related hooks which it should be allowed
access to.

Introduce XSM_HW_PRIV, and use it to mark many of the physdev_op and
platform_op.  The hwdom is allowed access for XSM_HW_PRIV.

Make XSM_HW_PRIV a new privilege level that is given to the hardware
domain, but is not exclusive.  The control domain can still execute
XSM_HW_PRIV commands.  This is a little questionable since it's unclear
how the control domain can meaningfully execute them.  But this approach
is chosen to maintain the increasing privileges and keep control domain
fully privileged.

Testing was performed with hardware+xenstore capabilities for dom0 and a
control dom3 booted from hyperlaunch.  The additional xenstore
permissions allowed hwdom+xenstore XSM_XS_PRIV which are necesary for
xenstore.

A traditional dom0 will be both privileged and hardware domain, so it
continues to have all accesses.

Why not XSM:Flask?  XSM:Flask is fine grain, and this aims to allow
coarse grain.  domUs are still domUs.  If capabilities are meant to be a
first class citizen, they should be usable by the default XSM policy.

Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
---
 xen/arch/arm/platform_hypercall.c |  2 +-
 xen/arch/x86/msi.c                |  2 +-
 xen/arch/x86/physdev.c            | 12 ++++++------
 xen/arch/x86/platform_hypercall.c |  2 +-
 xen/drivers/passthrough/pci.c     |  5 +++--
 xen/drivers/pci/physdev.c         |  2 +-
 xen/include/xsm/dummy.h           | 20 ++++++++++++--------
 xen/include/xsm/xsm.h             |  1 +
 8 files changed, 26 insertions(+), 20 deletions(-)

diff --git a/xen/arch/arm/platform_hypercall.c b/xen/arch/arm/platform_hypercall.c
index ac55622426..a84596ae3a 100644
--- a/xen/arch/arm/platform_hypercall.c
+++ b/xen/arch/arm/platform_hypercall.c
@@ -35,7 +35,7 @@ long do_platform_op(XEN_GUEST_HANDLE_PARAM(xen_platform_op_t) u_xenpf_op)
     if ( d == NULL )
         return -ESRCH;
 
-    ret = xsm_platform_op(XSM_PRIV, op->cmd);
+    ret = xsm_platform_op(XSM_HW_PRIV, op->cmd);
     if ( ret )
         return ret;
 
diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c
index 5389bc0867..30801d980c 100644
--- a/xen/arch/x86/msi.c
+++ b/xen/arch/x86/msi.c
@@ -1360,7 +1360,7 @@ int pci_restore_msi_state(struct pci_dev *pdev)
     if ( !use_msi )
         return -EOPNOTSUPP;
 
-    ret = xsm_resource_setup_pci(XSM_PRIV,
+    ret = xsm_resource_setup_pci(XSM_HW_PRIV,
                                 (pdev->seg << 16) | (pdev->bus << 8) |
                                 pdev->devfn);
     if ( ret )
diff --git a/xen/arch/x86/physdev.c b/xen/arch/x86/physdev.c
index 4dfa1c0191..ce1ba41fa3 100644
--- a/xen/arch/x86/physdev.c
+++ b/xen/arch/x86/physdev.c
@@ -358,7 +358,7 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         ret = -EFAULT;
         if ( copy_from_guest(&apic, arg, 1) != 0 )
             break;
-        ret = xsm_apic(XSM_PRIV, currd, cmd);
+        ret = xsm_apic(XSM_HW_PRIV, currd, cmd);
         if ( ret )
             break;
         ret = ioapic_guest_read(apic.apic_physbase, apic.reg, &apic.value);
@@ -372,7 +372,7 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         ret = -EFAULT;
         if ( copy_from_guest(&apic, arg, 1) != 0 )
             break;
-        ret = xsm_apic(XSM_PRIV, currd, cmd);
+        ret = xsm_apic(XSM_HW_PRIV, currd, cmd);
         if ( ret )
             break;
         ret = ioapic_guest_write(apic.apic_physbase, apic.reg, apic.value);
@@ -388,7 +388,7 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 
         /* Use the APIC check since this dummy hypercall should still only
          * be called by the domain with access to program the ioapic */
-        ret = xsm_apic(XSM_PRIV, currd, cmd);
+        ret = xsm_apic(XSM_HW_PRIV, currd, cmd);
         if ( ret )
             break;
 
@@ -490,7 +490,7 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         if ( copy_from_guest(&dev, arg, 1) )
             ret = -EFAULT;
         else
-            ret = xsm_resource_setup_pci(XSM_PRIV,
+            ret = xsm_resource_setup_pci(XSM_HW_PRIV,
                                          (dev.seg << 16) | (dev.bus << 8) |
                                          dev.devfn) ?:
                   pci_prepare_msix(dev.seg, dev.bus, dev.devfn,
@@ -501,7 +501,7 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
     case PHYSDEVOP_pci_mmcfg_reserved: {
         struct physdev_pci_mmcfg_reserved info;
 
-        ret = xsm_resource_setup_misc(XSM_PRIV);
+        ret = xsm_resource_setup_misc(XSM_HW_PRIV);
         if ( ret )
             break;
 
@@ -567,7 +567,7 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         if ( setup_gsi.gsi < 0 || setup_gsi.gsi >= nr_irqs_gsi )
             break;
 
-        ret = xsm_resource_setup_gsi(XSM_PRIV, setup_gsi.gsi);
+        ret = xsm_resource_setup_gsi(XSM_HW_PRIV, setup_gsi.gsi);
         if ( ret )
             break;
 
diff --git a/xen/arch/x86/platform_hypercall.c b/xen/arch/x86/platform_hypercall.c
index 90abd3197f..8efb4ad05f 100644
--- a/xen/arch/x86/platform_hypercall.c
+++ b/xen/arch/x86/platform_hypercall.c
@@ -228,7 +228,7 @@ ret_t do_platform_op(
     if ( op->interface_version != XENPF_INTERFACE_VERSION )
         return -EACCES;
 
-    ret = xsm_platform_op(XSM_PRIV, op->cmd);
+    ret = xsm_platform_op(XSM_HW_PRIV, op->cmd);
     if ( ret )
         return ret;
 
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 3edcfa8a04..9de7f0d358 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -672,7 +672,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
     else
         type = "device";
 
-    ret = xsm_resource_plug_pci(XSM_PRIV, (seg << 16) | (bus << 8) | devfn);
+    ret = xsm_resource_plug_pci(XSM_HW_PRIV, (seg << 16) | (bus << 8) | devfn);
     if ( ret )
         return ret;
 
@@ -824,7 +824,8 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn)
     struct pci_dev *pdev;
     int ret;
 
-    ret = xsm_resource_unplug_pci(XSM_PRIV, (seg << 16) | (bus << 8) | devfn);
+    ret = xsm_resource_unplug_pci(XSM_HW_PRIV,
+                                  (seg << 16) | (bus << 8) | devfn);
     if ( ret )
         return ret;
 
diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
index 0161a85e1e..c223611dfb 100644
--- a/xen/drivers/pci/physdev.c
+++ b/xen/drivers/pci/physdev.c
@@ -86,7 +86,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
                         dev_reset.dev.bus,
                         dev_reset.dev.devfn);
 
-        ret = xsm_resource_setup_pci(XSM_PRIV, sbdf.sbdf);
+        ret = xsm_resource_setup_pci(XSM_HW_PRIV, sbdf.sbdf);
         if ( ret )
             break;
 
diff --git a/xen/include/xsm/dummy.h b/xen/include/xsm/dummy.h
index 9227205fcd..d8df3f66c4 100644
--- a/xen/include/xsm/dummy.h
+++ b/xen/include/xsm/dummy.h
@@ -94,6 +94,10 @@ static always_inline int xsm_default_action(
         if ( target && evaluate_nospec(src->target == target) )
             return 0;
         fallthrough;
+    case XSM_HW_PRIV:
+        if ( action == XSM_HW_PRIV && is_hardware_domain(src) )
+            return 0;
+        fallthrough;
     case XSM_PRIV:
         if ( is_control_domain(src) )
             return 0;
@@ -275,7 +279,7 @@ static XSM_INLINE int cf_check xsm_console_io(
     if ( cmd == CONSOLEIO_write )
         return xsm_default_action(XSM_HOOK, d, NULL);
 #endif
-    return xsm_default_action(XSM_PRIV, d, NULL);
+    return xsm_default_action(XSM_HW_PRIV, d, NULL);
 }
 
 static XSM_INLINE int cf_check xsm_profile(
@@ -455,33 +459,33 @@ static XSM_INLINE int cf_check xsm_resource_unplug_core(XSM_DEFAULT_VOID)
 static XSM_INLINE int cf_check xsm_resource_plug_pci(
     XSM_DEFAULT_ARG uint32_t machine_bdf)
 {
-    XSM_ASSERT_ACTION(XSM_PRIV);
+    XSM_ASSERT_ACTION(XSM_HW_PRIV);
     return xsm_default_action(action, current->domain, NULL);
 }
 
 static XSM_INLINE int cf_check xsm_resource_unplug_pci(
     XSM_DEFAULT_ARG uint32_t machine_bdf)
 {
-    XSM_ASSERT_ACTION(XSM_PRIV);
+    XSM_ASSERT_ACTION(XSM_HW_PRIV);
     return xsm_default_action(action, current->domain, NULL);
 }
 
 static XSM_INLINE int cf_check xsm_resource_setup_pci(
     XSM_DEFAULT_ARG uint32_t machine_bdf)
 {
-    XSM_ASSERT_ACTION(XSM_PRIV);
+    XSM_ASSERT_ACTION(XSM_HW_PRIV);
     return xsm_default_action(action, current->domain, NULL);
 }
 
 static XSM_INLINE int cf_check xsm_resource_setup_gsi(XSM_DEFAULT_ARG int gsi)
 {
-    XSM_ASSERT_ACTION(XSM_PRIV);
+    XSM_ASSERT_ACTION(XSM_HW_PRIV);
     return xsm_default_action(action, current->domain, NULL);
 }
 
 static XSM_INLINE int cf_check xsm_resource_setup_misc(XSM_DEFAULT_VOID)
 {
-    XSM_ASSERT_ACTION(XSM_PRIV);
+    XSM_ASSERT_ACTION(XSM_HW_PRIV);
     return xsm_default_action(action, current->domain, NULL);
 }
 
@@ -673,7 +677,7 @@ static XSM_INLINE int cf_check xsm_mem_sharing(XSM_DEFAULT_ARG struct domain *d)
 
 static XSM_INLINE int cf_check xsm_platform_op(XSM_DEFAULT_ARG uint32_t op)
 {
-    XSM_ASSERT_ACTION(XSM_PRIV);
+    XSM_ASSERT_ACTION(XSM_HW_PRIV);
     return xsm_default_action(action, current->domain, NULL);
 }
 
@@ -701,7 +705,7 @@ static XSM_INLINE int cf_check xsm_mem_sharing_op(
 static XSM_INLINE int cf_check xsm_apic(
     XSM_DEFAULT_ARG struct domain *d, int cmd)
 {
-    XSM_ASSERT_ACTION(XSM_PRIV);
+    XSM_ASSERT_ACTION(XSM_HW_PRIV);
     return xsm_default_action(action, d, NULL);
 }
 
diff --git a/xen/include/xsm/xsm.h b/xen/include/xsm/xsm.h
index 24acc16125..264db4d8ee 100644
--- a/xen/include/xsm/xsm.h
+++ b/xen/include/xsm/xsm.h
@@ -36,6 +36,7 @@ enum xsm_default {
     XSM_DM_PRIV,  /* Device model can perform on its target domain */
     XSM_TARGET,   /* Can perform on self or your target domain */
     XSM_PRIV,     /* Privileged - normally restricted to dom0 */
+    XSM_HW_PRIV,  /* Hardware Privileged - normally restricted to dom0/hwdom */
     XSM_XS_PRIV,  /* Xenstore domain - can do some privileged operations */
     XSM_OTHER     /* Something more complex */
 };
-- 
2.49.0

[PATCH 2/4] xsm/silo: Support hwdom/control domains

[Jason Andryuk]

In a disaggregated environment, dom0 is split into Control, Hardware,
and Xenstore domains, along with domUs.  The is_control_domain() check
is not sufficient to handle all these cases.  Add is_priv_domain() to
support allowing for the various domains.

The purpose of SILO mode is to prevent domUs from interacting with each
other.  But dom0 was allowed to communicate with domUs to provide
services.  As the disaggregation of dom0, Control, Hardware and Xenstore
are all service domains that need to communicate with other domains.

To provide xenstore connections, the Xenstore domain must be allowed to
connect via grants and event channels.  Xenstore domain must also be
allowed to connect to Control and Hardware to provide xenstore to them.

Hardware domain will provide PV devices to domains, so it must be
allowed to connect to domains.

That leaves Control.  Xenstore and Hardware would already allow access
to Control, so it can obtain services that way.  Control should be
"privileged", which would mean it can make the connections.  But with
Xenstore and Hardware providing their services to domUs, there may not
be a reason to allow Control to use grants or event channels with domUs.
Still, Control is privileged, so it should be allowed to do something if
it chooses.  Establishing a grant, or event channel requires action on
both sides, so allow for the possibility.  This does open up an argo
wildcard ring from domUs, FWIW.

This silo check is for grants, event channels and argo.  The dummy
policy handles other calls, so Hardware is prevented from foreign
mapping Control's memory with that.

Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
---
v2:
Add xenstore domain
Expand commit message
Remove always_inline
---
 xen/xsm/silo.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/xen/xsm/silo.c b/xen/xsm/silo.c
index b89b364287..db48705674 100644
--- a/xen/xsm/silo.c
+++ b/xen/xsm/silo.c
@@ -20,6 +20,12 @@
 #define XSM_NO_WRAPPERS
 #include <xsm/dummy.h>
 
+static bool is_priv_domain(const struct domain *d)
+{
+    return is_xenstore_domain(d) || is_hardware_domain(d) ||
+           is_control_domain(d);
+}
+
 /*
  * Check if inter-domain communication is allowed.
  * Return true when pass check.
@@ -29,8 +35,8 @@ static bool silo_mode_dom_check(const struct domain *ldom,
 {
     const struct domain *currd = current->domain;
 
-    return (is_control_domain(currd) || is_control_domain(ldom) ||
-            is_control_domain(rdom) || ldom == rdom);
+    return (is_priv_domain(currd) || is_priv_domain(ldom) ||
+            is_priv_domain(rdom) || ldom == rdom);
 }
 
 static int cf_check silo_evtchn_unbound(
-- 
2.49.0

[PATCH 3/4] xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model

[Jason Andryuk]

To add more flexibility in system configuration add the new
DOMAIN_CAPS_DEVICE_MODEL flag and XEN_DOMCTL_CDF_device_model.

Thie new flag corresponds to allowing XSM_DM_PRIV for the domain.  This
will enable running device model emulators (QEMU) from the assigne
domain for multiple target domains.

Stubdoms assign target allowing the stubdom to serve as the device
model for a single domain.  This new flag allows the single domain to
provide emulators for multiple guests.

The specific scenario is a disaggregated system with the hardware domain
providing device models for muitple guest domains.

The OCaml code needs the flag added in as well.

Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
---
 tools/ocaml/libs/xc/xenctrl.ml          |  1 +
 tools/ocaml/libs/xc/xenctrl.mli         |  1 +
 xen/arch/arm/domain.c                   |  3 ++-
 xen/common/device-tree/dom0less-build.c |  3 +++
 xen/common/domain.c                     |  3 ++-
 xen/include/public/bootfdt.h            | 12 ++++++++++--
 xen/include/public/domctl.h             |  4 +++-
 xen/include/xen/sched.h                 |  9 +++++++++
 xen/include/xsm/dummy.h                 |  2 ++
 9 files changed, 33 insertions(+), 5 deletions(-)

diff --git a/tools/ocaml/libs/xc/xenctrl.ml b/tools/ocaml/libs/xc/xenctrl.ml
index 2690f9a923..ef0c5dca2a 100644
--- a/tools/ocaml/libs/xc/xenctrl.ml
+++ b/tools/ocaml/libs/xc/xenctrl.ml
@@ -70,6 +70,7 @@ type domain_create_flag =
   | CDF_IOMMU
   | CDF_NESTED_VIRT
   | CDF_VPMU
+  | CDF_DEVICE_MODEL
 
 type domain_create_iommu_opts =
   | IOMMU_NO_SHAREPT
diff --git a/tools/ocaml/libs/xc/xenctrl.mli b/tools/ocaml/libs/xc/xenctrl.mli
index febbe1f6ae..c0156fa5c6 100644
--- a/tools/ocaml/libs/xc/xenctrl.mli
+++ b/tools/ocaml/libs/xc/xenctrl.mli
@@ -63,6 +63,7 @@ type domain_create_flag =
   | CDF_IOMMU
   | CDF_NESTED_VIRT
   | CDF_VPMU
+  | CDF_DEVICE_MODEL
 
 type domain_create_iommu_opts =
   | IOMMU_NO_SHAREPT
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 45aeb8bddc..12fda0762f 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -612,7 +612,8 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
     unsigned int max_vcpus;
     unsigned int flags_required = (XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap);
     unsigned int flags_optional = (XEN_DOMCTL_CDF_iommu | XEN_DOMCTL_CDF_vpmu |
-                                   XEN_DOMCTL_CDF_xs_domain );
+                                   XEN_DOMCTL_CDF_xs_domain |
+                                   XEN_DOMCTL_CDF_device_model);
     unsigned int sve_vl_bits = sve_decode_vl(config->arch.sve_vl);
 
     if ( (config->flags & ~flags_optional) != flags_required )
diff --git a/xen/common/device-tree/dom0less-build.c b/xen/common/device-tree/dom0less-build.c
index 3d503c6973..993ff32f5e 100644
--- a/xen/common/device-tree/dom0less-build.c
+++ b/xen/common/device-tree/dom0less-build.c
@@ -884,6 +884,9 @@ void __init create_domUs(void)
                 d_cfg.flags |= XEN_DOMCTL_CDF_xs_domain;
                 d_cfg.max_evtchn_port = -1;
             }
+
+            if ( val & DOMAIN_CAPS_DEVICE_MODEL )
+                d_cfg.flags |= XEN_DOMCTL_CDF_device_model;
         }
 
         if ( dt_find_property(node, "xen,static-mem", NULL) )
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 153cd75340..ab2c8f864d 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -721,7 +721,8 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
          ~(XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
            XEN_DOMCTL_CDF_s3_integrity | XEN_DOMCTL_CDF_oos_off |
            XEN_DOMCTL_CDF_xs_domain | XEN_DOMCTL_CDF_iommu |
-           XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu) )
+           XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu |
+           XEN_DOMCTL_CDF_device_model) )
     {
         dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
         return -EINVAL;
diff --git a/xen/include/public/bootfdt.h b/xen/include/public/bootfdt.h
index 86c46b42a9..c6b5afc76a 100644
--- a/xen/include/public/bootfdt.h
+++ b/xen/include/public/bootfdt.h
@@ -25,7 +25,15 @@
 #define DOMAIN_CAPS_HARDWARE (1U << 1)
 /* Xenstore domain. */
 #define DOMAIN_CAPS_XENSTORE (1U << 2)
-#define DOMAIN_CAPS_MASK     (DOMAIN_CAPS_CONTROL | DOMAIN_CAPS_HARDWARE | \
-                              DOMAIN_CAPS_XENSTORE)
+/*
+ * Device model capability allows the use of the dm_op hypercalls to provide
+ * the device model emulation (run QEMU) for other domains.  This is a
+ * subset of the Control capability which can be granted to the
+ * Hardware domain for running QEMU.
+ */
+#define DOMAIN_CAPS_DEVICE_MODEL (1U << 3)
+
+#define DOMAIN_CAPS_MASK     (DOMAIN_CAPS_CONTROL  | DOMAIN_CAPS_HARDWARE | \
+                              DOMAIN_CAPS_XENSTORE | DOMAIN_CAPS_DEVICE_MODEL )
 
 #endif /* __XEN_PUBLIC_BOOTFDT_H__ */
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 5b2063eed9..2280489be2 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -66,9 +66,11 @@ struct xen_domctl_createdomain {
 #define XEN_DOMCTL_CDF_nested_virt    (1U << _XEN_DOMCTL_CDF_nested_virt)
 /* Should we expose the vPMU to the guest? */
 #define XEN_DOMCTL_CDF_vpmu           (1U << 7)
+/* Allow domain to provide device model for multiple other domains */
+#define XEN_DOMCTL_CDF_device_model   (1U << 8)
 
 /* Max XEN_DOMCTL_CDF_* constant.  Used for ABI checking. */
-#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_vpmu
+#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_device_model
 
     uint32_t flags;
 
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index fe53d4fab7..9863603d93 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -1148,6 +1148,15 @@ static always_inline bool is_control_domain(const struct domain *d)
     return evaluate_nospec(d->is_privileged);
 }
 
+/* This check is for functionality specific to a device model domain */
+static always_inline bool is_dm_domain(const struct domain *d)
+{
+    if ( IS_ENABLED(CONFIG_PV_SHIM_EXCLUSIVE) )
+        return false;
+
+    return evaluate_nospec(d->options & XEN_DOMCTL_CDF_device_model);
+}
+
 #define VM_ASSIST(d, t) (test_bit(VMASST_TYPE_ ## t, &(d)->vm_assist))
 
 static always_inline bool is_pv_domain(const struct domain *d)
diff --git a/xen/include/xsm/dummy.h b/xen/include/xsm/dummy.h
index d8df3f66c4..477fadaefd 100644
--- a/xen/include/xsm/dummy.h
+++ b/xen/include/xsm/dummy.h
@@ -91,6 +91,8 @@ static always_inline int xsm_default_action(
             return 0;
         fallthrough;
     case XSM_DM_PRIV:
+        if ( is_dm_domain(src) )
+            return 0;
         if ( target && evaluate_nospec(src->target == target) )
             return 0;
         fallthrough;
-- 
2.49.0

[PATCH 4/4] xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

[Jason Andryuk]

Allow the hwdom to access the console, and to access physical
information about the system.

xenconsoled can read Xen's dmesg.  If it's in hwdom, then that
permission would be required.

SYSCTL_physinfo is mainly to silence xl messages:

$ xl list
libxl: error: libxl_utils.c:818:libxl_cpu_bitmap_alloc: failed to retrieve the maximum number of cpus

Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
---
This is not strictly needed.
---
 xen/common/sysctl.c     |  2 +-
 xen/include/xsm/dummy.h | 14 ++++++++++++--
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/xen/common/sysctl.c b/xen/common/sysctl.c
index c2d99ae12e..89d5176f4d 100644
--- a/xen/common/sysctl.c
+++ b/xen/common/sysctl.c
@@ -42,7 +42,7 @@ long do_sysctl(XEN_GUEST_HANDLE_PARAM(xen_sysctl_t) u_sysctl)
     if ( op->interface_version != XEN_SYSCTL_INTERFACE_VERSION )
         return -EACCES;
 
-    ret = xsm_sysctl(XSM_PRIV, op->cmd);
+    ret = xsm_sysctl(XSM_OTHER, op->cmd);
     if ( ret )
         return ret;
 
diff --git a/xen/include/xsm/dummy.h b/xen/include/xsm/dummy.h
index 477fadaefd..5e806dc241 100644
--- a/xen/include/xsm/dummy.h
+++ b/xen/include/xsm/dummy.h
@@ -188,8 +188,18 @@ static XSM_INLINE int cf_check xsm_domctl(
 
 static XSM_INLINE int cf_check xsm_sysctl(XSM_DEFAULT_ARG int cmd)
 {
-    XSM_ASSERT_ACTION(XSM_PRIV);
-    return xsm_default_action(action, current->domain, NULL);
+    XSM_ASSERT_ACTION(XSM_OTHER);
+    switch ( cmd )
+    {
+    case XEN_SYSCTL_readconsole:
+        return xsm_default_action(XSM_HW_PRIV, current->domain, NULL);
+    case XEN_SYSCTL_physinfo:
+        if ( is_hardware_domain(current->domain) )
+            return xsm_default_action(XSM_HW_PRIV, current->domain, NULL);
+        fallthrough;
+    default:
+        return xsm_default_action(XSM_PRIV, current->domain, NULL);
+    }
 }
 
 static XSM_INLINE int cf_check xsm_readconsole(XSM_DEFAULT_ARG uint32_t clear)
-- 
2.49.0

Re: [PATCH 1/4] xen/xsm: Add XSM_HW_PRIV

[Jason Andryuk]

On 2025-06-11 09:02, Jan Beulich wrote:
> On 11.06.2025 00:57, Jason Andryuk wrote:
>> Xen includes disctinct concepts of a control domain (privileged) and a
>> hardware domain, but there is only a single XSM_PRIV check.  For dom0
>> this is not an issue as they are one and the same.
>>
>> With hyperlaunch and its build capabilities, a non-privileged hwdom and a
>> privileged control domain should be possible.  Today the hwdom fails the
>> XSM_PRIV checks for hardware-related hooks which it should be allowed
>> access to.
>>
>> Introduce XSM_HW_PRIV, and use it to mark many of the physdev_op and
>> platform_op.  The hwdom is allowed access for XSM_HW_PRIV.
>>
>> Make XSM_HW_PRIV a new privilege level that is given to the hardware
>> domain, but is not exclusive.  The control domain can still execute
>> XSM_HW_PRIV commands.  This is a little questionable since it's unclear
>> how the control domain can meaningfully execute them.  But this approach
>> is chosen to maintain the increasing privileges and keep control domain
>> fully privileged.
> 
> I consider this conceptually wrong. "Control" aiui refers to software
> (e.g. VMs or system-wide settings), but there ought to be a (pretty?)
> clear boundary between control and hardware domains, imo. As to
> "pretty" - should any overlap be necessary (xms_machine_memory_map()
> comes to mind), such would need handling specially then, I think. At
> the same time: The more of an overlap there is, the less clear it is
> why the two want/need separating in the first place.

So you are in favor of splitting control and hardware into distinct 
sets?  I am okay with this.  I implemented that originally, but I 
started doubting it.  Mainly, should control be denied any permission?

We aren't using the toolstack to build domains - dom0less or Hyperlaunch 
handles that.  This avoids issues that might arise from running the 
toolstack.

Thanks for your feedback.

-Jason

Re: [PATCH 2/4] xsm/silo: Support hwdom/control domains

[Jason Andryuk]

On 2025-06-11 09:17, Jan Beulich wrote:
> On 11.06.2025 00:57, Jason Andryuk wrote:
>> In a disaggregated environment, dom0 is split into Control, Hardware,
>> and Xenstore domains, along with domUs.  The is_control_domain() check
>> is not sufficient to handle all these cases.  Add is_priv_domain() to
>> support allowing for the various domains.
>>
>> The purpose of SILO mode is to prevent domUs from interacting with each
>> other.  But dom0 was allowed to communicate with domUs to provide
>> services.  As the disaggregation of dom0, Control, Hardware and Xenstore
>> are all service domains that need to communicate with other domains.
>>
>> To provide xenstore connections, the Xenstore domain must be allowed to
>> connect via grants and event channels.  Xenstore domain must also be
>> allowed to connect to Control and Hardware to provide xenstore to them.
> 
> Are you suggesting that SILO at present is incompatible with a Xenstore
> domain? silo_mode_dom_check() in its original form has no special
> precautions, after all.

Yes, it is incompatible with the current silo_mode_dom_check().  Only 
Control domain is allowed to use grants and event channels with a domU. 
A Xenstore domain would be denied.

Xenstore stubdom only exists for x86 today.  My limited attempts to run 
xenstored in an dedicated Xenstore ARM Linux domain have failed.

>> Hardware domain will provide PV devices to domains, so it must be
>> allowed to connect to domains.
> 
> As a built-in policy, isn't this already going too far? There could
> conceivably be configurations with only pass-through devices in use, in
> which case neither grants nor the event channels operations intercepted
> by SILO would be required.

Such a domain wouldn't have any PV devices configured?  I don't think 
this changes anything compared to today.

Both sides need to be configured and opt-in.  Hardware is a system 
domain, so it should be possible to allow grants and event channels. 
But they won't be used unless configured.

>> That leaves Control.  Xenstore and Hardware would already allow access
>> to Control, so it can obtain services that way.  Control should be
>> "privileged", which would mean it can make the connections.  But with
>> Xenstore and Hardware providing their services to domUs, there may not
>> be a reason to allow Control to use grants or event channels with domUs.
>> Still, Control is privileged, so it should be allowed to do something if
>> it chooses.  Establishing a grant, or event channel requires action on
>> both sides, so allow for the possibility.  This does open up an argo
>> wildcard ring from domUs, FWIW.
> 
> Along the lines of my reply to patch 1, I think Hardware and Control
> need to have a pretty strong boundary between them. It's hard to see,
> for example, whether grant map/copy/transfer would indeed make sense
> between the two.

The Hardware domain might provide a PV device to Control?

I've tested removing control:
static bool is_priv_domain(const struct domain *d)
{
     return is_xenstore_domain(d) || is_hardware_domain(d);
}

And that works in my limited ARM dom0less testing.  The toolstack isn't 
really exercised in that case.  It seems strange that the privileged 
control domain is *not* allowed though.

> Similarly I'm not convinced a strong boundary isn't also needed
> between Xenstore and Hardware.

If hardware is providing PV devices to domains, it will need access to 
Xenstore.  I don't see how you can get around it.

I tried to explain this in the first paragraph.  SILO's purpose was to 
isolate domUs from each other, but allow it to access dom0.  dom0 
embodies the control, hardware, and xenstore capabilities.  So as a 
first approximation, each of Control, Hardware, and Xenstore should be 
allowed to communicate with domUs.

domUs needs to communicate with Xenstore and Hardware for PV devices.

Xenstore provides Xenstore access to Hardware.

Control would want Xenstore access.

I don't know if this helps, but here's a table:

     | CTL | HW  | XS  | domU
----------------------------
CTL |     |  ?  |  y  |  ?
HW  |  ?  |     |  y  |  y
XS  |  y  |  y  |     |  y
domU|  ?  |  y  |  y  |  x

Control and Hardware would be y if we allow PV devices

Control and domUs - I don't have an immediate rational for them.  Except 
that Control is privileged.  I've been running xenconsoled in Hardware. 
If xenconsoled is in Control, then access would be required.

>> --- a/xen/xsm/silo.c
>> +++ b/xen/xsm/silo.c
>> @@ -20,6 +20,12 @@
>>   #define XSM_NO_WRAPPERS
>>   #include <xsm/dummy.h>
>>   
>> +static bool is_priv_domain(const struct domain *d)
>> +{
>> +    return is_xenstore_domain(d) || is_hardware_domain(d) ||
>> +           is_control_domain(d);
>> +}
> 
> This construct expands to two evaluate_nospec(), which likely isn't
> wanted. Some open-coding may be pretty much unavoidable here.

Thanks, yes, good point.

> (I'm
> surprised it's not three, i.e. I find it odd that is_xenstore_domain()
> doesn't also use that guard.)

It looks okay to me.  There were only 2 uses until I added a 3rd in the 
dom0less code.  The XSM check has evaluate_nospec() and the other 2 uses 
aren't security critical - Setting a domain info flag, and __init code 
for dom0less.  Maybe moving the evaluate_nospec() would be safer in case 
use grows in the future, but it looks okay to me today.

Regards,
Jason

Re: [PATCH 3/4] xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model

[Jason Andryuk]

On 2025-06-11 09:24, Jan Beulich wrote:
> On 11.06.2025 00:57, Jason Andryuk wrote:
>> To add more flexibility in system configuration add the new
>> DOMAIN_CAPS_DEVICE_MODEL flag and XEN_DOMCTL_CDF_device_model.
>>
>> Thie new flag corresponds to allowing XSM_DM_PRIV for the domain.  This
>> will enable running device model emulators (QEMU) from the assigne
>> domain for multiple target domains.
>>
>> Stubdoms assign target allowing the stubdom to serve as the device
>> model for a single domain.  This new flag allows the single domain to
>> provide emulators for multiple guests.
>>
>> The specific scenario is a disaggregated system with the hardware domain
>> providing device models for muitple guest domains.
> 
> Why the hardware domain? Unless a DM also needs access to some of the
> physical hardware, it ought to run in a separate domain. Conceivably
> such a domain could service multiply guests, so maybe the "single
> target" concept presently used for stubdom simply needed extending?

One configuration is the hardware domain running QEMU for the 
virtio-gpu.  In an earlier iteration, I allowed XSM_DM_PRIV for 
is_hardware_domain().  Rightfully, there was some questioning of that 
hardcoding.  Adding a new flag allows it to be configurable.

Maybe target could be extended.  I was thinking that could be left for 
the stubdom case as it is today.  i.e. a 1-1 device model.  But a 1-N 
case could be handled this way.

Today dom0 XSM_DM_PRIV falls through to is_control_domain().  The idea 
was place a new check directly corresponding to XSM_DM_PRIV.

Regards,
Jason

Re: [PATCH 4/4] xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

[Jason Andryuk]

On 2025-06-11 09:27, Jan Beulich wrote:
> On 11.06.2025 00:57, Jason Andryuk wrote:
>> Allow the hwdom to access the console, and to access physical
>> information about the system.
>>
>> xenconsoled can read Xen's dmesg.  If it's in hwdom, then that
>> permission would be required.
> 
> Why would xenconsoled run in the hardware domain? It's purely a software
> construct, isn't it? As a daemon, putting it in the control domain may
> make sense. Otherwise it probably ought to go in a service domain.

My approach has been to transform dom0 into the hardware domain and add 
a new control domain.  xenconsoled was left running in the hardware domain.

I suppose it could move.  Maybe that would be fine?  I haven't tried. 
The Hyperlaunch code populates the console grants to point at the 
hardware domain, and I just followed that.

One aspect of why I left most things running in the Hardware domain was 
to not run things in the Control domain.  If Control is the highest 
privileged entity, we'd rather run software in lower privileged places. 
Especially something like xenconsoled which is receiving data from the 
domUs.

Running in a service domain is a good suggestion, but we haven't made it 
that far.

Regards,
Jason

Re: [PATCH 0/4] XSM changes for split hardware / control domain

[Jason Andryuk]

On 2025-06-11 09:28, Jan Beulich wrote:
> On 11.06.2025 00:57, Jason Andryuk wrote:
>> Theses are the broad changes needed for a split hardware / control
>> domain.
>>
>> An earlier posting gave device_model privileges to hardware domain.  For
>> this posting, it was split out into a new capability.  This way the
>> operator can choose where to run the device models without making the
>> hardware domain have the permissions.
>>
>> The first patch add XSM_HW_PRIV for the hardware hypercalls.  Unlike the
>> first posting, the control domain can call these hypercalls even though
>> it doesn't really make sense.  The idea was to keep the control domain
>> all powerful from an XSM perspective.
>>
>> SILO is changed to allow control, hardwware or xenstore to service
>> domUs.  Xenstore and hardware will use grants for PV interfaces.
>> Control wouldn't typically provide PV interfaces to domUs, but it is
>> given the permision to do so.  Again, to keep control all powerful.
>>
>> xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo this is not strictly
>> needed.  xenconsoled could read Xen's dmesg.  If it's in hwdom, then
>> that permission would be required.  SYSCTL_physinfo is mainly to silence
>> xl messages, which are mostly cosmetic.
>>
>> Jason Andryuk (4):
>>    xen/xsm: Add XSM_HW_PRIV
>>    xsm/silo: Support hwdom/control domains
>>    xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model
>>    xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo
> 
> Overall I can't help the impression that this level of disaggregation simply
> requires the use of Flask.

I have thought about that.  The problem with Flask is the complexity of 
the security server.  We don't want to have to deal with all that code. 
A fixed policy is easier for our coverage testing.

Exposing separate control, hardware and xenstore capabilities, it makes 
sense for the default policy to function with them.  This would be a 
coarse level of functionality, and Flask would remain for fine-grain and 
MAC enforcement.

Regards,
Jason

Re: [PATCH 3/4] xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model

[Christian Lindig]

Acked-by: Christian Lindig <christian.lindig@cloud.com>

> On 10 Jun 2025, at 23:57, Jason Andryuk <jason.andryuk@amd.com> wrote:
> 
> To add more flexibility in system configuration add the new
> DOMAIN_CAPS_DEVICE_MODEL flag and XEN_DOMCTL_CDF_device_model.
> 
> Thie new flag corresponds to allowing XSM_DM_PRIV for the domain.  This
> will enable running device model emulators (QEMU) from the assigne
> domain for multiple target domains.
> 
> Stubdoms assign target allowing the stubdom to serve as the device
> model for a single domain.  This new flag allows the single domain to
> provide emulators for multiple guests.
> 
> The specific scenario is a disaggregated system with the hardware domain
> providing device models for muitple guest domains.
> 
> The OCaml code needs the flag added in as well.
> 
> Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
> ---
> tools/ocaml/libs/xc/xenctrl.ml          |  1 +
> tools/ocaml/libs/xc/xenctrl.mli         |  1 +
> xen/arch/arm/domain.c                   |  3 ++-
> xen/common/device-tree/dom0less-build.c |  3 +++
> xen/common/domain.c                     |  3 ++-
> xen/include/public/bootfdt.h            | 12 ++++++++++--
> xen/include/public/domctl.h             |  4 +++-
> xen/include/xen/sched.h                 |  9 +++++++++
> xen/include/xsm/dummy.h                 |  2 ++
> 9 files changed, 33 insertions(+), 5 deletions(-)
> 
> diff --git a/tools/ocaml/libs/xc/xenctrl.ml b/tools/ocaml/libs/xc/xenctrl.ml
> index 2690f9a923..ef0c5dca2a 100644
> --- a/tools/ocaml/libs/xc/xenctrl.ml
> +++ b/tools/ocaml/libs/xc/xenctrl.ml
> @@ -70,6 +70,7 @@ type domain_create_flag =
>   | CDF_IOMMU
>   | CDF_NESTED_VIRT
>   | CDF_VPMU
> +  | CDF_DEVICE_MODEL
> 
> type domain_create_iommu_opts =
>   | IOMMU_NO_SHAREPT
> diff --git a/tools/ocaml/libs/xc/xenctrl.mli b/tools/ocaml/libs/xc/xenctrl.mli
> index febbe1f6ae..c0156fa5c6 100644
> --- a/tools/ocaml/libs/xc/xenctrl.mli
> +++ b/tools/ocaml/libs/xc/xenctrl.mli
> @@ -63,6 +63,7 @@ type domain_create_flag =
>   | CDF_IOMMU
>   | CDF_NESTED_VIRT
>   | CDF_VPMU
> +  | CDF_DEVICE_MODEL
> 
> type domain_create_iommu_opts =
>   | IOMMU_NO_SHAREPT
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 45aeb8bddc..12fda0762f 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -612,7 +612,8 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
>     unsigned int max_vcpus;
>     unsigned int flags_required = (XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap);
>     unsigned int flags_optional = (XEN_DOMCTL_CDF_iommu | XEN_DOMCTL_CDF_vpmu |
> -                                   XEN_DOMCTL_CDF_xs_domain );
> +                                   XEN_DOMCTL_CDF_xs_domain |
> +                                   XEN_DOMCTL_CDF_device_model);
>     unsigned int sve_vl_bits = sve_decode_vl(config->arch.sve_vl);
> 
>     if ( (config->flags & ~flags_optional) != flags_required )
> diff --git a/xen/common/device-tree/dom0less-build.c b/xen/common/device-tree/dom0less-build.c
> index 3d503c6973..993ff32f5e 100644
> --- a/xen/common/device-tree/dom0less-build.c
> +++ b/xen/common/device-tree/dom0less-build.c
> @@ -884,6 +884,9 @@ void __init create_domUs(void)
>                 d_cfg.flags |= XEN_DOMCTL_CDF_xs_domain;
>                 d_cfg.max_evtchn_port = -1;
>             }
> +
> +            if ( val & DOMAIN_CAPS_DEVICE_MODEL )
> +                d_cfg.flags |= XEN_DOMCTL_CDF_device_model;
>         }
> 
>         if ( dt_find_property(node, "xen,static-mem", NULL) )
> diff --git a/xen/common/domain.c b/xen/common/domain.c
> index 153cd75340..ab2c8f864d 100644
> --- a/xen/common/domain.c
> +++ b/xen/common/domain.c
> @@ -721,7 +721,8 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
>          ~(XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
>            XEN_DOMCTL_CDF_s3_integrity | XEN_DOMCTL_CDF_oos_off |
>            XEN_DOMCTL_CDF_xs_domain | XEN_DOMCTL_CDF_iommu |
> -           XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu) )
> +           XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu |
> +           XEN_DOMCTL_CDF_device_model) )
>     {
>         dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
>         return -EINVAL;
> diff --git a/xen/include/public/bootfdt.h b/xen/include/public/bootfdt.h
> index 86c46b42a9..c6b5afc76a 100644
> --- a/xen/include/public/bootfdt.h
> +++ b/xen/include/public/bootfdt.h
> @@ -25,7 +25,15 @@
> #define DOMAIN_CAPS_HARDWARE (1U << 1)
> /* Xenstore domain. */
> #define DOMAIN_CAPS_XENSTORE (1U << 2)
> -#define DOMAIN_CAPS_MASK     (DOMAIN_CAPS_CONTROL | DOMAIN_CAPS_HARDWARE | \
> -                              DOMAIN_CAPS_XENSTORE)
> +/*
> + * Device model capability allows the use of the dm_op hypercalls to provide
> + * the device model emulation (run QEMU) for other domains.  This is a
> + * subset of the Control capability which can be granted to the
> + * Hardware domain for running QEMU.
> + */
> +#define DOMAIN_CAPS_DEVICE_MODEL (1U << 3)
> +
> +#define DOMAIN_CAPS_MASK     (DOMAIN_CAPS_CONTROL  | DOMAIN_CAPS_HARDWARE | \
> +                              DOMAIN_CAPS_XENSTORE | DOMAIN_CAPS_DEVICE_MODEL )
> 
> #endif /* __XEN_PUBLIC_BOOTFDT_H__ */
> diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
> index 5b2063eed9..2280489be2 100644
> --- a/xen/include/public/domctl.h
> +++ b/xen/include/public/domctl.h
> @@ -66,9 +66,11 @@ struct xen_domctl_createdomain {
> #define XEN_DOMCTL_CDF_nested_virt    (1U << _XEN_DOMCTL_CDF_nested_virt)
> /* Should we expose the vPMU to the guest? */
> #define XEN_DOMCTL_CDF_vpmu           (1U << 7)
> +/* Allow domain to provide device model for multiple other domains */
> +#define XEN_DOMCTL_CDF_device_model   (1U << 8)
> 
> /* Max XEN_DOMCTL_CDF_* constant.  Used for ABI checking. */
> -#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_vpmu
> +#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_device_model
> 
>     uint32_t flags;
> 
> diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
> index fe53d4fab7..9863603d93 100644
> --- a/xen/include/xen/sched.h
> +++ b/xen/include/xen/sched.h
> @@ -1148,6 +1148,15 @@ static always_inline bool is_control_domain(const struct domain *d)
>     return evaluate_nospec(d->is_privileged);
> }
> 
> +/* This check is for functionality specific to a device model domain */
> +static always_inline bool is_dm_domain(const struct domain *d)
> +{
> +    if ( IS_ENABLED(CONFIG_PV_SHIM_EXCLUSIVE) )
> +        return false;
> +
> +    return evaluate_nospec(d->options & XEN_DOMCTL_CDF_device_model);
> +}
> +
> #define VM_ASSIST(d, t) (test_bit(VMASST_TYPE_ ## t, &(d)->vm_assist))
> 
> static always_inline bool is_pv_domain(const struct domain *d)
> diff --git a/xen/include/xsm/dummy.h b/xen/include/xsm/dummy.h
> index d8df3f66c4..477fadaefd 100644
> --- a/xen/include/xsm/dummy.h
> +++ b/xen/include/xsm/dummy.h
> @@ -91,6 +91,8 @@ static always_inline int xsm_default_action(
>             return 0;
>         fallthrough;
>     case XSM_DM_PRIV:
> +        if ( is_dm_domain(src) )
> +            return 0;
>         if ( target && evaluate_nospec(src->target == target) )
>             return 0;
>         fallthrough;
> -- 
> 2.49.0
> 

Re: [PATCH 1/4] xen/xsm: Add XSM_HW_PRIV

[Jan Beulich]

On 11.06.2025 00:57, Jason Andryuk wrote:
> Xen includes disctinct concepts of a control domain (privileged) and a
> hardware domain, but there is only a single XSM_PRIV check.  For dom0
> this is not an issue as they are one and the same.
> 
> With hyperlaunch and its build capabilities, a non-privileged hwdom and a
> privileged control domain should be possible.  Today the hwdom fails the
> XSM_PRIV checks for hardware-related hooks which it should be allowed
> access to.
> 
> Introduce XSM_HW_PRIV, and use it to mark many of the physdev_op and
> platform_op.  The hwdom is allowed access for XSM_HW_PRIV.
> 
> Make XSM_HW_PRIV a new privilege level that is given to the hardware
> domain, but is not exclusive.  The control domain can still execute
> XSM_HW_PRIV commands.  This is a little questionable since it's unclear
> how the control domain can meaningfully execute them.  But this approach
> is chosen to maintain the increasing privileges and keep control domain
> fully privileged.

I consider this conceptually wrong. "Control" aiui refers to software
(e.g. VMs or system-wide settings), but there ought to be a (pretty?)
clear boundary between control and hardware domains, imo. As to
"pretty" - should any overlap be necessary (xms_machine_memory_map()
comes to mind), such would need handling specially then, I think. At
the same time: The more of an overlap there is, the less clear it is
why the two want/need separating in the first place.

Jan

Re: [PATCH 2/4] xsm/silo: Support hwdom/control domains

[Jan Beulich]

On 11.06.2025 00:57, Jason Andryuk wrote:
> In a disaggregated environment, dom0 is split into Control, Hardware,
> and Xenstore domains, along with domUs.  The is_control_domain() check
> is not sufficient to handle all these cases.  Add is_priv_domain() to
> support allowing for the various domains.
> 
> The purpose of SILO mode is to prevent domUs from interacting with each
> other.  But dom0 was allowed to communicate with domUs to provide
> services.  As the disaggregation of dom0, Control, Hardware and Xenstore
> are all service domains that need to communicate with other domains.
> 
> To provide xenstore connections, the Xenstore domain must be allowed to
> connect via grants and event channels.  Xenstore domain must also be
> allowed to connect to Control and Hardware to provide xenstore to them.

Are you suggesting that SILO at present is incompatible with a Xenstore
domain? silo_mode_dom_check() in its original form has no special
precautions, after all.

> Hardware domain will provide PV devices to domains, so it must be
> allowed to connect to domains.

As a built-in policy, isn't this already going too far? There could
conceivably be configurations with only pass-through devices in use, in
which case neither grants nor the event channels operations intercepted
by SILO would be required.

> That leaves Control.  Xenstore and Hardware would already allow access
> to Control, so it can obtain services that way.  Control should be
> "privileged", which would mean it can make the connections.  But with
> Xenstore and Hardware providing their services to domUs, there may not
> be a reason to allow Control to use grants or event channels with domUs.
> Still, Control is privileged, so it should be allowed to do something if
> it chooses.  Establishing a grant, or event channel requires action on
> both sides, so allow for the possibility.  This does open up an argo
> wildcard ring from domUs, FWIW.

Along the lines of my reply to patch 1, I think Hardware and Control
need to have a pretty strong boundary between them. It's hard to see,
for example, whether grant map/copy/transfer would indeed make sense
between the two.

Similarly I'm not convinced a strong boundary isn't also needed
between Xenstore and Hardware.

> --- a/xen/xsm/silo.c
> +++ b/xen/xsm/silo.c
> @@ -20,6 +20,12 @@
>  #define XSM_NO_WRAPPERS
>  #include <xsm/dummy.h>
>  
> +static bool is_priv_domain(const struct domain *d)
> +{
> +    return is_xenstore_domain(d) || is_hardware_domain(d) ||
> +           is_control_domain(d);
> +}

This construct expands to two evaluate_nospec(), which likely isn't
wanted. Some open-coding may be pretty much unavoidable here. (I'm
surprised it's not three, i.e. I find it odd that is_xenstore_domain()
doesn't also use that guard.)

Jan

Re: [PATCH 3/4] xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model

[Jan Beulich]

On 11.06.2025 00:57, Jason Andryuk wrote:
> To add more flexibility in system configuration add the new
> DOMAIN_CAPS_DEVICE_MODEL flag and XEN_DOMCTL_CDF_device_model.
> 
> Thie new flag corresponds to allowing XSM_DM_PRIV for the domain.  This
> will enable running device model emulators (QEMU) from the assigne
> domain for multiple target domains.
> 
> Stubdoms assign target allowing the stubdom to serve as the device
> model for a single domain.  This new flag allows the single domain to
> provide emulators for multiple guests.
> 
> The specific scenario is a disaggregated system with the hardware domain
> providing device models for muitple guest domains.

Why the hardware domain? Unless a DM also needs access to some of the
physical hardware, it ought to run in a separate domain. Conceivably
such a domain could service multiply guests, so maybe the "single
target" concept presently used for stubdom simply needed extending?

Jan

Re: [PATCH 4/4] xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

[Jan Beulich]

On 11.06.2025 00:57, Jason Andryuk wrote:
> Allow the hwdom to access the console, and to access physical
> information about the system.
> 
> xenconsoled can read Xen's dmesg.  If it's in hwdom, then that
> permission would be required.

Why would xenconsoled run in the hardware domain? It's purely a software
construct, isn't it? As a daemon, putting it in the control domain may
make sense. Otherwise it probably ought to go in a service domain.

Jan

Re: [PATCH 0/4] XSM changes for split hardware / control domain

[Jan Beulich]

On 11.06.2025 00:57, Jason Andryuk wrote:
> Theses are the broad changes needed for a split hardware / control
> domain.
> 
> An earlier posting gave device_model privileges to hardware domain.  For
> this posting, it was split out into a new capability.  This way the
> operator can choose where to run the device models without making the
> hardware domain have the permissions.
> 
> The first patch add XSM_HW_PRIV for the hardware hypercalls.  Unlike the
> first posting, the control domain can call these hypercalls even though
> it doesn't really make sense.  The idea was to keep the control domain
> all powerful from an XSM perspective.
> 
> SILO is changed to allow control, hardwware or xenstore to service
> domUs.  Xenstore and hardware will use grants for PV interfaces.
> Control wouldn't typically provide PV interfaces to domUs, but it is
> given the permision to do so.  Again, to keep control all powerful.
> 
> xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo this is not strictly
> needed.  xenconsoled could read Xen's dmesg.  If it's in hwdom, then
> that permission would be required.  SYSCTL_physinfo is mainly to silence
> xl messages, which are mostly cosmetic.
> 
> Jason Andryuk (4):
>   xen/xsm: Add XSM_HW_PRIV
>   xsm/silo: Support hwdom/control domains
>   xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model
>   xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

Overall I can't help the impression that this level of disaggregation simply
requires the use of Flask.

Jan

Re: [PATCH 0/4] XSM changes for split hardware / control domain

[Jan Beulich]

On 11.06.2025 07:08, Jason Andryuk wrote:
> On 2025-06-11 09:28, Jan Beulich wrote:
>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>> Theses are the broad changes needed for a split hardware / control
>>> domain.
>>>
>>> An earlier posting gave device_model privileges to hardware domain.  For
>>> this posting, it was split out into a new capability.  This way the
>>> operator can choose where to run the device models without making the
>>> hardware domain have the permissions.
>>>
>>> The first patch add XSM_HW_PRIV for the hardware hypercalls.  Unlike the
>>> first posting, the control domain can call these hypercalls even though
>>> it doesn't really make sense.  The idea was to keep the control domain
>>> all powerful from an XSM perspective.
>>>
>>> SILO is changed to allow control, hardwware or xenstore to service
>>> domUs.  Xenstore and hardware will use grants for PV interfaces.
>>> Control wouldn't typically provide PV interfaces to domUs, but it is
>>> given the permision to do so.  Again, to keep control all powerful.
>>>
>>> xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo this is not strictly
>>> needed.  xenconsoled could read Xen's dmesg.  If it's in hwdom, then
>>> that permission would be required.  SYSCTL_physinfo is mainly to silence
>>> xl messages, which are mostly cosmetic.
>>>
>>> Jason Andryuk (4):
>>>    xen/xsm: Add XSM_HW_PRIV
>>>    xsm/silo: Support hwdom/control domains
>>>    xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model
>>>    xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo
>>
>> Overall I can't help the impression that this level of disaggregation simply
>> requires the use of Flask.
> 
> I have thought about that.  The problem with Flask is the complexity of 
> the security server.  We don't want to have to deal with all that code. 
> A fixed policy is easier for our coverage testing.
> 
> Exposing separate control, hardware and xenstore capabilities, it makes 
> sense for the default policy to function with them.

Yet as indicated in replies to individual patches - the boundaries between
the three aren't clear, which imo goes (to some degree at least) against the
purpose of "disaggregation".

Jan

>  This would be a 
> coarse level of functionality, and Flask would remain for fine-grain and 
> MAC enforcement.
> 
> Regards,
> Jason

Re: [PATCH 1/4] xen/xsm: Add XSM_HW_PRIV

[Jan Beulich]

On 11.06.2025 05:13, Jason Andryuk wrote:
> On 2025-06-11 09:02, Jan Beulich wrote:
>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>> Xen includes disctinct concepts of a control domain (privileged) and a
>>> hardware domain, but there is only a single XSM_PRIV check.  For dom0
>>> this is not an issue as they are one and the same.
>>>
>>> With hyperlaunch and its build capabilities, a non-privileged hwdom and a
>>> privileged control domain should be possible.  Today the hwdom fails the
>>> XSM_PRIV checks for hardware-related hooks which it should be allowed
>>> access to.
>>>
>>> Introduce XSM_HW_PRIV, and use it to mark many of the physdev_op and
>>> platform_op.  The hwdom is allowed access for XSM_HW_PRIV.
>>>
>>> Make XSM_HW_PRIV a new privilege level that is given to the hardware
>>> domain, but is not exclusive.  The control domain can still execute
>>> XSM_HW_PRIV commands.  This is a little questionable since it's unclear
>>> how the control domain can meaningfully execute them.  But this approach
>>> is chosen to maintain the increasing privileges and keep control domain
>>> fully privileged.
>>
>> I consider this conceptually wrong. "Control" aiui refers to software
>> (e.g. VMs or system-wide settings), but there ought to be a (pretty?)
>> clear boundary between control and hardware domains, imo. As to
>> "pretty" - should any overlap be necessary (xms_machine_memory_map()
>> comes to mind), such would need handling specially then, I think. At
>> the same time: The more of an overlap there is, the less clear it is
>> why the two want/need separating in the first place.
> 
> So you are in favor of splitting control and hardware into distinct 
> sets?  I am okay with this.  I implemented that originally, but I 
> started doubting it.  Mainly, should control be denied any permission?

Yes, imo: Fundamentally for anything the hardware domain is supposed to
be doing. Yet as indicated in other replies to this series - boundaries
aren't always as clear as they ought to be for a clean separation.

> We aren't using the toolstack to build domains - dom0less or Hyperlaunch 
> handles that.  This avoids issues that might arise from running the 
> toolstack.

IOW you don't have a control domain there in the first place?

Jan

Re: [PATCH 2/4] xsm/silo: Support hwdom/control domains

[Jan Beulich]

On 11.06.2025 06:20, Jason Andryuk wrote:
> On 2025-06-11 09:17, Jan Beulich wrote:
>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>> In a disaggregated environment, dom0 is split into Control, Hardware,
>>> and Xenstore domains, along with domUs.  The is_control_domain() check
>>> is not sufficient to handle all these cases.  Add is_priv_domain() to
>>> support allowing for the various domains.
>>>
>>> The purpose of SILO mode is to prevent domUs from interacting with each
>>> other.  But dom0 was allowed to communicate with domUs to provide
>>> services.  As the disaggregation of dom0, Control, Hardware and Xenstore
>>> are all service domains that need to communicate with other domains.
>>>
>>> To provide xenstore connections, the Xenstore domain must be allowed to
>>> connect via grants and event channels.  Xenstore domain must also be
>>> allowed to connect to Control and Hardware to provide xenstore to them.
>>
>> Are you suggesting that SILO at present is incompatible with a Xenstore
>> domain? silo_mode_dom_check() in its original form has no special
>> precautions, after all.
> 
> Yes, it is incompatible with the current silo_mode_dom_check().  Only 
> Control domain is allowed to use grants and event channels with a domU. 
> A Xenstore domain would be denied.
> 
> Xenstore stubdom only exists for x86 today.  My limited attempts to run 
> xenstored in an dedicated Xenstore ARM Linux domain have failed.

This may want sorting independently first. Once sorted, the requirements
here may become more clear.

>>> Hardware domain will provide PV devices to domains, so it must be
>>> allowed to connect to domains.
>>
>> As a built-in policy, isn't this already going too far? There could
>> conceivably be configurations with only pass-through devices in use, in
>> which case neither grants nor the event channels operations intercepted
>> by SILO would be required.
> 
> Such a domain wouldn't have any PV devices configured?

Indeed, that's my point: Why would Hardware then have a need to be
allowed to connect to domains.

>  I don't think this changes anything compared to today.

I don't think I see what you mean to tell me with this. What we're
discussing here is the effect of the separation you're suggesting,
which necessarily is different from what we have today.

> Both sides need to be configured and opt-in.  Hardware is a system 
> domain, so it should be possible to allow grants and event channels. 
> But they won't be used unless configured.

"Won't be used" isn't enough, imo. Isn't disaggregation about proper
isolation, i.e. to guarantee that unwanted interactions can't occur?

>>> That leaves Control.  Xenstore and Hardware would already allow access
>>> to Control, so it can obtain services that way.  Control should be
>>> "privileged", which would mean it can make the connections.  But with
>>> Xenstore and Hardware providing their services to domUs, there may not
>>> be a reason to allow Control to use grants or event channels with domUs.
>>> Still, Control is privileged, so it should be allowed to do something if
>>> it chooses.  Establishing a grant, or event channel requires action on
>>> both sides, so allow for the possibility.  This does open up an argo
>>> wildcard ring from domUs, FWIW.
>>
>> Along the lines of my reply to patch 1, I think Hardware and Control
>> need to have a pretty strong boundary between them. It's hard to see,
>> for example, whether grant map/copy/transfer would indeed make sense
>> between the two.
> 
> The Hardware domain might provide a PV device to Control?
> 
> I've tested removing control:
> static bool is_priv_domain(const struct domain *d)
> {
>      return is_xenstore_domain(d) || is_hardware_domain(d);
> }
> 
> And that works in my limited ARM dom0less testing.  The toolstack isn't 
> really exercised in that case.  It seems strange that the privileged 
> control domain is *not* allowed though.

With the intended separation, there's (imo) not going to be any
all-mighty domain anymore. Neither Hardware nor Control.

>> Similarly I'm not convinced a strong boundary isn't also needed
>> between Xenstore and Hardware.
> 
> If hardware is providing PV devices to domains, it will need access to 
> Xenstore.  I don't see how you can get around it.
> 
> I tried to explain this in the first paragraph.  SILO's purpose was to 
> isolate domUs from each other, but allow it to access dom0.  dom0 
> embodies the control, hardware, and xenstore capabilities.  So as a 
> first approximation, each of Control, Hardware, and Xenstore should be 
> allowed to communicate with domUs.

Yes. Yet what to permit between the three special entities is far less
clear. Hence why I'm unconvinced this can be expressed by SILO, and
would rather require Flask.

> domUs needs to communicate with Xenstore and Hardware for PV devices.
> 
> Xenstore provides Xenstore access to Hardware.
> 
> Control would want Xenstore access.
> 
> I don't know if this helps, but here's a table:
> 
>      | CTL | HW  | XS  | domU
> ----------------------------
> CTL |     |  ?  |  y  |  ?
> HW  |  ?  |     |  y  |  y
> XS  |  y  |  y  |     |  y
> domU|  ?  |  y  |  y  |  x
> 
> Control and Hardware would be y if we allow PV devices
> 
> Control and domUs - I don't have an immediate rational for them.  Except 
> that Control is privileged.  I've been running xenconsoled in Hardware. 
> If xenconsoled is in Control, then access would be required.

Perhaps some clarification is first need about what Control really is
(and is not). It is sole the domain to create other domains. But beyond
that things become unclear. E.g. xenconsoled may not belong into either
Hardware or Control.

>>> --- a/xen/xsm/silo.c
>>> +++ b/xen/xsm/silo.c
>>> @@ -20,6 +20,12 @@
>>>   #define XSM_NO_WRAPPERS
>>>   #include <xsm/dummy.h>
>>>   
>>> +static bool is_priv_domain(const struct domain *d)
>>> +{
>>> +    return is_xenstore_domain(d) || is_hardware_domain(d) ||
>>> +           is_control_domain(d);
>>> +}
>>
>> This construct expands to two evaluate_nospec(), which likely isn't
>> wanted. Some open-coding may be pretty much unavoidable here.
> 
> Thanks, yes, good point.
> 
>> (I'm
>> surprised it's not three, i.e. I find it odd that is_xenstore_domain()
>> doesn't also use that guard.)
> 
> It looks okay to me.  There were only 2 uses until I added a 3rd in the 
> dom0less code.  The XSM check has evaluate_nospec() and the other 2 uses 
> aren't security critical - Setting a domain info flag, and __init code 
> for dom0less.  Maybe moving the evaluate_nospec() would be safer in case 
> use grows in the future, but it looks okay to me today.

When some of the hardening was first introduced, actual use sites were
indeed taken into account. That wasn't quite right though, I think. Any
such construct ought to be safe to use anywhere. For uses with clearly
no concerns towards speculative abuse, a 2nd lightweight form of such
constructs should then exist, imo. As to your use of "security critical":
I'm not convinced you what mean is covering the potential of speculative
abuse of involved code paths.

Jan

Re: [PATCH 2/4] xsm/silo: Support hwdom/control domains

[Jason Andryuk]

On 2025-06-12 03:52, Jan Beulich wrote:
> On 11.06.2025 06:20, Jason Andryuk wrote:
>> On 2025-06-11 09:17, Jan Beulich wrote:
>>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>>> In a disaggregated environment, dom0 is split into Control, Hardware,
>>>> and Xenstore domains, along with domUs.  The is_control_domain() check
>>>> is not sufficient to handle all these cases.  Add is_priv_domain() to
>>>> support allowing for the various domains.
>>>>
>>>> The purpose of SILO mode is to prevent domUs from interacting with each
>>>> other.  But dom0 was allowed to communicate with domUs to provide
>>>> services.  As the disaggregation of dom0, Control, Hardware and Xenstore
>>>> are all service domains that need to communicate with other domains.
>>>>
>>>> To provide xenstore connections, the Xenstore domain must be allowed to
>>>> connect via grants and event channels.  Xenstore domain must also be
>>>> allowed to connect to Control and Hardware to provide xenstore to them.
>>>
>>> Are you suggesting that SILO at present is incompatible with a Xenstore
>>> domain? silo_mode_dom_check() in its original form has no special
>>> precautions, after all.
>>
>> Yes, it is incompatible with the current silo_mode_dom_check().  Only
>> Control domain is allowed to use grants and event channels with a domU.
>> A Xenstore domain would be denied.
>>
>> Xenstore stubdom only exists for x86 today.  My limited attempts to run
>> xenstored in an dedicated Xenstore ARM Linux domain have failed.
> 
> This may want sorting independently first. Once sorted, the requirements
> here may become more clear.

HW+XS-> xenstore works
CTL+XS or XS -> the domain's console just stops.  vCPUs are in Linux cpu 
idle.  I haven't figured out more.  This required some Linux changes to 
query the capabilities since XS isn't exposed and ARM assumes initial 
domain implies HW + CTL.  It's orthogonal to my goals, so I haven't 
looked too hard.

>>>> Hardware domain will provide PV devices to domains, so it must be
>>>> allowed to connect to domains.
>>>
>>> As a built-in policy, isn't this already going too far? There could
>>> conceivably be configurations with only pass-through devices in use, in
>>> which case neither grants nor the event channels operations intercepted
>>> by SILO would be required.
>>
>> Such a domain wouldn't have any PV devices configured?
> 
> Indeed, that's my point: Why would Hardware then have a need to be
> allowed to connect to domains.
> 
>>   I don't think this changes anything compared to today.
> 
> I don't think I see what you mean to tell me with this. What we're
> discussing here is the effect of the separation you're suggesting,
> which necessarily is different from what we have today.
>
>> Both sides need to be configured and opt-in.  Hardware is a system
>> domain, so it should be possible to allow grants and event channels.
>> But they won't be used unless configured.
> 
> "Won't be used" isn't enough, imo. Isn't disaggregation about proper
> isolation, i.e. to guarantee that unwanted interactions can't occur?

Disaggregation is the separation of components.  The security policy 
applied is related but distinct.

"Won't be used" is how dummy and SILO (with respect to dom0) devices 
work today

dummy -> cooperating domUs can communicate
SILO  -> domUs cannot communicate
Flask -> configurable, but typically strict limits to explicit 
communication channels

SILO today doesn't deny communication between a domU without PV devices 
and dom0 - they just aren't configured.  I'm saying that would be the 
same with a split hardware domain.  PV devices just aren't configured, 
but there is no mandatory denial.  SILO only enforces mandatory denial 
between domUs today.

If you want mandatory enforcement, Flask is the correct choice. 
Everything would be explicitly configured.  Some domains could 
communicate with hwdom and others could not.

For SILO, a split hardware domain would be allowed communication with a 
domU if configured by the administrator.  I see this as comparable to 
configuring a domU to access dom0 PV backends today.

>>>> That leaves Control.  Xenstore and Hardware would already allow access
>>>> to Control, so it can obtain services that way.  Control should be
>>>> "privileged", which would mean it can make the connections.  But with
>>>> Xenstore and Hardware providing their services to domUs, there may not
>>>> be a reason to allow Control to use grants or event channels with domUs.
>>>> Still, Control is privileged, so it should be allowed to do something if
>>>> it chooses.  Establishing a grant, or event channel requires action on
>>>> both sides, so allow for the possibility.  This does open up an argo
>>>> wildcard ring from domUs, FWIW.
>>>
>>> Along the lines of my reply to patch 1, I think Hardware and Control
>>> need to have a pretty strong boundary between them. It's hard to see,
>>> for example, whether grant map/copy/transfer would indeed make sense
>>> between the two.
>>
>> The Hardware domain might provide a PV device to Control?
>>
>> I've tested removing control:
>> static bool is_priv_domain(const struct domain *d)
>> {
>>       return is_xenstore_domain(d) || is_hardware_domain(d);
>> }
>>
>> And that works in my limited ARM dom0less testing.  The toolstack isn't
>> really exercised in that case.  It seems strange that the privileged
>> control domain is *not* allowed though.
> 
> With the intended separation, there's (imo) not going to be any
> all-mighty domain anymore. Neither Hardware nor Control.
> 
>>> Similarly I'm not convinced a strong boundary isn't also needed
>>> between Xenstore and Hardware.
>>
>> If hardware is providing PV devices to domains, it will need access to
>> Xenstore.  I don't see how you can get around it.
>>
>> I tried to explain this in the first paragraph.  SILO's purpose was to
>> isolate domUs from each other, but allow it to access dom0.  dom0
>> embodies the control, hardware, and xenstore capabilities.  So as a
>> first approximation, each of Control, Hardware, and Xenstore should be
>> allowed to communicate with domUs.
> 
> Yes. Yet what to permit between the three special entities is far less
> clear. Hence why I'm unconvinced this can be expressed by SILO, and
> would rather require Flask.
> 
>> domUs needs to communicate with Xenstore and Hardware for PV devices.
>>
>> Xenstore provides Xenstore access to Hardware.
>>
>> Control would want Xenstore access.
>>
>> I don't know if this helps, but here's a table:
>>
>>       | CTL | HW  | XS  | domU
>> ----------------------------
>> CTL |     |  ?  |  y  |  ?
>> HW  |  ?  |     |  y  |  y
>> XS  |  y  |  y  |     |  y
>> domU|  ?  |  y  |  y  |  x
>>
>> Control and Hardware would be y if we allow PV devices
>>
>> Control and domUs - I don't have an immediate rational for them.  Except
>> that Control is privileged.  I've been running xenconsoled in Hardware.
>> If xenconsoled is in Control, then access would be required.
> 
> Perhaps some clarification is first need about what Control really is
> (and is not). It is sole the domain to create other domains. But beyond
> that things become unclear. E.g. xenconsoled may not belong into either
> Hardware or Control.
> 
>>>> --- a/xen/xsm/silo.c
>>>> +++ b/xen/xsm/silo.c
>>>> @@ -20,6 +20,12 @@
>>>>    #define XSM_NO_WRAPPERS
>>>>    #include <xsm/dummy.h>
>>>>    
>>>> +static bool is_priv_domain(const struct domain *d)
>>>> +{
>>>> +    return is_xenstore_domain(d) || is_hardware_domain(d) ||
>>>> +           is_control_domain(d);
>>>> +}
>>>
>>> This construct expands to two evaluate_nospec(), which likely isn't
>>> wanted. Some open-coding may be pretty much unavoidable here.
>>
>> Thanks, yes, good point.
>>
>>> (I'm
>>> surprised it's not three, i.e. I find it odd that is_xenstore_domain()
>>> doesn't also use that guard.)
>>
>> It looks okay to me.  There were only 2 uses until I added a 3rd in the
>> dom0less code.  The XSM check has evaluate_nospec() and the other 2 uses
>> aren't security critical - Setting a domain info flag, and __init code
>> for dom0less.  Maybe moving the evaluate_nospec() would be safer in case
>> use grows in the future, but it looks okay to me today.
> 
> When some of the hardening was first introduced, actual use sites were
> indeed taken into account. That wasn't quite right though, I think. Any
> such construct ought to be safe to use anywhere. For uses with clearly
> no concerns towards speculative abuse, a 2nd lightweight form of such
> constructs should then exist, imo. As to your use of "security critical":
> I'm not convinced you what mean is covering the potential of speculative
> abuse of involved code paths.

I can't parse this last sentence, and I think it's your main point.

XSM -> don't speculate around permission checks.  That's what I meant by 
"security critical".

The __init code is inaccessible to users, so it doesn't matter.

         if ( is_xenstore_domain(d) )
             continue;

getdomaininfo sets a flag, so I don't see this making a security 
difference.  It's not controlling loads or code paths.

(is_xenstore_domain(d)          ? XEN_DOMINF_xs_domain : 0) |

Regards,
Jason

Re: [PATCH 1/4] xen/xsm: Add XSM_HW_PRIV

[Jason Andryuk]

On 2025-06-12 03:36, Jan Beulich wrote:
> On 11.06.2025 05:13, Jason Andryuk wrote:
>> On 2025-06-11 09:02, Jan Beulich wrote:
>>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>>> Xen includes disctinct concepts of a control domain (privileged) and a
>>>> hardware domain, but there is only a single XSM_PRIV check.  For dom0
>>>> this is not an issue as they are one and the same.
>>>>
>>>> With hyperlaunch and its build capabilities, a non-privileged hwdom and a
>>>> privileged control domain should be possible.  Today the hwdom fails the
>>>> XSM_PRIV checks for hardware-related hooks which it should be allowed
>>>> access to.
>>>>
>>>> Introduce XSM_HW_PRIV, and use it to mark many of the physdev_op and
>>>> platform_op.  The hwdom is allowed access for XSM_HW_PRIV.
>>>>
>>>> Make XSM_HW_PRIV a new privilege level that is given to the hardware
>>>> domain, but is not exclusive.  The control domain can still execute
>>>> XSM_HW_PRIV commands.  This is a little questionable since it's unclear
>>>> how the control domain can meaningfully execute them.  But this approach
>>>> is chosen to maintain the increasing privileges and keep control domain
>>>> fully privileged.
>>>
>>> I consider this conceptually wrong. "Control" aiui refers to software
>>> (e.g. VMs or system-wide settings), but there ought to be a (pretty?)
>>> clear boundary between control and hardware domains, imo. As to
>>> "pretty" - should any overlap be necessary (xms_machine_memory_map()
>>> comes to mind), such would need handling specially then, I think. At
>>> the same time: The more of an overlap there is, the less clear it is
>>> why the two want/need separating in the first place.
>>
>> So you are in favor of splitting control and hardware into distinct
>> sets?  I am okay with this.  I implemented that originally, but I
>> started doubting it.  Mainly, should control be denied any permission?
> 
> Yes, imo: Fundamentally for anything the hardware domain is supposed to
> be doing. 

Ok.

> Yet as indicated in other replies to this series - boundaries
> aren't always as clear as they ought to be for a clean separation.

Agreed.

>> We aren't using the toolstack to build domains - dom0less or Hyperlaunch
>> handles that.  This avoids issues that might arise from running the
>> toolstack.
> 
> IOW you don't have a control domain there in the first place?

I have a domain with d->is_privileged == true.  We don't create more 
domains with it though, which was your other email's definition of the 
control domain.  But it can pause and unpause domains.

Regards,
Jason

Re: [PATCH 2/4] xsm/silo: Support hwdom/control domains

[Jason Andryuk]

On 2025-06-12 12:56, Jason Andryuk wrote:
> On 2025-06-12 03:52, Jan Beulich wrote:
>> On 11.06.2025 06:20, Jason Andryuk wrote:
>>> On 2025-06-11 09:17, Jan Beulich wrote:
>>>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>>>> In a disaggregated environment, dom0 is split into Control, Hardware,
>>>>> and Xenstore domains, along with domUs.  The is_control_domain() check
>>>>> is not sufficient to handle all these cases.  Add is_priv_domain() to
>>>>> support allowing for the various domains.
>>>>>
>>>>> The purpose of SILO mode is to prevent domUs from interacting with 
>>>>> each
>>>>> other.  But dom0 was allowed to communicate with domUs to provide
>>>>> services.  As the disaggregation of dom0, Control, Hardware and 
>>>>> Xenstore
>>>>> are all service domains that need to communicate with other domains.
>>>>>
>>>>> To provide xenstore connections, the Xenstore domain must be 
>>>>> allowed to
>>>>> connect via grants and event channels.  Xenstore domain must also be
>>>>> allowed to connect to Control and Hardware to provide xenstore to 
>>>>> them.
>>>>
>>>> Are you suggesting that SILO at present is incompatible with a Xenstore
>>>> domain? silo_mode_dom_check() in its original form has no special
>>>> precautions, after all.
>>>
>>> Yes, it is incompatible with the current silo_mode_dom_check().  Only
>>> Control domain is allowed to use grants and event channels with a domU.
>>> A Xenstore domain would be denied.
>>>
>>> Xenstore stubdom only exists for x86 today.  My limited attempts to run
>>> xenstored in an dedicated Xenstore ARM Linux domain have failed.
>>
>> This may want sorting independently first. Once sorted, the requirements
>> here may become more clear.
> 
> HW+XS-> xenstore works
> CTL+XS or XS -> the domain's console just stops.  vCPUs are in Linux cpu 
> idle.  I haven't figured out more.  This required some Linux changes to 
> query the capabilities since XS isn't exposed and ARM assumes initial 
> domain implies HW + CTL.  It's orthogonal to my goals, so I haven't 
> looked too hard.

I got standalone Xenstore working on ARM.  Linux was blocking in 
xs_reset_watches() - the Xenstore domain needs to skip that function 
like xen_initial_domain().

This is with SILO's check as:
static bool is_priv_domain(const struct domain *d)
{
     return evaluate_nospec((d->options & XEN_DOMCTL_CDF_xs_domain) ||
                            d == hardware_domain);
}

Regards,
Jason

Re: [PATCH 2/4] xsm/silo: Support hwdom/control domains

[Jan Beulich]

On 12.06.2025 18:56, Jason Andryuk wrote:
> On 2025-06-12 03:52, Jan Beulich wrote:
>> On 11.06.2025 06:20, Jason Andryuk wrote:
>>> On 2025-06-11 09:17, Jan Beulich wrote:
>>>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>>>> --- a/xen/xsm/silo.c
>>>>> +++ b/xen/xsm/silo.c
>>>>> @@ -20,6 +20,12 @@
>>>>>    #define XSM_NO_WRAPPERS
>>>>>    #include <xsm/dummy.h>
>>>>>    
>>>>> +static bool is_priv_domain(const struct domain *d)
>>>>> +{
>>>>> +    return is_xenstore_domain(d) || is_hardware_domain(d) ||
>>>>> +           is_control_domain(d);
>>>>> +}
>>>>
>>>> This construct expands to two evaluate_nospec(), which likely isn't
>>>> wanted. Some open-coding may be pretty much unavoidable here.
>>>
>>> Thanks, yes, good point.
>>>
>>>> (I'm
>>>> surprised it's not three, i.e. I find it odd that is_xenstore_domain()
>>>> doesn't also use that guard.)
>>>
>>> It looks okay to me.  There were only 2 uses until I added a 3rd in the
>>> dom0less code.  The XSM check has evaluate_nospec() and the other 2 uses
>>> aren't security critical - Setting a domain info flag, and __init code
>>> for dom0less.  Maybe moving the evaluate_nospec() would be safer in case
>>> use grows in the future, but it looks okay to me today.
>>
>> When some of the hardening was first introduced, actual use sites were
>> indeed taken into account. That wasn't quite right though, I think. Any
>> such construct ought to be safe to use anywhere. For uses with clearly
>> no concerns towards speculative abuse, a 2nd lightweight form of such
>> constructs should then exist, imo. As to your use of "security critical":
>> I'm not convinced you what mean is covering the potential of speculative
>> abuse of involved code paths.
> 
> I can't parse this last sentence, and I think it's your main point.

Oh, sorry - the "you" and "what" ought to have swapped places.

> XSM -> don't speculate around permission checks.  That's what I meant by 
> "security critical".
> 
> The __init code is inaccessible to users, so it doesn't matter.
> 
>          if ( is_xenstore_domain(d) )
>              continue;
> 
> getdomaininfo sets a flag, so I don't see this making a security 
> difference.  It's not controlling loads or code paths.

Right, but this is what I said should imo not have been done: Make a
predicate speculation-safe (or not) based on its present uses. It's
imo more likely than not that a new use being added won't result in
the predicate being looked at, re-considering its safety for the new
use.

And indeed there's a 3rd use, in xsm_default_action():

    case XSM_XS_PRIV:
        if ( action == XSM_XS_PRIV &&
             evaluate_nospec(is_xenstore_domain(src)) )
            return 0;
        fallthrough;

It should not have been necessary to open-code the speculation safety
here, just like such isn't required a few lines later:

    case XSM_PRIV:
        if ( is_control_domain(src) )
            return 0;
        return -EPERM;

I am, btw, also not convinced the uses of evaluate_nospec() are fully
correct here, in that they apply to only part of the if() conditions.
For "action == XSM_XS_PRIV" it's okay as long as
- the function is indeed inlined, and
- the function argument is compile-time constant.
For "target" the same applies, but there is more room there for the
latter of the constraints to not be met. The argument in favor of
the present arrangements likely was that our main concern here is
with the "success" paths. Yet such argumentation would again be
dependent upon all call sites fitting the assumption that on the
"failure" paths there would be nothing critical that follows.

Jan

Re: [PATCH 3/4] xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model

[Stefano Stabellini]

On Wed, 11 Jun 2025, Jan Beulich wrote:
> On 11.06.2025 00:57, Jason Andryuk wrote:
> > To add more flexibility in system configuration add the new
> > DOMAIN_CAPS_DEVICE_MODEL flag and XEN_DOMCTL_CDF_device_model.
> > 
> > Thie new flag corresponds to allowing XSM_DM_PRIV for the domain.  This
> > will enable running device model emulators (QEMU) from the assigne
> > domain for multiple target domains.
> > 
> > Stubdoms assign target allowing the stubdom to serve as the device
> > model for a single domain.  This new flag allows the single domain to
> > provide emulators for multiple guests.
> > 
> > The specific scenario is a disaggregated system with the hardware domain
> > providing device models for muitple guest domains.
> 
> Why the hardware domain? Unless a DM also needs access to some of the
> physical hardware, it ought to run in a separate domain. Conceivably
> such a domain could service multiply guests, so maybe the "single
> target" concept presently used for stubdom simply needed extending?

Not necessarily. While it is possible to have driver domains, it is not
the default configuration.

In a default configuration, the hardware domain gets all the hardware by
default and therefore will also run the PV backends and Virtio backends.
The Virtio backends require DM hypercalls. Let me elaborate further.

In the datacenter, we have Dom0 typically with all the hardware, the
backends (PV and Virtio), and also the toolstack. Then all other domains
are created dynamically by the toolstack. Driver domains are possible
but not very common.

In automotive/embedded, the total number of domains is static, so we can
create them using dom0less. We don't need the toolstack to create VMs.
Also, we have safety concerns, so we want to take away as much
privileges as possible from Dom0. This is easy because thanks to
dom0less, we don't need the toolstack and we don't need to create VMs
dynamically.

So the model is that Dom0 becomes the hardware domain: it has all the
drivers and backends but it is not privileged in the sense of
creating/destroying other VMs. If a user wants to have Dom0 "super
powers", they can create an optional Control Domain. The Control Domain
is expected to be tiny, such as XTF or Zephyr. It will have the ability
that Dom0 used to have but without the drivers. From a privilege
perspective, the Control Domain could create additional VMs, but in
automotive/embedded it is not expected to be a use-case because the
total number of VMs is still static. 

So your point about driver domains. Yes, one can have driver domains the
same way that one can have driver domains in the datacenter but it is
not the default. The new default for embedded is what I described above
and I think it is a very widely applicable concept across industries:
automotive, industrial, robotics, etc. and also across vendors: AMD,
Xilinx, Renesas, EPAM, ARM, etc.

Re: [PATCH 4/4] xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

[Stefano Stabellini]

On Wed, 11 Jun 2025, Jason Andryuk wrote:
> On 2025-06-11 09:27, Jan Beulich wrote:
> > On 11.06.2025 00:57, Jason Andryuk wrote:
> > > Allow the hwdom to access the console, and to access physical
> > > information about the system.
> > > 
> > > xenconsoled can read Xen's dmesg.  If it's in hwdom, then that
> > > permission would be required.
> > 
> > Why would xenconsoled run in the hardware domain? It's purely a software
> > construct, isn't it? As a daemon, putting it in the control domain may
> > make sense. Otherwise it probably ought to go in a service domain.
> 
> My approach has been to transform dom0 into the hardware domain and add a new
> control domain.  xenconsoled was left running in the hardware domain.

I think we should keep xenconsoled in the hardware domain because the
control domain should be just optional. (However, one could say that with
Denis' recent changes xenconsoled is also optional because one can use
console hypercalls or emulators (PL011, NS16550) for all DomUs.)



> I suppose it could move.  Maybe that would be fine?  I haven't tried. The
> Hyperlaunch code populates the console grants to point at the hardware domain,
> and I just followed that.
> 
> One aspect of why I left most things running in the Hardware domain was to not
> run things in the Control domain.  If Control is the highest privileged
> entity, we'd rather run software in lower privileged places. Especially
> something like xenconsoled which is receiving data from the domUs.

Yes, I agree with Jason. It is a bad idea to run xenconsoled in the
Control Domain because the Control Domain is meant to be safe from
interference. We want to keep the number of potential vehicles for
interference down to a minimum and shared memory between Control Domain
and DomUs is certainly a vehicle for interference.

Re: [PATCH 0/4] XSM changes for split hardware / control domain

[Stefano Stabellini]

On Thu, 12 Jun 2025, Jan Beulich wrote:
> On 11.06.2025 07:08, Jason Andryuk wrote:
> > On 2025-06-11 09:28, Jan Beulich wrote:
> >> On 11.06.2025 00:57, Jason Andryuk wrote:
> >>> Theses are the broad changes needed for a split hardware / control
> >>> domain.
> >>>
> >>> An earlier posting gave device_model privileges to hardware domain.  For
> >>> this posting, it was split out into a new capability.  This way the
> >>> operator can choose where to run the device models without making the
> >>> hardware domain have the permissions.
> >>>
> >>> The first patch add XSM_HW_PRIV for the hardware hypercalls.  Unlike the
> >>> first posting, the control domain can call these hypercalls even though
> >>> it doesn't really make sense.  The idea was to keep the control domain
> >>> all powerful from an XSM perspective.
> >>>
> >>> SILO is changed to allow control, hardwware or xenstore to service
> >>> domUs.  Xenstore and hardware will use grants for PV interfaces.
> >>> Control wouldn't typically provide PV interfaces to domUs, but it is
> >>> given the permision to do so.  Again, to keep control all powerful.
> >>>
> >>> xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo this is not strictly
> >>> needed.  xenconsoled could read Xen's dmesg.  If it's in hwdom, then
> >>> that permission would be required.  SYSCTL_physinfo is mainly to silence
> >>> xl messages, which are mostly cosmetic.
> >>>
> >>> Jason Andryuk (4):
> >>>    xen/xsm: Add XSM_HW_PRIV
> >>>    xsm/silo: Support hwdom/control domains
> >>>    xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model
> >>>    xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo
> >>
> >> Overall I can't help the impression that this level of disaggregation simply
> >> requires the use of Flask.
> > 
> > I have thought about that.  The problem with Flask is the complexity of 
> > the security server.  We don't want to have to deal with all that code. 
> > A fixed policy is easier for our coverage testing.
> > 
> > Exposing separate control, hardware and xenstore capabilities, it makes 
> > sense for the default policy to function with them.
> 
> Yet as indicated in replies to individual patches - the boundaries between
> the three aren't clear, which imo goes (to some degree at least) against the
> purpose of "disaggregation".

At a high level things are clear. We have a static number of VMs, so
with Dom0less we don't need domain creation capabilities. Dom0 can be
de-privileged and becomes the Hardware Domain. To allow monitoring and
other privileged operations an optional Control Domain might be present
(or not).

- The Control Domain has d->is_privileged == true but has no direct
  access to hardware by default. With Dom0less ability to create all the
  VMs at boot, the Control Domain is not required for domain creation,
  so it is just optional.

- The Hardware Domain has all hardware by default but d->is_privileged
  == false. The Hardware Domain should be as close to a DomU as possible
  in terms of privileges but it must be able to run PV backends and
  Virtio backends.

- Xenstored will typically be in the Hardware Domain to enable PV
  backends, especially as the Control Domain is optional. It could also
  live in its own separate domain to speed up the boot. If a user
  chooses to deploy a Control Domain, xenstored could also run there.
  It could be entirely missing if a user only deploys Virtio. It makes
  sense to provide flexibility on this.

We have other instances of XSM policies without Flask, such as SILO. I
think the industrial/automotive use-case is clear enough and shared
among many Xen community members so I think it makes sense to optimize a
policy for it without having to involving Flask because Flask's
complexity would cause trouble in terms of coverage testing and
validation.

Re: [PATCH 3/4] xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 3866 bytes --]

On 6/13/25 18:47, Stefano Stabellini wrote:
> On Wed, 11 Jun 2025, Jan Beulich wrote:
>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>> To add more flexibility in system configuration add the new
>>> DOMAIN_CAPS_DEVICE_MODEL flag and XEN_DOMCTL_CDF_device_model.
>>>
>>> Thie new flag corresponds to allowing XSM_DM_PRIV for the domain.  This
>>> will enable running device model emulators (QEMU) from the assigne
>>> domain for multiple target domains.
>>>
>>> Stubdoms assign target allowing the stubdom to serve as the device
>>> model for a single domain.  This new flag allows the single domain to
>>> provide emulators for multiple guests.
>>>
>>> The specific scenario is a disaggregated system with the hardware domain
>>> providing device models for muitple guest domains.
>>
>> Why the hardware domain? Unless a DM also needs access to some of the
>> physical hardware, it ought to run in a separate domain. Conceivably
>> such a domain could service multiply guests, so maybe the "single
>> target" concept presently used for stubdom simply needed extending?
> 
> Not necessarily. While it is possible to have driver domains, it is not
> the default configuration.
> 
> In a default configuration, the hardware domain gets all the hardware by
> default and therefore will also run the PV backends and Virtio backends.
> The Virtio backends require DM hypercalls. Let me elaborate further.
> 
> In the datacenter, we have Dom0 typically with all the hardware, the
> backends (PV and Virtio), and also the toolstack. Then all other domains
> are created dynamically by the toolstack. Driver domains are possible
> but not very common.
> 
> In automotive/embedded, the total number of domains is static, so we can
> create them using dom0less. We don't need the toolstack to create VMs.
> Also, we have safety concerns, so we want to take away as much
> privileges as possible from Dom0. This is easy because thanks to
> dom0less, we don't need the toolstack and we don't need to create VMs
> dynamically.
> 
> So the model is that Dom0 becomes the hardware domain: it has all the
> drivers and backends but it is not privileged in the sense of
> creating/destroying other VMs. If a user wants to have Dom0 "super
> powers", they can create an optional Control Domain. The Control Domain
> is expected to be tiny, such as XTF or Zephyr. It will have the ability
> that Dom0 used to have but without the drivers. From a privilege
> perspective, the Control Domain could create additional VMs, but in
> automotive/embedded it is not expected to be a use-case because the
> total number of VMs is still static. 
> 
> So your point about driver domains. Yes, one can have driver domains the
> same way that one can have driver domains in the datacenter but it is
> not the default. The new default for embedded is what I described above
> and I think it is a very widely applicable concept across industries:
> automotive, industrial, robotics, etc. and also across vendors: AMD,
> Xilinx, Renesas, EPAM, ARM, etc.

I think the benefits of this are much reduced as long as the hardware
domain is not strongly isolated from the other domains, in the sense that
the hardware domain being able to compromise other domains is not
considered a security vulnerability.  Specifically, in safety-critical
scenarios the hardware domain (which, to the best of my understanding,
generally runs Linux) must not be able to compromise any of the safety-
critical domains.

This is, of course, achievable, but my understanding is that it isn't
something guaranteed by upstream Xen.  Rather, each user must ensure
it by assigning any hardware that could compromise Xen to the control
domain or a quarantine domain.

Could this be included in documentation?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 3/4] xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model

[Stefano Stabellini]

On Fri, 13 Jun 2025, Demi Marie Obenour wrote:
> On 6/13/25 18:47, Stefano Stabellini wrote:
> > On Wed, 11 Jun 2025, Jan Beulich wrote:
> >> On 11.06.2025 00:57, Jason Andryuk wrote:
> >>> To add more flexibility in system configuration add the new
> >>> DOMAIN_CAPS_DEVICE_MODEL flag and XEN_DOMCTL_CDF_device_model.
> >>>
> >>> Thie new flag corresponds to allowing XSM_DM_PRIV for the domain.  This
> >>> will enable running device model emulators (QEMU) from the assigne
> >>> domain for multiple target domains.
> >>>
> >>> Stubdoms assign target allowing the stubdom to serve as the device
> >>> model for a single domain.  This new flag allows the single domain to
> >>> provide emulators for multiple guests.
> >>>
> >>> The specific scenario is a disaggregated system with the hardware domain
> >>> providing device models for muitple guest domains.
> >>
> >> Why the hardware domain? Unless a DM also needs access to some of the
> >> physical hardware, it ought to run in a separate domain. Conceivably
> >> such a domain could service multiply guests, so maybe the "single
> >> target" concept presently used for stubdom simply needed extending?
> > 
> > Not necessarily. While it is possible to have driver domains, it is not
> > the default configuration.
> > 
> > In a default configuration, the hardware domain gets all the hardware by
> > default and therefore will also run the PV backends and Virtio backends.
> > The Virtio backends require DM hypercalls. Let me elaborate further.
> > 
> > In the datacenter, we have Dom0 typically with all the hardware, the
> > backends (PV and Virtio), and also the toolstack. Then all other domains
> > are created dynamically by the toolstack. Driver domains are possible
> > but not very common.
> > 
> > In automotive/embedded, the total number of domains is static, so we can
> > create them using dom0less. We don't need the toolstack to create VMs.
> > Also, we have safety concerns, so we want to take away as much
> > privileges as possible from Dom0. This is easy because thanks to
> > dom0less, we don't need the toolstack and we don't need to create VMs
> > dynamically.
> > 
> > So the model is that Dom0 becomes the hardware domain: it has all the
> > drivers and backends but it is not privileged in the sense of
> > creating/destroying other VMs. If a user wants to have Dom0 "super
> > powers", they can create an optional Control Domain. The Control Domain
> > is expected to be tiny, such as XTF or Zephyr. It will have the ability
> > that Dom0 used to have but without the drivers. From a privilege
> > perspective, the Control Domain could create additional VMs, but in
> > automotive/embedded it is not expected to be a use-case because the
> > total number of VMs is still static. 
> > 
> > So your point about driver domains. Yes, one can have driver domains the
> > same way that one can have driver domains in the datacenter but it is
> > not the default. The new default for embedded is what I described above
> > and I think it is a very widely applicable concept across industries:
> > automotive, industrial, robotics, etc. and also across vendors: AMD,
> > Xilinx, Renesas, EPAM, ARM, etc.
> 
> I think the benefits of this are much reduced as long as the hardware
> domain is not strongly isolated from the other domains, in the sense that
> the hardware domain being able to compromise other domains is not
> considered a security vulnerability.  Specifically, in safety-critical
> scenarios the hardware domain (which, to the best of my understanding,
> generally runs Linux) must not be able to compromise any of the safety-
> critical domains.
> 
> This is, of course, achievable, but my understanding is that it isn't
> something guaranteed by upstream Xen.  Rather, each user must ensure
> it by assigning any hardware that could compromise Xen to the control
> domain or a quarantine domain.
> 
> Could this be included in documentation?

Yes, I agree that it should be included in the documentation.

Re: [PATCH 3/4] xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model

[Jan Beulich]

On 14.06.2025 00:47, Stefano Stabellini wrote:
> On Wed, 11 Jun 2025, Jan Beulich wrote:
>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>> To add more flexibility in system configuration add the new
>>> DOMAIN_CAPS_DEVICE_MODEL flag and XEN_DOMCTL_CDF_device_model.
>>>
>>> Thie new flag corresponds to allowing XSM_DM_PRIV for the domain.  This
>>> will enable running device model emulators (QEMU) from the assigne
>>> domain for multiple target domains.
>>>
>>> Stubdoms assign target allowing the stubdom to serve as the device
>>> model for a single domain.  This new flag allows the single domain to
>>> provide emulators for multiple guests.
>>>
>>> The specific scenario is a disaggregated system with the hardware domain
>>> providing device models for muitple guest domains.
>>
>> Why the hardware domain? Unless a DM also needs access to some of the
>> physical hardware, it ought to run in a separate domain. Conceivably
>> such a domain could service multiply guests, so maybe the "single
>> target" concept presently used for stubdom simply needed extending?
> 
> Not necessarily. While it is possible to have driver domains, it is not
> the default configuration.
> 
> In a default configuration, the hardware domain gets all the hardware by
> default and therefore will also run the PV backends and Virtio backends.
> The Virtio backends require DM hypercalls. Let me elaborate further.
> 
> In the datacenter, we have Dom0 typically with all the hardware, the
> backends (PV and Virtio), and also the toolstack. Then all other domains
> are created dynamically by the toolstack. Driver domains are possible
> but not very common.
> 
> In automotive/embedded, the total number of domains is static, so we can
> create them using dom0less. We don't need the toolstack to create VMs.
> Also, we have safety concerns, so we want to take away as much
> privileges as possible from Dom0.

At least purely by the wording, this ...

> This is easy because thanks to
> dom0less, we don't need the toolstack and we don't need to create VMs
> dynamically.
> 
> So the model is that Dom0 becomes the hardware domain: it has all the
> drivers and backends but it is not privileged in the sense of
> creating/destroying other VMs. If a user wants to have Dom0 "super
> powers", they can create an optional Control Domain. The Control Domain
> is expected to be tiny, such as XTF or Zephyr. It will have the ability
> that Dom0 used to have but without the drivers. From a privilege
> perspective, the Control Domain could create additional VMs, but in
> automotive/embedded it is not expected to be a use-case because the
> total number of VMs is still static. 
> 
> So your point about driver domains. Yes, one can have driver domains the
> same way that one can have driver domains in the datacenter but it is
> not the default.

... kind of contradicts this: Running e.g. qemu in Dom0 gives Dom0 quite
a bit of extra privilege. (And no, the term "driver domain" does not
describe a domain running DMs, imo.)

Jan

> The new default for embedded is what I described above
> and I think it is a very widely applicable concept across industries:
> automotive, industrial, robotics, etc. and also across vendors: AMD,
> Xilinx, Renesas, EPAM, ARM, etc.

Re: [PATCH 4/4] xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

[Jan Beulich]

On 14.06.2025 00:51, Stefano Stabellini wrote:
> On Wed, 11 Jun 2025, Jason Andryuk wrote:
>> On 2025-06-11 09:27, Jan Beulich wrote:
>>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>>> Allow the hwdom to access the console, and to access physical
>>>> information about the system.
>>>>
>>>> xenconsoled can read Xen's dmesg.  If it's in hwdom, then that
>>>> permission would be required.
>>>
>>> Why would xenconsoled run in the hardware domain? It's purely a software
>>> construct, isn't it? As a daemon, putting it in the control domain may
>>> make sense. Otherwise it probably ought to go in a service domain.
>>
>> My approach has been to transform dom0 into the hardware domain and add a new
>> control domain.  xenconsoled was left running in the hardware domain.
> 
> I think we should keep xenconsoled in the hardware domain because the
> control domain should be just optional. (However, one could say that with
> Denis' recent changes xenconsoled is also optional because one can use
> console hypercalls or emulators (PL011, NS16550) for all DomUs.)
> 
> 
> 
>> I suppose it could move.  Maybe that would be fine?  I haven't tried. The
>> Hyperlaunch code populates the console grants to point at the hardware domain,
>> and I just followed that.
>>
>> One aspect of why I left most things running in the Hardware domain was to not
>> run things in the Control domain.  If Control is the highest privileged
>> entity, we'd rather run software in lower privileged places. Especially
>> something like xenconsoled which is receiving data from the domUs.
> 
> Yes, I agree with Jason. It is a bad idea to run xenconsoled in the
> Control Domain because the Control Domain is meant to be safe from
> interference. We want to keep the number of potential vehicles for
> interference down to a minimum and shared memory between Control Domain
> and DomUs is certainly a vehicle for interference.

As much as it is when xenconsoled runs in the hardware domain? Especially
if the hardware domain is also running e.g. PV backends or qemu instances?

Jan

Re: [PATCH 4/4] xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

[Stefano Stabellini]

On Mon, 16 Jun 2025, Jan Beulich wrote:
> On 14.06.2025 00:51, Stefano Stabellini wrote:
> > On Wed, 11 Jun 2025, Jason Andryuk wrote:
> >> On 2025-06-11 09:27, Jan Beulich wrote:
> >>> On 11.06.2025 00:57, Jason Andryuk wrote:
> >>>> Allow the hwdom to access the console, and to access physical
> >>>> information about the system.
> >>>>
> >>>> xenconsoled can read Xen's dmesg.  If it's in hwdom, then that
> >>>> permission would be required.
> >>>
> >>> Why would xenconsoled run in the hardware domain? It's purely a software
> >>> construct, isn't it? As a daemon, putting it in the control domain may
> >>> make sense. Otherwise it probably ought to go in a service domain.
> >>
> >> My approach has been to transform dom0 into the hardware domain and add a new
> >> control domain.  xenconsoled was left running in the hardware domain.
> > 
> > I think we should keep xenconsoled in the hardware domain because the
> > control domain should be just optional. (However, one could say that with
> > Denis' recent changes xenconsoled is also optional because one can use
> > console hypercalls or emulators (PL011, NS16550) for all DomUs.)
> > 
> > 
> > 
> >> I suppose it could move.  Maybe that would be fine?  I haven't tried. The
> >> Hyperlaunch code populates the console grants to point at the hardware domain,
> >> and I just followed that.
> >>
> >> One aspect of why I left most things running in the Hardware domain was to not
> >> run things in the Control domain.  If Control is the highest privileged
> >> entity, we'd rather run software in lower privileged places. Especially
> >> something like xenconsoled which is receiving data from the domUs.
> > 
> > Yes, I agree with Jason. It is a bad idea to run xenconsoled in the
> > Control Domain because the Control Domain is meant to be safe from
> > interference. We want to keep the number of potential vehicles for
> > interference down to a minimum and shared memory between Control Domain
> > and DomUs is certainly a vehicle for interference.
> 
> As much as it is when xenconsoled runs in the hardware domain? Especially
> if the hardware domain is also running e.g. PV backends or qemu instances?

It looks like you are thinking of the possible
interference from the Hardware Domain to the Control Domain via
xenconsoled, correct?

If that is the case, good thinking. I can see that you have really
understood the essence of the problem we are trying to solve.

That is not an issue because the Control Domain shouldn't use PV
console. Instead, it should use the console hypercall, or the
PL011/NS16550 emulators in Xen.

Re: [PATCH 3/4] xen: Add DOMAIN_CAPS_DEVICE_MODEL & XEN_DOMCTL_CDF_device_model

[Stefano Stabellini]

On Mon, 16 Jun 2025, Jan Beulich wrote:
> On 14.06.2025 00:47, Stefano Stabellini wrote:
> > On Wed, 11 Jun 2025, Jan Beulich wrote:
> >> On 11.06.2025 00:57, Jason Andryuk wrote:
> >>> To add more flexibility in system configuration add the new
> >>> DOMAIN_CAPS_DEVICE_MODEL flag and XEN_DOMCTL_CDF_device_model.
> >>>
> >>> Thie new flag corresponds to allowing XSM_DM_PRIV for the domain.  This
> >>> will enable running device model emulators (QEMU) from the assigne
> >>> domain for multiple target domains.
> >>>
> >>> Stubdoms assign target allowing the stubdom to serve as the device
> >>> model for a single domain.  This new flag allows the single domain to
> >>> provide emulators for multiple guests.
> >>>
> >>> The specific scenario is a disaggregated system with the hardware domain
> >>> providing device models for muitple guest domains.
> >>
> >> Why the hardware domain? Unless a DM also needs access to some of the
> >> physical hardware, it ought to run in a separate domain. Conceivably
> >> such a domain could service multiply guests, so maybe the "single
> >> target" concept presently used for stubdom simply needed extending?
> > 
> > Not necessarily. While it is possible to have driver domains, it is not
> > the default configuration.
> > 
> > In a default configuration, the hardware domain gets all the hardware by
> > default and therefore will also run the PV backends and Virtio backends.
> > The Virtio backends require DM hypercalls. Let me elaborate further.
> > 
> > In the datacenter, we have Dom0 typically with all the hardware, the
> > backends (PV and Virtio), and also the toolstack. Then all other domains
> > are created dynamically by the toolstack. Driver domains are possible
> > but not very common.
> > 
> > In automotive/embedded, the total number of domains is static, so we can
> > create them using dom0less. We don't need the toolstack to create VMs.
> > Also, we have safety concerns, so we want to take away as much
> > privileges as possible from Dom0.
> 
> At least purely by the wording, this ...
> 
> > This is easy because thanks to
> > dom0less, we don't need the toolstack and we don't need to create VMs
> > dynamically.
> > 
> > So the model is that Dom0 becomes the hardware domain: it has all the
> > drivers and backends but it is not privileged in the sense of
> > creating/destroying other VMs. If a user wants to have Dom0 "super
> > powers", they can create an optional Control Domain. The Control Domain
> > is expected to be tiny, such as XTF or Zephyr. It will have the ability
> > that Dom0 used to have but without the drivers. From a privilege
> > perspective, the Control Domain could create additional VMs, but in
> > automotive/embedded it is not expected to be a use-case because the
> > total number of VMs is still static. 
> > 
> > So your point about driver domains. Yes, one can have driver domains the
> > same way that one can have driver domains in the datacenter but it is
> > not the default.
> 
> ... kind of contradicts this: Running e.g. qemu in Dom0 gives Dom0 quite
> a bit of extra privilege.

Yes, in an ideal world that would not be necessary. However, in
automotive Virtio has become the standard. While there are efforts
ongoing to rework the Virtio protocol to have a better security/safety
profile, we need to provide something that works today. Even PV drivers
are not perfect in that regard because I don't think we can claim they
are free from interference but that is another topic.

In order to provide something that works today, we need to have support
for virtio backends in the hardware domain. Like you said, that gives
quite a bit of extra privilege to the hardware domain which is not
acceptable when targeting a "safe" VM such as the Control Domain.

Thus, we have another series to restrict DM and foreign mapping
hypercalls from being able to target "safe" domains. In other words, the
patch series will prevent the hardware domain from being able to target
the Control Domain or another DomU configured as "safe" with DM
hypercalls or foreign mapping hypercalls.

It will be up to the user to decide which domUs the harwdare domain will
be able to target. That way, the user will still be able to configure
one or more VMs are completely protected from interference from the
hardware domain at the cost of having no (traditional) virtio devices.

Re: [PATCH 4/4] xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

[Jan Beulich]

On 17.06.2025 02:10, Stefano Stabellini wrote:
> On Mon, 16 Jun 2025, Jan Beulich wrote:
>> On 14.06.2025 00:51, Stefano Stabellini wrote:
>>> On Wed, 11 Jun 2025, Jason Andryuk wrote:
>>>> On 2025-06-11 09:27, Jan Beulich wrote:
>>>>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>>>>> Allow the hwdom to access the console, and to access physical
>>>>>> information about the system.
>>>>>>
>>>>>> xenconsoled can read Xen's dmesg.  If it's in hwdom, then that
>>>>>> permission would be required.
>>>>>
>>>>> Why would xenconsoled run in the hardware domain? It's purely a software
>>>>> construct, isn't it? As a daemon, putting it in the control domain may
>>>>> make sense. Otherwise it probably ought to go in a service domain.
>>>>
>>>> My approach has been to transform dom0 into the hardware domain and add a new
>>>> control domain.  xenconsoled was left running in the hardware domain.
>>>
>>> I think we should keep xenconsoled in the hardware domain because the
>>> control domain should be just optional. (However, one could say that with
>>> Denis' recent changes xenconsoled is also optional because one can use
>>> console hypercalls or emulators (PL011, NS16550) for all DomUs.)
>>>
>>>
>>>
>>>> I suppose it could move.  Maybe that would be fine?  I haven't tried. The
>>>> Hyperlaunch code populates the console grants to point at the hardware domain,
>>>> and I just followed that.
>>>>
>>>> One aspect of why I left most things running in the Hardware domain was to not
>>>> run things in the Control domain.  If Control is the highest privileged
>>>> entity, we'd rather run software in lower privileged places. Especially
>>>> something like xenconsoled which is receiving data from the domUs.
>>>
>>> Yes, I agree with Jason. It is a bad idea to run xenconsoled in the
>>> Control Domain because the Control Domain is meant to be safe from
>>> interference. We want to keep the number of potential vehicles for
>>> interference down to a minimum and shared memory between Control Domain
>>> and DomUs is certainly a vehicle for interference.
>>
>> As much as it is when xenconsoled runs in the hardware domain? Especially
>> if the hardware domain is also running e.g. PV backends or qemu instances?
> 
> It looks like you are thinking of the possible
> interference from the Hardware Domain to the Control Domain via
> xenconsoled, correct?

More like interference with the system as a whole, which simply includes
Control.

> If that is the case, good thinking. I can see that you have really
> understood the essence of the problem we are trying to solve.
> 
> That is not an issue because the Control Domain shouldn't use PV
> console. Instead, it should use the console hypercall, or the
> PL011/NS16550 emulators in Xen.

Well. I think the underlying concept of Control Domain being highly
privileged needs more general discussion. As indicated elsewhere, I
didn't think disaggregation (whichever way done) would leave any
domain with effectively full privilege. I wonder what others think.

Jan

Re: [PATCH 4/4] xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

[Stefano Stabellini]

On Tue, 17 Jun 2025, Jan Beulich wrote:
> On 17.06.2025 02:10, Stefano Stabellini wrote:
> > On Mon, 16 Jun 2025, Jan Beulich wrote:
> >> On 14.06.2025 00:51, Stefano Stabellini wrote:
> >>> On Wed, 11 Jun 2025, Jason Andryuk wrote:
> >>>> On 2025-06-11 09:27, Jan Beulich wrote:
> >>>>> On 11.06.2025 00:57, Jason Andryuk wrote:
> >>>>>> Allow the hwdom to access the console, and to access physical
> >>>>>> information about the system.
> >>>>>>
> >>>>>> xenconsoled can read Xen's dmesg.  If it's in hwdom, then that
> >>>>>> permission would be required.
> >>>>>
> >>>>> Why would xenconsoled run in the hardware domain? It's purely a software
> >>>>> construct, isn't it? As a daemon, putting it in the control domain may
> >>>>> make sense. Otherwise it probably ought to go in a service domain.
> >>>>
> >>>> My approach has been to transform dom0 into the hardware domain and add a new
> >>>> control domain.  xenconsoled was left running in the hardware domain.
> >>>
> >>> I think we should keep xenconsoled in the hardware domain because the
> >>> control domain should be just optional. (However, one could say that with
> >>> Denis' recent changes xenconsoled is also optional because one can use
> >>> console hypercalls or emulators (PL011, NS16550) for all DomUs.)
> >>>
> >>>
> >>>
> >>>> I suppose it could move.  Maybe that would be fine?  I haven't tried. The
> >>>> Hyperlaunch code populates the console grants to point at the hardware domain,
> >>>> and I just followed that.
> >>>>
> >>>> One aspect of why I left most things running in the Hardware domain was to not
> >>>> run things in the Control domain.  If Control is the highest privileged
> >>>> entity, we'd rather run software in lower privileged places. Especially
> >>>> something like xenconsoled which is receiving data from the domUs.
> >>>
> >>> Yes, I agree with Jason. It is a bad idea to run xenconsoled in the
> >>> Control Domain because the Control Domain is meant to be safe from
> >>> interference. We want to keep the number of potential vehicles for
> >>> interference down to a minimum and shared memory between Control Domain
> >>> and DomUs is certainly a vehicle for interference.
> >>
> >> As much as it is when xenconsoled runs in the hardware domain? Especially
> >> if the hardware domain is also running e.g. PV backends or qemu instances?
> > 
> > It looks like you are thinking of the possible
> > interference from the Hardware Domain to the Control Domain via
> > xenconsoled, correct?
> 
> More like interference with the system as a whole, which simply includes
> Control.
> 
> > If that is the case, good thinking. I can see that you have really
> > understood the essence of the problem we are trying to solve.
> > 
> > That is not an issue because the Control Domain shouldn't use PV
> > console. Instead, it should use the console hypercall, or the
> > PL011/NS16550 emulators in Xen.
> 
> Well. I think the underlying concept of Control Domain being highly
> privileged needs more general discussion. As indicated elsewhere, I
> didn't think disaggregation (whichever way done) would leave any
> domain with effectively full privilege. I wonder what others think.

Keep in mind that the threat model here is different from the
datacenter. 

But the Control Domain is optional. If the user doesn't want it, the
user can avoid it.

Even on a fully static system (no VM creation), it is convenient to have
a domain that can monitor the others and trigger domain reset (we are
reimplementing domain reboot to be more like a soft reset so that the VM
doesn't need to be destroyed and recreated). As an example, the Control
Domain could be used to monitor a non-safe domain such as Android,
detect an Android crash, and trigger an Android reboot.

Re: [PATCH 4/4] xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

[Jan Beulich]

On 19.06.2025 02:36, Stefano Stabellini wrote:
> On Tue, 17 Jun 2025, Jan Beulich wrote:
>> On 17.06.2025 02:10, Stefano Stabellini wrote:
>>> On Mon, 16 Jun 2025, Jan Beulich wrote:
>>>> On 14.06.2025 00:51, Stefano Stabellini wrote:
>>>>> On Wed, 11 Jun 2025, Jason Andryuk wrote:
>>>>>> On 2025-06-11 09:27, Jan Beulich wrote:
>>>>>>> On 11.06.2025 00:57, Jason Andryuk wrote:
>>>>>>>> Allow the hwdom to access the console, and to access physical
>>>>>>>> information about the system.
>>>>>>>>
>>>>>>>> xenconsoled can read Xen's dmesg.  If it's in hwdom, then that
>>>>>>>> permission would be required.
>>>>>>>
>>>>>>> Why would xenconsoled run in the hardware domain? It's purely a software
>>>>>>> construct, isn't it? As a daemon, putting it in the control domain may
>>>>>>> make sense. Otherwise it probably ought to go in a service domain.
>>>>>>
>>>>>> My approach has been to transform dom0 into the hardware domain and add a new
>>>>>> control domain.  xenconsoled was left running in the hardware domain.
>>>>>
>>>>> I think we should keep xenconsoled in the hardware domain because the
>>>>> control domain should be just optional. (However, one could say that with
>>>>> Denis' recent changes xenconsoled is also optional because one can use
>>>>> console hypercalls or emulators (PL011, NS16550) for all DomUs.)
>>>>>
>>>>>
>>>>>
>>>>>> I suppose it could move.  Maybe that would be fine?  I haven't tried. The
>>>>>> Hyperlaunch code populates the console grants to point at the hardware domain,
>>>>>> and I just followed that.
>>>>>>
>>>>>> One aspect of why I left most things running in the Hardware domain was to not
>>>>>> run things in the Control domain.  If Control is the highest privileged
>>>>>> entity, we'd rather run software in lower privileged places. Especially
>>>>>> something like xenconsoled which is receiving data from the domUs.
>>>>>
>>>>> Yes, I agree with Jason. It is a bad idea to run xenconsoled in the
>>>>> Control Domain because the Control Domain is meant to be safe from
>>>>> interference. We want to keep the number of potential vehicles for
>>>>> interference down to a minimum and shared memory between Control Domain
>>>>> and DomUs is certainly a vehicle for interference.
>>>>
>>>> As much as it is when xenconsoled runs in the hardware domain? Especially
>>>> if the hardware domain is also running e.g. PV backends or qemu instances?
>>>
>>> It looks like you are thinking of the possible
>>> interference from the Hardware Domain to the Control Domain via
>>> xenconsoled, correct?
>>
>> More like interference with the system as a whole, which simply includes
>> Control.
>>
>>> If that is the case, good thinking. I can see that you have really
>>> understood the essence of the problem we are trying to solve.
>>>
>>> That is not an issue because the Control Domain shouldn't use PV
>>> console. Instead, it should use the console hypercall, or the
>>> PL011/NS16550 emulators in Xen.
>>
>> Well. I think the underlying concept of Control Domain being highly
>> privileged needs more general discussion. As indicated elsewhere, I
>> didn't think disaggregation (whichever way done) would leave any
>> domain with effectively full privilege. I wonder what others think.
> 
> Keep in mind that the threat model here is different from the
> datacenter. 
> 
> But the Control Domain is optional. If the user doesn't want it, the
> user can avoid it.
> 
> Even on a fully static system (no VM creation), it is convenient to have
> a domain that can monitor the others and trigger domain reset (we are
> reimplementing domain reboot to be more like a soft reset so that the VM
> doesn't need to be destroyed and recreated).

Suggesting that in such an environment Control should have no permission
to create domains. This would avoid various threats, including e.g.
massive amounts of dynamic memory allocation.

> As an example, the Control
> Domain could be used to monitor a non-safe domain such as Android,
> detect an Android crash, and trigger an Android reboot.

Yet at the same time it would have to be prevented from interfering with
any of the critical domains.

Altogether this doesn't sound like "highest privilege" to me then.

Jan

Re: [PATCH 4/4] xsm/dummy: Allow hwdom SYSCTL_readconsole/physinfo

[Stefano Stabellini]

On Fri, 20 Jun 2025, Jan Beulich wrote:
> On 19.06.2025 02:36, Stefano Stabellini wrote:
> > On Tue, 17 Jun 2025, Jan Beulich wrote:
> >> On 17.06.2025 02:10, Stefano Stabellini wrote:
> >>> On Mon, 16 Jun 2025, Jan Beulich wrote:
> >>>> On 14.06.2025 00:51, Stefano Stabellini wrote:
> >>>>> On Wed, 11 Jun 2025, Jason Andryuk wrote:
> >>>>>> On 2025-06-11 09:27, Jan Beulich wrote:
> >>>>>>> On 11.06.2025 00:57, Jason Andryuk wrote:
> >>>>>>>> Allow the hwdom to access the console, and to access physical
> >>>>>>>> information about the system.
> >>>>>>>>
> >>>>>>>> xenconsoled can read Xen's dmesg.  If it's in hwdom, then that
> >>>>>>>> permission would be required.
> >>>>>>>
> >>>>>>> Why would xenconsoled run in the hardware domain? It's purely a software
> >>>>>>> construct, isn't it? As a daemon, putting it in the control domain may
> >>>>>>> make sense. Otherwise it probably ought to go in a service domain.
> >>>>>>
> >>>>>> My approach has been to transform dom0 into the hardware domain and add a new
> >>>>>> control domain.  xenconsoled was left running in the hardware domain.
> >>>>>
> >>>>> I think we should keep xenconsoled in the hardware domain because the
> >>>>> control domain should be just optional. (However, one could say that with
> >>>>> Denis' recent changes xenconsoled is also optional because one can use
> >>>>> console hypercalls or emulators (PL011, NS16550) for all DomUs.)
> >>>>>
> >>>>>
> >>>>>
> >>>>>> I suppose it could move.  Maybe that would be fine?  I haven't tried. The
> >>>>>> Hyperlaunch code populates the console grants to point at the hardware domain,
> >>>>>> and I just followed that.
> >>>>>>
> >>>>>> One aspect of why I left most things running in the Hardware domain was to not
> >>>>>> run things in the Control domain.  If Control is the highest privileged
> >>>>>> entity, we'd rather run software in lower privileged places. Especially
> >>>>>> something like xenconsoled which is receiving data from the domUs.
> >>>>>
> >>>>> Yes, I agree with Jason. It is a bad idea to run xenconsoled in the
> >>>>> Control Domain because the Control Domain is meant to be safe from
> >>>>> interference. We want to keep the number of potential vehicles for
> >>>>> interference down to a minimum and shared memory between Control Domain
> >>>>> and DomUs is certainly a vehicle for interference.
> >>>>
> >>>> As much as it is when xenconsoled runs in the hardware domain? Especially
> >>>> if the hardware domain is also running e.g. PV backends or qemu instances?
> >>>
> >>> It looks like you are thinking of the possible
> >>> interference from the Hardware Domain to the Control Domain via
> >>> xenconsoled, correct?
> >>
> >> More like interference with the system as a whole, which simply includes
> >> Control.
> >>
> >>> If that is the case, good thinking. I can see that you have really
> >>> understood the essence of the problem we are trying to solve.
> >>>
> >>> That is not an issue because the Control Domain shouldn't use PV
> >>> console. Instead, it should use the console hypercall, or the
> >>> PL011/NS16550 emulators in Xen.
> >>
> >> Well. I think the underlying concept of Control Domain being highly
> >> privileged needs more general discussion. As indicated elsewhere, I
> >> didn't think disaggregation (whichever way done) would leave any
> >> domain with effectively full privilege. I wonder what others think.
> > 
> > Keep in mind that the threat model here is different from the
> > datacenter. 
> > 
> > But the Control Domain is optional. If the user doesn't want it, the
> > user can avoid it.
> > 
> > Even on a fully static system (no VM creation), it is convenient to have
> > a domain that can monitor the others and trigger domain reset (we are
> > reimplementing domain reboot to be more like a soft reset so that the VM
> > doesn't need to be destroyed and recreated).
> 
> Suggesting that in such an environment Control should have no permission
> to create domains. This would avoid various threats, including e.g.
> massive amounts of dynamic memory allocation.

+1


> > As an example, the Control
> > Domain could be used to monitor a non-safe domain such as Android,
> > detect an Android crash, and trigger an Android reboot.
> 
> Yet at the same time it would have to be prevented from interfering with
> any of the critical domains.
> 
> Altogether this doesn't sound like "highest privilege" to me then.

You are right. We should be more precise with our wording.