[PATCH v5 0/5] ARM Nested Virt Support

[PATCH v5 0/5] ARM Nested Virt Support

[Eric Auger]

Now that ARM nested virt has landed in kvm/next, let's turn the series
into a PATCH series. The linux header update was made against kvm/next.

For gaining virt functionality in KVM accelerated L1, The host needs to
be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
with: -machine virt,virtualization=on.

This series can be found at:
https://github.com/eauger/qemu/tree/v10.0.0-nv-v5

Original version from Miguel:
[1] https://lore.kernel.org/all/20230227163718.62003-1-miguel.luis@oracle.com/
version from Haibo:
[2] https://lore.kernel.org/qemu-devel/cover.1617281290.git.haibo.xu@linaro.org/

History:
v4 -> v5:
- rebase on top of v10.0.0

v3 -> v4:
- fix: only set maint_irq if vms->virt

v2 -> v3:
- KVM EL2 only is set if virtualization option is set
- fixes regression with virtualization=off
- Add checks against unsupported GIC configs until the kernel does

Eric Auger (1):
  linux-headers: Update against kvm/next

Haibo Xu (4):
  hw/arm: Allow setting KVM vGIC maintenance IRQ
  target/arm/kvm: Add helper to detect EL2 when using KVM
  target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported
  hw/arm/virt: Allow virt extensions with KVM

 include/hw/intc/arm_gicv3_common.h          |  1 +
 include/standard-headers/linux/virtio_pci.h |  1 +
 linux-headers/asm-arm64/kvm.h               |  9 +++++----
 linux-headers/linux/kvm.h                   |  3 +++
 linux-headers/linux/vhost.h                 |  4 ++--
 target/arm/kvm_arm.h                        |  7 +++++++
 hw/arm/virt.c                               | 13 ++++++++++++-
 hw/intc/arm_gicv3_common.c                  |  1 +
 hw/intc/arm_gicv3_kvm.c                     | 21 +++++++++++++++++++++
 target/arm/kvm-stub.c                       |  5 +++++
 target/arm/kvm.c                            | 21 +++++++++++++++++++++
 11 files changed, 79 insertions(+), 7 deletions(-)

-- 
2.49.0

[PATCH v5 1/5] linux-headers: Update against kvm/next

[Eric Auger]

This is a linux header update against kvm/next (e9f17038d814)

especially targeted to get nested virt enablement.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
---
 include/standard-headers/linux/virtio_pci.h | 1 +
 linux-headers/asm-arm64/kvm.h               | 9 +++++----
 linux-headers/linux/kvm.h                   | 3 +++
 linux-headers/linux/vhost.h                 | 4 ++--
 4 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/include/standard-headers/linux/virtio_pci.h b/include/standard-headers/linux/virtio_pci.h
index 91fec6f502..09e964e6ee 100644
--- a/include/standard-headers/linux/virtio_pci.h
+++ b/include/standard-headers/linux/virtio_pci.h
@@ -246,6 +246,7 @@ struct virtio_pci_cfg_cap {
 #define VIRTIO_ADMIN_CMD_LIST_USE	0x1
 
 /* Admin command group type. */
+#define VIRTIO_ADMIN_GROUP_TYPE_SELF	0x0
 #define VIRTIO_ADMIN_GROUP_TYPE_SRIOV	0x1
 
 /* Transitional device admin command. */
diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h
index 4e6aff08df..f4d9baafa1 100644
--- a/linux-headers/asm-arm64/kvm.h
+++ b/linux-headers/asm-arm64/kvm.h
@@ -419,10 +419,11 @@ enum {
 
 /* Device Control API on vcpu fd */
 #define KVM_ARM_VCPU_PMU_V3_CTRL	0
-#define   KVM_ARM_VCPU_PMU_V3_IRQ	0
-#define   KVM_ARM_VCPU_PMU_V3_INIT	1
-#define   KVM_ARM_VCPU_PMU_V3_FILTER	2
-#define   KVM_ARM_VCPU_PMU_V3_SET_PMU	3
+#define   KVM_ARM_VCPU_PMU_V3_IRQ		0
+#define   KVM_ARM_VCPU_PMU_V3_INIT		1
+#define   KVM_ARM_VCPU_PMU_V3_FILTER		2
+#define   KVM_ARM_VCPU_PMU_V3_SET_PMU		3
+#define   KVM_ARM_VCPU_PMU_V3_SET_NR_COUNTERS	4
 #define KVM_ARM_VCPU_TIMER_CTRL		1
 #define   KVM_ARM_VCPU_TIMER_IRQ_VTIMER		0
 #define   KVM_ARM_VCPU_TIMER_IRQ_PTIMER		1
diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 99cc82a275..e50334fc52 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -923,6 +923,9 @@ struct kvm_enable_cap {
 #define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237
 #define KVM_CAP_X86_GUEST_MODE 238
 #define KVM_CAP_ARM_WRITABLE_IMP_ID_REGS 239
+#define KVM_CAP_ARM_EL2 240
+#define KVM_CAP_ARM_EL2_E2H0 241
+#define KVM_CAP_RISCV_MP_STATE_RESET 242
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
diff --git a/linux-headers/linux/vhost.h b/linux-headers/linux/vhost.h
index b95dd84eef..d4b3e2ae13 100644
--- a/linux-headers/linux/vhost.h
+++ b/linux-headers/linux/vhost.h
@@ -28,10 +28,10 @@
 
 /* Set current process as the (exclusive) owner of this file descriptor.  This
  * must be called before any other vhost command.  Further calls to
- * VHOST_OWNER_SET fail until VHOST_OWNER_RESET is called. */
+ * VHOST_SET_OWNER fail until VHOST_RESET_OWNER is called. */
 #define VHOST_SET_OWNER _IO(VHOST_VIRTIO, 0x01)
 /* Give up ownership, and reset the device to default values.
- * Allows subsequent call to VHOST_OWNER_SET to succeed. */
+ * Allows subsequent call to VHOST_SET_OWNER to succeed. */
 #define VHOST_RESET_OWNER _IO(VHOST_VIRTIO, 0x02)
 
 /* Set up/modify memory layout */
-- 
2.49.0

[PATCH v5 2/5] hw/arm: Allow setting KVM vGIC maintenance IRQ

[Eric Auger]

From: Haibo Xu <haibo.xu@linaro.org>

Allow virt arm machine to set the interrupt ID for the KVM
GIC maintenance interrupt.

This setting must be done before the KVM_DEV_ARM_VGIC_CTRL_INIT
hence the choice to perform the setting in the GICv3 realize
instead of proceeding the same way as kvm_arm_pmu_set_irq().

Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
Signed-off-by: Eric Auger <eric.auger@redhat.com>

---

v3 -> v4:
- only set maint_irq if vms->virt

v2 -> v3:
- tweak the commit message and explain why we do not proceed
  the same way as kvm_arm_pmu_set_irq (Peter)

v1 -> v2:
- [Miguel] replaced the has_virt_extensions by the maintenance irq
  intid property. [Eric] restored kvm_device_check_attr and
  kvm_device_access standard usage and conditionally call those
  if the prop is set.
---
 include/hw/intc/arm_gicv3_common.h |  1 +
 hw/arm/virt.c                      |  3 +++
 hw/intc/arm_gicv3_common.c         |  1 +
 hw/intc/arm_gicv3_kvm.c            | 21 +++++++++++++++++++++
 4 files changed, 26 insertions(+)

diff --git a/include/hw/intc/arm_gicv3_common.h b/include/hw/intc/arm_gicv3_common.h
index a3d6a0e507..c18503869f 100644
--- a/include/hw/intc/arm_gicv3_common.h
+++ b/include/hw/intc/arm_gicv3_common.h
@@ -231,6 +231,7 @@ struct GICv3State {
     uint32_t num_cpu;
     uint32_t num_irq;
     uint32_t revision;
+    uint32_t maint_irq;
     bool lpi_enable;
     bool nmi_support;
     bool security_extn;
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 9a6cd085a3..675ddeec14 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -831,6 +831,9 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)
                                          OBJECT(mem), &error_fatal);
                 qdev_prop_set_bit(vms->gic, "has-lpi", true);
             }
+        } else if (vms->virt) {
+            qdev_prop_set_uint32(vms->gic, "maintenance-interrupt-id",
+                                 ARCH_GIC_MAINT_IRQ);
         }
     } else {
         if (!kvm_irqchip_in_kernel()) {
diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c
index 1cee68193c..e438d8c042 100644
--- a/hw/intc/arm_gicv3_common.c
+++ b/hw/intc/arm_gicv3_common.c
@@ -612,6 +612,7 @@ static const Property arm_gicv3_common_properties[] = {
     DEFINE_PROP_BOOL("has-lpi", GICv3State, lpi_enable, 0),
     DEFINE_PROP_BOOL("has-nmi", GICv3State, nmi_support, 0),
     DEFINE_PROP_BOOL("has-security-extensions", GICv3State, security_extn, 0),
+    DEFINE_PROP_UINT32("maintenance-interrupt-id", GICv3State, maint_irq, 0),
     /*
      * Compatibility property: force 8 bits of physical priority, even
      * if the CPU being emulated should have fewer.
diff --git a/hw/intc/arm_gicv3_kvm.c b/hw/intc/arm_gicv3_kvm.c
index 3be3bf6c28..b30aac7aee 100644
--- a/hw/intc/arm_gicv3_kvm.c
+++ b/hw/intc/arm_gicv3_kvm.c
@@ -22,6 +22,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "hw/intc/arm_gicv3_common.h"
+#include "hw/arm/virt.h"
 #include "qemu/error-report.h"
 #include "qemu/module.h"
 #include "system/kvm.h"
@@ -825,6 +826,26 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp)
         return;
     }
 
+    if (s->maint_irq) {
+        int ret;
+
+        ret = kvm_device_check_attr(s->dev_fd,
+                                    KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0);
+        if (!ret) {
+            error_setg_errno(errp, errno,
+                             "VGICv3 setting maintenance IRQ is not "
+                             "supported by this host kernel");
+            return;
+        }
+
+        ret = kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ, 0,
+                                &s->maint_irq, true, errp);
+        if (ret) {
+            error_setg_errno(errp, errno, "Failed to set VGIC maintenance IRQ");
+            return;
+       }
+    }
+
     multiple_redist_region_allowed =
         kvm_device_check_attr(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_ADDR,
                               KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION);
-- 
2.49.0

[PATCH v5 3/5] target/arm/kvm: Add helper to detect EL2 when using KVM

[Eric Auger]

From: Haibo Xu <haibo.xu@linaro.org>

Introduce query support for KVM_CAP_ARM_EL2.

Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 target/arm/kvm_arm.h  | 7 +++++++
 target/arm/kvm-stub.c | 5 +++++
 target/arm/kvm.c      | 5 +++++
 3 files changed, 17 insertions(+)

diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
index c4178d1327..445cf9a0a3 100644
--- a/target/arm/kvm_arm.h
+++ b/target/arm/kvm_arm.h
@@ -190,6 +190,13 @@ bool kvm_arm_sve_supported(void);
  */
 bool kvm_arm_mte_supported(void);
 
+/**
+ * kvm_arm_el2_supported:
+ *
+ * Returns true if KVM can enable EL2 and false otherwise.
+ */
+bool kvm_arm_el2_supported(void);
+
 /**
  * kvm_arm_get_max_vm_ipa_size:
  * @ms: Machine state handle
diff --git a/target/arm/kvm-stub.c b/target/arm/kvm-stub.c
index 34e57fab01..c93462c5b9 100644
--- a/target/arm/kvm-stub.c
+++ b/target/arm/kvm-stub.c
@@ -47,6 +47,11 @@ bool kvm_arm_mte_supported(void)
     return false;
 }
 
+bool kvm_arm_el2_supported(void)
+{
+    return false;
+}
+
 /*
  * These functions should never actually be called without KVM support.
  */
diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index a2791aa866..ad6c80dd49 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -1769,6 +1769,11 @@ bool kvm_arm_aarch32_supported(void)
     return kvm_check_extension(kvm_state, KVM_CAP_ARM_EL1_32BIT);
 }
 
+bool kvm_arm_el2_supported(void)
+{
+    return kvm_check_extension(kvm_state, KVM_CAP_ARM_EL2);
+}
+
 bool kvm_arm_sve_supported(void)
 {
     return kvm_check_extension(kvm_state, KVM_CAP_ARM_SVE);
-- 
2.49.0

[PATCH v5 4/5] target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported

[Eric Auger]

From: Haibo Xu <haibo.xu@linaro.org>

KVM_CAP_ARM_EL2 must be supported by the cpu to enable ARM_FEATURE_EL2.
In case the host does support NV, expose the feature.

Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

---
v2 -> v3:
- check pmu->has_el2 on kvm_arch_init_vcpu() when setting
  KVM_ARM_VCPU_HAS_EL2 feature (Peter)

v1 -> v2:
- remove isar_feature_aa64_aa32_el2 modif in target/arm/cpu.h
  [Richard] and use el2_supported in kvm_arch_init_vcpu
---
 target/arm/kvm.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index ad6c80dd49..495c8246c2 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -227,6 +227,7 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
      */
     int fdarray[3];
     bool sve_supported;
+    bool el2_supported;
     bool pmu_supported = false;
     uint64_t features = 0;
     int err;
@@ -246,6 +247,14 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
         init.features[0] |= 1 << KVM_ARM_VCPU_SVE;
     }
 
+    /*
+     * Ask for EL2 if supported.
+     */
+    el2_supported = kvm_arm_el2_supported();
+    if (el2_supported) {
+        init.features[0] |= 1 << KVM_ARM_VCPU_HAS_EL2;
+    }
+
     /*
      * Ask for Pointer Authentication if supported, so that we get
      * the unsanitized field values for AA64ISAR1_EL1.
@@ -429,6 +438,10 @@ static bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
     features |= 1ULL << ARM_FEATURE_AARCH64;
     features |= 1ULL << ARM_FEATURE_GENERIC_TIMER;
 
+    if (el2_supported) {
+        features |= 1ULL << ARM_FEATURE_EL2;
+    }
+
     ahcf->features = features;
 
     return true;
@@ -1889,6 +1902,9 @@ int kvm_arch_init_vcpu(CPUState *cs)
         cpu->kvm_init_features[0] |= (1 << KVM_ARM_VCPU_PTRAUTH_ADDRESS |
                                       1 << KVM_ARM_VCPU_PTRAUTH_GENERIC);
     }
+    if (cpu->has_el2 && kvm_arm_el2_supported()) {
+        cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_HAS_EL2;
+    }
 
     /* Do KVM_ARM_VCPU_INIT ioctl */
     ret = kvm_arm_vcpu_init(cpu);
-- 
2.49.0

[PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

[Eric Auger]

From: Haibo Xu <haibo.xu@linaro.org>

Up to now virt support on guest has been only supported with TCG.
Now it becomes feasible to use it with KVM acceleration.

Also check only in-kernel GICv3 is used along with KVM EL2.

Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

---
v2 -> v3:
- check gic version/in-kernel implementation when kvm el2 is set (Peter)

v1 -> v2:
- fixed test ordering: virt && ((kvm && !kvm_el2) || hvf) [Richard]
- tweeked the commit title & message
---
 hw/arm/virt.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 675ddeec14..06a57380b8 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -795,6 +795,13 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem)
     default:
         g_assert_not_reached();
     }
+
+    if (kvm_enabled() && vms->virt &&
+        (revision != 3 || !kvm_irqchip_in_kernel())) {
+        error_report("KVM EL2 only is supported with in-kernel GICv3");
+        exit(1);
+    }
+
     vms->gic = qdev_new(gictype);
     qdev_prop_set_uint32(vms->gic, "revision", revision);
     qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus);
@@ -2208,7 +2215,8 @@ static void machvirt_init(MachineState *machine)
         exit(1);
     }
 
-    if (vms->virt && (kvm_enabled() || hvf_enabled())) {
+    if (vms->virt &&
+        ((kvm_enabled() && !kvm_arm_el2_supported()) || hvf_enabled())) {
         error_report("mach-virt: %s does not support providing "
                      "Virtualization extensions to the guest CPU",
                      current_accel_name());
-- 
2.49.0

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Marc Zyngier]

Hi Eric,

On Tue, 27 May 2025 07:24:32 +0100,
Eric Auger <eric.auger@redhat.com> wrote:
> 
> Now that ARM nested virt has landed in kvm/next, let's turn the series
> into a PATCH series. The linux header update was made against kvm/next.
> 
> For gaining virt functionality in KVM accelerated L1, The host needs to
> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
> with: -machine virt,virtualization=on.

Thanks for respinning this series.

Do you have any plan to support the non-VHE version of the NV support
(as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser
hypervisors (such as *cough* Xen *cough*), which completely rely on
HCR_EL2.E2H being 0?


Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Eric Auger]

Hi Marc,

On 5/27/25 9:39 AM, Marc Zyngier wrote:
> Hi Eric,
>
> On Tue, 27 May 2025 07:24:32 +0100,
> Eric Auger <eric.auger@redhat.com> wrote:
>> Now that ARM nested virt has landed in kvm/next, let's turn the series
>> into a PATCH series. The linux header update was made against kvm/next.
>>
>> For gaining virt functionality in KVM accelerated L1, The host needs to
>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
>> with: -machine virt,virtualization=on.
> Thanks for respinning this series.
>
> Do you have any plan to support the non-VHE version of the NV support
> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser
> hypervisors (such as *cough* Xen *cough*), which completely rely on
> HCR_EL2.E2H being 0?

yes I can definitively extend the series to support
KVM_ARM_VCPU_HAS_EL2_E2H0.

Eric


>
>
> Thanks,
>
> 	M.
>

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Miguel Luis]

Hi Eric,

> On 27 May 2025, at 06:24, Eric Auger <eric.auger@redhat.com> wrote:
> 
> Now that ARM nested virt has landed in kvm/next, let's turn the series
> into a PATCH series. The linux header update was made against kvm/next.
> 
> For gaining virt functionality in KVM accelerated L1, The host needs to
> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
> with: -machine virt,virtualization=on.

For the current kvmarm/next the guest also needs “kvm-arm.mode=nested” I believe.

> 
> This series can be found at:
> https://github.com/eauger/qemu/tree/v10.0.0-nv-v5
> 

Thanks for the respin.

Cheers
Miguel

> Original version from Miguel:
> [1] https://lore.kernel.org/all/20230227163718.62003-1-miguel.luis@oracle.com/
> version from Haibo:
> [2] https://lore.kernel.org/qemu-devel/cover.1617281290.git.haibo.xu@linaro.org/
> 
> History:
> v4 -> v5:
> - rebase on top of v10.0.0
> 
> v3 -> v4:
> - fix: only set maint_irq if vms->virt
> 
> v2 -> v3:
> - KVM EL2 only is set if virtualization option is set
> - fixes regression with virtualization=off
> - Add checks against unsupported GIC configs until the kernel does
> 
> Eric Auger (1):
>  linux-headers: Update against kvm/next
> 
> Haibo Xu (4):
>  hw/arm: Allow setting KVM vGIC maintenance IRQ
>  target/arm/kvm: Add helper to detect EL2 when using KVM
>  target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported
>  hw/arm/virt: Allow virt extensions with KVM
> 
> include/hw/intc/arm_gicv3_common.h          |  1 +
> include/standard-headers/linux/virtio_pci.h |  1 +
> linux-headers/asm-arm64/kvm.h               |  9 +++++----
> linux-headers/linux/kvm.h                   |  3 +++
> linux-headers/linux/vhost.h                 |  4 ++--
> target/arm/kvm_arm.h                        |  7 +++++++
> hw/arm/virt.c                               | 13 ++++++++++++-
> hw/intc/arm_gicv3_common.c                  |  1 +
> hw/intc/arm_gicv3_kvm.c                     | 21 +++++++++++++++++++++
> target/arm/kvm-stub.c                       |  5 +++++
> target/arm/kvm.c                            | 21 +++++++++++++++++++++
> 11 files changed, 79 insertions(+), 7 deletions(-)
> 
> -- 
> 2.49.0
> 

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Miguel Luis]

Hi Marc,

> On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote:
> 
> Hi Eric,
> 
> On Tue, 27 May 2025 07:24:32 +0100,
> Eric Auger <eric.auger@redhat.com> wrote:
>> 
>> Now that ARM nested virt has landed in kvm/next, let's turn the series
>> into a PATCH series. The linux header update was made against kvm/next.
>> 
>> For gaining virt functionality in KVM accelerated L1, The host needs to
>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
>> with: -machine virt,virtualization=on.
> 
> Thanks for respinning this series.
> 
> Do you have any plan to support the non-VHE version of the NV support
> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser
> hypervisors (such as *cough* Xen *cough*), which completely rely on
> HCR_EL2.E2H being 0?
> 

Something that pops up is early_kvm_mode_cfg trying to handle nested mode
while KVM_ARM_VCPU_HAS_EL2_E2H0 is set.

Miguel

> 
> Thanks,
> 
> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Marc Zyngier]

On Tue, 27 May 2025 12:33:23 +0100,
Miguel Luis <miguel.luis@oracle.com> wrote:
> 
> Hi Eric,
> 
> > On 27 May 2025, at 06:24, Eric Auger <eric.auger@redhat.com> wrote:
> > 
> > Now that ARM nested virt has landed in kvm/next, let's turn the series
> > into a PATCH series. The linux header update was made against kvm/next.
> > 
> > For gaining virt functionality in KVM accelerated L1, The host needs to
> > be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
> > with: -machine virt,virtualization=on.
> 
> For the current kvmarm/next the guest also needs
> “kvm-arm.mode=nested” I believe.

No, unless you want the guest itself to be NV-capable.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Marc Zyngier]

On Tue, 27 May 2025 12:40:35 +0100,
Miguel Luis <miguel.luis@oracle.com> wrote:
> 
> Hi Marc,
> 
> > On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > Hi Eric,
> > 
> > On Tue, 27 May 2025 07:24:32 +0100,
> > Eric Auger <eric.auger@redhat.com> wrote:
> >> 
> >> Now that ARM nested virt has landed in kvm/next, let's turn the series
> >> into a PATCH series. The linux header update was made against kvm/next.
> >> 
> >> For gaining virt functionality in KVM accelerated L1, The host needs to
> >> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
> >> with: -machine virt,virtualization=on.
> > 
> > Thanks for respinning this series.
> > 
> > Do you have any plan to support the non-VHE version of the NV support
> > (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser
> > hypervisors (such as *cough* Xen *cough*), which completely rely on
> > HCR_EL2.E2H being 0?
> > 
> 
> Something that pops up is early_kvm_mode_cfg trying to handle nested mode
> while KVM_ARM_VCPU_HAS_EL2_E2H0 is set.

Care to elaborate?

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Miguel Luis]

> On 27 May 2025, at 12:01, Marc Zyngier <maz@kernel.org> wrote:
> 
> On Tue, 27 May 2025 12:33:23 +0100,
> Miguel Luis <miguel.luis@oracle.com> wrote:
>> 
>> Hi Eric,
>> 
>>> On 27 May 2025, at 06:24, Eric Auger <eric.auger@redhat.com> wrote:
>>> 
>>> Now that ARM nested virt has landed in kvm/next, let's turn the series
>>> into a PATCH series. The linux header update was made against kvm/next.
>>> 
>>> For gaining virt functionality in KVM accelerated L1, The host needs to
>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
>>> with: -machine virt,virtualization=on.
>> 
>> For the current kvmarm/next the guest also needs
>> “kvm-arm.mode=nested” I believe.
> 
> No, unless you want the guest itself to be NV-capable.
> 

Correct, I got carried away with some mode combinations. Maybe we should depict
here more broadly how NV might be used with different mode combinations. I'll
think about this further ahead.

As far this series go I couldn't found any issue booting a L1 guest with
virtualization=on and a L2 guest with virtualization=off.

Miguel

> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Eric Auger]

Hi Miguel, Marc,

On 5/27/25 2:54 PM, Miguel Luis wrote:
>
>> On 27 May 2025, at 12:01, Marc Zyngier <maz@kernel.org> wrote:
>>
>> On Tue, 27 May 2025 12:33:23 +0100,
>> Miguel Luis <miguel.luis@oracle.com> wrote:
>>> Hi Eric,
>>>
>>>> On 27 May 2025, at 06:24, Eric Auger <eric.auger@redhat.com> wrote:
>>>>
>>>> Now that ARM nested virt has landed in kvm/next, let's turn the series
>>>> into a PATCH series. The linux header update was made against kvm/next.
>>>>
>>>> For gaining virt functionality in KVM accelerated L1, The host needs to
>>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
>>>> with: -machine virt,virtualization=on.
>>> For the current kvmarm/next the guest also needs
>>> “kvm-arm.mode=nested” I believe.
>> No, unless you want the guest itself to be NV-capable.
Effectively you don't need "kvm-arm.mode=nested" except if you want want
multiple level of nesting.
>>
> Correct, I got carried away with some mode combinations. Maybe we should depict
> here more broadly how NV might be used with different mode combinations. I'll
> think about this further ahead.
>
> As far this series go I couldn't found any issue booting a L1 guest with
> virtualization=on and a L2 guest with virtualization=off.

on my end I tested with various untouched L2 guests (debian, fed, rhel)
in 4kB/4kB/4KB page size mode (host, L1, L2). Those configs were successful.

with 64kB/64kB/64kB configs I am less lucky atm. One one machine I
cannot boot L1 with virtualization=on. On the other I can boot L1 but
cannot boot L2.

Trying my best to debug a little bit further with my setup. Anyway if
somebody else can try 64kB configs, it would help to confirm whether
there are pending issues. I don't think they are related to this qemu
integration series though.

Thanks

Eric
>
> Miguel
>
>> M.
>>
>> -- 
>> Without deviation from the norm, progress is not possible.
>

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Miguel Luis]

> On 27 May 2025, at 12:02, Marc Zyngier <maz@kernel.org> wrote:
> 
> On Tue, 27 May 2025 12:40:35 +0100,
> Miguel Luis <miguel.luis@oracle.com> wrote:
>> 
>> Hi Marc,
>> 
>>> On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote:
>>> 
>>> Hi Eric,
>>> 
>>> On Tue, 27 May 2025 07:24:32 +0100,
>>> Eric Auger <eric.auger@redhat.com> wrote:
>>>> 
>>>> Now that ARM nested virt has landed in kvm/next, let's turn the series
>>>> into a PATCH series. The linux header update was made against kvm/next.
>>>> 
>>>> For gaining virt functionality in KVM accelerated L1, The host needs to
>>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
>>>> with: -machine virt,virtualization=on.
>>> 
>>> Thanks for respinning this series.
>>> 
>>> Do you have any plan to support the non-VHE version of the NV support
>>> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser
>>> hypervisors (such as *cough* Xen *cough*), which completely rely on
>>> HCR_EL2.E2H being 0?
>>> 
>> 
>> Something that pops up is early_kvm_mode_cfg trying to handle nested mode
>> while KVM_ARM_VCPU_HAS_EL2_E2H0 is set.
> 
> Care to elaborate?
> 

Say host is booted in nested mode (kvm-arm.mode=nested) and host's KVM supports
both KVM_CAP_ARM_EL2 and KVM_CAP_ARM_E2H0.

A L1 guest boots setting both KVM_ARM_VCPU_HAS_EL2 and
KVM_ARM_VCPU_HAS_EL2_E2H0 and guest kernel's command line state
kvm-arm.mode=nested.

This splats the kernel from early_kvm_mode_cfg along a malformed early option
message.

This is not related to this qemu integration though, should you take it somewhere else,
please let me know?

Thanks
Miguel

> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Marc Zyngier]

On Tue, 27 May 2025 14:24:31 +0100,
Miguel Luis <miguel.luis@oracle.com> wrote:
> 
> 
> 
> > On 27 May 2025, at 12:02, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > On Tue, 27 May 2025 12:40:35 +0100,
> > Miguel Luis <miguel.luis@oracle.com> wrote:
> >> 
> >> Hi Marc,
> >> 
> >>> On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote:
> >>> 
> >>> Hi Eric,
> >>> 
> >>> On Tue, 27 May 2025 07:24:32 +0100,
> >>> Eric Auger <eric.auger@redhat.com> wrote:
> >>>> 
> >>>> Now that ARM nested virt has landed in kvm/next, let's turn the series
> >>>> into a PATCH series. The linux header update was made against kvm/next.
> >>>> 
> >>>> For gaining virt functionality in KVM accelerated L1, The host needs to
> >>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
> >>>> with: -machine virt,virtualization=on.
> >>> 
> >>> Thanks for respinning this series.
> >>> 
> >>> Do you have any plan to support the non-VHE version of the NV support
> >>> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser
> >>> hypervisors (such as *cough* Xen *cough*), which completely rely on
> >>> HCR_EL2.E2H being 0?
> >>> 
> >> 
> >> Something that pops up is early_kvm_mode_cfg trying to handle nested mode
> >> while KVM_ARM_VCPU_HAS_EL2_E2H0 is set.
> > 
> > Care to elaborate?
> > 
> 
> Say host is booted in nested mode (kvm-arm.mode=nested) and host's KVM supports
> both KVM_CAP_ARM_EL2 and KVM_CAP_ARM_E2H0.
> 
> A L1 guest boots setting both KVM_ARM_VCPU_HAS_EL2 and
> KVM_ARM_VCPU_HAS_EL2_E2H0 and guest kernel's command line state
> kvm-arm.mode=nested.
>
> This splats the kernel from early_kvm_mode_cfg along a malformed early option
> message.

BEBKAC. You are asking for nested on a (virtual) machine that doesn't
support it, and the kernel tells you so with a warning. Try the same
thing on a physical machine that doesn't have NV, and observe the
result.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Marc Zyngier]

On Tue, 27 May 2025 14:11:41 +0100,
Eric Auger <eric.auger@redhat.com> wrote:
> 
> Hi Miguel, Marc,
> 
> On 5/27/25 2:54 PM, Miguel Luis wrote:
> >
> >> On 27 May 2025, at 12:01, Marc Zyngier <maz@kernel.org> wrote:
> >>
> >> On Tue, 27 May 2025 12:33:23 +0100,
> >> Miguel Luis <miguel.luis@oracle.com> wrote:
> >>> Hi Eric,
> >>>
> >>>> On 27 May 2025, at 06:24, Eric Auger <eric.auger@redhat.com> wrote:
> >>>>
> >>>> Now that ARM nested virt has landed in kvm/next, let's turn the series
> >>>> into a PATCH series. The linux header update was made against kvm/next.
> >>>>
> >>>> For gaining virt functionality in KVM accelerated L1, The host needs to
> >>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
> >>>> with: -machine virt,virtualization=on.
> >>> For the current kvmarm/next the guest also needs
> >>> “kvm-arm.mode=nested” I believe.
> >> No, unless you want the guest itself to be NV-capable.
> Effectively you don't need "kvm-arm.mode=nested" except if you want want
> multiple level of nesting.

You need kvm-arm.mode=nested on the command-line of a kernel that acts
as a host exposing KVM_CAP_ARM_EL2. This is obviously a recursive
property, but that's for the user to decide if they want it or not.

> > Correct, I got carried away with some mode combinations. Maybe we should depict
> > here more broadly how NV might be used with different mode combinations. I'll
> > think about this further ahead.
> >
> > As far this series go I couldn't found any issue booting a L1 guest with
> > virtualization=on and a L2 guest with virtualization=off.
> 
> on my end I tested with various untouched L2 guests (debian, fed, rhel)
> in 4kB/4kB/4KB page size mode (host, L1, L2). Those configs were successful.

I'm doing 4/4/4 (x1e) and 16/16/4 (m2). I'm also doing 4/4/4/4/4, but
that's not very reliable yet,

> with 64kB/64kB/64kB configs I am less lucky atm. One one machine I
> cannot boot L1 with virtualization=on. On the other I can boot L1 but
> cannot boot L2.

I think x1e supports 64k pages, but I'd need to try. Not exactly the
highest priority on my list of things to debug.

> Trying my best to debug a little bit further with my setup. Anyway if
> somebody else can try 64kB configs, it would help to confirm whether
> there are pending issues. I don't think they are related to this qemu
> integration series though.

Yeah, definitely a kernel issue IMO.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Miguel Luis]

Hi Marc,

> On 27 May 2025, at 13:46, Marc Zyngier <maz@kernel.org> wrote:
> 
> On Tue, 27 May 2025 14:24:31 +0100,
> Miguel Luis <miguel.luis@oracle.com> wrote:
>> 
>> 
>> 
>>> On 27 May 2025, at 12:02, Marc Zyngier <maz@kernel.org> wrote:
>>> 
>>> On Tue, 27 May 2025 12:40:35 +0100,
>>> Miguel Luis <miguel.luis@oracle.com> wrote:
>>>> 
>>>> Hi Marc,
>>>> 
>>>>> On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote:
>>>>> 
>>>>> Hi Eric,
>>>>> 
>>>>> On Tue, 27 May 2025 07:24:32 +0100,
>>>>> Eric Auger <eric.auger@redhat.com> wrote:
>>>>>> 
>>>>>> Now that ARM nested virt has landed in kvm/next, let's turn the series
>>>>>> into a PATCH series. The linux header update was made against kvm/next.
>>>>>> 
>>>>>> For gaining virt functionality in KVM accelerated L1, The host needs to
>>>>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
>>>>>> with: -machine virt,virtualization=on.
>>>>> 
>>>>> Thanks for respinning this series.
>>>>> 
>>>>> Do you have any plan to support the non-VHE version of the NV support
>>>>> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser
>>>>> hypervisors (such as *cough* Xen *cough*), which completely rely on
>>>>> HCR_EL2.E2H being 0?
>>>>> 
>>>> 
>>>> Something that pops up is early_kvm_mode_cfg trying to handle nested mode
>>>> while KVM_ARM_VCPU_HAS_EL2_E2H0 is set.
>>> 
>>> Care to elaborate?
>>> 
>> 
>> Say host is booted in nested mode (kvm-arm.mode=nested) and host's KVM supports
>> both KVM_CAP_ARM_EL2 and KVM_CAP_ARM_E2H0.
>> 
>> A L1 guest boots setting both KVM_ARM_VCPU_HAS_EL2 and
>> KVM_ARM_VCPU_HAS_EL2_E2H0 and guest kernel's command line state
>> kvm-arm.mode=nested.
>> 
>> This splats the kernel from early_kvm_mode_cfg along a malformed early option
>> message.
> 
> BEBKAC. You are asking for nested on a (virtual) machine that doesn't
> support it, and the kernel tells you so with a warning. Try the same
> thing on a physical machine that doesn't have NV, and observe the
> result.
> 

Ack.

I find trying them a great way to improve resilience.
I’ve tried the scenarios below which have similar results on the guest:

1.
Host: kvm-arm.mode=nested

L1 Guest: kvm-arm.mode=nvhe setting both
KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0

Result on the guest: No early_kvm_mode_cfg splat, boot proceeds, ends up in a hard lockup splat.

2.
Host: kvm-arm.mode=nested

L1 Guest: kvm-arm.mode=nested setting both
KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0

Result on the guest: Splat at early_kvm_mode_cfg, boot proceeds, ends up in hard lockup splat.

Does this means there’s a default fallback mode in which nv gets on when 
kvm-arm.mode fed to the guest kernel cmdline differs from the expected?

Miguel

> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Marc Zyngier]

On Tue, 27 May 2025 16:55:32 +0100,
Miguel Luis <miguel.luis@oracle.com> wrote:
> 
> Hi Marc,
> 
> > On 27 May 2025, at 13:46, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > On Tue, 27 May 2025 14:24:31 +0100,
> > Miguel Luis <miguel.luis@oracle.com> wrote:
> >> 
> >> 
> >> 
> >>> On 27 May 2025, at 12:02, Marc Zyngier <maz@kernel.org> wrote:
> >>> 
> >>> On Tue, 27 May 2025 12:40:35 +0100,
> >>> Miguel Luis <miguel.luis@oracle.com> wrote:
> >>>> 
> >>>> Hi Marc,
> >>>> 
> >>>>> On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote:
> >>>>> 
> >>>>> Hi Eric,
> >>>>> 
> >>>>> On Tue, 27 May 2025 07:24:32 +0100,
> >>>>> Eric Auger <eric.auger@redhat.com> wrote:
> >>>>>> 
> >>>>>> Now that ARM nested virt has landed in kvm/next, let's turn the series
> >>>>>> into a PATCH series. The linux header update was made against kvm/next.
> >>>>>> 
> >>>>>> For gaining virt functionality in KVM accelerated L1, The host needs to
> >>>>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
> >>>>>> with: -machine virt,virtualization=on.
> >>>>> 
> >>>>> Thanks for respinning this series.
> >>>>> 
> >>>>> Do you have any plan to support the non-VHE version of the NV support
> >>>>> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser
> >>>>> hypervisors (such as *cough* Xen *cough*), which completely rely on
> >>>>> HCR_EL2.E2H being 0?
> >>>>> 
> >>>> 
> >>>> Something that pops up is early_kvm_mode_cfg trying to handle nested mode
> >>>> while KVM_ARM_VCPU_HAS_EL2_E2H0 is set.
> >>> 
> >>> Care to elaborate?
> >>> 
> >> 
> >> Say host is booted in nested mode (kvm-arm.mode=nested) and host's KVM supports
> >> both KVM_CAP_ARM_EL2 and KVM_CAP_ARM_E2H0.
> >> 
> >> A L1 guest boots setting both KVM_ARM_VCPU_HAS_EL2 and
> >> KVM_ARM_VCPU_HAS_EL2_E2H0 and guest kernel's command line state
> >> kvm-arm.mode=nested.
> >> 
> >> This splats the kernel from early_kvm_mode_cfg along a malformed early option
> >> message.
> > 
> > BEBKAC. You are asking for nested on a (virtual) machine that doesn't
> > support it, and the kernel tells you so with a warning. Try the same
> > thing on a physical machine that doesn't have NV, and observe the
> > result.
> > 
> 
> Ack.
> 
> I find trying them a great way to improve resilience.
> I’ve tried the scenarios below which have similar results on the guest:
> 
> 1.
> Host: kvm-arm.mode=nested
> 
> L1 Guest: kvm-arm.mode=nvhe setting both
> KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0
> 
> Result on the guest: No early_kvm_mode_cfg splat, boot proceeds, ends up in a hard lockup splat.

Setting kvm-arm.mode=nvhe when KVM_ARM_VCPU_HAS_EL2_E2H0 is set is a
tautology. The very definition of nVHE is that HCR_EL2.E2H=0.

>
> 2.
> Host: kvm-arm.mode=nested
> 
> L1 Guest: kvm-arm.mode=nested setting both
> KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0
> 
> Result on the guest: Splat at early_kvm_mode_cfg, boot proceeds, ends up in hard lockup splat.

I don't see any of these lockups with kvmtool. See this:

	https://pastebin.com/uyYzsBHc

for an example of a boot with both capabilities set and the nonsense
"nested" on the command-line (your #2).

> Does this means there’s a default fallback mode in which nv gets on when 
> kvm-arm.mode fed to the guest kernel cmdline differs from the expected?

I don't understand your question. We have two modes of operation:

- HAS_EL2 enables NV on the host, and additionally enables recursive
  NV. As a consequence, HCR_EL2.E2H is RES1. This is how NV will be
  supported long term.

- HAS_EL2_E2H0 restricts the above by not exposing NV to the guest,
  and enforcing HCR_EL2.E2H to be RES0. I expect this to gradually be
  removed from implementations, and eventually disappear.

As you can see, there is no "fallback mode". You pick the mode you
want based on the guest you want to run and the capabilities of the
hardware.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Miguel Luis]

> On 27 May 2025, at 16:52, Marc Zyngier <maz@kernel.org> wrote:
> 
> On Tue, 27 May 2025 16:55:32 +0100,
> Miguel Luis <miguel.luis@oracle.com> wrote:
>> 
>> Hi Marc,
>> 
>>> On 27 May 2025, at 13:46, Marc Zyngier <maz@kernel.org> wrote:
>>> 
>>> On Tue, 27 May 2025 14:24:31 +0100,
>>> Miguel Luis <miguel.luis@oracle.com> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On 27 May 2025, at 12:02, Marc Zyngier <maz@kernel.org> wrote:
>>>>> 
>>>>> On Tue, 27 May 2025 12:40:35 +0100,
>>>>> Miguel Luis <miguel.luis@oracle.com> wrote:
>>>>>> 
>>>>>> Hi Marc,
>>>>>> 
>>>>>>> On 27 May 2025, at 07:39, Marc Zyngier <maz@kernel.org> wrote:
>>>>>>> 
>>>>>>> Hi Eric,
>>>>>>> 
>>>>>>> On Tue, 27 May 2025 07:24:32 +0100,
>>>>>>> Eric Auger <eric.auger@redhat.com> wrote:
>>>>>>>> 
>>>>>>>> Now that ARM nested virt has landed in kvm/next, let's turn the series
>>>>>>>> into a PATCH series. The linux header update was made against kvm/next.
>>>>>>>> 
>>>>>>>> For gaining virt functionality in KVM accelerated L1, The host needs to
>>>>>>>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
>>>>>>>> with: -machine virt,virtualization=on.
>>>>>>> 
>>>>>>> Thanks for respinning this series.
>>>>>>> 
>>>>>>> Do you have any plan to support the non-VHE version of the NV support
>>>>>>> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser
>>>>>>> hypervisors (such as *cough* Xen *cough*), which completely rely on
>>>>>>> HCR_EL2.E2H being 0?
>>>>>>> 
>>>>>> 
>>>>>> Something that pops up is early_kvm_mode_cfg trying to handle nested mode
>>>>>> while KVM_ARM_VCPU_HAS_EL2_E2H0 is set.
>>>>> 
>>>>> Care to elaborate?
>>>>> 
>>>> 
>>>> Say host is booted in nested mode (kvm-arm.mode=nested) and host's KVM supports
>>>> both KVM_CAP_ARM_EL2 and KVM_CAP_ARM_E2H0.
>>>> 
>>>> A L1 guest boots setting both KVM_ARM_VCPU_HAS_EL2 and
>>>> KVM_ARM_VCPU_HAS_EL2_E2H0 and guest kernel's command line state
>>>> kvm-arm.mode=nested.
>>>> 
>>>> This splats the kernel from early_kvm_mode_cfg along a malformed early option
>>>> message.
>>> 
>>> BEBKAC. You are asking for nested on a (virtual) machine that doesn't
>>> support it, and the kernel tells you so with a warning. Try the same
>>> thing on a physical machine that doesn't have NV, and observe the
>>> result.
>>> 
>> 
>> Ack.
>> 
>> I find trying them a great way to improve resilience.
>> I’ve tried the scenarios below which have similar results on the guest:
>> 
>> 1.
>> Host: kvm-arm.mode=nested
>> 
>> L1 Guest: kvm-arm.mode=nvhe setting both
>> KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0
>> 
>> Result on the guest: No early_kvm_mode_cfg splat, boot proceeds, ends up in a hard lockup splat.
> 
> Setting kvm-arm.mode=nvhe when KVM_ARM_VCPU_HAS_EL2_E2H0 is set is a
> tautology. The very definition of nVHE is that HCR_EL2.E2H=0.
> 
>> 
>> 2.
>> Host: kvm-arm.mode=nested
>> 
>> L1 Guest: kvm-arm.mode=nested setting both
>> KVM_ARM_VCPU_HAS_EL2 and KVM_ARM_VCPU_HAS_EL2_E2H0
>> 
>> Result on the guest: Splat at early_kvm_mode_cfg, boot proceeds, ends up in hard lockup splat.
> 
> I don't see any of these lockups with kvmtool. See this:
> 
> https://pastebin.com/uyYzsBHc

Could you try bigger values for -c and check whether you can reproduce the issue?

> 
> for an example of a boot with both capabilities set and the nonsense
> "nested" on the command-line (your #2).
> 
>> Does this means there’s a default fallback mode in which nv gets on when 
>> kvm-arm.mode fed to the guest kernel cmdline differs from the expected?
> 
> I don't understand your question. We have two modes of operation:
> 
> - HAS_EL2 enables NV on the host, and additionally enables recursive
>  NV. As a consequence, HCR_EL2.E2H is RES1. This is how NV will be
>  supported long term.
> 
> - HAS_EL2_E2H0 restricts the above by not exposing NV to the guest,
>  and enforcing HCR_EL2.E2H to be RES0. I expect this to gradually be
>  removed from implementations, and eventually disappear.
> 
> As you can see, there is no "fallback mode". You pick the mode you
> want based on the guest you want to run and the capabilities of the
> hardware.
> 

I’m now suspecting the lockups might have a different reason than guest’s mode.

Thanks
Miguel

> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Marc Zyngier]

On Wed, 28 May 2025 00:52:40 +0100,
Miguel Luis <miguel.luis@oracle.com> wrote:
> 
> 
> 
> > On 27 May 2025, at 16:52, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > On Tue, 27 May 2025 16:55:32 +0100,
> > Miguel Luis <miguel.luis@oracle.com> wrote:
> >> 
 >> Result on the guest: Splat at early_kvm_mode_cfg, boot proceeds, ends up in hard lockup splat.
> > 
> > I don't see any of these lockups with kvmtool. See this:
> > 
> > https://pastebin.com/uyYzsBHc
> 
> Could you try bigger values for -c and check whether you can reproduce the issue?

How about 512, which is the maximum we support, and works for me with
an oversubscribing ratio of 42:1? More importantly, how about *you*
analyse the issue and report something that is actually actionable?

> I’m now suspecting the lockups might have a different reason than guest’s mode.

I'm shocked! :)

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 1425 bytes --]

Eric Auger <eric.auger@redhat.com> writes:

> From: Haibo Xu <haibo.xu@linaro.org>
>
> Up to now virt support on guest has been only supported with TCG.
> Now it becomes feasible to use it with KVM acceleration.
>
> Also check only in-kernel GICv3 is used along with KVM EL2.
>
> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

Hi!  From what I can tell, this will produce an error on hosts that
don't support nested virtualization when QEMU is invoked with -accel
kvm:tcg -machine virtualization=on, but I don't think that's the ideal
behaviour.  It would make more sense for it to fall back to the first
permitted accel option that does support running the machine as
configured, so if hardware nested virtualization is not supported, it
should fall back to TCG.

I maintain an OS development environment that includes scripts for
running images in QEMU, where running KVM on those images is a
requirement.  Currently, those scripts simply force TCG on aarch64.
With this change, to take advantage of KVM NV support, I'd have to try
to identify in the script whether NV would be supported.  QEMU would be
in a much better position to determine this and fall back to TCG if it's
unsupported, like how the -accel option with multiple values usually
works.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

[Eric Auger]

Hi Alyssa,
On 6/17/25 4:17 PM, Alyssa Ross wrote:
> Eric Auger <eric.auger@redhat.com> writes:
>
>> From: Haibo Xu <haibo.xu@linaro.org>
>>
>> Up to now virt support on guest has been only supported with TCG.
>> Now it becomes feasible to use it with KVM acceleration.
>>
>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>
>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> Hi!  From what I can tell, this will produce an error on hosts that
> don't support nested virtualization when QEMU is invoked with -accel
> kvm:tcg -machine virtualization=on, but I don't think that's the ideal
Thanks for reporting this issue. This is not an expected behavior. I
will respin hopefully this week so I will have give it a try.

What kind of error msg do you get?

Thanks

Eric
> behaviour.  It would make more sense for it to fall back to the first
> permitted accel option that does support running the machine as
> configured, so if hardware nested virtualization is not supported, it
> should fall back to TCG.
>
> I maintain an OS development environment that includes scripts for
> running images in QEMU, where running KVM on those images is a
> requirement.  Currently, those scripts simply force TCG on aarch64.
> With this change, to take advantage of KVM NV support, I'd have to try
> to identify in the script whether NV would be supported.  QEMU would be
> in a much better position to determine this and fall back to TCG if it's
> unsupported, like how the -accel option with multiple values usually
> works.

Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

[Eric Auger]

Hi Alyssa,

On 6/17/25 4:17 PM, Alyssa Ross wrote:
> Eric Auger <eric.auger@redhat.com> writes:
>
>> From: Haibo Xu <haibo.xu@linaro.org>
>>
>> Up to now virt support on guest has been only supported with TCG.
>> Now it becomes feasible to use it with KVM acceleration.
>>
>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>
>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> Hi!  From what I can tell, this will produce an error on hosts that
> don't support nested virtualization when QEMU is invoked with -accel
> kvm:tcg -machine virtualization=on, but I don't think that's the ideal
> behaviour.  It would make more sense for it to fall back to the first
> permitted accel option that does support running the machine as
> configured, so if hardware nested virtualization is not supported, it
> should fall back to TCG.

Hum I think I misundertood your point.
I guess you meant you get "qemu-system-aarch64: mach-virt: KVM does not
support providing Virtualization extensions to the guest CPU" which
sounds the usual behavior of qemu in case we attempt to run a feature on
a host that does not support it.


>
> I maintain an OS development environment that includes scripts for
> running images in QEMU, where running KVM on those images is a
> requirement.  Currently, those scripts simply force TCG on aarch64.
> With this change, to take advantage of KVM NV support, I'd have to try
> to identify in the script whether NV would be supported.  QEMU would be
> in a much better position to determine this and fall back to TCG if it's
> unsupported, like how the -accel option with multiple values usually
> works.
Switching to another accel mode is a policy that is not implemented by
qemu. There is no such situation where qemu would switch from kvm accel
mode to TCG. You need to implement this policy in your test env I think

Thanks

Eric

Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

[Miguel Luis]

Hi Alyssa,

> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
> 
> Eric Auger <eric.auger@redhat.com> writes:
> 
>> From: Haibo Xu <haibo.xu@linaro.org>
>> 
>> Up to now virt support on guest has been only supported with TCG.
>> Now it becomes feasible to use it with KVM acceleration.
>> 
>> Also check only in-kernel GICv3 is used along with KVM EL2.
>> 
>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> 
> Hi!  From what I can tell, this will produce an error on hosts that
> don't support nested virtualization when QEMU is invoked with -accel
> kvm:tcg

I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
kvm:tcg’ with cpu ‘max’ ?

But more importantly, is this what you’re referring to?

Although,

> -machine virtualization=on,

should work for both '-accel kvm’ and ‘-accel tcg’.

> but I don't think that's the ideal
> behaviour.  It would make more sense for it to fall back to the first
> permitted accel option that does support running the machine as
> configured, so if hardware nested virtualization is not supported, it
> should fall back to TCG.
> 
> I maintain an OS development environment that includes scripts for
> running images in QEMU, where running KVM on those images is a
> requirement.  Currently, those scripts simply force TCG on aarch64.
> With this change, to take advantage of KVM NV support, I'd have to try
> to identify in the script whether NV would be supported.  QEMU would be
> in a much better position to determine this and fall back to TCG if it's
> unsupported, like how the -accel option with multiple values usually
> works.

Thanks,
Miguel

Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

[Eric Auger]

On 6/17/25 5:23 PM, Miguel Luis wrote:
> Hi Alyssa,
>
>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>
>> Eric Auger <eric.auger@redhat.com> writes:
>>
>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>
>>> Up to now virt support on guest has been only supported with TCG.
>>> Now it becomes feasible to use it with KVM acceleration.
>>>
>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>
>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>> Hi!  From what I can tell, this will produce an error on hosts that
>> don't support nested virtualization when QEMU is invoked with -accel
>> kvm:tcg
> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
> kvm:tcg’ with cpu ‘max’ ?
Does it exist?
qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg

Alyssa, didn't you mean -accel kvm or --accel tcg
>
> But more importantly, is this what you’re referring to?
>
> Although,
>
>> -machine virtualization=on,
> should work for both '-accel kvm’ and ‘-accel tcg’.
>
>> but I don't think that's the ideal
>> behaviour.  It would make more sense for it to fall back to the first
>> permitted accel option that does support running the machine as
>> configured, so if hardware nested virtualization is not supported, it
>> should fall back to TCG.
>>
>> I maintain an OS development environment that includes scripts for
>> running images in QEMU, where running KVM on those images is a
>> requirement.  Currently, those scripts simply force TCG on aarch64.
>> With this change, to take advantage of KVM NV support, I'd have to try
>> to identify in the script whether NV would be supported.  QEMU would be
>> in a much better position to determine this and fall back to TCG if it's
>> unsupported, like how the -accel option with multiple values usually
>> works.
> Thanks,
> Miguel

Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

[Miguel Luis]

Hi Eric,

> On 17 Jun 2025, at 15:41, Eric Auger <eric.auger@redhat.com> wrote:
> 
> 
> 
> On 6/17/25 5:23 PM, Miguel Luis wrote:
>> Hi Alyssa,
>> 
>>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>> 
>>> Eric Auger <eric.auger@redhat.com> writes:
>>> 
>>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>> 
>>>> Up to now virt support on guest has been only supported with TCG.
>>>> Now it becomes feasible to use it with KVM acceleration.
>>>> 
>>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>> 
>>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>> Hi!  From what I can tell, this will produce an error on hosts that
>>> don't support nested virtualization when QEMU is invoked with -accel
>>> kvm:tcg
>> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
>> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
>> kvm:tcg’ with cpu ‘max’ ?
> Does it exist?
> qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg

Maybe Alyssa is referring to ‘-M virt,accel=kvm:tcg,virtualization=on,gic-version=3’ ?

The above didn’t triggered any error. Anyhow if the above does what Alyssa’s saying 
we would just be missing the check for || !tcg_enabled() in this patch, I believe.

Miguel

> 
> Alyssa, didn't you mean -accel kvm or --accel tcg
>> 
>> But more importantly, is this what you’re referring to?
>> 
>> Although,
>> 
>>> -machine virtualization=on,
>> should work for both '-accel kvm’ and ‘-accel tcg’.
>> 
>>> but I don't think that's the ideal
>>> behaviour.  It would make more sense for it to fall back to the first
>>> permitted accel option that does support running the machine as
>>> configured, so if hardware nested virtualization is not supported, it
>>> should fall back to TCG.
>>> 
>>> I maintain an OS development environment that includes scripts for
>>> running images in QEMU, where running KVM on those images is a
>>> requirement.  Currently, those scripts simply force TCG on aarch64.
>>> With this change, to take advantage of KVM NV support, I'd have to try
>>> to identify in the script whether NV would be supported.  QEMU would be
>>> in a much better position to determine this and fall back to TCG if it's
>>> unsupported, like how the -accel option with multiple values usually
>>> works.
>> Thanks,
>> Miguel

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Eric Auger]

Hi Marc, Miguel,

On 5/27/25 9:39 AM, Marc Zyngier wrote:
> Hi Eric,
>
> On Tue, 27 May 2025 07:24:32 +0100,
> Eric Auger <eric.auger@redhat.com> wrote:
>> Now that ARM nested virt has landed in kvm/next, let's turn the series
>> into a PATCH series. The linux header update was made against kvm/next.
>>
>> For gaining virt functionality in KVM accelerated L1, The host needs to
>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
>> with: -machine virt,virtualization=on.
> Thanks for respinning this series.
>
> Do you have any plan to support the non-VHE version of the NV support
> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser
> hypervisors (such as *cough* Xen *cough*), which completely rely on
> HCR_EL2.E2H being 0?
While respinning I'd rather not support KVM_ARM_VCPU_HAS_EL2_E2H0 in
next respin. The code is mostly reviewed and I would rather add support
for this in a separate series. It will most probably rely on a change of
the virtualization property type to support a "restricted" enum value.

Thanks

Eric
>
>
> Thanks,
>
> 	M.
>

Re: [PATCH v5 0/5] ARM Nested Virt Support

[Miguel Luis]

[-- Attachment #1: Type: text/plain, Size: 1834 bytes --]



________________________________
De: Eric Auger <eric.auger@redhat.com>
Enviado: quinta-feira, junho 19, 2025 8:19:47 da manhã
Para: Marc Zyngier <maz@kernel.org>
Cc: eric.auger.pro@gmail.com <eric.auger.pro@gmail.com>; qemu-devel@nongnu.org <qemu-devel@nongnu.org>; qemu-arm@nongnu.org <qemu-arm@nongnu.org>; Miguel Luis <miguel.luis@oracle.com>; peter.maydell@linaro.org <peter.maydell@linaro.org>; richard.henderson@linaro.org <richard.henderson@linaro.org>; gkulkarni@amperecomputing.com <gkulkarni@amperecomputing.com>; gankulkarni@os.amperecomputing.com <gankulkarni@os.amperecomputing.com>
Assunto: Re: [PATCH v5 0/5] ARM Nested Virt Support

Hi Marc, Miguel,

On 5/27/25 9:39 AM, Marc Zyngier wrote:
> Hi Eric,
>
> On Tue, 27 May 2025 07:24:32 +0100,
> Eric Auger <eric.auger@redhat.com> wrote:
>> Now that ARM nested virt has landed in kvm/next, let's turn the series
>> into a PATCH series. The linux header update was made against kvm/next.
>>
>> For gaining virt functionality in KVM accelerated L1, The host needs to
>> be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked
>> with: -machine virt,virtualization=on.
> Thanks for respinning this series.
>
> Do you have any plan to support the non-VHE version of the NV support
> (as advertised by KVM_CAP_ARM_EL2_E2H0)? It would allow running lesser
> hypervisors (such as *cough* Xen *cough*), which completely rely on
> HCR_EL2.E2H being 0?
While respinning I'd rather not support KVM_ARM_VCPU_HAS_EL2_E2H0 in
next respin. The code is mostly reviewed and I would rather add support
for this in a separate series. It will most probably rely on a change of
the virtualization property type to support a "restricted" enum value.

Ack!
Thank you.

Miguel


Thanks

Eric
>
>
> Thanks,
>
>        M.
>



[-- Attachment #2: Type: text/html, Size: 3287 bytes --]

Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

[Eric Auger]

Hi Miguel,

On 6/17/25 5:50 PM, Miguel Luis wrote:
> Hi Eric,
>
>> On 17 Jun 2025, at 15:41, Eric Auger <eric.auger@redhat.com> wrote:
>>
>>
>>
>> On 6/17/25 5:23 PM, Miguel Luis wrote:
>>> Hi Alyssa,
>>>
>>>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>>>
>>>> Eric Auger <eric.auger@redhat.com> writes:
>>>>
>>>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>>>
>>>>> Up to now virt support on guest has been only supported with TCG.
>>>>> Now it becomes feasible to use it with KVM acceleration.
>>>>>
>>>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>>>
>>>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>>> Hi!  From what I can tell, this will produce an error on hosts that
>>>> don't support nested virtualization when QEMU is invoked with -accel
>>>> kvm:tcg
>>> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
>>> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
>>> kvm:tcg’ with cpu ‘max’ ?
>> Does it exist?
>> qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg
> Maybe Alyssa is referring to ‘-M virt,accel=kvm:tcg,virtualization=on,gic-version=3’ ?
>
> The above didn’t triggered any error. Anyhow if the above does what Alyssa’s saying 
> we would just be missing the check for || !tcg_enabled() in this patch, I believe.
You meant you would report an error only if !tcg_enabled()?

In practice tcg_enabled() does not seem to be set when trying kvm first.

Refering to the ancestor of this feature - which I did not know I
acknowledge-,
https://patchwork.ozlabs.org/project/qemu-devel/patch/1275954730-8196-20-git-send-email-aliguori@us.ibm.com/
it is said:

kvm|tcg tries kvm and fallbacks on tcg if kvm support isn't available

It does not state that each option is evaluated with kvm_enabled() and in case it fails enabling one because the host does not support it, it falls back to TCG.

I checked for instance with already integrated MTE option and if the host does not support it, it does not offer the capability to fall back to TCG.

Adding Peter in TO to get his feeling on this.

Thanks

Eric

Besides

>
> Miguel
>
>> Alyssa, didn't you mean -accel kvm or --accel tcg
>>> But more importantly, is this what you’re referring to?
>>>
>>> Although,
>>>
>>>> -machine virtualization=on,
>>> should work for both '-accel kvm’ and ‘-accel tcg’.
>>>
>>>> but I don't think that's the ideal
>>>> behaviour.  It would make more sense for it to fall back to the first
>>>> permitted accel option that does support running the machine as
>>>> configured, so if hardware nested virtualization is not supported, it
>>>> should fall back to TCG.
>>>>
>>>> I maintain an OS development environment that includes scripts for
>>>> running images in QEMU, where running KVM on those images is a
>>>> requirement.  Currently, those scripts simply force TCG on aarch64.
>>>> With this change, to take advantage of KVM NV support, I'd have to try
>>>> to identify in the script whether NV would be supported.  QEMU would be
>>>> in a much better position to determine this and fall back to TCG if it's
>>>> unsupported, like how the -accel option with multiple values usually
>>>> works.
>>> Thanks,
>>> Miguel
>

Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

[Eric Auger]

Hi,

On 6/17/25 5:50 PM, Miguel Luis wrote:
> Hi Eric,
>
>> On 17 Jun 2025, at 15:41, Eric Auger <eric.auger@redhat.com> wrote:
>>
>>
>>
>> On 6/17/25 5:23 PM, Miguel Luis wrote:
>>> Hi Alyssa,
>>>
>>>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>>>
>>>> Eric Auger <eric.auger@redhat.com> writes:
>>>>
>>>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>>>
>>>>> Up to now virt support on guest has been only supported with TCG.
>>>>> Now it becomes feasible to use it with KVM acceleration.
>>>>>
>>>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>>>
>>>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>>> Hi!  From what I can tell, this will produce an error on hosts that
>>>> don't support nested virtualization when QEMU is invoked with -accel
>>>> kvm:tcg
>>> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
>>> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
>>> kvm:tcg’ with cpu ‘max’ ?
>> Does it exist?
>> qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg
> Maybe Alyssa is referring to ‘-M virt,accel=kvm:tcg,virtualization=on,gic-version=3’ ?
>
> The above didn’t triggered any error. Anyhow if the above does what Alyssa’s saying 
> we would just be missing the check for || !tcg_enabled() in this patch, I believe.

After discussion with Paolo, the lack of the CAP should be detected
earlier in kvm_init/kvm_arch_init to allow the fallback to TCG.
in target/arm/kvm.c kvm_arch_init() some generic caps are checked but
none of them are related to machine settings and this code is virt arm
machine agnostic.

I checked and adding

    if (object_dynamic_cast(OBJECT(ms), TYPE_VIRT_MACHINE)) {
        VirtMachineState *vms = VIRT_MACHINE(ms);

        if (vms->virt && !kvm_arm_el2_supported()) {
            error_report("KVM does not support nested virtualization");
            ret = -EINVAL;
        }
    }

at the end of the function would do the job. But as I said previously
this is not done for other virt arm machine options that are accel
specific or require special KVM caps (secure, mte for instance) so it
would be a change in the approach.

As pointed out before fallback spirit was rather: "KVM isn't enabled"
than for "KVM doesn't support a specific feature".

Thanks

Eric




it's purely the first accelerator that says it can work
1:03
<https://redhat-internal.slack.com/archives/C04KFKV2SE9/p1750331005242709>
pbonzini
Where "can work" is only based on the host.


>
> Miguel
>
>> Alyssa, didn't you mean -accel kvm or --accel tcg
>>> But more importantly, is this what you’re referring to?
>>>
>>> Although,
>>>
>>>> -machine virtualization=on,
>>> should work for both '-accel kvm’ and ‘-accel tcg’.
>>>
>>>> but I don't think that's the ideal
>>>> behaviour.  It would make more sense for it to fall back to the first
>>>> permitted accel option that does support running the machine as
>>>> configured, so if hardware nested virtualization is not supported, it
>>>> should fall back to TCG.
>>>>
>>>> I maintain an OS development environment that includes scripts for
>>>> running images in QEMU, where running KVM on those images is a
>>>> requirement.  Currently, those scripts simply force TCG on aarch64.
>>>> With this change, to take advantage of KVM NV support, I'd have to try
>>>> to identify in the script whether NV would be supported.  QEMU would be
>>>> in a much better position to determine this and fall back to TCG if it's
>>>> unsupported, like how the -accel option with multiple values usually
>>>> works.
>>> Thanks,
>>> Miguel
>

Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 2830 bytes --]

Eric Auger <eric.auger@redhat.com> writes:

> Hi,
>
> On 6/17/25 5:50 PM, Miguel Luis wrote:
>> Hi Eric,
>>
>>> On 17 Jun 2025, at 15:41, Eric Auger <eric.auger@redhat.com> wrote:
>>>
>>>
>>>
>>> On 6/17/25 5:23 PM, Miguel Luis wrote:
>>>> Hi Alyssa,
>>>>
>>>>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>>>>
>>>>> Eric Auger <eric.auger@redhat.com> writes:
>>>>>
>>>>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>>>>
>>>>>> Up to now virt support on guest has been only supported with TCG.
>>>>>> Now it becomes feasible to use it with KVM acceleration.
>>>>>>
>>>>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>>>>
>>>>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>>>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>>>> Hi!  From what I can tell, this will produce an error on hosts that
>>>>> don't support nested virtualization when QEMU is invoked with -accel
>>>>> kvm:tcg
>>>> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
>>>> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
>>>> kvm:tcg’ with cpu ‘max’ ?
>>> Does it exist?
>>> qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg
>> Maybe Alyssa is referring to ‘-M virt,accel=kvm:tcg,virtualization=on,gic-version=3’ ?
>>
>> The above didn’t triggered any error. Anyhow if the above does what Alyssa’s saying 
>> we would just be missing the check for || !tcg_enabled() in this patch, I believe.
>
> After discussion with Paolo, the lack of the CAP should be detected
> earlier in kvm_init/kvm_arch_init to allow the fallback to TCG.
> in target/arm/kvm.c kvm_arch_init() some generic caps are checked but
> none of them are related to machine settings and this code is virt arm
> machine agnostic.
>
> I checked and adding
>
>     if (object_dynamic_cast(OBJECT(ms), TYPE_VIRT_MACHINE)) {
>         VirtMachineState *vms = VIRT_MACHINE(ms);
>
>         if (vms->virt && !kvm_arm_el2_supported()) {
>             error_report("KVM does not support nested virtualization");
>             ret = -EINVAL;
>         }
>     }
>
> at the end of the function would do the job. But as I said previously
> this is not done for other virt arm machine options that are accel
> specific or require special KVM caps (secure, mte for instance) so it
> would be a change in the approach.

Right — if it's not done for other options it doesn't need to be done
here either; I wasn't aware of that.  It would be nice if QEMU worked
differently, but that's certainly not in scope of this series.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [PATCH v5 5/5] hw/arm/virt: Allow virt extensions with KVM

[Miguel Luis]

Hi Eric,

> On 19 Jun 2025, at 13:29, Eric Auger <eric.auger@redhat.com> wrote:
> 
> Hi,
> 
> On 6/17/25 5:50 PM, Miguel Luis wrote:
>> Hi Eric,
>> 
>>> On 17 Jun 2025, at 15:41, Eric Auger <eric.auger@redhat.com> wrote:
>>> 
>>> 
>>> 
>>> On 6/17/25 5:23 PM, Miguel Luis wrote:
>>>> Hi Alyssa,
>>>> 
>>>>> On 17 Jun 2025, at 14:17, Alyssa Ross <hi@alyssa.is> wrote:
>>>>> 
>>>>> Eric Auger <eric.auger@redhat.com> writes:
>>>>> 
>>>>>> From: Haibo Xu <haibo.xu@linaro.org>
>>>>>> 
>>>>>> Up to now virt support on guest has been only supported with TCG.
>>>>>> Now it becomes feasible to use it with KVM acceleration.
>>>>>> 
>>>>>> Also check only in-kernel GICv3 is used along with KVM EL2.
>>>>>> 
>>>>>> Signed-off-by: Haibo Xu <haibo.xu@linaro.org>
>>>>>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
>>>>> Hi!  From what I can tell, this will produce an error on hosts that
>>>>> don't support nested virtualization when QEMU is invoked with -accel
>>>>> kvm:tcg
>>>> I didn’t know '-acell kvm:tcg’ could be used as a fallback mechanism between
>>>> acceleration modes. May I ask whether do you manage the ‘-cpu’ type for ‘-accel
>>>> kvm:tcg’ with cpu ‘max’ ?
>>> Does it exist?
>>> qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg
>> Maybe Alyssa is referring to ‘-M virt,accel=kvm:tcg,virtualization=on,gic-version=3’ ?
>> 
>> The above didn’t triggered any error. Anyhow if the above does what Alyssa’s saying 
>> we would just be missing the check for || !tcg_enabled() in this patch, I believe.
> 
> After discussion with Paolo, the lack of the CAP should be detected
> earlier in kvm_init/kvm_arch_init to allow the fallback to TCG.
> in target/arm/kvm.c kvm_arch_init() some generic caps are checked but
> none of them are related to machine settings and this code is virt arm
> machine agnostic.
> 
> I checked and adding
> 
>     if (object_dynamic_cast(OBJECT(ms), TYPE_VIRT_MACHINE)) {
>         VirtMachineState *vms = VIRT_MACHINE(ms);
> 
>         if (vms->virt && !kvm_arm_el2_supported()) {
>             error_report("KVM does not support nested virtualization");
>             ret = -EINVAL;
>         }
>     }
> 
> at the end of the function would do the job. But as I said previously
> this is not done for other virt arm machine options that are accel
> specific or require special KVM caps (secure, mte for instance) so it
> would be a change in the approach.
> 

Got it.

> As pointed out before fallback spirit was rather: "KVM isn't enabled"
> than for "KVM doesn't support a specific feature".
> 

Thank you for getting to the bottom of it.

Miguel

> Thanks
> 
> Eric
> 
> 
> 
> 
> it's purely the first accelerator that says it can work
> 1:03
> <https://redhat-internal.slack.com/archives/C04KFKV2SE9/p1750331005242709>
> pbonzini
> Where "can work" is only based on the host.
> 
> 
>> 
>> Miguel
>> 
>>> Alyssa, didn't you mean -accel kvm or --accel tcg
>>>> But more importantly, is this what you’re referring to?
>>>> 
>>>> Although,
>>>> 
>>>>> -machine virtualization=on,
>>>> should work for both '-accel kvm’ and ‘-accel tcg’.
>>>> 
>>>>> but I don't think that's the ideal
>>>>> behaviour.  It would make more sense for it to fall back to the first
>>>>> permitted accel option that does support running the machine as
>>>>> configured, so if hardware nested virtualization is not supported, it
>>>>> should fall back to TCG.
>>>>> 
>>>>> I maintain an OS development environment that includes scripts for
>>>>> running images in QEMU, where running KVM on those images is a
>>>>> requirement.  Currently, those scripts simply force TCG on aarch64.
>>>>> With this change, to take advantage of KVM NV support, I'd have to try
>>>>> to identify in the script whether NV would be supported.  QEMU would be
>>>>> in a much better position to determine this and fall back to TCG if it's
>>>>> unsupported, like how the -accel option with multiple values usually
>>>>> works.
>>>> Thanks,
>>>> Miguel