[Qemu-devel] [PATCH RFC v2 0/2] Enable cp15 timers with user space gic & kvm

[Qemu-devel] [PATCH RFC v2 0/2] Enable cp15 timers with user space gic & kvm

[Alexander Graf]

When running with KVM enabled, you can choose between emulating the
gic in kernel or user space. If the kernel supports in-kernel virtualization
of the interrupt controller, it will default to that. If not, if will
default to user space emulation.

Unfortunately when running in user mode gic emulation, we miss out on
timer events which are only available from kernel space.

This patch set leverages the new kernel/user space pending line synchronization
for those timer events.

rfc1 -> rfc2:

  - use local variable for ARM_CPU
  - remove bear trap
  - move timer warning to gic device

Alexander Graf (2):
  linux-headers: update
  ARM: KVM: Enable in-kernel timers with user space gic

 hw/intc/arm_gic.c             |  7 +++++++
 include/sysemu/kvm.h          | 11 +++++++++++
 kvm-all.c                     |  5 +++++
 kvm-stub.c                    |  5 +++++
 linux-headers/asm-arm/kvm.h   |  1 +
 linux-headers/asm-arm64/kvm.h |  1 +
 linux-headers/linux/kvm.h     |  6 ++++++
 target-arm/cpu.h              |  3 +++
 target-arm/kvm.c              | 20 ++++++++++++++++++++
 9 files changed, 59 insertions(+)

-- 
1.8.5.6

[Qemu-devel] [PATCH RFC v2 1/2] linux-headers: update

[Alexander Graf]

This patch updates the Linux headers to include the in-progress user
space ARM timer patches. It is explicitly RFC, as the patches are not
merged yet.
---
 linux-headers/asm-arm/kvm.h   | 1 +
 linux-headers/asm-arm64/kvm.h | 1 +
 linux-headers/linux/kvm.h     | 6 ++++++
 3 files changed, 8 insertions(+)

diff --git a/linux-headers/asm-arm/kvm.h b/linux-headers/asm-arm/kvm.h
index 541268c..5d58ec2 100644
--- a/linux-headers/asm-arm/kvm.h
+++ b/linux-headers/asm-arm/kvm.h
@@ -105,6 +105,7 @@ struct kvm_debug_exit_arch {
 };
 
 struct kvm_sync_regs {
+	__u8 timer_irq_level;
 };
 
 struct kvm_arch_memory_slot {
diff --git a/linux-headers/asm-arm64/kvm.h b/linux-headers/asm-arm64/kvm.h
index fd5a276..0e1cbd1 100644
--- a/linux-headers/asm-arm64/kvm.h
+++ b/linux-headers/asm-arm64/kvm.h
@@ -143,6 +143,7 @@ struct kvm_debug_exit_arch {
 #define KVM_GUESTDBG_USE_HW		(1 << 17)
 
 struct kvm_sync_regs {
+	__u8 timer_irq_level;
 };
 
 struct kvm_arch_memory_slot {
diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 4806e06..737113c 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -870,6 +870,7 @@ struct kvm_ppc_smmu_info {
 #define KVM_CAP_S390_USER_INSTR0 130
 #define KVM_CAP_MSI_DEVID 131
 #define KVM_CAP_PPC_HTM 132
+#define KVM_CAP_ARM_TIMER 133
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -1327,4 +1328,9 @@ struct kvm_assigned_msix_entry {
 #define KVM_X2APIC_API_USE_32BIT_IDS            (1ULL << 0)
 #define KVM_X2APIC_API_DISABLE_BROADCAST_QUIRK  (1ULL << 1)
 
+/* Available with KVM_CAP_ARM_TIMER */
+
+/* Bits for run->arm_timer.timesource */
+#define KVM_ARM_TIMER_VTIMER           (1 << 0)
+
 #endif /* __LINUX_KVM_H */
-- 
1.8.5.6

[Qemu-devel] [PATCH RFC v2 2/2] ARM: KVM: Enable in-kernel timers with user space gic

[Alexander Graf]

When running with KVM enabled, you can choose between emulating the
gic in kernel or user space. If the kernel supports in-kernel virtualization
of the interrupt controller, it will default to that. If not, if will
default to user space emulation.

Unfortunately when running in user mode gic emulation, we miss out on
timer events which are only available from kernel space. This patch leverages
the new kernel/user space pending line synchronization for those timer events.

Signed-off-by: Alexander Graf <agraf@suse.de>

---

rfc1 -> rfc2:

  - use local variable for ARM_CPU
  - remove bear trap
  - move timer warning to gic device
---
 hw/intc/arm_gic.c    |  7 +++++++
 include/sysemu/kvm.h | 11 +++++++++++
 kvm-all.c            |  5 +++++
 kvm-stub.c           |  5 +++++
 target-arm/cpu.h     |  3 +++
 target-arm/kvm.c     | 20 ++++++++++++++++++++
 6 files changed, 51 insertions(+)

diff --git a/hw/intc/arm_gic.c b/hw/intc/arm_gic.c
index 521aac3..1f3aacf 100644
--- a/hw/intc/arm_gic.c
+++ b/hw/intc/arm_gic.c
@@ -25,6 +25,7 @@
 #include "qom/cpu.h"
 #include "qemu/log.h"
 #include "trace.h"
+#include "sysemu/kvm.h"
 
 //#define DEBUG_GIC
 
@@ -1428,6 +1429,12 @@ static void arm_gic_realize(DeviceState *dev, Error **errp)
         return;
     }
 
+    if (kvm_enabled() && !kvm_arm_supports_timer()) {
+            error_setg(errp, "KVM with user space irqchip only works when the "
+                             "host kernel supports KVM_CAP_ARM_TIMER");
+            return;
+    }
+
     /* This creates distributor and main CPU interface (s->cpuiomem[0]) */
     gic_init_irqs_and_mmio(s, gic_set_irq, gic_ops);
 
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index df67cc0..9715fee 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -227,6 +227,17 @@ int kvm_init_vcpu(CPUState *cpu);
 int kvm_cpu_exec(CPUState *cpu);
 int kvm_destroy_vcpu(CPUState *cpu);
 
+/**
+ * kvm_arm_supports_timer
+ *
+ * Not all KVM implementations support notifications for the CP15 timers to
+ * user space. This function indicates whether the current KVM implementation
+ * does support them.
+ *
+ * Returns: true if KVM supports using ARM core timers from user space
+ */
+bool kvm_arm_supports_timer(void);
+
 #ifdef NEED_CPU_H
 #include "cpu.h"
 
diff --git a/kvm-all.c b/kvm-all.c
index 330219e..8d4696c 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -2194,6 +2194,11 @@ int kvm_has_intx_set_mask(void)
     return kvm_state->intx_set_mask;
 }
 
+bool kvm_arm_supports_timer(void)
+{
+    return kvm_check_extension(kvm_state, KVM_CAP_ARM_TIMER);
+}
+
 #ifdef KVM_CAP_SET_GUEST_DEBUG
 struct kvm_sw_breakpoint *kvm_find_sw_breakpoint(CPUState *cpu,
                                                  target_ulong pc)
diff --git a/kvm-stub.c b/kvm-stub.c
index b1b6b96..a4d408b 100644
--- a/kvm-stub.c
+++ b/kvm-stub.c
@@ -157,4 +157,9 @@ bool kvm_has_free_slot(MachineState *ms)
 {
     return false;
 }
+
+bool kvm_arm_supports_timer(void)
+{
+    return false;
+}
 #endif
diff --git a/target-arm/cpu.h b/target-arm/cpu.h
index ca5c849..2c379a3 100644
--- a/target-arm/cpu.h
+++ b/target-arm/cpu.h
@@ -659,6 +659,9 @@ struct ARMCPU {
 
     ARMELChangeHook *el_change_hook;
     void *el_change_hook_opaque;
+
+    /* Used to synchronize KVM and QEMU timer levels */
+    uint8_t timer_irq_level;
 };
 
 static inline ARMCPU *arm_env_get_cpu(CPUARMState *env)
diff --git a/target-arm/kvm.c b/target-arm/kvm.c
index c00b94e..c5f0d37 100644
--- a/target-arm/kvm.c
+++ b/target-arm/kvm.c
@@ -527,6 +527,26 @@ void kvm_arch_pre_run(CPUState *cs, struct kvm_run *run)
 
 MemTxAttrs kvm_arch_post_run(CPUState *cs, struct kvm_run *run)
 {
+    ARMCPU *cpu;
+    bool vtimer_high;
+
+    if (kvm_irqchip_in_kernel()) {
+        /*
+         * We only need to sync timer states with user-space interrupt
+         * controllers, so return early and save cycles if we don't.
+         */
+        return MEMTXATTRS_UNSPECIFIED;
+    }
+
+    cpu = ARM_CPU(cs);
+
+    /* Synchronize our internal vtimer irq line with the kvm one */
+    if (run->s.regs.timer_irq_level != cpu->timer_irq_level) {
+        vtimer_high = run->s.regs.timer_irq_level & KVM_ARM_TIMER_VTIMER;
+        qemu_set_irq(cpu->gt_timer_outputs[GTIMER_VIRT], vtimer_high ? 1 : 0);
+        cpu->timer_irq_level = run->s.regs.timer_irq_level;
+    }
+
     return MEMTXATTRS_UNSPECIFIED;
 }
 
-- 
1.8.5.6

Re: [Qemu-devel] [PATCH RFC v2 2/2] ARM: KVM: Enable in-kernel timers with user space gic

[Peter Maydell]

On 14 November 2016 at 14:32, Alexander Graf <agraf@suse.de> wrote:
> When running with KVM enabled, you can choose between emulating the
> gic in kernel or user space. If the kernel supports in-kernel virtualization
> of the interrupt controller, it will default to that. If not, if will
> default to user space emulation.
>
> Unfortunately when running in user mode gic emulation, we miss out on
> timer events which are only available from kernel space. This patch leverages
> the new kernel/user space pending line synchronization for those timer events.
>
> Signed-off-by: Alexander Graf <agraf@suse.de>

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>

thanks
-- PMM

Re: [Qemu-devel] [PATCH RFC v2 2/2] ARM: KVM: Enable in-kernel timers with user space gic

[Andrew Jones]

On Tue, Dec 13, 2016 at 01:20:50PM +0000, Peter Maydell wrote:
> On 14 November 2016 at 14:32, Alexander Graf <agraf@suse.de> wrote:
> > When running with KVM enabled, you can choose between emulating the
> > gic in kernel or user space. If the kernel supports in-kernel virtualization
> > of the interrupt controller, it will default to that. If not, if will
> > default to user space emulation.
> >
> > Unfortunately when running in user mode gic emulation, we miss out on
> > timer events which are only available from kernel space. This patch leverages
> > the new kernel/user space pending line synchronization for those timer events.
> >
> > Signed-off-by: Alexander Graf <agraf@suse.de>
> 
> Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
>

Hi everyone,

I probably missed a refresh of this patch, but as I didn't see anything,
I picked this one up today in order to test the KVM support recently
merged. Tweaking this patch a bit to fit the new ABI allowed me to
instantiate a KVM guest without the in-kernel irqchip (tested on a
mustang). So, FWIW, this is report of a successful test. Is there a
refreshed version of this patch someone can point me to, which I should
test instead?

Thanks,
drew

Re: [Qemu-devel] [PATCH RFC v2 2/2] ARM: KVM: Enable in-kernel timers with user space gic

[Alexander Graf]

On 06/26/2017 05:03 PM, Andrew Jones wrote:
> On Tue, Dec 13, 2016 at 01:20:50PM +0000, Peter Maydell wrote:
>> On 14 November 2016 at 14:32, Alexander Graf <agraf@suse.de> wrote:
>>> When running with KVM enabled, you can choose between emulating the
>>> gic in kernel or user space. If the kernel supports in-kernel virtualization
>>> of the interrupt controller, it will default to that. If not, if will
>>> default to user space emulation.
>>>
>>> Unfortunately when running in user mode gic emulation, we miss out on
>>> timer events which are only available from kernel space. This patch leverages
>>> the new kernel/user space pending line synchronization for those timer events.
>>>
>>> Signed-off-by: Alexander Graf <agraf@suse.de>
>> Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
>>
> Hi everyone,
>
> I probably missed a refresh of this patch, but as I didn't see anything,
> I picked this one up today in order to test the KVM support recently
> merged. Tweaking this patch a bit to fit the new ABI allowed me to
> instantiate a KVM guest without the in-kernel irqchip (tested on a
> mustang). So, FWIW, this is report of a successful test. Is there a
> refreshed version of this patch someone can point me to, which I should
> test instead?

Sorry, this did fall the cracks way too many times now. I've sent a 
respin that hopefully is slightly more future proof than this RFC :)

If your tests passed with this patch, please extend them to also cover 
SMP support, as that was broken with this RFC.


Alex

Re: [Qemu-devel] [PATCH RFC v2 2/2] ARM: KVM: Enable in-kernel timers with user space gic

[Andrew Jones]

On Mon, Jun 26, 2017 at 11:32:10PM +0200, Alexander Graf wrote:
> On 06/26/2017 05:03 PM, Andrew Jones wrote:
> > On Tue, Dec 13, 2016 at 01:20:50PM +0000, Peter Maydell wrote:
> > > On 14 November 2016 at 14:32, Alexander Graf <agraf@suse.de> wrote:
> > > > When running with KVM enabled, you can choose between emulating the
> > > > gic in kernel or user space. If the kernel supports in-kernel virtualization
> > > > of the interrupt controller, it will default to that. If not, if will
> > > > default to user space emulation.
> > > > 
> > > > Unfortunately when running in user mode gic emulation, we miss out on
> > > > timer events which are only available from kernel space. This patch leverages
> > > > the new kernel/user space pending line synchronization for those timer events.
> > > > 
> > > > Signed-off-by: Alexander Graf <agraf@suse.de>
> > > Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
> > > 
> > Hi everyone,
> > 
> > I probably missed a refresh of this patch, but as I didn't see anything,
> > I picked this one up today in order to test the KVM support recently
> > merged. Tweaking this patch a bit to fit the new ABI allowed me to
> > instantiate a KVM guest without the in-kernel irqchip (tested on a
> > mustang). So, FWIW, this is report of a successful test. Is there a
> > refreshed version of this patch someone can point me to, which I should
> > test instead?
> 
> Sorry, this did fall the cracks way too many times now. I've sent a respin
> that hopefully is slightly more future proof than this RFC :)
> 
> If your tests passed with this patch, please extend them to also cover SMP
> support, as that was broken with this RFC.

Indeed. I retested with the old version and now see that secondaries were
not booting, but they are with the new version.

Thanks,
drew