[PATCH RFC v1 00/10] Add loongarch kvm accel support

[PATCH RFC v1 00/10] Add loongarch kvm accel support

[Tianrui Zhao]

This series add loongarch kvm support, mainly implement
some interfaces used by kvm such as kvm_arch_get/set_regs,
kvm_arch_handle_exit, kvm_loongarch_set_interrupt, etc.

Currently, we are able to boot LoongArch KVM Linux Guests.
In loongarch VM, mmio devices and iocsr devices are emulated
in user space such as APIC, IPI, pci devices, etc, other
hardwares such as MMU, timer and csr are emulated in kernel.

It is based on temporarily unaccepted linux kvm:
https://github.com/loongson/linux-loongarch-kvm
And We will remove the RFC flag until the linux kvm patches
are merged.

The running environment of LoongArch virt machine:
1. Get the linux source by the above mentioned link.
   git checkout kvm-loongarch
   make ARCH=loongarch CROSS_COMPILE=loongarch64-unknown-linux-gnu- loongson3_defconfig
   make ARCH=loongarch CROSS_COMPILE=loongarch64-unknown-linux-gnu-
2. Get the qemu source: https://github.com/loongson/qemu
   git checkout kvm-loongarch
   ./configure --target-list="loongarch64-softmmu"  --enable-kvm
   make
3. Get uefi bios of LoongArch virt machine:
   Link: https://github.com/tianocore/edk2-platforms/tree/master/Platform/Loongson/LoongArchQemuPkg#readme
4. Also you can access the binary files we have already build:
   https://github.com/yangxiaojuan-loongson/qemu-binary

The command to boot loongarch virt machine:
   $ qemu-system-loongarch64 -machine virt -m 4G -cpu la464 \
   -smp 1 -bios QEMU_EFI.fd -kernel vmlinuz.efi -initrd ramdisk \
   -serial stdio   -monitor telnet:localhost:4495,server,nowait \
   -append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" \
   --nographic

Tianrui Zhao (10):
  linux-headers: Add KVM headers for loongarch
  target/loongarch: Define some kvm_arch interfaces
  target/loongarch: Supplement vcpu env initial when vcpu reset
  target/loongarch: Implement kvm get/set registers
  target/loongarch: Implement kvm_arch_init function
  target/loongarch: Implement kvm_arch_init_vcpu
  target/loongarch: Implement kvm_arch_handle_exit
  target/loongarch: Implement set vcpu intr for kvm
  target/loongarch: Add kvm-stub.c
  target/loongarch: Add kvm file into meson build

 linux-headers/asm-loongarch/kvm.h |  99 ++++++
 linux-headers/linux/kvm.h         |   9 +
 meson.build                       |   2 +
 target/loongarch/cpu.c            |  21 +-
 target/loongarch/cpu.h            |   3 +
 target/loongarch/kvm-stub.c       |  17 +
 target/loongarch/kvm.c            | 545 ++++++++++++++++++++++++++++++
 target/loongarch/kvm_loongarch.h  |  14 +
 target/loongarch/meson.build      |   1 +
 9 files changed, 706 insertions(+), 5 deletions(-)
 create mode 100644 linux-headers/asm-loongarch/kvm.h
 create mode 100644 target/loongarch/kvm-stub.c
 create mode 100644 target/loongarch/kvm.c
 create mode 100644 target/loongarch/kvm_loongarch.h

-- 
2.31.1

[PATCH RFC v1 01/10] linux-headers: Add KVM headers for loongarch

[Tianrui Zhao]

Add asm-loongarch/kvm.h for loongarch KVM, and update
the linux/kvm.h about loongarch part. The structures in
the header are used as kvm_ioctl arguments.

Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
---
 linux-headers/asm-loongarch/kvm.h | 99 +++++++++++++++++++++++++++++++
 linux-headers/linux/kvm.h         |  9 +++
 2 files changed, 108 insertions(+)
 create mode 100644 linux-headers/asm-loongarch/kvm.h

diff --git a/linux-headers/asm-loongarch/kvm.h b/linux-headers/asm-loongarch/kvm.h
new file mode 100644
index 0000000000..6420f59f9e
--- /dev/null
+++ b/linux-headers/asm-loongarch/kvm.h
@@ -0,0 +1,99 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (C) 2023 Loongson Technology Corporation Limited
+ */
+
+#ifndef __UAPI_ASM_LOONGARCH_KVM_H
+#define __UAPI_ASM_LOONGARCH_KVM_H
+
+#include <linux/types.h>
+
+/*
+ * KVM Loongarch specific structures and definitions.
+ */
+
+#define __KVM_HAVE_READONLY_MEM
+
+#define KVM_COALESCED_MMIO_PAGE_OFFSET 1
+
+/*
+ * for KVM_GET_REGS and KVM_SET_REGS
+ */
+struct kvm_regs {
+	/* out (KVM_GET_REGS) / in (KVM_SET_REGS) */
+	__u64 gpr[32];
+	__u64 pc;
+};
+
+/*
+ * for KVM_GET_FPU and KVM_SET_FPU
+ */
+struct kvm_fpu {
+	__u32 fcsr;
+	__u32 none;
+	__u64 fcc;    /* 8x8 */
+	struct kvm_fpureg {
+		__u64 val64[4];
+	} fpr[32];
+};
+
+/*
+ * For LoongArch, we use KVM_SET_ONE_REG and KVM_GET_ONE_REG to access various
+ * registers.  The id field is broken down as follows:
+ *
+ *  bits[63..52] - As per linux/kvm.h
+ *  bits[51..32] - Must be zero.
+ *  bits[31..16] - Register set.
+ *
+ * Register set = 0: GP registers from kvm_regs (see definitions below).
+ *
+ * Register set = 1: CSR registers.
+ *
+ * Register set = 2: KVM specific registers (see definitions below).
+ *
+ * Register set = 3: FPU / SIMD registers (see definitions below).
+ *
+ * Other sets registers may be added in the future.  Each set would
+ * have its own identifier in bits[31..16].
+ */
+
+#define KVM_REG_LOONGARCH_GP		(KVM_REG_LOONGARCH | 0x00000ULL)
+#define KVM_REG_LOONGARCH_CSR		(KVM_REG_LOONGARCH | 0x10000ULL)
+#define KVM_REG_LOONGARCH_KVM		(KVM_REG_LOONGARCH | 0x20000ULL)
+#define KVM_REG_LOONGARCH_FPU		(KVM_REG_LOONGARCH | 0x30000ULL)
+#define KVM_REG_LOONGARCH_MASK		(KVM_REG_LOONGARCH | 0x30000ULL)
+#define KVM_CSR_IDX_MASK		(0x10000 - 1)
+
+/*
+ * KVM_REG_LOONGARCH_KVM - KVM specific control registers.
+ */
+
+#define KVM_REG_LOONGARCH_COUNTER	(KVM_REG_LOONGARCH_KVM | KVM_REG_SIZE_U64 | 3)
+#define KVM_REG_LOONGARCH_VCPU_RESET	(KVM_REG_LOONGARCH_KVM | KVM_REG_SIZE_U64 | 4)
+
+struct kvm_debug_exit_arch {
+};
+
+/* for KVM_SET_GUEST_DEBUG */
+struct kvm_guest_debug_arch {
+};
+
+/* definition of registers in kvm_run */
+struct kvm_sync_regs {
+};
+
+/* dummy definition */
+struct kvm_sregs {
+};
+
+struct kvm_loongarch_interrupt {
+	/* in */
+	__u32 cpu;
+	__u32 irq;
+};
+
+#define KVM_NR_IRQCHIPS		1
+#define KVM_IRQCHIP_NUM_PINS	64
+#define KVM_MAX_CORES		256
+
+#endif /* __UAPI_ASM_LOONGARCH_KVM_H */
diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 1e2c16cfe3..8d72bb6810 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -264,6 +264,7 @@ struct kvm_xen_exit {
 #define KVM_EXIT_RISCV_SBI        35
 #define KVM_EXIT_RISCV_CSR        36
 #define KVM_EXIT_NOTIFY           37
+#define KVM_EXIT_LOONGARCH_IOCSR  38
 
 /* For KVM_EXIT_INTERNAL_ERROR */
 /* Emulate instruction failed. */
@@ -336,6 +337,13 @@ struct kvm_run {
 			__u32 len;
 			__u8  is_write;
 		} mmio;
+		/* KVM_EXIT_LOONGARCH_IOCSR */
+		struct {
+			__u64 phys_addr;
+			__u8  data[8];
+			__u32 len;
+			__u8  is_write;
+		} iocsr_io;
 		/* KVM_EXIT_HYPERCALL */
 		struct {
 			__u64 nr;
@@ -1343,6 +1351,7 @@ struct kvm_dirty_tlb {
 #define KVM_REG_ARM64		0x6000000000000000ULL
 #define KVM_REG_MIPS		0x7000000000000000ULL
 #define KVM_REG_RISCV		0x8000000000000000ULL
+#define KVM_REG_LOONGARCH	0x9000000000000000ULL
 
 #define KVM_REG_SIZE_SHIFT	52
 #define KVM_REG_SIZE_MASK	0x00f0000000000000ULL
-- 
2.31.1

[PATCH RFC v1 02/10] target/loongarch: Define some kvm_arch interfaces

[Tianrui Zhao]

Define some functions in target/loongarch/kvm.c, such as
kvm_arch_put_registers, kvm_arch_get_registers and
kvm_arch_handle_exit, etc. which are needed by kvm/kvm-all.c.
Now the most functions has no content and they will be
implemented in the next patches.

Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
---
 target/loongarch/kvm.c | 134 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 134 insertions(+)
 create mode 100644 target/loongarch/kvm.c

diff --git a/target/loongarch/kvm.c b/target/loongarch/kvm.c
new file mode 100644
index 0000000000..7c11c20c39
--- /dev/null
+++ b/target/loongarch/kvm.c
@@ -0,0 +1,134 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * QEMU LoongArch KVM
+ *
+ * Copyright (c) 2023 Loongson Technology Corporation Limited
+ */
+
+#include "qemu/osdep.h"
+#include <sys/ioctl.h>
+#include <linux/kvm.h>
+
+#include "qemu/timer.h"
+#include "qemu/error-report.h"
+#include "qemu/main-loop.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/kvm.h"
+#include "sysemu/kvm_int.h"
+#include "hw/pci/pci.h"
+#include "exec/memattrs.h"
+#include "exec/address-spaces.h"
+#include "hw/boards.h"
+#include "hw/irq.h"
+#include "qemu/log.h"
+#include "hw/loader.h"
+#include "migration/migration.h"
+#include "sysemu/runstate.h"
+#include "cpu-csr.h"
+#include "kvm_loongarch.h"
+
+#undef DEBUG_KVM
+#ifdef DEBUG_KVM
+#define DPRINTF(fmt, ...) printf("%s: " fmt, __func__, ## __VA_ARGS__)
+#else
+#define DPRINTF(fmt, ...) do {} while (0)
+#endif
+
+static bool cap_has_mp_state;
+const KVMCapabilityInfo kvm_arch_required_capabilities[] = {
+    KVM_CAP_LAST_INFO
+};
+
+int kvm_arch_get_registers(CPUState *cs)
+{
+    return 0;
+}
+int kvm_arch_put_registers(CPUState *cs, int level)
+{
+    return 0;
+}
+
+int kvm_arch_init_vcpu(CPUState *cs)
+{
+    return 0;
+}
+
+int kvm_arch_destroy_vcpu(CPUState *cs)
+{
+    return 0;
+}
+
+unsigned long kvm_arch_vcpu_id(CPUState *cs)
+{
+    return cs->cpu_index;
+}
+
+int kvm_arch_release_virq_post(int virq)
+{
+    return 0;
+}
+
+int kvm_arch_msi_data_to_gsi(uint32_t data)
+{
+    abort();
+}
+
+int kvm_arch_fixup_msi_route(struct kvm_irq_routing_entry *route,
+                             uint64_t address, uint32_t data, PCIDevice *dev)
+{
+    return 0;
+}
+
+int kvm_arch_add_msi_route_post(struct kvm_irq_routing_entry *route,
+                                int vector, PCIDevice *dev)
+{
+    return 0;
+}
+
+void kvm_arch_init_irq_routing(KVMState *s)
+{
+}
+
+int kvm_arch_init(MachineState *ms, KVMState *s)
+{
+    return 0;
+}
+
+int kvm_arch_irqchip_create(KVMState *s)
+{
+    return 0;
+}
+
+void kvm_arch_pre_run(CPUState *cs, struct kvm_run *run)
+{
+}
+
+MemTxAttrs kvm_arch_post_run(CPUState *cs, struct kvm_run *run)
+{
+    return MEMTXATTRS_UNSPECIFIED;
+}
+
+int kvm_arch_process_async_events(CPUState *cs)
+{
+    return cs->halted;
+}
+
+bool kvm_arch_stop_on_emulation_error(CPUState *cs)
+{
+    DPRINTF("%s\n", __func__);
+    return true;
+}
+
+bool kvm_arch_cpu_check_are_resettable(void)
+{
+    return true;
+}
+
+int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
+{
+    return 0;
+}
+
+void kvm_arch_accel_class_init(ObjectClass *oc)
+{
+}
-- 
2.31.1

[PATCH RFC v1 03/10] target/loongarch: Supplement vcpu env initial when vcpu reset

[Tianrui Zhao]

Supplement vcpu env initial when vcpu reset, including
init vcpu mp_state value to KVM_MP_STATE_RUNNABLE and
init vcpu CSR_CPUID,CSR_TID to cpu->cpu_index.

Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
---
 target/loongarch/cpu.c | 3 +++
 target/loongarch/cpu.h | 2 ++
 2 files changed, 5 insertions(+)

diff --git a/target/loongarch/cpu.c b/target/loongarch/cpu.c
index 97e6579f6a..600c00bbf2 100644
--- a/target/loongarch/cpu.c
+++ b/target/loongarch/cpu.c
@@ -486,10 +486,12 @@ static void loongarch_cpu_reset_hold(Object *obj)
 
     env->CSR_ESTAT = env->CSR_ESTAT & (~MAKE_64BIT_MASK(0, 2));
     env->CSR_RVACFG = FIELD_DP64(env->CSR_RVACFG, CSR_RVACFG, RBITS, 0);
+    env->CSR_CPUID = cs->cpu_index;
     env->CSR_TCFG = FIELD_DP64(env->CSR_TCFG, CSR_TCFG, EN, 0);
     env->CSR_LLBCTL = FIELD_DP64(env->CSR_LLBCTL, CSR_LLBCTL, KLO, 0);
     env->CSR_TLBRERA = FIELD_DP64(env->CSR_TLBRERA, CSR_TLBRERA, ISTLBR, 0);
     env->CSR_MERRCTL = FIELD_DP64(env->CSR_MERRCTL, CSR_MERRCTL, ISMERR, 0);
+    env->CSR_TID = cs->cpu_index;
 
     env->CSR_PRCFG3 = FIELD_DP64(env->CSR_PRCFG3, CSR_PRCFG3, TLB_TYPE, 2);
     env->CSR_PRCFG3 = FIELD_DP64(env->CSR_PRCFG3, CSR_PRCFG3, MTLB_ENTRY, 63);
@@ -510,6 +512,7 @@ static void loongarch_cpu_reset_hold(Object *obj)
 
     restore_fp_status(env);
     cs->exception_index = -1;
+    env->mp_state = KVM_MP_STATE_RUNNABLE;
 }
 
 static void loongarch_cpu_disas_set_info(CPUState *s, disassemble_info *info)
diff --git a/target/loongarch/cpu.h b/target/loongarch/cpu.h
index e11c875188..86b9f26d60 100644
--- a/target/loongarch/cpu.h
+++ b/target/loongarch/cpu.h
@@ -288,6 +288,7 @@ typedef struct CPUArchState {
     uint64_t CSR_PWCH;
     uint64_t CSR_STLBPS;
     uint64_t CSR_RVACFG;
+    uint64_t CSR_CPUID;
     uint64_t CSR_PRCFG1;
     uint64_t CSR_PRCFG2;
     uint64_t CSR_PRCFG3;
@@ -328,6 +329,7 @@ typedef struct CPUArchState {
     MemoryRegion iocsr_mem;
     bool load_elf;
     uint64_t elf_address;
+    uint32_t mp_state;
 #endif
 } CPULoongArchState;
 
-- 
2.31.1

[PATCH RFC v1 04/10] target/loongarch: Implement kvm get/set registers

[Tianrui Zhao]

Implement kvm_arch_get/set_registers interfaces, many regs
can be get/set in the function, such as core regs, csr regs,
fpu regs, mp state, etc.

Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
---
 target/loongarch/kvm.c | 353 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 351 insertions(+), 2 deletions(-)

diff --git a/target/loongarch/kvm.c b/target/loongarch/kvm.c
index 7c11c20c39..74d499b675 100644
--- a/target/loongarch/kvm.c
+++ b/target/loongarch/kvm.c
@@ -39,13 +39,362 @@ const KVMCapabilityInfo kvm_arch_required_capabilities[] = {
     KVM_CAP_LAST_INFO
 };
 
+static int kvm_loongarch_get_regs_core(CPUState *cs)
+{
+    int ret = 0;
+    int i;
+    struct kvm_regs regs;
+    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
+    CPULoongArchState *env = &cpu->env;
+
+    /* Get the current register set as KVM seems it */
+    ret = kvm_vcpu_ioctl(cs, KVM_GET_REGS, &regs);
+    if (ret < 0) {
+        return ret;
+    }
+
+    for (i = 0; i < 32; i++) {
+        env->gpr[i] = regs.gpr[i];
+    }
+
+    env->pc = regs.pc;
+    return ret;
+}
+
+static int kvm_loongarch_put_regs_core(CPUState *cs)
+{
+    int ret = 0;
+    int i;
+    struct kvm_regs regs;
+    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
+    CPULoongArchState *env = &cpu->env;
+
+    /* Set the registers based on QEMU's view of things */
+    for (i = 0; i < 32; i++) {
+        regs.gpr[i] = env->gpr[i];
+    }
+
+    regs.pc = env->pc;
+    ret = kvm_vcpu_ioctl(cs, KVM_SET_REGS, &regs);
+    return ret;
+}
+
+static inline int kvm_larch_getq(CPUState *cs, uint64_t reg_id,
+                                 uint64_t *addr)
+{
+    struct kvm_one_reg csrreg = {
+        .id = reg_id,
+        .addr = (uintptr_t)addr
+    };
+
+    return kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &csrreg);
+}
+static inline int kvm_larch_putq(CPUState *cs, uint64_t reg_id,
+                                 uint64_t *addr)
+{
+    struct kvm_one_reg csrreg = {
+        .id = reg_id,
+        .addr = (uintptr_t)addr
+    };
+
+    return kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &csrreg);
+}
+
+#define LOONGARCH_CSR_64(_R, _S)  \
+        (KVM_REG_LOONGARCH_CSR | KVM_REG_SIZE_U64 | (8 * (_R) + (_S)))
+
+#define KVM_IOC_CSRID(id) LOONGARCH_CSR_64(id, 0)
+
+#define KVM_GET_ONE_UREG64(cs, ret, regidx, addr)              \
+    ({                                                         \
+        err = kvm_larch_getq(cs, KVM_IOC_CSRID(regidx), addr); \
+        if (err < 0) {                                         \
+            ret = err;                                         \
+            DPRINTF("%s: Failed to get regidx 0x%x             \
+                    err:%d\n", __func__, regidx, err);         \
+        }                                                      \
+    })
+
+#define KVM_PUT_ONE_UREG64(cs, ret, regidx, addr)              \
+    ({                                                         \
+        err = kvm_larch_putq(cs, KVM_IOC_CSRID(regidx), addr); \
+        if (err < 0) {                                         \
+            ret = err;                                         \
+            DPRINTF("%s: Failed to put regidx 0x%x             \
+                    err:%d\n", __func__, regidx, err);         \
+        }                                                      \
+    })
+
+static int kvm_loongarch_get_csr(CPUState *cs)
+{
+    int err, ret = 0;
+    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
+    CPULoongArchState *env = &cpu->env;
+
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_CRMD, &env->CSR_CRMD);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_PRMD, &env->CSR_PRMD);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_EUEN, &env->CSR_EUEN);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_MISC, &env->CSR_MISC);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_ECFG, &env->CSR_ECFG);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_ESTAT, &env->CSR_ESTAT);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_ERA, &env->CSR_ERA);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_BADV, &env->CSR_BADV);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_BADI, &env->CSR_BADI);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_EENTRY, &env->CSR_EENTRY);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBIDX, &env->CSR_TLBIDX);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBEHI, &env->CSR_TLBEHI);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBELO0, &env->CSR_TLBELO0);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBELO1, &env->CSR_TLBELO1);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_ASID, &env->CSR_ASID);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_PGDL, &env->CSR_PGDL);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_PGDH, &env->CSR_PGDH);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_PGD, &env->CSR_PGD);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_PWCL, &env->CSR_PWCL);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_PWCH, &env->CSR_PWCH);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_STLBPS, &env->CSR_STLBPS);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_RVACFG, &env->CSR_RVACFG);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_PRCFG1, &env->CSR_PRCFG1);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_PRCFG2, &env->CSR_PRCFG2);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_PRCFG3, &env->CSR_PRCFG3);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(0), &env->CSR_SAVE[0]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(1), &env->CSR_SAVE[1]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(2), &env->CSR_SAVE[2]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(3), &env->CSR_SAVE[3]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(4), &env->CSR_SAVE[4]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(5), &env->CSR_SAVE[5]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(6), &env->CSR_SAVE[6]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(7), &env->CSR_SAVE[7]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TID, &env->CSR_TID);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_CNTC, &env->CSR_CNTC);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TICLR, &env->CSR_TICLR);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_LLBCTL, &env->CSR_LLBCTL);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_IMPCTL1, &env->CSR_IMPCTL1);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_IMPCTL2, &env->CSR_IMPCTL2);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRENTRY, &env->CSR_TLBRENTRY);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRBADV, &env->CSR_TLBRBADV);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRERA, &env->CSR_TLBRERA);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRSAVE, &env->CSR_TLBRSAVE);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRELO0, &env->CSR_TLBRELO0);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRELO1, &env->CSR_TLBRELO1);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBREHI, &env->CSR_TLBREHI);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRPRMD, &env->CSR_TLBRPRMD);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_DMW(0), &env->CSR_DMW[0]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_DMW(1), &env->CSR_DMW[1]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_DMW(2), &env->CSR_DMW[2]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_DMW(3), &env->CSR_DMW[3]);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TVAL, &env->CSR_TVAL);
+    KVM_GET_ONE_UREG64(cs, ret, LOONGARCH_CSR_TCFG, &env->CSR_TCFG);
+
+    return ret;
+}
+
+static int kvm_loongarch_put_csr(CPUState *cs)
+{
+    int err, ret = 0;
+    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
+    CPULoongArchState *env = &cpu->env;
+
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_CRMD, &env->CSR_CRMD);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_PRMD, &env->CSR_PRMD);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_EUEN, &env->CSR_EUEN);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_MISC, &env->CSR_MISC);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_ECFG, &env->CSR_ECFG);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_ESTAT, &env->CSR_ESTAT);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_ERA, &env->CSR_ERA);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_BADV, &env->CSR_BADV);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_BADI, &env->CSR_BADI);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_EENTRY, &env->CSR_EENTRY);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBIDX, &env->CSR_TLBIDX);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBEHI, &env->CSR_TLBEHI);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBELO0, &env->CSR_TLBELO0);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBELO1, &env->CSR_TLBELO1);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_ASID, &env->CSR_ASID);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_PGDL, &env->CSR_PGDL);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_PGDH, &env->CSR_PGDH);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_PGD, &env->CSR_PGD);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_PWCL, &env->CSR_PWCL);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_PWCH, &env->CSR_PWCH);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_STLBPS, &env->CSR_STLBPS);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_RVACFG, &env->CSR_RVACFG);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_CPUID, &env->CSR_CPUID);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_PRCFG1, &env->CSR_PRCFG1);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_PRCFG2, &env->CSR_PRCFG2);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_PRCFG3, &env->CSR_PRCFG3);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(0), &env->CSR_SAVE[0]);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(1), &env->CSR_SAVE[1]);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(2), &env->CSR_SAVE[2]);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(3), &env->CSR_SAVE[3]);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(4), &env->CSR_SAVE[4]);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(5), &env->CSR_SAVE[5]);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(6), &env->CSR_SAVE[6]);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_SAVE(7), &env->CSR_SAVE[7]);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TID, &env->CSR_TID);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_CNTC, &env->CSR_CNTC);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TICLR, &env->CSR_TICLR);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_LLBCTL, &env->CSR_LLBCTL);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_IMPCTL1, &env->CSR_IMPCTL1);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_IMPCTL2, &env->CSR_IMPCTL2);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRENTRY, &env->CSR_TLBRENTRY);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRBADV, &env->CSR_TLBRBADV);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRERA, &env->CSR_TLBRERA);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRSAVE, &env->CSR_TLBRSAVE);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRELO0, &env->CSR_TLBRELO0);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRELO1, &env->CSR_TLBRELO1);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBREHI, &env->CSR_TLBREHI);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TLBRPRMD, &env->CSR_TLBRPRMD);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_DMW(0), &env->CSR_DMW[0]);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_DMW(1), &env->CSR_DMW[1]);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_DMW(2), &env->CSR_DMW[2]);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_DMW(3), &env->CSR_DMW[3]);
+    /*
+     * timer cfg must be put at last since it is used to enable
+     * guest timer
+     */
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TVAL, &env->CSR_TVAL);
+    KVM_PUT_ONE_UREG64(cs, ret, LOONGARCH_CSR_TCFG, &env->CSR_TCFG);
+    return ret;
+}
+
+static int kvm_loongarch_get_regs_fp(CPUState *cs)
+{
+    int ret, i;
+    struct kvm_fpu fpu;
+
+    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
+    CPULoongArchState *env = &cpu->env;
+
+    ret = kvm_vcpu_ioctl(cs, KVM_GET_FPU, &fpu);
+    if (ret < 0) {
+        DPRINTF("%s: Failed to get FPU (%d)\n", __func__, err);
+        return ret;
+    }
+
+    env->fcsr0 = fpu.fcsr;
+    for (i = 0; i < 32; i++) {
+        env->fpr[i] = fpu.fpr[i].val64[0];
+    }
+    for (i = 0; i < 8; i++) {
+        env->cf[i] = fpu.fcc & 0xFF;
+        fpu.fcc = fpu.fcc >> 8;
+    }
+
+    return ret;
+}
+
+static int kvm_loongarch_put_regs_fp(CPUState *cs)
+{
+    int ret, i;
+    struct kvm_fpu fpu;
+
+    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
+    CPULoongArchState *env = &cpu->env;
+
+    fpu.fcsr = env->fcsr0;
+    fpu.fcc = 0;
+    for (i = 0; i < 32; i++) {
+        fpu.fpr[i].val64[0] = env->fpr[i];
+    }
+
+    for (i = 0; i < 8; i++) {
+        fpu.fcc |= env->cf[i] << (8 * i);
+    }
+
+    ret = kvm_vcpu_ioctl(cs, KVM_SET_FPU, &fpu);
+    if (ret < 0) {
+        DPRINTF("%s: Failed to put FPU (%d)\n", __func__, err);
+    }
+
+    return ret;
+}
+
+static int kvm_loongarch_get_mpstate(CPUState *cs)
+{
+    int ret = 0;
+    struct kvm_mp_state mp_state;
+    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
+    CPULoongArchState *env = &cpu->env;
+
+    if (cap_has_mp_state) {
+        ret = kvm_vcpu_ioctl(cs, KVM_GET_MP_STATE, &mp_state);
+        if (ret) {
+            fprintf(stderr, "%s: failed to get MP_STATE %d/%s\n",
+                    __func__, ret, strerror(-ret));
+            return ret;
+        }
+        env->mp_state = mp_state.mp_state;
+    }
+
+    return ret;
+}
+
+static int kvm_loongarch_put_mpstate(CPUState *cs)
+{
+    int ret = 0;
+
+    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
+    CPULoongArchState *env = &cpu->env;
+
+    struct kvm_mp_state mp_state = {
+        .mp_state = env->mp_state
+    };
+
+    if (cap_has_mp_state) {
+        ret = kvm_vcpu_ioctl(cs, KVM_SET_MP_STATE, &mp_state);
+        if (ret) {
+            fprintf(stderr, "%s: failed to set MP_STATE %d/%s\n",
+                    __func__, ret, strerror(-ret));
+        }
+    }
+
+    return ret;
+}
+
 int kvm_arch_get_registers(CPUState *cs)
 {
-    return 0;
+    int ret;
+
+    ret = kvm_loongarch_get_regs_core(cs);
+    if (ret) {
+        return ret;
+    }
+
+    ret = kvm_loongarch_get_csr(cs);
+    if (ret) {
+        return ret;
+    }
+
+    ret = kvm_loongarch_get_regs_fp(cs);
+    if (ret) {
+        return ret;
+    }
+
+    ret = kvm_loongarch_get_mpstate(cs);
+
+    return ret;
 }
+
 int kvm_arch_put_registers(CPUState *cs, int level)
 {
-    return 0;
+    int ret;
+
+    ret = kvm_loongarch_put_regs_core(cs);
+    if (ret) {
+        return ret;
+    }
+
+    ret = kvm_loongarch_put_csr(cs);
+    if (ret) {
+        return ret;
+    }
+
+    ret = kvm_loongarch_put_regs_fp(cs);
+    if (ret) {
+        return ret;
+    }
+
+    ret = kvm_loongarch_put_mpstate(cs);
+
+    return ret;
 }
 
 int kvm_arch_init_vcpu(CPUState *cs)
-- 
2.31.1

[PATCH RFC v1 05/10] target/loongarch: Implement kvm_arch_init function

[Tianrui Zhao]

Implement the kvm_arch_init of loongarch, in the function, the
KVM_CAP_MP_STATE cap is checked by kvm ioctl.

Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
---
 target/loongarch/kvm.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/target/loongarch/kvm.c b/target/loongarch/kvm.c
index 74d499b675..53f495a588 100644
--- a/target/loongarch/kvm.c
+++ b/target/loongarch/kvm.c
@@ -440,6 +440,7 @@ void kvm_arch_init_irq_routing(KVMState *s)
 
 int kvm_arch_init(MachineState *ms, KVMState *s)
 {
+    cap_has_mp_state = kvm_check_extension(s, KVM_CAP_MP_STATE);
     return 0;
 }
 
-- 
2.31.1

[PATCH RFC v1 06/10] target/loongarch: Implement kvm_arch_init_vcpu

[Tianrui Zhao]

Implement kvm_arch_init_vcpu interface for loongarch,
in this function, we register VM change state handler.
And when VM state changes to running, the counter value
should be put into kvm to keep consistent with kvm,
and when state change to stop, counter value should be
refreshed from kvm.

Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
---
 target/loongarch/cpu.h |  1 +
 target/loongarch/kvm.c | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/target/loongarch/cpu.h b/target/loongarch/cpu.h
index 86b9f26d60..473d9986f3 100644
--- a/target/loongarch/cpu.h
+++ b/target/loongarch/cpu.h
@@ -350,6 +350,7 @@ struct ArchCPU {
 
     /* 'compatible' string for this CPU for Linux device trees */
     const char *dtb_compatible;
+    uint64_t counter_value;
 };
 
 #define TYPE_LOONGARCH_CPU "loongarch-cpu"
diff --git a/target/loongarch/kvm.c b/target/loongarch/kvm.c
index 53f495a588..f8772bbb27 100644
--- a/target/loongarch/kvm.c
+++ b/target/loongarch/kvm.c
@@ -397,8 +397,31 @@ int kvm_arch_put_registers(CPUState *cs, int level)
     return ret;
 }
 
+static void kvm_loongarch_vm_stage_change(void *opaque, bool running,
+                                          RunState state)
+{
+    int ret;
+    CPUState *cs = opaque;
+    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
+
+    if (running) {
+        ret = kvm_larch_putq(cs, KVM_REG_LOONGARCH_COUNTER,
+                             &cpu->counter_value);
+        if (ret < 0) {
+            printf("%s: Failed to put counter_value (%d)\n", __func__, ret);
+        }
+    } else {
+        ret = kvm_larch_getq(cs, KVM_REG_LOONGARCH_COUNTER,
+                             &cpu->counter_value);
+        if (ret < 0) {
+            printf("%s: Failed to get counter_value (%d)\n", __func__, ret);
+        }
+    }
+}
+
 int kvm_arch_init_vcpu(CPUState *cs)
 {
+    qemu_add_vm_change_state_handler(kvm_loongarch_vm_stage_change, cs);
     return 0;
 }
 
-- 
2.31.1

[PATCH RFC v1 07/10] target/loongarch: Implement kvm_arch_handle_exit

[Tianrui Zhao]

Implement kvm_arch_handle_exit for loongarch. In this
function, the KVM_EXIT_LOONGARCH_IOCSR is handled,
we read or write the iocsr address space by the addr,
length and is_write argument in kvm_run.

Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
---
 target/loongarch/kvm.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/target/loongarch/kvm.c b/target/loongarch/kvm.c
index f8772bbb27..4ce343d276 100644
--- a/target/loongarch/kvm.c
+++ b/target/loongarch/kvm.c
@@ -499,7 +499,29 @@ bool kvm_arch_cpu_check_are_resettable(void)
 
 int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
 {
-    return 0;
+    int ret = 0;
+    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
+    CPULoongArchState *env = &cpu->env;
+    MemTxAttrs attrs = {};
+
+    attrs.requester_id = env_cpu(env)->cpu_index;
+
+    DPRINTF("%s\n", __func__);
+    switch (run->exit_reason) {
+    case KVM_EXIT_LOONGARCH_IOCSR:
+        address_space_rw(&env->address_space_iocsr,
+                         run->iocsr_io.phys_addr,
+                         attrs,
+                         run->iocsr_io.data,
+                         run->iocsr_io.len,
+                         run->iocsr_io.is_write);
+        break;
+    default:
+        ret = -1;
+        fprintf(stderr, "KVM: unknown exit reason %d\n", run->exit_reason);
+        break;
+    }
+    return ret;
 }
 
 void kvm_arch_accel_class_init(ObjectClass *oc)
-- 
2.31.1

[PATCH RFC v1 08/10] target/loongarch: Implement set vcpu intr for kvm

[Tianrui Zhao]

Implement loongarch kvm set vcpu interrupt interface,
when a irq is set in vcpu, we use the KVM_INTERRUPT
ioctl to set intr into kvm.

Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
---
 target/loongarch/cpu.c           | 18 +++++++++++++-----
 target/loongarch/kvm.c           | 16 ++++++++++++++++
 target/loongarch/kvm_loongarch.h | 13 +++++++++++++
 3 files changed, 42 insertions(+), 5 deletions(-)
 create mode 100644 target/loongarch/kvm_loongarch.h

diff --git a/target/loongarch/cpu.c b/target/loongarch/cpu.c
index 600c00bbf2..d0b8f652a1 100644
--- a/target/loongarch/cpu.c
+++ b/target/loongarch/cpu.c
@@ -18,6 +18,11 @@
 #include "cpu-csr.h"
 #include "sysemu/reset.h"
 #include "tcg/tcg.h"
+#include "sysemu/kvm.h"
+#include "kvm_loongarch.h"
+#ifdef CONFIG_KVM
+#include <linux/kvm.h>
+#endif
 
 const char * const regnames[32] = {
     "r0", "r1", "r2", "r3", "r4", "r5", "r6", "r7",
@@ -104,12 +109,15 @@ void loongarch_cpu_set_irq(void *opaque, int irq, int level)
         return;
     }
 
-    env->CSR_ESTAT = deposit64(env->CSR_ESTAT, irq, 1, level != 0);
-
-    if (FIELD_EX64(env->CSR_ESTAT, CSR_ESTAT, IS)) {
-        cpu_interrupt(cs, CPU_INTERRUPT_HARD);
+    if (kvm_enabled()) {
+        kvm_loongarch_set_interrupt(cpu, irq, level);
     } else {
-        cpu_reset_interrupt(cs, CPU_INTERRUPT_HARD);
+        env->CSR_ESTAT = deposit64(env->CSR_ESTAT, irq, 1, level != 0);
+        if (FIELD_EX64(env->CSR_ESTAT, CSR_ESTAT, IS)) {
+            cpu_interrupt(cs, CPU_INTERRUPT_HARD);
+        } else {
+            cpu_reset_interrupt(cs, CPU_INTERRUPT_HARD);
+        }
     }
 }
 
diff --git a/target/loongarch/kvm.c b/target/loongarch/kvm.c
index 4ce343d276..ef71062070 100644
--- a/target/loongarch/kvm.c
+++ b/target/loongarch/kvm.c
@@ -524,6 +524,22 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
     return ret;
 }
 
+int kvm_loongarch_set_interrupt(LoongArchCPU *cpu, int irq, int level)
+{
+    struct kvm_loongarch_interrupt intr;
+    CPUState *cs = CPU(cpu);
+
+    intr.cpu = cs->cpu_index;
+    if (level) {
+        intr.irq = irq;
+    } else {
+        intr.irq = -irq;
+    }
+
+    DPRINTF("%s:  IRQ: %d\n", __func__, intr.irq);
+    return kvm_vcpu_ioctl(cs, KVM_INTERRUPT, &intr);
+}
+
 void kvm_arch_accel_class_init(ObjectClass *oc)
 {
 }
diff --git a/target/loongarch/kvm_loongarch.h b/target/loongarch/kvm_loongarch.h
new file mode 100644
index 0000000000..cdef980eec
--- /dev/null
+++ b/target/loongarch/kvm_loongarch.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * QEMU LoongArch kvm interface
+ *
+ * Copyright (c) 2023 Loongson Technology Corporation Limited
+ */
+
+#ifndef QEMU_KVM_LOONGARCH_H
+#define QEMU_KVM_LOONGARCH_H
+
+int  kvm_loongarch_set_interrupt(LoongArchCPU *cpu, int irq, int level);
+
+#endif
-- 
2.31.1

[PATCH RFC v1 09/10] target/loongarch: Add kvm-stub.c

[Tianrui Zhao]

Add kvm-stub.c for loongarch, there are two stub functions:
kvm_loongarch_reset_vcpu and kvm_loongarch_set_interrupt.

Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
---
 target/loongarch/kvm-stub.c      | 17 +++++++++++++++++
 target/loongarch/kvm_loongarch.h |  1 +
 2 files changed, 18 insertions(+)
 create mode 100644 target/loongarch/kvm-stub.c

diff --git a/target/loongarch/kvm-stub.c b/target/loongarch/kvm-stub.c
new file mode 100644
index 0000000000..e28827ee07
--- /dev/null
+++ b/target/loongarch/kvm-stub.c
@@ -0,0 +1,17 @@
+/*
+ * QEMU KVM LoongArch specific function stubs
+ *
+ * Copyright (c) 2023 Loongson Technology Corporation Limited
+ */
+#include "qemu/osdep.h"
+#include "cpu.h"
+
+void kvm_loongarch_reset_vcpu(LoongArchCPU *cpu)
+{
+    abort();
+}
+
+void kvm_loongarch_set_interrupt(LoongArchCPU *cpu, int irq, int level)
+{
+    abort();
+}
diff --git a/target/loongarch/kvm_loongarch.h b/target/loongarch/kvm_loongarch.h
index cdef980eec..c03f4bef0f 100644
--- a/target/loongarch/kvm_loongarch.h
+++ b/target/loongarch/kvm_loongarch.h
@@ -8,6 +8,7 @@
 #ifndef QEMU_KVM_LOONGARCH_H
 #define QEMU_KVM_LOONGARCH_H
 
+void kvm_loongarch_reset_vcpu(LoongArchCPU *cpu);
 int  kvm_loongarch_set_interrupt(LoongArchCPU *cpu, int irq, int level);
 
 #endif
-- 
2.31.1

[PATCH RFC v1 10/10] target/loongarch: Add kvm file into meson build

[Tianrui Zhao]

Add kvm.c and kvm-stub.c into meson.build to compile
it when kvm is configed. Meanwhile in meson.build,
we set the kvm_targets to loongarch64-softmmu when
the cpu is loongarch.

Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
---
 meson.build                  | 2 ++
 target/loongarch/meson.build | 1 +
 2 files changed, 3 insertions(+)

diff --git a/meson.build b/meson.build
index 29f8644d6d..9a4bce4add 100644
--- a/meson.build
+++ b/meson.build
@@ -114,6 +114,8 @@ elif cpu in ['mips', 'mips64']
   kvm_targets = ['mips-softmmu', 'mipsel-softmmu', 'mips64-softmmu', 'mips64el-softmmu']
 elif cpu in ['riscv']
   kvm_targets = ['riscv32-softmmu', 'riscv64-softmmu']
+elif cpu in ['loongarch64']
+  kvm_targets = ['loongarch64-softmmu']
 else
   kvm_targets = []
 endif
diff --git a/target/loongarch/meson.build b/target/loongarch/meson.build
index 9293a8ab78..0c6cec81e4 100644
--- a/target/loongarch/meson.build
+++ b/target/loongarch/meson.build
@@ -26,6 +26,7 @@ loongarch_softmmu_ss.add(files(
 
 common_ss.add(when: 'CONFIG_LOONGARCH_DIS', if_true: [files('disas.c'), gen])
 
+loongarch_ss.add(when: 'CONFIG_KVM', if_true: files('kvm.c'), if_false: files('kvm-stub.c'))
 loongarch_ss.add_all(when: 'CONFIG_TCG', if_true: [loongarch_tcg_ss])
 
 target_arch += {'loongarch': loongarch_ss}
-- 
2.31.1

Re: [PATCH RFC v1 01/10] linux-headers: Add KVM headers for loongarch

[Cornelia Huck]

On Thu, Apr 20 2023, Tianrui Zhao <zhaotianrui@loongson.cn> wrote:

> Add asm-loongarch/kvm.h for loongarch KVM, and update
> the linux/kvm.h about loongarch part. The structures in
> the header are used as kvm_ioctl arguments.

Just a procedural note: It's probably best to explicitly mark this as a
placeholder patch until you can replace it with a full headers update.

>
> Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
> ---
>  linux-headers/asm-loongarch/kvm.h | 99 +++++++++++++++++++++++++++++++
>  linux-headers/linux/kvm.h         |  9 +++
>  2 files changed, 108 insertions(+)
>  create mode 100644 linux-headers/asm-loongarch/kvm.h

Re: [PATCH RFC v1 07/10] target/loongarch: Implement kvm_arch_handle_exit

[Philippe Mathieu-Daudé]

On 20/4/23 11:36, Tianrui Zhao wrote:
> Implement kvm_arch_handle_exit for loongarch. In this
> function, the KVM_EXIT_LOONGARCH_IOCSR is handled,
> we read or write the iocsr address space by the addr,
> length and is_write argument in kvm_run.
> 
> Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
> ---
>   target/loongarch/kvm.c | 24 +++++++++++++++++++++++-
>   1 file changed, 23 insertions(+), 1 deletion(-)
> 
> diff --git a/target/loongarch/kvm.c b/target/loongarch/kvm.c
> index f8772bbb27..4ce343d276 100644
> --- a/target/loongarch/kvm.c
> +++ b/target/loongarch/kvm.c
> @@ -499,7 +499,29 @@ bool kvm_arch_cpu_check_are_resettable(void)
>   
>   int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
>   {
> -    return 0;
> +    int ret = 0;
> +    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
> +    CPULoongArchState *env = &cpu->env;
> +    MemTxAttrs attrs = {};
> +
> +    attrs.requester_id = env_cpu(env)->cpu_index;
> +
> +    DPRINTF("%s\n", __func__);

Please use trace events instead of DPRINTF(), as we are trying to remove
these.

> +    switch (run->exit_reason) {
> +    case KVM_EXIT_LOONGARCH_IOCSR:
> +        address_space_rw(&env->address_space_iocsr,
> +                         run->iocsr_io.phys_addr,
> +                         attrs,
> +                         run->iocsr_io.data,
> +                         run->iocsr_io.len,
> +                         run->iocsr_io.is_write);
> +        break;
> +    default:
> +        ret = -1;
> +        fprintf(stderr, "KVM: unknown exit reason %d\n", run->exit_reason);

Would warn_report() be more appropriate here?

> +        break;
> +    }
> +    return ret;
>   }
>   
>   void kvm_arch_accel_class_init(ObjectClass *oc)

Re: [PATCH RFC v1 09/10] target/loongarch: Add kvm-stub.c

[Philippe Mathieu-Daudé]

On 20/4/23 11:36, Tianrui Zhao wrote:
> Add kvm-stub.c for loongarch, there are two stub functions:
> kvm_loongarch_reset_vcpu and kvm_loongarch_set_interrupt.
> 
> Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
> ---
>   target/loongarch/kvm-stub.c      | 17 +++++++++++++++++
>   target/loongarch/kvm_loongarch.h |  1 +
>   2 files changed, 18 insertions(+)
>   create mode 100644 target/loongarch/kvm-stub.c
> 
> diff --git a/target/loongarch/kvm-stub.c b/target/loongarch/kvm-stub.c
> new file mode 100644
> index 0000000000..e28827ee07
> --- /dev/null
> +++ b/target/loongarch/kvm-stub.c
> @@ -0,0 +1,17 @@
> +/*
> + * QEMU KVM LoongArch specific function stubs
> + *
> + * Copyright (c) 2023 Loongson Technology Corporation Limited
> + */
> +#include "qemu/osdep.h"
> +#include "cpu.h"
> +
> +void kvm_loongarch_reset_vcpu(LoongArchCPU *cpu)

Where is kvm_loongarch_reset_vcpu() called?

> +{
> +    abort();
> +}
> +
> +void kvm_loongarch_set_interrupt(LoongArchCPU *cpu, int irq, int level)
> +{
> +    abort();

Please use g_assert_not_reached() which display more useful informations.

> +}

Add this stub in the previous patch "Implement set vcpu intr for kvm".

> diff --git a/target/loongarch/kvm_loongarch.h b/target/loongarch/kvm_loongarch.h
> index cdef980eec..c03f4bef0f 100644
> --- a/target/loongarch/kvm_loongarch.h
> +++ b/target/loongarch/kvm_loongarch.h
> @@ -8,6 +8,7 @@
>   #ifndef QEMU_KVM_LOONGARCH_H
>   #define QEMU_KVM_LOONGARCH_H
>   
> +void kvm_loongarch_reset_vcpu(LoongArchCPU *cpu);
>   int  kvm_loongarch_set_interrupt(LoongArchCPU *cpu, int irq, int level);
>   
>   #endif

Re: [PATCH RFC v1 07/10] target/loongarch: Implement kvm_arch_handle_exit

[Tianrui Zhao]

在 2023年04月20日 18:00, Philippe Mathieu-Daudé 写道:
> On 20/4/23 11:36, Tianrui Zhao wrote:
>> Implement kvm_arch_handle_exit for loongarch. In this
>> function, the KVM_EXIT_LOONGARCH_IOCSR is handled,
>> we read or write the iocsr address space by the addr,
>> length and is_write argument in kvm_run.
>>
>> Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
>> ---
>>   target/loongarch/kvm.c | 24 +++++++++++++++++++++++-
>>   1 file changed, 23 insertions(+), 1 deletion(-)
>>
>> diff --git a/target/loongarch/kvm.c b/target/loongarch/kvm.c
>> index f8772bbb27..4ce343d276 100644
>> --- a/target/loongarch/kvm.c
>> +++ b/target/loongarch/kvm.c
>> @@ -499,7 +499,29 @@ bool kvm_arch_cpu_check_are_resettable(void)
>>     int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
>>   {
>> -    return 0;
>> +    int ret = 0;
>> +    LoongArchCPU *cpu = LOONGARCH_CPU(cs);
>> +    CPULoongArchState *env = &cpu->env;
>> +    MemTxAttrs attrs = {};
>> +
>> +    attrs.requester_id = env_cpu(env)->cpu_index;
>> +
>> +    DPRINTF("%s\n", __func__);
>
> Please use trace events instead of DPRINTF(), as we are trying to remove
> these.
Thanks, I will replace it with trace events.

Thanks
Tianrui Zhao
>
>> +    switch (run->exit_reason) {
>> +    case KVM_EXIT_LOONGARCH_IOCSR:
>> +        address_space_rw(&env->address_space_iocsr,
>> +                         run->iocsr_io.phys_addr,
>> +                         attrs,
>> +                         run->iocsr_io.data,
>> +                         run->iocsr_io.len,
>> +                         run->iocsr_io.is_write);
>> +        break;
>> +    default:
>> +        ret = -1;
>> +        fprintf(stderr, "KVM: unknown exit reason %d\n", 
>> run->exit_reason);
>
> Would warn_report() be more appropriate here?
Thanks, I will use warn_report() here.

Thanks
Tianrui Zhao
>
>> +        break;
>> +    }
>> +    return ret;
>>   }
>>     void kvm_arch_accel_class_init(ObjectClass *oc)

Re: [PATCH RFC v1 09/10] target/loongarch: Add kvm-stub.c

[Tianrui Zhao]

在 2023年04月20日 18:04, Philippe Mathieu-Daudé 写道:
> On 20/4/23 11:36, Tianrui Zhao wrote:
>> Add kvm-stub.c for loongarch, there are two stub functions:
>> kvm_loongarch_reset_vcpu and kvm_loongarch_set_interrupt.
>>
>> Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
>> ---
>>   target/loongarch/kvm-stub.c      | 17 +++++++++++++++++
>>   target/loongarch/kvm_loongarch.h |  1 +
>>   2 files changed, 18 insertions(+)
>>   create mode 100644 target/loongarch/kvm-stub.c
>>
>> diff --git a/target/loongarch/kvm-stub.c b/target/loongarch/kvm-stub.c
>> new file mode 100644
>> index 0000000000..e28827ee07
>> --- /dev/null
>> +++ b/target/loongarch/kvm-stub.c
>> @@ -0,0 +1,17 @@
>> +/*
>> + * QEMU KVM LoongArch specific function stubs
>> + *
>> + * Copyright (c) 2023 Loongson Technology Corporation Limited
>> + */
>> +#include "qemu/osdep.h"
>> +#include "cpu.h"
>> +
>> +void kvm_loongarch_reset_vcpu(LoongArchCPU *cpu)
>
> Where is kvm_loongarch_reset_vcpu() called?
Thanks and nowhere called this function, I will remove it.

Thanks
Tianrui Zhao
>
>> +{
>> +    abort();
>> +}
>> +
>> +void kvm_loongarch_set_interrupt(LoongArchCPU *cpu, int irq, int level)
>> +{
>> +    abort();
>
> Please use g_assert_not_reached() which display more useful informations.
Thanks, I will use the g_assert_not_reached() to replace it.

Thanks
Tianrui Zhao
>
>> +}
>
> Add this stub in the previous patch "Implement set vcpu intr for kvm".
Thanks, I will move this stub function into previous patch "Implement 
set vcpu intr for kvm".

Thanks
Tianrui Zhao
>
>> diff --git a/target/loongarch/kvm_loongarch.h 
>> b/target/loongarch/kvm_loongarch.h
>> index cdef980eec..c03f4bef0f 100644
>> --- a/target/loongarch/kvm_loongarch.h
>> +++ b/target/loongarch/kvm_loongarch.h
>> @@ -8,6 +8,7 @@
>>   #ifndef QEMU_KVM_LOONGARCH_H
>>   #define QEMU_KVM_LOONGARCH_H
>>   +void kvm_loongarch_reset_vcpu(LoongArchCPU *cpu);
>>   int  kvm_loongarch_set_interrupt(LoongArchCPU *cpu, int irq, int 
>> level);
>>     #endif

Re: [PATCH RFC v1 01/10] linux-headers: Add KVM headers for loongarch

[Tianrui Zhao]

在 2023年04月20日 17:49, Cornelia Huck 写道:
> On Thu, Apr 20 2023, Tianrui Zhao <zhaotianrui@loongson.cn> wrote:
>
>> Add asm-loongarch/kvm.h for loongarch KVM, and update
>> the linux/kvm.h about loongarch part. The structures in
>> the header are used as kvm_ioctl arguments.
> Just a procedural note: It's probably best to explicitly mark this as a
> placeholder patch until you can replace it with a full headers update.
Thanks, I will mark this as a placeholder patch until it can be merged.

Thanks
Tianrui Zhao
>
>> Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
>> ---
>>   linux-headers/asm-loongarch/kvm.h | 99 +++++++++++++++++++++++++++++++
>>   linux-headers/linux/kvm.h         |  9 +++
>>   2 files changed, 108 insertions(+)
>>   create mode 100644 linux-headers/asm-loongarch/kvm.h

Re: [PATCH RFC v1 01/10] linux-headers: Add KVM headers for loongarch

[Tianrui Zhao]

在 2023年04月20日 20:59, Tianrui Zhao 写道:
>
>
> 在 2023年04月20日 17:49, Cornelia Huck 写道:
>> On Thu, Apr 20 2023, Tianrui Zhao <zhaotianrui@loongson.cn> wrote:
>>
>>> Add asm-loongarch/kvm.h for loongarch KVM, and update
>>> the linux/kvm.h about loongarch part. The structures in
>>> the header are used as kvm_ioctl arguments.
>> Just a procedural note: It's probably best to explicitly mark this as a
>> placeholder patch until you can replace it with a full headers update.
> Thanks, I will mark this as a placeholder patch until it can be merged.
>
> Thanks
> Tianrui Zhao
I will use update-linux-headers.sh to generate the kvm related headers 
until the linux kvm patches are accepted. And now this patch is only a 
placeholder to show some kvm structures and macros for reviewers.

Thanks
Tianrui Zhao
>>
>>> Signed-off-by: Tianrui Zhao <zhaotianrui@loongson.cn>
>>> ---
>>>   linux-headers/asm-loongarch/kvm.h | 99 
>>> +++++++++++++++++++++++++++++++
>>>   linux-headers/linux/kvm.h         |  9 +++
>>>   2 files changed, 108 insertions(+)
>>>   create mode 100644 linux-headers/asm-loongarch/kvm.h
>