[PATCH 0/2] Add support for ARMv8 CPUs to kvmtool

[PATCH 0/2] Add support for ARMv8 CPUs to kvmtool

[Will Deacon]

Hello again,

These two patches add support for ARMv8 processors running an AArch64 instance
of kvm to kvmtool. Both AArch32 and AArch64 guests are supported and, in the
case of the latter, the guest page size may be either 64k or 4k.

This depends on the ARM updates series I just posted:

  https://lists.cs.columbia.edu/pipermail/kvmarm/2013-January/004505.html

Feedback welcome,

Will


Will Deacon (2):
  kvm tools: add support for ARMv8 processors
  kvm tools: arm: align guest memory buffer to maximum page size

 tools/kvm/Makefile                                 |  14 +-
 tools/kvm/arm/aarch32/include/kvm/kvm-arch.h       |  20 +--
 .../kvm/arm/aarch32/include/kvm/kvm-config-arch.h  |   8 ++
 tools/kvm/arm/aarch64/cortex-a57.c                 |  95 ++++++++++++
 tools/kvm/arm/aarch64/include/kvm/barrier.h        |   8 ++
 tools/kvm/arm/aarch64/include/kvm/kvm-arch.h       |  17 +++
 .../kvm/arm/aarch64/include/kvm/kvm-config-arch.h  |  10 ++
 tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h   |  13 ++
 tools/kvm/arm/aarch64/kvm-cpu.c                    | 160 +++++++++++++++++++++
 tools/kvm/arm/fdt.c                                |   2 +-
 tools/kvm/arm/include/arm-common/kvm-arch.h        |  32 ++++-
 .../include/{kvm => arm-common}/kvm-config-arch.h  |   8 +-
 tools/kvm/arm/kvm.c                                |  26 +++-
 13 files changed, 381 insertions(+), 32 deletions(-)
 create mode 100644 tools/kvm/arm/aarch32/include/kvm/kvm-config-arch.h
 create mode 100644 tools/kvm/arm/aarch64/cortex-a57.c
 create mode 100644 tools/kvm/arm/aarch64/include/kvm/barrier.h
 create mode 100644 tools/kvm/arm/aarch64/include/kvm/kvm-arch.h
 create mode 100644 tools/kvm/arm/aarch64/include/kvm/kvm-config-arch.h
 create mode 100644 tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
 create mode 100644 tools/kvm/arm/aarch64/kvm-cpu.c
 rename tools/kvm/arm/include/{kvm => arm-common}/kvm-config-arch.h (61%)

-- 
1.8.0

[PATCH 1/2] kvm tools: add support for ARMv8 processors

[Will Deacon]

This patch adds support for ARMv8 processors (more specifically,
Cortex-A57) to kvmtool. Both AArch64 and AArch32 guests are supported,
so the existing AArch32 code is slightly restructured to allow for
re-use of much of the current code.

The implementation closely follows the ARMv7 code and reuses much of the
work written there.

Tested-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
---
 tools/kvm/Makefile                                 |  14 +-
 tools/kvm/arm/aarch32/include/kvm/kvm-arch.h       |  20 +--
 .../kvm/arm/aarch32/include/kvm/kvm-config-arch.h  |   8 ++
 tools/kvm/arm/aarch64/cortex-a57.c                 |  95 ++++++++++++
 tools/kvm/arm/aarch64/include/kvm/barrier.h        |   8 ++
 tools/kvm/arm/aarch64/include/kvm/kvm-arch.h       |  17 +++
 .../kvm/arm/aarch64/include/kvm/kvm-config-arch.h  |  10 ++
 tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h   |  13 ++
 tools/kvm/arm/aarch64/kvm-cpu.c                    | 160 +++++++++++++++++++++
 tools/kvm/arm/fdt.c                                |   2 +-
 tools/kvm/arm/include/arm-common/kvm-arch.h        |  22 ++-
 .../include/{kvm => arm-common}/kvm-config-arch.h  |   8 +-
 tools/kvm/arm/kvm.c                                |   2 +-
 13 files changed, 353 insertions(+), 26 deletions(-)
 create mode 100644 tools/kvm/arm/aarch32/include/kvm/kvm-config-arch.h
 create mode 100644 tools/kvm/arm/aarch64/cortex-a57.c
 create mode 100644 tools/kvm/arm/aarch64/include/kvm/barrier.h
 create mode 100644 tools/kvm/arm/aarch64/include/kvm/kvm-arch.h
 create mode 100644 tools/kvm/arm/aarch64/include/kvm/kvm-config-arch.h
 create mode 100644 tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
 create mode 100644 tools/kvm/arm/aarch64/kvm-cpu.c
 rename tools/kvm/arm/include/{kvm => arm-common}/kvm-config-arch.h (61%)

diff --git a/tools/kvm/Makefile b/tools/kvm/Makefile
index 33aa4d8..0c59faa 100644
--- a/tools/kvm/Makefile
+++ b/tools/kvm/Makefile
@@ -103,7 +103,7 @@ OBJS	+= virtio/mmio.o
 
 # Translate uname -m into ARCH string
 ARCH ?= $(shell uname -m | sed -e s/i.86/i386/ -e s/ppc.*/powerpc/ \
-	  -e s/armv7.*/arm/)
+	  -e s/armv7.*/arm/ -e s/aarch64.*/arm64/)
 
 ifeq ($(ARCH),i386)
 	ARCH         := x86
@@ -174,6 +174,18 @@ ifeq ($(ARCH), arm)
 	OTHEROBJS	+= $(LIBFDT_OBJS)
 endif
 
+# ARM64
+ifeq ($(ARCH), arm64)
+	DEFINES		+= -DCONFIG_ARM64
+	OBJS		+= $(OBJS_ARM_COMMON)
+	OBJS		+= arm/aarch64/cortex-a57.o
+	OBJS		+= arm/aarch64/kvm-cpu.o
+	ARCH_INCLUDE	:= $(HDRS_ARM_COMMON)
+	ARCH_INCLUDE	+= -Iarm/aarch64/include
+	CFLAGS		+= -I../../scripts/dtc/libfdt
+	OTHEROBJS	+= $(LIBFDT_OBJS)
+endif
+
 ###
 
 ifeq (,$(ARCH_INCLUDE))
diff --git a/tools/kvm/arm/aarch32/include/kvm/kvm-arch.h b/tools/kvm/arm/aarch32/include/kvm/kvm-arch.h
index ca79b24..1632e3c 100644
--- a/tools/kvm/arm/aarch32/include/kvm/kvm-arch.h
+++ b/tools/kvm/arm/aarch32/include/kvm/kvm-arch.h
@@ -1,28 +1,12 @@
 #ifndef KVM__KVM_ARCH_H
 #define KVM__KVM_ARCH_H
 
-#include <linux/const.h>
-
-#define ARM_LOMAP_MMIO_AREA	_AC(0x00000000, UL)
-#define ARM_LOMAP_AXI_AREA	_AC(0x40000000, UL)
-#define ARM_LOMAP_MEMORY_AREA	_AC(0x80000000, UL)
-#define ARM_LOMAP_MAX_MEMORY	_AC(0x7fffffff, UL)
-
 #define ARM_GIC_DIST_SIZE	0x1000
-#define ARM_GIC_DIST_BASE	(ARM_LOMAP_AXI_AREA - ARM_GIC_DIST_SIZE)
 #define ARM_GIC_CPUI_SIZE	0x2000
-#define ARM_GIC_CPUI_BASE	(ARM_GIC_DIST_BASE - ARM_GIC_CPUI_SIZE)
-
-#define ARM_KERN_OFFSET		0x8000
-
-#define ARM_VIRTIO_MMIO_SIZE	(ARM_GIC_DIST_BASE - ARM_LOMAP_MMIO_AREA)
-#define ARM_PCI_MMIO_SIZE	(ARM_LOMAP_MEMORY_AREA - ARM_LOMAP_AXI_AREA)
 
-#define ARM_MEMORY_AREA		ARM_LOMAP_MEMORY_AREA
-#define ARM_MAX_MEMORY		ARM_LOMAP_MAX_MEMORY
+#define ARM_KERN_OFFSET(...)	0x8000
 
-#define KVM_PCI_MMIO_AREA	ARM_LOMAP_AXI_AREA
-#define KVM_VIRTIO_MMIO_AREA	ARM_LOMAP_MMIO_AREA
+#define ARM_MAX_MEMORY(...)	ARM_LOMAP_MAX_MEMORY
 
 #include "arm-common/kvm-arch.h"
 
diff --git a/tools/kvm/arm/aarch32/include/kvm/kvm-config-arch.h b/tools/kvm/arm/aarch32/include/kvm/kvm-config-arch.h
new file mode 100644
index 0000000..acf0d23
--- /dev/null
+++ b/tools/kvm/arm/aarch32/include/kvm/kvm-config-arch.h
@@ -0,0 +1,8 @@
+#ifndef KVM__KVM_CONFIG_ARCH_H
+#define KVM__KVM_CONFIG_ARCH_H
+
+#define ARM_OPT_ARCH_RUN(...)
+
+#include "arm-common/kvm-config-arch.h"
+
+#endif /* KVM__KVM_CONFIG_ARCH_H */
diff --git a/tools/kvm/arm/aarch64/cortex-a57.c b/tools/kvm/arm/aarch64/cortex-a57.c
new file mode 100644
index 0000000..4fd11ba
--- /dev/null
+++ b/tools/kvm/arm/aarch64/cortex-a57.c
@@ -0,0 +1,95 @@
+#include "kvm/fdt.h"
+#include "kvm/kvm.h"
+#include "kvm/kvm-cpu.h"
+#include "kvm/util.h"
+
+#include "arm-common/gic.h"
+
+#include <linux/byteorder.h>
+#include <linux/types.h>
+
+#define CPU_NAME_MAX_LEN 8
+static void generate_cpu_nodes(void *fdt, struct kvm *kvm)
+{
+	int cpu;
+
+	_FDT(fdt_begin_node(fdt, "cpus"));
+	_FDT(fdt_property_cell(fdt, "#address-cells", 0x1));
+	_FDT(fdt_property_cell(fdt, "#size-cells", 0x0));
+
+	for (cpu = 0; cpu < kvm->nrcpus; ++cpu) {
+		char cpu_name[CPU_NAME_MAX_LEN];
+
+		if (kvm->cpus[cpu]->cpu_type != KVM_ARM_TARGET_CORTEX_A57) {
+			pr_warning("Ignoring unknown type for CPU %d\n", cpu);
+			continue;
+		}
+
+		snprintf(cpu_name, CPU_NAME_MAX_LEN, "cpu@%d", cpu);
+
+		_FDT(fdt_begin_node(fdt, cpu_name));
+		_FDT(fdt_property_string(fdt, "device_type", "cpu"));
+		_FDT(fdt_property_string(fdt, "compatible", "arm,cortex-a57"));
+
+		if (kvm->nrcpus > 1)
+			_FDT(fdt_property_string(fdt, "enable-method", "psci"));
+
+		_FDT(fdt_property_cell(fdt, "reg", cpu));
+		_FDT(fdt_end_node(fdt));
+	}
+
+	_FDT(fdt_end_node(fdt));
+}
+
+static void generate_timer_nodes(void *fdt, struct kvm *kvm)
+{
+	u32 cpu_mask = (((1 << kvm->nrcpus) - 1) << GIC_FDT_IRQ_PPI_CPU_SHIFT) \
+		       & GIC_FDT_IRQ_PPI_CPU_MASK;
+	u32 irq_prop[] = {
+		cpu_to_fdt32(GIC_FDT_IRQ_TYPE_PPI),
+		cpu_to_fdt32(13),
+		cpu_to_fdt32(cpu_mask | GIC_FDT_IRQ_FLAGS_EDGE_LO_HI),
+
+		cpu_to_fdt32(GIC_FDT_IRQ_TYPE_PPI),
+		cpu_to_fdt32(14),
+		cpu_to_fdt32(cpu_mask | GIC_FDT_IRQ_FLAGS_EDGE_LO_HI),
+
+		cpu_to_fdt32(GIC_FDT_IRQ_TYPE_PPI),
+		cpu_to_fdt32(11),
+		cpu_to_fdt32(cpu_mask | GIC_FDT_IRQ_FLAGS_EDGE_LO_HI),
+
+		cpu_to_fdt32(GIC_FDT_IRQ_TYPE_PPI),
+		cpu_to_fdt32(10),
+		cpu_to_fdt32(cpu_mask | GIC_FDT_IRQ_FLAGS_EDGE_LO_HI),
+	};
+
+	_FDT(fdt_begin_node(fdt, "timer"));
+	_FDT(fdt_property_string(fdt, "compatible", "arm,armv8-timer"));
+	_FDT(fdt_property(fdt, "interrupts", irq_prop, sizeof(irq_prop)));
+	_FDT(fdt_end_node(fdt));
+}
+
+static void generate_fdt_nodes(void *fdt, struct kvm *kvm, u32 gic_phandle)
+{
+	generate_cpu_nodes(fdt, kvm);
+	gic__generate_fdt_nodes(fdt, gic_phandle);
+	generate_timer_nodes(fdt, kvm);
+}
+
+
+static int cortex_a57__vcpu_init(struct kvm_cpu *vcpu)
+{
+	vcpu->generate_fdt_nodes = generate_fdt_nodes;
+	return 0;
+}
+
+static struct kvm_arm_target target_cortex_a57 = {
+	.id	= KVM_ARM_TARGET_CORTEX_A57,
+	.init	= cortex_a57__vcpu_init,
+};
+
+static int cortex_a57__core_init(struct kvm *kvm)
+{
+	return kvm_cpu__register_kvm_arm_target(&target_cortex_a57);
+}
+core_init(cortex_a57__core_init);
diff --git a/tools/kvm/arm/aarch64/include/kvm/barrier.h b/tools/kvm/arm/aarch64/include/kvm/barrier.h
new file mode 100644
index 0000000..97ab252
--- /dev/null
+++ b/tools/kvm/arm/aarch64/include/kvm/barrier.h
@@ -0,0 +1,8 @@
+#ifndef KVM__KVM_BARRIER_H
+#define KVM__KVM_BARRIER_H
+
+#define mb()	asm volatile ("dmb ish"		: : : "memory")
+#define rmb()	asm volatile ("dmb ishld"	: : : "memory")
+#define wmb()	asm volatile ("dmb ishst"	: : : "memory")
+
+#endif /* KVM__KVM_BARRIER_H */
diff --git a/tools/kvm/arm/aarch64/include/kvm/kvm-arch.h b/tools/kvm/arm/aarch64/include/kvm/kvm-arch.h
new file mode 100644
index 0000000..2f08a26
--- /dev/null
+++ b/tools/kvm/arm/aarch64/include/kvm/kvm-arch.h
@@ -0,0 +1,17 @@
+#ifndef KVM__KVM_ARCH_H
+#define KVM__KVM_ARCH_H
+
+#define ARM_GIC_DIST_SIZE	0x10000
+#define ARM_GIC_CPUI_SIZE	0x10000
+
+#define ARM_KERN_OFFSET(kvm)	((kvm)->cfg.arch.aarch32_guest	?	\
+				0x8000				:	\
+				0x80000)
+
+#define ARM_MAX_MEMORY(kvm)	((kvm)->cfg.arch.aarch32_guest	?	\
+				ARM_LOMAP_MAX_MEMORY		:	\
+				ARM_HIMAP_MAX_MEMORY)
+
+#include "arm-common/kvm-arch.h"
+
+#endif /* KVM__KVM_ARCH_H */
diff --git a/tools/kvm/arm/aarch64/include/kvm/kvm-config-arch.h b/tools/kvm/arm/aarch64/include/kvm/kvm-config-arch.h
new file mode 100644
index 0000000..89860ae
--- /dev/null
+++ b/tools/kvm/arm/aarch64/include/kvm/kvm-config-arch.h
@@ -0,0 +1,10 @@
+#ifndef KVM__KVM_CONFIG_ARCH_H
+#define KVM__KVM_CONFIG_ARCH_H
+
+#define ARM_OPT_ARCH_RUN(cfg)						\
+	OPT_BOOLEAN('\0', "aarch32", &(cfg)->aarch32_guest,		\
+			"Run AArch32 guest"),
+
+#include "arm-common/kvm-config-arch.h"
+
+#endif /* KVM__KVM_CONFIG_ARCH_H */
diff --git a/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h b/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
new file mode 100644
index 0000000..d85c583
--- /dev/null
+++ b/tools/kvm/arm/aarch64/include/kvm/kvm-cpu-arch.h
@@ -0,0 +1,13 @@
+#ifndef KVM__KVM_CPU_ARCH_H
+#define KVM__KVM_CPU_ARCH_H
+
+#include "kvm/kvm.h"
+
+#include "arm-common/kvm-cpu-arch.h"
+
+#define ARM_VCPU_FEATURE_FLAGS(kvm, cpuid)	{				\
+	[0] = ((!!(cpuid) << KVM_ARM_VCPU_POWER_OFF) |				\
+	       (!!(kvm)->cfg.arch.aarch32_guest << KVM_ARM_VCPU_EL1_32BIT))	\
+}
+
+#endif /* KVM__KVM_CPU_ARCH_H */
diff --git a/tools/kvm/arm/aarch64/kvm-cpu.c b/tools/kvm/arm/aarch64/kvm-cpu.c
new file mode 100644
index 0000000..2eb06ea
--- /dev/null
+++ b/tools/kvm/arm/aarch64/kvm-cpu.c
@@ -0,0 +1,160 @@
+#include "kvm/kvm-cpu.h"
+#include "kvm/kvm.h"
+
+#include <asm/ptrace.h>
+
+#define COMPAT_PSR_F_BIT	0x00000040
+#define COMPAT_PSR_I_BIT	0x00000080
+#define COMPAT_PSR_MODE_SVC	0x00000013
+
+#define ARM64_CORE_REG(x)	(KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
+				 KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
+
+static void reset_vcpu_aarch32(struct kvm_cpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_one_reg reg;
+	u64 data;
+
+	reg.addr = (u64)&data;
+
+	/* pstate = all interrupts masked */
+	data	= COMPAT_PSR_I_BIT | COMPAT_PSR_F_BIT | COMPAT_PSR_MODE_SVC;
+	reg.id	= ARM64_CORE_REG(regs.pstate);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (spsr[EL1])");
+
+	/* Secondary cores are stopped awaiting PSCI wakeup */
+	if (vcpu->cpu_id != 0)
+		return;
+
+	/* r0 = 0 */
+	data	= 0;
+	reg.id	= ARM64_CORE_REG(regs.regs[0]);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (r0)");
+
+	/* r1 = machine type (-1) */
+	data	= -1;
+	reg.id	= ARM64_CORE_REG(regs.regs[1]);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (r1)");
+
+	/* r2 = physical address of the device tree blob */
+	data	= kvm->arch.dtb_guest_start;
+	reg.id	= ARM64_CORE_REG(regs.regs[2]);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (r2)");
+
+	/* pc = start of kernel image */
+	data	= kvm->arch.kern_guest_start;
+	reg.id	= ARM64_CORE_REG(regs.pc);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (pc)");
+}
+
+static void reset_vcpu_aarch64(struct kvm_cpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_one_reg reg;
+	u64 data;
+
+	reg.addr = (u64)&data;
+
+	/* pstate = all interrupts masked */
+	data	= PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL1h;
+	reg.id	= ARM64_CORE_REG(regs.pstate);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (spsr[EL1])");
+
+	/* x1...x3 = 0 */
+	data	= 0;
+	reg.id	= ARM64_CORE_REG(regs.regs[1]);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (x1)");
+
+	reg.id	= ARM64_CORE_REG(regs.regs[2]);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (x2)");
+
+	reg.id	= ARM64_CORE_REG(regs.regs[3]);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (x3)");
+
+	/* Secondary cores are stopped awaiting PSCI wakeup */
+	if (vcpu->cpu_id == 0) {
+		/* x0 = physical address of the device tree blob */
+		data	= kvm->arch.dtb_guest_start;
+		reg.id	= ARM64_CORE_REG(regs.regs[0]);
+		if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+			die_perror("KVM_SET_ONE_REG failed (x0)");
+
+		/* pc = start of kernel image */
+		data	= kvm->arch.kern_guest_start;
+		reg.id	= ARM64_CORE_REG(regs.pc);
+		if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+			die_perror("KVM_SET_ONE_REG failed (pc)");
+	}
+}
+
+void kvm_cpu__reset_vcpu(struct kvm_cpu *vcpu)
+{
+	if (vcpu->kvm->cfg.arch.aarch32_guest)
+		return reset_vcpu_aarch32(vcpu);
+	else
+		return reset_vcpu_aarch64(vcpu);
+}
+
+void kvm_cpu__show_code(struct kvm_cpu *vcpu)
+{
+	struct kvm_one_reg reg;
+	unsigned long data;
+
+	reg.addr = (u64)&data;
+
+	printf("*pc:\n");
+	reg.id = ARM64_CORE_REG(regs.pc);
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
+		die("KVM_GET_ONE_REG failed (show_code @ PC)");
+
+	kvm__dump_mem(vcpu->kvm, data, 32);
+	printf("\n");
+
+	printf("*lr:\n");
+	reg.id = ARM64_CORE_REG(regs.regs[30]);
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
+		die("KVM_GET_ONE_REG failed (show_code @ LR)");
+
+	kvm__dump_mem(vcpu->kvm, data, 32);
+	printf("\n");
+}
+
+void kvm_cpu__show_registers(struct kvm_cpu *vcpu)
+{
+	struct kvm_one_reg reg;
+	unsigned long data;
+	int debug_fd = kvm_cpu__get_debug_fd();
+
+	reg.addr = (u64)&data;
+	dprintf(debug_fd, "\n Registers:\n");
+
+	reg.id		= ARM64_CORE_REG(regs.pc);
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
+		die("KVM_GET_ONE_REG failed (pc)");
+	dprintf(debug_fd, " PC:    0x%lx\n", data);
+
+	reg.id		= ARM64_CORE_REG(regs.pstate);
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
+		die("KVM_GET_ONE_REG failed (pstate)");
+	dprintf(debug_fd, " PSTATE:    0x%lx\n", data);
+
+	reg.id		= ARM64_CORE_REG(sp_el1);
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
+		die("KVM_GET_ONE_REG failed (sp_el1)");
+	dprintf(debug_fd, " SP_EL1:    0x%lx\n", data);
+
+	reg.id		= ARM64_CORE_REG(regs.regs[30]);
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
+		die("KVM_GET_ONE_REG failed (lr)");
+	dprintf(debug_fd, " LR:    0x%lx\n", data);
+}
diff --git a/tools/kvm/arm/fdt.c b/tools/kvm/arm/fdt.c
index 6c12e79..20e0308 100644
--- a/tools/kvm/arm/fdt.c
+++ b/tools/kvm/arm/fdt.c
@@ -178,7 +178,7 @@ int load_flat_binary(struct kvm *kvm, int fd_kernel, int fd_initrd,
 	 */
 	limit = kvm->ram_start + min(kvm->ram_size, (u64)SZ_256M) - 1;
 
-	pos = kvm->ram_start + ARM_KERN_OFFSET;
+	pos = kvm->ram_start + ARM_KERN_OFFSET(kvm);
 	kvm->arch.kern_guest_start = host_to_guest_flat(kvm, pos);
 	if (read_image(fd_kernel, &pos, limit) == -ENOMEM)
 		die("kernel image too big to contain in guest memory.");
diff --git a/tools/kvm/arm/include/arm-common/kvm-arch.h b/tools/kvm/arm/include/arm-common/kvm-arch.h
index 798af59..46ee7e2 100644
--- a/tools/kvm/arm/include/arm-common/kvm-arch.h
+++ b/tools/kvm/arm/include/arm-common/kvm-arch.h
@@ -1,11 +1,29 @@
 #ifndef ARM_COMMON__KVM_ARCH_H
 #define ARM_COMMON__KVM_ARCH_H
 
-#define VIRTIO_DEFAULT_TRANS	VIRTIO_MMIO
-
 #include <stdbool.h>
+#include <linux/const.h>
 #include <linux/types.h>
 
+#define ARM_MMIO_AREA		_AC(0x0000000000000000, UL)
+#define ARM_AXI_AREA		_AC(0x0000000040000000, UL)
+#define ARM_MEMORY_AREA		_AC(0x0000000080000000, UL)
+
+#define ARM_LOMAP_MAX_MEMORY	((1ULL << 32) - ARM_MEMORY_AREA)
+#define ARM_HIMAP_MAX_MEMORY	((1ULL << 40) - ARM_MEMORY_AREA)
+
+#define ARM_GIC_DIST_BASE	(ARM_AXI_AREA - ARM_GIC_DIST_SIZE)
+#define ARM_GIC_CPUI_BASE	(ARM_GIC_DIST_BASE - ARM_GIC_CPUI_SIZE)
+#define ARM_GIC_SIZE		(ARM_GIC_DIST_SIZE + ARM_GIC_CPUI_SIZE)
+
+#define ARM_VIRTIO_MMIO_SIZE	(ARM_AXI_AREA - ARM_GIC_SIZE)
+#define ARM_PCI_MMIO_SIZE	(ARM_MEMORY_AREA - ARM_AXI_AREA)
+
+#define KVM_PCI_MMIO_AREA	ARM_AXI_AREA
+#define KVM_VIRTIO_MMIO_AREA	ARM_MMIO_AREA
+
+#define VIRTIO_DEFAULT_TRANS	VIRTIO_MMIO
+
 static inline bool arm_addr_in_virtio_mmio_region(u64 phys_addr)
 {
 	u64 limit = KVM_VIRTIO_MMIO_AREA + ARM_VIRTIO_MMIO_SIZE;
diff --git a/tools/kvm/arm/include/kvm/kvm-config-arch.h b/tools/kvm/arm/include/arm-common/kvm-config-arch.h
similarity index 61%
rename from tools/kvm/arm/include/kvm/kvm-config-arch.h
rename to tools/kvm/arm/include/arm-common/kvm-config-arch.h
index f63f302..7ac6f6e 100644
--- a/tools/kvm/arm/include/kvm/kvm-config-arch.h
+++ b/tools/kvm/arm/include/arm-common/kvm-config-arch.h
@@ -1,15 +1,17 @@
-#ifndef KVM__KVM_CONFIG_ARCH_H
-#define KVM__KVM_CONFIG_ARCH_H
+#ifndef ARM_COMMON__KVM_CONFIG_ARCH_H
+#define ARM_COMMON__KVM_CONFIG_ARCH_H
 
 #include "kvm/parse-options.h"
 
 struct kvm_config_arch {
 	const char *dump_dtb_filename;
+	bool aarch32_guest;
 };
 
 #define OPT_ARCH_RUN(pfx, cfg)						\
 	pfx,								\
+	ARM_OPT_ARCH_RUN(cfg)						\
 	OPT_STRING('\0', "dump-dtb", &(cfg)->dump_dtb_filename,		\
 		   ".dtb file", "Dump generated .dtb to specified file"),
 
-#endif /* KVM__KVM_CONFIG_ARCH_H */
+#endif /* ARM_COMMON__KVM_CONFIG_ARCH_H */
diff --git a/tools/kvm/arm/kvm.c b/tools/kvm/arm/kvm.c
index b10c857..9eff927 100644
--- a/tools/kvm/arm/kvm.c
+++ b/tools/kvm/arm/kvm.c
@@ -57,7 +57,7 @@ void kvm__arch_set_cmdline(char *cmdline, bool video)
 void kvm__arch_init(struct kvm *kvm, const char *hugetlbfs_path, u64 ram_size)
 {
 	/* Allocate guest memory. */
-	kvm->ram_size = min(ram_size, (u64)ARM_MAX_MEMORY);
+	kvm->ram_size = min(ram_size, (u64)ARM_MAX_MEMORY(kvm));
 	kvm->ram_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path, kvm->ram_size);
 	if (kvm->ram_start == MAP_FAILED)
 		die("Failed to map %lld bytes for guest memory (%d)",
-- 
1.8.0

[PATCH 2/2] kvm tools: arm: align guest memory buffer to maximum page size

[Will Deacon]

If we're running a guest with a larger page size than the host,
interesting things start to happen when communicating via a virtio-mmio
device because the idea of buffer alignment between the guest and the
host will be off by the misalignment of the guest memory buffer allocated
by the host. This causes things like the index field of vring.used to
be accessed at different addresses on the guest and the host, leading
to deadlock.

Fix this problem by allocating guest memory aligned to the maximum
possible page size for the architecture (64K).

Signed-off-by: Will Deacon <will.deacon@arm.com>
---
 tools/kvm/arm/include/arm-common/kvm-arch.h | 10 ++++++++++
 tools/kvm/arm/kvm.c                         | 24 ++++++++++++++++++------
 2 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/tools/kvm/arm/include/arm-common/kvm-arch.h b/tools/kvm/arm/include/arm-common/kvm-arch.h
index 46ee7e2..7860e17 100644
--- a/tools/kvm/arm/include/arm-common/kvm-arch.h
+++ b/tools/kvm/arm/include/arm-common/kvm-arch.h
@@ -37,6 +37,16 @@ static inline bool arm_addr_in_pci_mmio_region(u64 phys_addr)
 }
 
 struct kvm_arch {
+	/*
+	 * We may have to align the guest memory for virtio, so keep the
+	 * original pointers here for munmap.
+	 */
+	void	*ram_alloc_start;
+	u64	ram_alloc_size;
+
+	/*
+	 * Guest addresses for memory layout.
+	 */
 	u64	memory_guest_start;
 	u64	kern_guest_start;
 	u64	initrd_guest_start;
diff --git a/tools/kvm/arm/kvm.c b/tools/kvm/arm/kvm.c
index 9eff927..1bcfce3 100644
--- a/tools/kvm/arm/kvm.c
+++ b/tools/kvm/arm/kvm.c
@@ -7,6 +7,7 @@
 
 #include <linux/kernel.h>
 #include <linux/kvm.h>
+#include <linux/sizes.h>
 
 struct kvm_ext kvm_req_ext[] = {
 	{ DEFINE_KVM_EXT(KVM_CAP_IRQCHIP) },
@@ -41,7 +42,7 @@ void kvm__init_ram(struct kvm *kvm)
 
 void kvm__arch_delete_ram(struct kvm *kvm)
 {
-	munmap(kvm->ram_start, kvm->ram_size);
+	munmap(kvm->arch.ram_alloc_start, kvm->arch.ram_alloc_size);
 }
 
 void kvm__arch_periodic_poll(struct kvm *kvm)
@@ -56,13 +57,24 @@ void kvm__arch_set_cmdline(char *cmdline, bool video)
 
 void kvm__arch_init(struct kvm *kvm, const char *hugetlbfs_path, u64 ram_size)
 {
-	/* Allocate guest memory. */
+	/*
+	 * Allocate guest memory. We must align out buffer to 64K to
+	 * correlate with the maximum guest page size for virtio-mmio.
+	 */
 	kvm->ram_size = min(ram_size, (u64)ARM_MAX_MEMORY(kvm));
-	kvm->ram_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path, kvm->ram_size);
-	if (kvm->ram_start == MAP_FAILED)
+	kvm->arch.ram_alloc_size = kvm->ram_size + SZ_64K;
+	kvm->arch.ram_alloc_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path,
+						kvm->arch.ram_alloc_size);
+
+	if (kvm->arch.ram_alloc_start == MAP_FAILED)
 		die("Failed to map %lld bytes for guest memory (%d)",
-		    kvm->ram_size, errno);
-	madvise(kvm->ram_start, kvm->ram_size, MADV_MERGEABLE);
+		    kvm->arch.ram_alloc_size, errno);
+
+	kvm->ram_start = (void *)ALIGN((unsigned long)kvm->arch.ram_alloc_start,
+					SZ_64K);
+
+	madvise(kvm->arch.ram_alloc_start, kvm->arch.ram_alloc_size,
+		MADV_MERGEABLE);
 
 	/* Initialise the virtual GIC. */
 	if (gic__init_irqchip(kvm))
-- 
1.8.0

Re: [PATCH 0/2] Add support for ARMv8 CPUs to kvmtool

[Pekka Enberg]

Hi Will,

On Mon, Jan 7, 2013 at 8:43 PM, Will Deacon <will.deacon@arm.com> wrote:
> Hello again,
>
> These two patches add support for ARMv8 processors running an AArch64 instance
> of kvm to kvmtool. Both AArch32 and AArch64 guests are supported and, in the
> case of the latter, the guest page size may be either 64k or 4k.
>
> This depends on the ARM updates series I just posted:
>
>   https://lists.cs.columbia.edu/pipermail/kvmarm/2013-January/004505.html
>
> Feedback welcome,

So will ARMv7 continue to work after this patch even without the ARM
updates? If so, I'm happy to merge the patch before the updates hit
v3.9.

                        Pekka

Re: [PATCH 0/2] Add support for ARMv8 CPUs to kvmtool

[Will Deacon]

On Wed, Jan 09, 2013 at 11:16:42AM +0000, Pekka Enberg wrote:
> Hi Will,

Hello,

> On Mon, Jan 7, 2013 at 8:43 PM, Will Deacon <will.deacon@arm.com> wrote:
> > These two patches add support for ARMv8 processors running an AArch64 instance
> > of kvm to kvmtool. Both AArch32 and AArch64 guests are supported and, in the
> > case of the latter, the guest page size may be either 64k or 4k.
> >
> > This depends on the ARM updates series I just posted:
> >
> >   https://lists.cs.columbia.edu/pipermail/kvmarm/2013-January/004505.html
> >
> > Feedback welcome,
> 
> So will ARMv7 continue to work after this patch even without the ARM
> updates? If so, I'm happy to merge the patch before the updates hit
> v3.9.

I think we're getting our wires crossed a bit here, so I'll try to explain
my madness:

	- Mainline ARM kernels cannot be booted by kvmtool yet, you
	  currently have to use my virt/mach branch from

	    git://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git

	  I hope to upstream this for 3.9 (it has gone through the necessary
	  review on the ARM kernel list).

	- As part of the review mentioned above, some kvmtool changes were
	  required to support SMP with the latest virt/mach. You just merged
	  these in my ARM updates series.

	- The ARMv8 kvmtool code builds on top of the series you just merged
	  and won't apply without it. Applying it won't have any adverse
	  effects on ARMv7 (AArch32). In fact, the two series sit together
	  on my kvmtool/arm branch from

	    git://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git

	  but I posted them separately to avoid holding up the ARM updates.

Does that answer you question or have I made things worse?!

Cheers,

Will

Re: [PATCH 0/2] Add support for ARMv8 CPUs to kvmtool

[Pekka Enberg]

On Wed, Jan 9, 2013 at 1:33 PM, Will Deacon <will.deacon@arm.com> wrote:
> I think we're getting our wires crossed a bit here, so I'll try to explain
> my madness:
>
>         - Mainline ARM kernels cannot be booted by kvmtool yet, you
>           currently have to use my virt/mach branch from
>
>             git://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git
>
>           I hope to upstream this for 3.9 (it has gone through the necessary
>           review on the ARM kernel list).
>
>         - As part of the review mentioned above, some kvmtool changes were
>           required to support SMP with the latest virt/mach. You just merged
>           these in my ARM updates series.
>
>         - The ARMv8 kvmtool code builds on top of the series you just merged
>           and won't apply without it. Applying it won't have any adverse
>           effects on ARMv7 (AArch32). In fact, the two series sit together
>           on my kvmtool/arm branch from
>
>             git://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git
>
>           but I posted them separately to avoid holding up the ARM updates.
>
> Does that answer you question or have I made things worse?!

Yes, it answers my question. I'll go ahead and apply the ARMv8 patches.