[PATCH 0/4] ARM updates for kvmtool

[PATCH 0/4] ARM updates for kvmtool

[Will Deacon]

Hello kvm hackers,

This patch series introduces some updates to the ARM (AArch32) kvm tools
code:

	- virtio mmio fixes to deal with guest page sizes != 4k (in
	  preparation for AArch64, which I will post separately).
	- .dtb dumping via the lkvm command line
	- Support for PSCI firmware as a replacement to the spin-table
	  based SMP boot code

The last option was implemented after discussion on the linux-arm-kernel
list when adding support for the mach-virt platform. I hope to upstream
the kernel-side part of the implementation for 3.9 and expect the kvm
bits to follow once that has been merged.

All feedback welcome.

Will


Will Deacon (4):
  kvm tools: virtio: remove hardcoded assumptions about guest page size
  kvm tools: pedantry: fix annoying typo
  kvm tools: arm: make .dtb dumping a command-line option
  kvm tools: arm: add support for PSCI firmware in place of spin-tables

 tools/kvm/Makefile                                 |  5 +-
 tools/kvm/arm/aarch32/cortex-a15.c                 |  8 +--
 tools/kvm/arm/aarch32/include/kvm/kvm-arch.h       |  1 -
 tools/kvm/arm/aarch32/include/kvm/kvm-cpu-arch.h   | 12 +++++
 tools/kvm/arm/aarch32/kvm-cpu.c                    | 59 ++++++++++------------
 tools/kvm/arm/aarch32/smp-pen.S                    | 39 --------------
 tools/kvm/arm/fdt.c                                | 54 +++++++-------------
 tools/kvm/arm/include/arm-common/gic.h             |  2 -
 tools/kvm/arm/include/arm-common/kvm-arch.h        |  5 --
 .../arm/include/{kvm => arm-common}/kvm-cpu-arch.h |  6 +--
 tools/kvm/arm/include/kvm/kvm-config-arch.h        |  8 +++
 tools/kvm/arm/kvm-cpu.c                            |  4 +-
 tools/kvm/arm/kvm.c                                |  1 +
 tools/kvm/arm/smp.c                                | 21 --------
 tools/kvm/include/kvm/virtio.h                     | 14 +----
 tools/kvm/kvm.c                                    |  2 +-
 tools/kvm/virtio/9p.c                              |  7 +--
 tools/kvm/virtio/balloon.c                         |  7 +--
 tools/kvm/virtio/blk.c                             |  7 +--
 tools/kvm/virtio/console.c                         |  7 +--
 tools/kvm/virtio/mmio.c                            |  8 +--
 tools/kvm/virtio/net.c                             |  8 +--
 tools/kvm/virtio/pci.c                             |  4 +-
 tools/kvm/virtio/rng.c                             |  7 +--
 tools/kvm/virtio/scsi.c                            |  7 +--
 25 files changed, 114 insertions(+), 189 deletions(-)
 create mode 100644 tools/kvm/arm/aarch32/include/kvm/kvm-cpu-arch.h
 delete mode 100644 tools/kvm/arm/aarch32/smp-pen.S
 rename tools/kvm/arm/include/{kvm => arm-common}/kvm-cpu-arch.h (87%)
 delete mode 100644 tools/kvm/arm/smp.c

-- 
1.8.0

[PATCH 1/4] kvm tools: virtio: remove hardcoded assumptions about guest page size

[Will Deacon]

virtio-based PCI devices deal only with 4k memory granules, making
direct use of the VIRTIO_PCI_VRING_ALIGN and VIRTIO_PCI_QUEUE_ADDR_SHIFT
constants when initialising the virtqueues for a device.

For MMIO-based devices, the guest page size is arbitrary and may differ
from that of the host (this is the case on AArch64, where both 4k and
64k pages are supported).

This patch fixes the virtio drivers to honour the guest page size passed
when configuring the virtio device and align the virtqueues accordingly.

Signed-off-by: Will Deacon <will.deacon@arm.com>
---
 tools/kvm/include/kvm/virtio.h | 14 ++------------
 tools/kvm/virtio/9p.c          |  7 ++++---
 tools/kvm/virtio/balloon.c     |  7 ++++---
 tools/kvm/virtio/blk.c         |  7 ++++---
 tools/kvm/virtio/console.c     |  7 ++++---
 tools/kvm/virtio/mmio.c        |  8 ++++----
 tools/kvm/virtio/net.c         |  8 ++++----
 tools/kvm/virtio/pci.c         |  4 +++-
 tools/kvm/virtio/rng.c         |  7 ++++---
 tools/kvm/virtio/scsi.c        |  7 ++++---
 10 files changed, 37 insertions(+), 39 deletions(-)

diff --git a/tools/kvm/include/kvm/virtio.h b/tools/kvm/include/kvm/virtio.h
index 5dc2544..924279b 100644
--- a/tools/kvm/include/kvm/virtio.h
+++ b/tools/kvm/include/kvm/virtio.h
@@ -43,17 +43,6 @@ static inline bool virt_queue__available(struct virt_queue *vq)
 	return vq->vring.avail->idx !=  vq->last_avail_idx;
 }
 
-/*
- * Warning: on 32-bit hosts, shifting pfn left may cause a truncation of pfn values
- * higher than 4GB - thus, pointing to the wrong area in guest virtual memory space
- * and breaking the virt queue which owns this pfn.
- */
-static inline void *guest_pfn_to_host(struct kvm *kvm, u32 pfn)
-{
-	return guest_flat_to_host(kvm, (unsigned long)pfn << VIRTIO_PCI_QUEUE_ADDR_SHIFT);
-}
-
-
 struct vring_used_elem *virt_queue__set_used_elem(struct virt_queue *queue, u32 head, u32 len);
 
 bool virtio_queue__should_signal(struct virt_queue *vq);
@@ -81,7 +70,8 @@ struct virtio_ops {
 	u8 *(*get_config)(struct kvm *kvm, void *dev);
 	u32 (*get_host_features)(struct kvm *kvm, void *dev);
 	void (*set_guest_features)(struct kvm *kvm, void *dev, u32 features);
-	int (*init_vq)(struct kvm *kvm, void *dev, u32 vq, u32 pfn);
+	int (*init_vq)(struct kvm *kvm, void *dev, u32 vq, u32 page_size,
+		       u32 align, u32 pfn);
 	int (*notify_vq)(struct kvm *kvm, void *dev, u32 vq);
 	int (*get_pfn_vq)(struct kvm *kvm, void *dev, u32 vq);
 	int (*get_size_vq)(struct kvm *kvm, void *dev, u32 vq);
diff --git a/tools/kvm/virtio/9p.c b/tools/kvm/virtio/9p.c
index 4665876..60865dd 100644
--- a/tools/kvm/virtio/9p.c
+++ b/tools/kvm/virtio/9p.c
@@ -1254,7 +1254,8 @@ static void set_guest_features(struct kvm *kvm, void *dev, u32 features)
 	p9dev->features = features;
 }
 
-static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
+static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
+		   u32 pfn)
 {
 	struct p9_dev *p9dev = dev;
 	struct p9_dev_job *job;
@@ -1265,10 +1266,10 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
 
 	queue		= &p9dev->vqs[vq];
 	queue->pfn	= pfn;
-	p		= guest_pfn_to_host(kvm, queue->pfn);
+	p		= guest_flat_to_host(kvm, queue->pfn * page_size);
 	job		= &p9dev->jobs[vq];
 
-	vring_init(&queue->vring, VIRTQUEUE_NUM, p, VIRTIO_PCI_VRING_ALIGN);
+	vring_init(&queue->vring, VIRTQUEUE_NUM, p, align);
 
 	*job		= (struct p9_dev_job) {
 		.vq		= queue,
diff --git a/tools/kvm/virtio/balloon.c b/tools/kvm/virtio/balloon.c
index 9edce87..d1b64fa 100644
--- a/tools/kvm/virtio/balloon.c
+++ b/tools/kvm/virtio/balloon.c
@@ -193,7 +193,8 @@ static void set_guest_features(struct kvm *kvm, void *dev, u32 features)
 	bdev->features = features;
 }
 
-static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
+static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
+		   u32 pfn)
 {
 	struct bln_dev *bdev = dev;
 	struct virt_queue *queue;
@@ -203,10 +204,10 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
 
 	queue		= &bdev->vqs[vq];
 	queue->pfn	= pfn;
-	p		= guest_pfn_to_host(kvm, queue->pfn);
+	p		= guest_flat_to_host(kvm, queue->pfn * page_size);
 
 	thread_pool__init_job(&bdev->jobs[vq], kvm, virtio_bln_do_io, queue);
-	vring_init(&queue->vring, VIRTIO_BLN_QUEUE_SIZE, p, VIRTIO_PCI_VRING_ALIGN);
+	vring_init(&queue->vring, VIRTIO_BLN_QUEUE_SIZE, p, align);
 
 	return 0;
 }
diff --git a/tools/kvm/virtio/blk.c b/tools/kvm/virtio/blk.c
index ec57e96..44ac44b 100644
--- a/tools/kvm/virtio/blk.c
+++ b/tools/kvm/virtio/blk.c
@@ -156,7 +156,8 @@ static void set_guest_features(struct kvm *kvm, void *dev, u32 features)
 	bdev->features = features;
 }
 
-static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
+static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
+		   u32 pfn)
 {
 	struct blk_dev *bdev = dev;
 	struct virt_queue *queue;
@@ -166,9 +167,9 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
 
 	queue		= &bdev->vqs[vq];
 	queue->pfn	= pfn;
-	p		= guest_pfn_to_host(kvm, queue->pfn);
+	p		= guest_flat_to_host(kvm, queue->pfn * page_size);
 
-	vring_init(&queue->vring, VIRTIO_BLK_QUEUE_SIZE, p, VIRTIO_PCI_VRING_ALIGN);
+	vring_init(&queue->vring, VIRTIO_BLK_QUEUE_SIZE, p, align);
 
 	return 0;
 }
diff --git a/tools/kvm/virtio/console.c b/tools/kvm/virtio/console.c
index 74b6ffd..b18d3a9 100644
--- a/tools/kvm/virtio/console.c
+++ b/tools/kvm/virtio/console.c
@@ -128,7 +128,8 @@ static void set_guest_features(struct kvm *kvm, void *dev, u32 features)
 	/* Unused */
 }
 
-static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
+static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
+		   u32 pfn)
 {
 	struct virt_queue *queue;
 	void *p;
@@ -139,9 +140,9 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
 
 	queue		= &cdev.vqs[vq];
 	queue->pfn	= pfn;
-	p		= guest_pfn_to_host(kvm, queue->pfn);
+	p		= guest_flat_to_host(kvm, queue->pfn * page_size);
 
-	vring_init(&queue->vring, VIRTIO_CONSOLE_QUEUE_SIZE, p, VIRTIO_PCI_VRING_ALIGN);
+	vring_init(&queue->vring, VIRTIO_CONSOLE_QUEUE_SIZE, p, align);
 
 	if (vq == VIRTIO_CONSOLE_TX_QUEUE)
 		thread_pool__init_job(&cdev.jobs[vq], kvm, virtio_console_handle_callback, queue);
diff --git a/tools/kvm/virtio/mmio.c b/tools/kvm/virtio/mmio.c
index f94c54f..bd30f37 100644
--- a/tools/kvm/virtio/mmio.c
+++ b/tools/kvm/virtio/mmio.c
@@ -163,25 +163,25 @@ static void virtio_mmio_config_out(u64 addr, void *data, u32 len,
 	case VIRTIO_MMIO_GUEST_PAGE_SIZE:
 		val = ioport__read32(data);
 		vmmio->hdr.guest_page_size = val;
-		/* FIXME: set guest page size */
 		break;
 	case VIRTIO_MMIO_QUEUE_NUM:
 		val = ioport__read32(data);
 		vmmio->hdr.queue_num = val;
-		/* FIXME: set vq size */
 		vdev->ops->set_size_vq(vmmio->kvm, vmmio->dev,
 				       vmmio->hdr.queue_sel, val);
 		break;
 	case VIRTIO_MMIO_QUEUE_ALIGN:
 		val = ioport__read32(data);
 		vmmio->hdr.queue_align = val;
-		/* FIXME: set used ring alignment */
 		break;
 	case VIRTIO_MMIO_QUEUE_PFN:
 		val = ioport__read32(data);
 		virtio_mmio_init_ioeventfd(vmmio->kvm, vdev, vmmio->hdr.queue_sel);
 		vdev->ops->init_vq(vmmio->kvm, vmmio->dev,
-				   vmmio->hdr.queue_sel, val);
+				   vmmio->hdr.queue_sel,
+				   vmmio->hdr.guest_page_size,
+				   vmmio->hdr.queue_align,
+				   val);
 		break;
 	case VIRTIO_MMIO_QUEUE_NOTIFY:
 		val = ioport__read32(data);
diff --git a/tools/kvm/virtio/net.c b/tools/kvm/virtio/net.c
index db77ab8..68bd107 100644
--- a/tools/kvm/virtio/net.c
+++ b/tools/kvm/virtio/net.c
@@ -320,7 +320,8 @@ static void set_guest_features(struct kvm *kvm, void *dev, u32 features)
 	ndev->features = features;
 }
 
-static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
+static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
+		   u32 pfn)
 {
 	struct vhost_vring_state state = { .index = vq };
 	struct vhost_vring_addr addr;
@@ -333,10 +334,9 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
 
 	queue		= &ndev->vqs[vq];
 	queue->pfn	= pfn;
-	p		= guest_pfn_to_host(kvm, queue->pfn);
+	p		= guest_flat_to_host(kvm, queue->pfn * page_size);
 
-	/* FIXME: respect pci and mmio vring alignment */
-	vring_init(&queue->vring, VIRTIO_NET_QUEUE_SIZE, p, VIRTIO_PCI_VRING_ALIGN);
+	vring_init(&queue->vring, VIRTIO_NET_QUEUE_SIZE, p, align);
 
 	if (ndev->vhost_fd == 0)
 		return 0;
diff --git a/tools/kvm/virtio/pci.c b/tools/kvm/virtio/pci.c
index 3d3e599..227d567 100644
--- a/tools/kvm/virtio/pci.c
+++ b/tools/kvm/virtio/pci.c
@@ -197,7 +197,9 @@ static bool virtio_pci__io_out(struct ioport *ioport, struct kvm *kvm, u16 port,
 	case VIRTIO_PCI_QUEUE_PFN:
 		val = ioport__read32(data);
 		virtio_pci__init_ioeventfd(kvm, vdev, vpci->queue_selector);
-		vdev->ops->init_vq(kvm, vpci->dev, vpci->queue_selector, val);
+		vdev->ops->init_vq(kvm, vpci->dev, vpci->queue_selector,
+				   1 << VIRTIO_PCI_QUEUE_ADDR_SHIFT,
+				   VIRTIO_PCI_VRING_ALIGN, val);
 		break;
 	case VIRTIO_PCI_QUEUE_SEL:
 		vpci->queue_selector = ioport__read16(data);
diff --git a/tools/kvm/virtio/rng.c b/tools/kvm/virtio/rng.c
index 2ddfb28..2ce8afd 100644
--- a/tools/kvm/virtio/rng.c
+++ b/tools/kvm/virtio/rng.c
@@ -86,7 +86,8 @@ static void virtio_rng_do_io(struct kvm *kvm, void *param)
 	rdev->vdev.ops->signal_vq(kvm, &rdev->vdev, vq - rdev->vqs);
 }
 
-static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
+static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
+		   u32 pfn)
 {
 	struct rng_dev *rdev = dev;
 	struct virt_queue *queue;
@@ -97,11 +98,11 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
 
 	queue		= &rdev->vqs[vq];
 	queue->pfn	= pfn;
-	p		= guest_pfn_to_host(kvm, queue->pfn);
+	p		= guest_flat_to_host(kvm, queue->pfn * page_size);
 
 	job = &rdev->jobs[vq];
 
-	vring_init(&queue->vring, VIRTIO_RNG_QUEUE_SIZE, p, VIRTIO_PCI_VRING_ALIGN);
+	vring_init(&queue->vring, VIRTIO_RNG_QUEUE_SIZE, p, align);
 
 	*job = (struct rng_dev_job) {
 		.vq	= queue,
diff --git a/tools/kvm/virtio/scsi.c b/tools/kvm/virtio/scsi.c
index 7ba76d0..05b2dc6 100644
--- a/tools/kvm/virtio/scsi.c
+++ b/tools/kvm/virtio/scsi.c
@@ -49,7 +49,8 @@ static void set_guest_features(struct kvm *kvm, void *dev, u32 features)
 	sdev->features = features;
 }
 
-static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
+static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 page_size, u32 align,
+		   u32 pfn)
 {
 	struct vhost_vring_state state = { .index = vq };
 	struct vhost_vring_addr addr;
@@ -62,9 +63,9 @@ static int init_vq(struct kvm *kvm, void *dev, u32 vq, u32 pfn)
 
 	queue		= &sdev->vqs[vq];
 	queue->pfn	= pfn;
-	p		= guest_pfn_to_host(kvm, queue->pfn);
+	p		= guest_flat_to_host(kvm, queue->pfn * page_size);
 
-	vring_init(&queue->vring, VIRTIO_SCSI_QUEUE_SIZE, p, VIRTIO_PCI_VRING_ALIGN);
+	vring_init(&queue->vring, VIRTIO_SCSI_QUEUE_SIZE, p, align);
 
 	if (sdev->vhost_fd == 0)
 		return 0;
-- 
1.8.0

[PATCH 2/4] kvm tools: pedantry: fix annoying typo

[Will Deacon]

s/extention/extension/

I should get out more...

Signed-off-by: Will Deacon <will.deacon@arm.com>
---
 tools/kvm/kvm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/kvm/kvm.c b/tools/kvm/kvm.c
index 3ea6339..a6b3c23 100644
--- a/tools/kvm/kvm.c
+++ b/tools/kvm/kvm.c
@@ -291,7 +291,7 @@ int kvm__init(struct kvm *kvm)
 	}
 
 	if (kvm__check_extensions(kvm)) {
-		pr_err("A required KVM extention is not supported by OS");
+		pr_err("A required KVM extension is not supported by OS");
 		ret = -ENOSYS;
 		goto err_vm_fd;
 	}
-- 
1.8.0

[PATCH 3/4] kvm tools: arm: make .dtb dumping a command-line option

[Will Deacon]

It can sometimes be useful to dump the .dtb file generated by kvmtool
when debugging a guest. Currently, this is achieved by rebuilding the
tool and changing some #defines, which is fairly clumsy to use.

This patch adds a new command-line option for ARM, allowing the dtb
to be dumped to a named file at runtime.

Signed-off-by: Will Deacon <will.deacon@arm.com>
---
 tools/kvm/arm/fdt.c                         | 18 ++++++------------
 tools/kvm/arm/include/kvm/kvm-config-arch.h |  8 ++++++++
 2 files changed, 14 insertions(+), 12 deletions(-)

diff --git a/tools/kvm/arm/fdt.c b/tools/kvm/arm/fdt.c
index c7f4b52..e52c10c 100644
--- a/tools/kvm/arm/fdt.c
+++ b/tools/kvm/arm/fdt.c
@@ -13,9 +13,6 @@
 #include <linux/kernel.h>
 #include <linux/sizes.h>
 
-#define DEBUG			0
-#define DEBUG_FDT_DUMP_FILE	"/tmp/kvmtool.dtb"
-
 static char kern_cmdline[COMMAND_LINE_SIZE];
 
 bool kvm__load_firmware(struct kvm *kvm, const char *firmware_filename)
@@ -28,25 +25,21 @@ int kvm__arch_setup_firmware(struct kvm *kvm)
 	return 0;
 }
 
-#if DEBUG
-static void dump_fdt(void *fdt)
+static void dump_fdt(const char *dtb_file, void *fdt)
 {
 	int count, fd;
 
-	fd = open(DEBUG_FDT_DUMP_FILE, O_CREAT | O_TRUNC | O_RDWR, 0666);
+	fd = open(dtb_file, O_CREAT | O_TRUNC | O_RDWR, 0666);
 	if (fd < 0)
-		die("Failed to write dtb to %s", DEBUG_FDT_DUMP_FILE);
+		die("Failed to write dtb to %s", dtb_file);
 
 	count = write(fd, fdt, FDT_MAX_SIZE);
 	if (count < 0)
 		die_perror("Failed to dump dtb");
 
-	pr_info("Wrote %d bytes to dtb %s\n", count, DEBUG_FDT_DUMP_FILE);
+	pr_info("Wrote %d bytes to dtb %s\n", count, dtb_file);
 	close(fd);
 }
-#else
-static void dump_fdt(void *fdt) { }
-#endif
 
 #define DEVICE_NAME_MAX_LEN 32
 static void generate_virtio_mmio_node(void *fdt, struct virtio_mmio *vmmio)
@@ -143,7 +136,8 @@ static int setup_fdt(struct kvm *kvm)
 	_FDT(fdt_open_into(fdt, fdt_dest, FDT_MAX_SIZE));
 	_FDT(fdt_pack(fdt_dest));
 
-	dump_fdt(fdt_dest);
+	if (kvm->cfg.arch.dump_dtb_filename)
+		dump_fdt(kvm->cfg.arch.dump_dtb_filename, fdt_dest);
 	return 0;
 }
 late_init(setup_fdt);
diff --git a/tools/kvm/arm/include/kvm/kvm-config-arch.h b/tools/kvm/arm/include/kvm/kvm-config-arch.h
index 60f61de..f63f302 100644
--- a/tools/kvm/arm/include/kvm/kvm-config-arch.h
+++ b/tools/kvm/arm/include/kvm/kvm-config-arch.h
@@ -1,7 +1,15 @@
 #ifndef KVM__KVM_CONFIG_ARCH_H
 #define KVM__KVM_CONFIG_ARCH_H
 
+#include "kvm/parse-options.h"
+
 struct kvm_config_arch {
+	const char *dump_dtb_filename;
 };
 
+#define OPT_ARCH_RUN(pfx, cfg)						\
+	pfx,								\
+	OPT_STRING('\0', "dump-dtb", &(cfg)->dump_dtb_filename,		\
+		   ".dtb file", "Dump generated .dtb to specified file"),
+
 #endif /* KVM__KVM_CONFIG_ARCH_H */
-- 
1.8.0

[PATCH 4/4] kvm tools: arm: add support for PSCI firmware in place of spin-tables

[Will Deacon]

ARM has recently published a document describing a firmware interface
for CPU power management, which can be used for booting secondary cores
on an SMP platform, amongst other things. As part of the mach-virt
upstreaming for the kernel (that is, the virtual platform targetted by
kvmtool), it was suggested that we use this interface instead of the
current spin-table based approach.

This patch implements PSCI support in kvmtool for ARM, removing a fair
amount of code in the process.

Signed-off-by: Will Deacon <will.deacon@arm.com>
---
 tools/kvm/Makefile                                 |  5 +-
 tools/kvm/arm/aarch32/cortex-a15.c                 |  8 +--
 tools/kvm/arm/aarch32/include/kvm/kvm-arch.h       |  1 -
 tools/kvm/arm/aarch32/include/kvm/kvm-cpu-arch.h   | 12 +++++
 tools/kvm/arm/aarch32/kvm-cpu.c                    | 59 ++++++++++------------
 tools/kvm/arm/aarch32/smp-pen.S                    | 39 --------------
 tools/kvm/arm/fdt.c                                | 36 +++++--------
 tools/kvm/arm/include/arm-common/gic.h             |  2 -
 tools/kvm/arm/include/arm-common/kvm-arch.h        |  5 --
 .../arm/include/{kvm => arm-common}/kvm-cpu-arch.h |  6 +--
 tools/kvm/arm/kvm-cpu.c                            |  4 +-
 tools/kvm/arm/kvm.c                                |  1 +
 tools/kvm/arm/smp.c                                | 21 --------
 13 files changed, 62 insertions(+), 137 deletions(-)
 create mode 100644 tools/kvm/arm/aarch32/include/kvm/kvm-cpu-arch.h
 delete mode 100644 tools/kvm/arm/aarch32/smp-pen.S
 rename tools/kvm/arm/include/{kvm => arm-common}/kvm-cpu-arch.h (87%)
 delete mode 100644 tools/kvm/arm/smp.c

diff --git a/tools/kvm/Makefile b/tools/kvm/Makefile
index a83dd10..33aa4d8 100644
--- a/tools/kvm/Makefile
+++ b/tools/kvm/Makefile
@@ -160,18 +160,15 @@ endif
 
 # ARM
 OBJS_ARM_COMMON		:= arm/fdt.o arm/gic.o arm/ioport.o arm/irq.o \
-			   arm/kvm.o arm/kvm-cpu.o arm/smp.o
+			   arm/kvm.o arm/kvm-cpu.o
 HDRS_ARM_COMMON		:= arm/include
 ifeq ($(ARCH), arm)
 	DEFINES		+= -DCONFIG_ARM
 	OBJS		+= $(OBJS_ARM_COMMON)
 	OBJS		+= arm/aarch32/cortex-a15.o
 	OBJS		+= arm/aarch32/kvm-cpu.o
-	OBJS       	+= arm/aarch32/smp-pen.o
 	ARCH_INCLUDE	:= $(HDRS_ARM_COMMON)
 	ARCH_INCLUDE	+= -Iarm/aarch32/include
-	ASFLAGS		+= -D__ASSEMBLY__
-	ASFLAGS		+= -I$(ARCH_INCLUDE)
 	CFLAGS		+= -march=armv7-a
 	CFLAGS		+= -I../../scripts/dtc/libfdt
 	OTHEROBJS	+= $(LIBFDT_OBJS)
diff --git a/tools/kvm/arm/aarch32/cortex-a15.c b/tools/kvm/arm/aarch32/cortex-a15.c
index eac0bb9..8031747 100644
--- a/tools/kvm/arm/aarch32/cortex-a15.c
+++ b/tools/kvm/arm/aarch32/cortex-a15.c
@@ -31,12 +31,8 @@ static void generate_cpu_nodes(void *fdt, struct kvm *kvm)
 		_FDT(fdt_property_string(fdt, "device_type", "cpu"));
 		_FDT(fdt_property_string(fdt, "compatible", "arm,cortex-a15"));
 
-		if (kvm->nrcpus > 1) {
-			_FDT(fdt_property_string(fdt, "enable-method",
-						 "spin-table"));
-			_FDT(fdt_property_cell(fdt, "cpu-release-addr",
-					       kvm->arch.smp_jump_guest_start));
-		}
+		if (kvm->nrcpus > 1)
+			_FDT(fdt_property_string(fdt, "enable-method", "psci"));
 
 		_FDT(fdt_property_cell(fdt, "reg", cpu));
 		_FDT(fdt_end_node(fdt));
diff --git a/tools/kvm/arm/aarch32/include/kvm/kvm-arch.h b/tools/kvm/arm/aarch32/include/kvm/kvm-arch.h
index f236895..ca79b24 100644
--- a/tools/kvm/arm/aarch32/include/kvm/kvm-arch.h
+++ b/tools/kvm/arm/aarch32/include/kvm/kvm-arch.h
@@ -15,7 +15,6 @@
 
 #define ARM_KERN_OFFSET		0x8000
 
-#define ARM_SMP_PEN_SIZE	PAGE_SIZE
 #define ARM_VIRTIO_MMIO_SIZE	(ARM_GIC_DIST_BASE - ARM_LOMAP_MMIO_AREA)
 #define ARM_PCI_MMIO_SIZE	(ARM_LOMAP_MEMORY_AREA - ARM_LOMAP_AXI_AREA)
 
diff --git a/tools/kvm/arm/aarch32/include/kvm/kvm-cpu-arch.h b/tools/kvm/arm/aarch32/include/kvm/kvm-cpu-arch.h
new file mode 100644
index 0000000..b9fda07
--- /dev/null
+++ b/tools/kvm/arm/aarch32/include/kvm/kvm-cpu-arch.h
@@ -0,0 +1,12 @@
+#ifndef KVM__KVM_CPU_ARCH_H
+#define KVM__KVM_CPU_ARCH_H
+
+#include "kvm/kvm.h"
+
+#include "arm-common/kvm-cpu-arch.h"
+
+#define ARM_VCPU_FEATURE_FLAGS(kvm, cpuid)	{			\
+	[0] = (!!(cpuid) << KVM_ARM_VCPU_POWER_OFF),			\
+}
+
+#endif /* KVM__KVM_CPU_ARCH_H */
diff --git a/tools/kvm/arm/aarch32/kvm-cpu.c b/tools/kvm/arm/aarch32/kvm-cpu.c
index f00a2f1..a528789 100644
--- a/tools/kvm/arm/aarch32/kvm-cpu.c
+++ b/tools/kvm/arm/aarch32/kvm-cpu.c
@@ -21,38 +21,33 @@ void kvm_cpu__reset_vcpu(struct kvm_cpu *vcpu)
 	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
 		die_perror("KVM_SET_ONE_REG failed (cpsr)");
 
-	if (vcpu->cpu_id == 0) {
-		/* r0 = 0 */
-		data	= 0;
-		reg.id	= ARM_CORE_REG(usr_regs.ARM_r0);
-		if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
-			die_perror("KVM_SET_ONE_REG failed (r0)");
-
-		/* r1 = machine type (-1) */
-		data	= -1;
-		reg.id	= ARM_CORE_REG(usr_regs.ARM_r1);
-		if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
-			die_perror("KVM_SET_ONE_REG failed (r1)");
-
-		/* r2 = physical address of the device tree blob */
-		data	= kvm->arch.dtb_guest_start;
-		reg.id	= ARM_CORE_REG(usr_regs.ARM_r2);
-		if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
-			die_perror("KVM_SET_ONE_REG failed (r2)");
-
-		/* pc = start of kernel image */
-		data	= kvm->arch.kern_guest_start;
-		reg.id	= ARM_CORE_REG(usr_regs.ARM_pc);
-		if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
-			die_perror("KVM_SET_ONE_REG failed (pc)");
-
-	} else {
-		/* Simply enter the pen */
-		data	= kvm->arch.smp_pen_guest_start;
-		reg.id	= ARM_CORE_REG(usr_regs.ARM_pc);
-		if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
-			die_perror("KVM_SET_ONE_REG failed (SMP pc)");
-	}
+	/* Secondary cores are stopped awaiting PSCI wakeup */
+	if (vcpu->cpu_id != 0)
+		return;
+
+	/* r0 = 0 */
+	data	= 0;
+	reg.id	= ARM_CORE_REG(usr_regs.ARM_r0);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (r0)");
+
+	/* r1 = machine type (-1) */
+	data	= -1;
+	reg.id	= ARM_CORE_REG(usr_regs.ARM_r1);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (r1)");
+
+	/* r2 = physical address of the device tree blob */
+	data	= kvm->arch.dtb_guest_start;
+	reg.id	= ARM_CORE_REG(usr_regs.ARM_r2);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (r2)");
+
+	/* pc = start of kernel image */
+	data	= kvm->arch.kern_guest_start;
+	reg.id	= ARM_CORE_REG(usr_regs.ARM_pc);
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg) < 0)
+		die_perror("KVM_SET_ONE_REG failed (pc)");
 }
 
 void kvm_cpu__show_code(struct kvm_cpu *vcpu)
diff --git a/tools/kvm/arm/aarch32/smp-pen.S b/tools/kvm/arm/aarch32/smp-pen.S
deleted file mode 100644
index 1e63c95..0000000
--- a/tools/kvm/arm/aarch32/smp-pen.S
+++ /dev/null
@@ -1,39 +0,0 @@
-#include "kvm/kvm-arch.h"
-
-#include "arm-common/gic.h"
-
-#define AARCH32_SMP_BAD_MAGIC	0xdeadc0de
-
-	.arm
-
-	.globl	smp_pen_start
-	.globl	smp_jump_addr
-	.globl	smp_pen_end
-
-	.align
-smp_pen_start:
-	@ Ensure that the CPU interface is enabled for the wfi wakeup
-	ldr	r0, =ARM_GIC_CPUI_BASE
-	mov	r1, #GIC_CPUI_CTLR_EN
-	str	r1, [r0]
-
-	@ Set the priority mask to accept any interrupt
-	mov	r1, #GIC_CPUI_PMR_MIN_PRIO
-	str	r1, [r0, #GIC_CPUI_OFF_PMR]
-
-	@ Now wait for the primary to poke us
-	adr	r0, smp_jump_addr
-	ldr	r1, =AARCH32_SMP_BAD_MAGIC
-	dsb
-1:	wfi
-	ldr	r2, [r0]
-	cmp	r1, r2
-	beq	1b
-	mov	pc, r2
-
-	.ltorg
-
-	.align
-smp_jump_addr:
-	.long	AARCH32_SMP_BAD_MAGIC
-smp_pen_end:
diff --git a/tools/kvm/arm/fdt.c b/tools/kvm/arm/fdt.c
index e52c10c..6c12e79 100644
--- a/tools/kvm/arm/fdt.c
+++ b/tools/kvm/arm/fdt.c
@@ -82,10 +82,6 @@ static int setup_fdt(struct kvm *kvm)
 
 	/* Create new tree without a reserve map */
 	_FDT(fdt_create(fdt, FDT_MAX_SIZE));
-	if (kvm->nrcpus > 1)
-		_FDT(fdt_add_reservemap_entry(fdt,
-					      kvm->arch.smp_pen_guest_start,
-					      ARM_SMP_PEN_SIZE));
 	_FDT(fdt_finish_reservemap(fdt));
 
 	/* Header */
@@ -129,6 +125,16 @@ static int setup_fdt(struct kvm *kvm)
 		dev_hdr = device__next_dev(dev_hdr);
 	}
 
+	/* PSCI firmware */
+	_FDT(fdt_begin_node(fdt, "psci"));
+	_FDT(fdt_property_string(fdt, "compatible", "arm,psci"));
+	_FDT(fdt_property_string(fdt, "method", "hvc"));
+	_FDT(fdt_property_cell(fdt, "cpu_suspend", KVM_PSCI_FN_CPU_SUSPEND));
+	_FDT(fdt_property_cell(fdt, "cpu_off", KVM_PSCI_FN_CPU_OFF));
+	_FDT(fdt_property_cell(fdt, "cpu_on", KVM_PSCI_FN_CPU_ON));
+	_FDT(fdt_property_cell(fdt, "migrate", KVM_PSCI_FN_MIGRATE));
+	_FDT(fdt_end_node(fdt));
+
 	/* Finalise. */
 	_FDT(fdt_end_node(fdt));
 	_FDT(fdt_finish(fdt));
@@ -157,7 +163,6 @@ static int read_image(int fd, void **pos, void *limit)
 
 #define FDT_ALIGN	SZ_2M
 #define INITRD_ALIGN	4
-#define SMP_PEN_ALIGN	PAGE_SIZE
 int load_flat_binary(struct kvm *kvm, int fd_kernel, int fd_initrd,
 		     const char *kernel_cmdline)
 {
@@ -168,8 +173,8 @@ int load_flat_binary(struct kvm *kvm, int fd_kernel, int fd_initrd,
 		die_perror("lseek");
 
 	/*
-	 * Linux requires the initrd, pen and dtb to be mapped inside
-	 * lowmem, so we can't just place them at the top of memory.
+	 * Linux requires the initrd and dtb to be mapped inside lowmem,
+	 * so we can't just place them at the top of memory.
 	 */
 	limit = kvm->ram_start + min(kvm->ram_size, (u64)SZ_256M) - 1;
 
@@ -186,24 +191,9 @@ int load_flat_binary(struct kvm *kvm, int fd_kernel, int fd_initrd,
 	/*
 	 * Now load backwards from the end of memory so the kernel
 	 * decompressor has plenty of space to work with. First up is
-	 * the SMP pen if we have more than one virtual CPU...
+	 * the device tree blob...
 	 */
 	pos = limit;
-	if (kvm->cfg.nrcpus > 1) {
-		pos -= (ARM_SMP_PEN_SIZE + SMP_PEN_ALIGN);
-		guest_addr = ALIGN(host_to_guest_flat(kvm, pos), SMP_PEN_ALIGN);
-		pos = guest_flat_to_host(kvm, guest_addr);
-		if (pos < kernel_end)
-			die("SMP pen overlaps with kernel image.");
-
-		kvm->arch.smp_pen_guest_start = guest_addr;
-		pr_info("Placing SMP pen at 0x%llx - 0x%llx",
-			kvm->arch.smp_pen_guest_start,
-			host_to_guest_flat(kvm, limit));
-		limit = pos;
-	}
-
-	/* ...now the device tree blob... */
 	pos -= (FDT_MAX_SIZE + FDT_ALIGN);
 	guest_addr = ALIGN(host_to_guest_flat(kvm, pos), FDT_ALIGN);
 	pos = guest_flat_to_host(kvm, guest_addr);
diff --git a/tools/kvm/arm/include/arm-common/gic.h b/tools/kvm/arm/include/arm-common/gic.h
index d6a18e1..850edc7 100644
--- a/tools/kvm/arm/include/arm-common/gic.h
+++ b/tools/kvm/arm/include/arm-common/gic.h
@@ -26,12 +26,10 @@
 #define GIC_MAX_CPUS			8
 #define GIC_MAX_IRQ			255
 
-#ifndef __ASSEMBLY__
 struct kvm;
 
 int gic__alloc_irqnum(void);
 int gic__init_irqchip(struct kvm *kvm);
 void gic__generate_fdt_nodes(void *fdt, u32 phandle);
 
-#endif /* __ASSEMBLY__ */
 #endif /* ARM_COMMON__GIC_H */
diff --git a/tools/kvm/arm/include/arm-common/kvm-arch.h b/tools/kvm/arm/include/arm-common/kvm-arch.h
index c910f07..798af59 100644
--- a/tools/kvm/arm/include/arm-common/kvm-arch.h
+++ b/tools/kvm/arm/include/arm-common/kvm-arch.h
@@ -3,8 +3,6 @@
 
 #define VIRTIO_DEFAULT_TRANS	VIRTIO_MMIO
 
-#ifndef __ASSEMBLY__
-
 #include <stdbool.h>
 #include <linux/types.h>
 
@@ -26,9 +24,6 @@ struct kvm_arch {
 	u64	initrd_guest_start;
 	u64	initrd_size;
 	u64	dtb_guest_start;
-	u64	smp_pen_guest_start;
-	u64	smp_jump_guest_start;
 };
 
-#endif /* __ASSEMBLY__ */
 #endif /* ARM_COMMON__KVM_ARCH_H */
diff --git a/tools/kvm/arm/include/kvm/kvm-cpu-arch.h b/tools/kvm/arm/include/arm-common/kvm-cpu-arch.h
similarity index 87%
rename from tools/kvm/arm/include/kvm/kvm-cpu-arch.h
rename to tools/kvm/arm/include/arm-common/kvm-cpu-arch.h
index f0aeca2..351fbe6 100644
--- a/tools/kvm/arm/include/kvm/kvm-cpu-arch.h
+++ b/tools/kvm/arm/include/arm-common/kvm-cpu-arch.h
@@ -1,5 +1,5 @@
-#ifndef KVM__KVM_CPU_ARCH_H
-#define KVM__KVM_CPU_ARCH_H
+#ifndef ARM_COMMON__KVM_CPU_ARCH_H
+#define ARM_COMMON__KVM_CPU_ARCH_H
 
 #include <linux/kvm.h>
 #include <pthread.h>
@@ -43,4 +43,4 @@ static inline bool kvm_cpu__emulate_io(struct kvm *kvm, u16 port, void *data,
 bool kvm_cpu__emulate_mmio(struct kvm *kvm, u64 phys_addr, u8 *data, u32 len,
 			   u8 is_write);
 
-#endif /* KVM__KVM_CPU_ARCH_H */
+#endif /* ARM_COMMON__KVM_CPU_ARCH_H */
diff --git a/tools/kvm/arm/kvm-cpu.c b/tools/kvm/arm/kvm-cpu.c
index 3b08e55..7a0eff45 100644
--- a/tools/kvm/arm/kvm-cpu.c
+++ b/tools/kvm/arm/kvm-cpu.c
@@ -33,7 +33,9 @@ struct kvm_cpu *kvm_cpu__arch_init(struct kvm *kvm, unsigned long cpu_id)
 	struct kvm_cpu *vcpu;
 	int coalesced_offset, mmap_size, err = -1;
 	unsigned int i;
-	struct kvm_vcpu_init vcpu_init = { };
+	struct kvm_vcpu_init vcpu_init = {
+		.features = ARM_VCPU_FEATURE_FLAGS(kvm, cpu_id)
+	};
 
 	vcpu = calloc(1, sizeof(struct kvm_cpu));
 	if (!vcpu)
diff --git a/tools/kvm/arm/kvm.c b/tools/kvm/arm/kvm.c
index bfce685..b10c857 100644
--- a/tools/kvm/arm/kvm.c
+++ b/tools/kvm/arm/kvm.c
@@ -11,6 +11,7 @@
 struct kvm_ext kvm_req_ext[] = {
 	{ DEFINE_KVM_EXT(KVM_CAP_IRQCHIP) },
 	{ DEFINE_KVM_EXT(KVM_CAP_ONE_REG) },
+	{ DEFINE_KVM_EXT(KVM_CAP_ARM_PSCI) },
 	{ 0, 0 },
 };
 
diff --git a/tools/kvm/arm/smp.c b/tools/kvm/arm/smp.c
deleted file mode 100644
index c1e59d2..0000000
--- a/tools/kvm/arm/smp.c
+++ /dev/null
@@ -1,21 +0,0 @@
-#include "kvm/kvm.h"
-
-extern u8 smp_pen_start, smp_pen_end, smp_jump_addr;
-
-static int smp_pen_init(struct kvm *kvm)
-{
-	unsigned long pen_size, pen_start, jump_offset;
-
-	if (!(kvm->nrcpus > 1))
-		return 0;
-
-	pen_size = &smp_pen_end - &smp_pen_start;
-	pen_start = kvm->arch.smp_pen_guest_start;
-	jump_offset = &smp_jump_addr - &smp_pen_start;
-
-	kvm->arch.smp_jump_guest_start = pen_start + jump_offset;
-	memcpy(guest_flat_to_host(kvm, pen_start), &smp_pen_start, pen_size);
-
-	return 0;
-}
-firmware_init(smp_pen_init);
-- 
1.8.0

Re: [PATCH 0/4] ARM updates for kvmtool

[Christoffer Dall]

On Mon, Jan 7, 2013 at 1:14 PM, Will Deacon <will.deacon@arm.com> wrote:
>   - virtio mmio fixes to deal with guest page sizes != 4k (in
>           preparation for AArch64, which I will post separately).
>         - .dtb dumping via the lkvm command line
>         - Support for PSCI firmware as a replacement to the spin-table
>           based SMP boot code
>
> The last option was implemented after discussion on the linux-arm-kernel
> list when adding support for the mach-virt platform.

I completely missed that, it would have been nice if the kvmarm list
were cc'ed on those discussions.

> I hope to upstream
> the kernel-side part of the implementation for 3.9 and expect the kvm
> bits to follow once that has been merged.
>
> All feedback welcome.

Very cool, I'm looking forward to trying this out, hopefully I'll find
cycles this week.

-Christoffer

Re: [PATCH 0/4] ARM updates for kvmtool

[Pekka Enberg]

On Mon, Jan 7, 2013 at 8:14 PM, Will Deacon <will.deacon@arm.com> wrote:
> Hello kvm hackers,
>
> This patch series introduces some updates to the ARM (AArch32) kvm tools
> code:
>
>         - virtio mmio fixes to deal with guest page sizes != 4k (in
>           preparation for AArch64, which I will post separately).
>         - .dtb dumping via the lkvm command line
>         - Support for PSCI firmware as a replacement to the spin-table
>           based SMP boot code
>
> The last option was implemented after discussion on the linux-arm-kernel
> list when adding support for the mach-virt platform. I hope to upstream
> the kernel-side part of the implementation for 3.9 and expect the kvm
> bits to follow once that has been merged.
>
> All feedback welcome.

Applied, thanks Will!

Re: [PATCH 0/4] ARM updates for kvmtool

[Will Deacon]

On Wed, Jan 09, 2013 at 11:13:58AM +0000, Pekka Enberg wrote:
> On Mon, Jan 7, 2013 at 8:14 PM, Will Deacon <will.deacon@arm.com> wrote:
> > Hello kvm hackers,
> >
> > This patch series introduces some updates to the ARM (AArch32) kvm tools
> > code:
> >
> >         - virtio mmio fixes to deal with guest page sizes != 4k (in
> >           preparation for AArch64, which I will post separately).
> >         - .dtb dumping via the lkvm command line
> >         - Support for PSCI firmware as a replacement to the spin-table
> >           based SMP boot code
> >
> > The last option was implemented after discussion on the linux-arm-kernel
> > list when adding support for the mach-virt platform. I hope to upstream
> > the kernel-side part of the implementation for 3.9 and expect the kvm
> > bits to follow once that has been merged.
> >
> > All feedback welcome.
> 
> Applied, thanks Will!

Great, cheers Pekka.

Will