[PATCH 0/5] Add support for ppc64le in kvmtool

[PATCH 0/5] Add support for ppc64le in kvmtool

[Balbir Singh]

Changelog v2: Add support for running a task on all cpus by
              Michael Ellerman. Large improvements to patch 2/4

This patchset adds support for ppc64le. As a part of the support 1/4 converts
key data structures in the fdt to big endian. 2/4 introduces h_set_mode call
to support little endian interrupt processing. This requires support to execute
and queue commands to a particular vcpu and hence a generic infrastructure
is added in patch 2/4. Patch 3/4 fixes a race condition found during exit.
Patch 4/4 adds support for fixing spapr_pci to support little endian guests
so that virtio-pci can be detected and virtio can work

This patchset was tested on x64 (on my laptop) and on a ppc64le system.

Balbir Singh (5):
  Add basic little endian support.
  Add basic infrastructure to run tasks on vCPUs
  Implement H_SET_MODE for ppc64le
  Fix a race during exit processing
  Implement spapr pci for little endian systems.

 arm/include/arm-common/kvm-cpu-arch.h |  1 +
 builtin-run.c                         |  3 ++
 include/kvm/kvm-cpu.h                 |  6 +++
 include/kvm/kvm.h                     |  1 +
 kvm-cpu.c                             | 73 +++++++++++++++++++++++++++++++++++
 kvm.c                                 |  5 +++
 mips/include/kvm/kvm-cpu-arch.h       |  1 +
 powerpc/include/kvm/kvm-cpu-arch.h    |  3 ++
 powerpc/kvm.c                         | 26 ++++++-------
 powerpc/spapr.h                       | 20 ++++++++--
 powerpc/spapr_hcall.c                 | 66 +++++++++++++++++++++++++++++++
 powerpc/spapr_pci.c                   | 25 ++++++------
 x86/include/kvm/kvm-cpu-arch.h        |  1 +
 13 files changed, 203 insertions(+), 28 deletions(-)

-- 
2.5.5

[PATCH 1/5] Add basic little endian support.

[Balbir Singh]

Currently kvmtool works well/was designed for big endian ppc64 systems.
This patch adds support for little endian systems

The system does not yet boot as support for h_set_mode is required to help
with exceptions in big endian mode -- first page fault. The support comes in
the next patch of the series

Signed-off-by: Balbir Singh <bsingharora@gmail.com>
---
 powerpc/kvm.c   | 24 ++++++++++++------------
 powerpc/spapr.h |  5 +++--
 2 files changed, 15 insertions(+), 14 deletions(-)

diff --git a/powerpc/kvm.c b/powerpc/kvm.c
index b4c3310..d147e0c 100644
--- a/powerpc/kvm.c
+++ b/powerpc/kvm.c
@@ -253,21 +253,21 @@ static void generate_segment_page_sizes(struct kvm_ppc_smmu_info *info, struct f
 		if (sps->page_shift == 0)
 			break;
 
-		*p++ = sps->page_shift;
-		*p++ = sps->slb_enc;
+		*p++ = cpu_to_be32(sps->page_shift);
+		*p++ = cpu_to_be32(sps->slb_enc);
 
 		for (j = 0; j < KVM_PPC_PAGE_SIZES_MAX_SZ; j++)
 			if (!info->sps[i].enc[j].page_shift)
 				break;
 
-		*p++ = j;	/* count of enc */
+		*p++ = cpu_to_be32(j);	/* count of enc */
 
 		for (j = 0; j < KVM_PPC_PAGE_SIZES_MAX_SZ; j++) {
 			if (!info->sps[i].enc[j].page_shift)
 				break;
 
-			*p++ = info->sps[i].enc[j].page_shift;
-			*p++ = info->sps[i].enc[j].pte_enc;
+			*p++ = cpu_to_be32(info->sps[i].enc[j].page_shift);
+			*p++ = cpu_to_be32(info->sps[i].enc[j].pte_enc);
 		}
 	}
 }
@@ -292,7 +292,7 @@ static int setup_fdt(struct kvm *kvm)
 	u8		staging_fdt[FDT_MAX_SIZE];
 	struct cpu_info *cpu_info = find_cpu_info(kvm);
 	struct fdt_prop segment_page_sizes;
-	u32 segment_sizes_1T[] = {0x1c, 0x28, 0xffffffff, 0xffffffff};
+	u32 segment_sizes_1T[] = {cpu_to_be32(0x1c), cpu_to_be32(0x28), 0xffffffff, 0xffffffff};
 
 	/* Generate an appropriate DT at kvm->arch.fdt_gra */
 	void *fdt_dest = guest_flat_to_host(kvm, kvm->arch.fdt_gra);
@@ -364,7 +364,7 @@ static int setup_fdt(struct kvm *kvm)
 	_FDT(fdt_property_cell(fdt, "#size-cells", 0x0));
 
 	for (i = 0; i < smp_cpus; i += SMT_THREADS) {
-		int32_t pft_size_prop[] = { 0, HPT_ORDER };
+		int32_t pft_size_prop[] = { 0, cpu_to_be32(HPT_ORDER) };
 		uint32_t servers_prop[SMT_THREADS];
 		uint32_t gservers_prop[SMT_THREADS * 2];
 		int threads = (smp_cpus - i) >= SMT_THREADS ? SMT_THREADS :
@@ -503,11 +503,11 @@ int kvm__arch_setup_firmware(struct kvm *kvm)
 	 */
 	uint32_t *rtas = guest_flat_to_host(kvm, kvm->arch.rtas_gra);
 
-	rtas[0] = 0x7c641b78;
-	rtas[1] = 0x3c600000;
-	rtas[2] = 0x6063f000;
-	rtas[3] = 0x44000022;
-	rtas[4] = 0x4e800020;
+	rtas[0] = cpu_to_be32(0x7c641b78);
+	rtas[1] = cpu_to_be32(0x3c600000);
+	rtas[2] = cpu_to_be32(0x6063f000);
+	rtas[3] = cpu_to_be32(0x44000022);
+	rtas[4] = cpu_to_be32(0x4e800020);
 	kvm->arch.rtas_size = 20;
 
 	pr_info("Set up %ld bytes of RTAS at 0x%lx\n",
diff --git a/powerpc/spapr.h b/powerpc/spapr.h
index 7a377d0..8b294d1 100644
--- a/powerpc/spapr.h
+++ b/powerpc/spapr.h
@@ -15,6 +15,7 @@
 #define __HW_SPAPR_H__
 
 #include <inttypes.h>
+#include <linux/byteorder.h>
 
 #include "kvm/kvm.h"
 #include "kvm/kvm-cpu.h"
@@ -80,12 +81,12 @@ int spapr_rtas_fdt_setup(struct kvm *kvm, void *fdt);
 
 static inline uint32_t rtas_ld(struct kvm *kvm, target_ulong phys, int n)
 {
-	return *((uint32_t *)guest_flat_to_host(kvm, phys + 4*n));
+	return cpu_to_be32(*((uint32_t *)guest_flat_to_host(kvm, phys + 4*n)));
 }
 
 static inline void rtas_st(struct kvm *kvm, target_ulong phys, int n, uint32_t val)
 {
-	*((uint32_t *)guest_flat_to_host(kvm, phys + 4*n)) = val;
+	*((uint32_t *)guest_flat_to_host(kvm, phys + 4*n)) = cpu_to_be32(val);
 }
 
 typedef void (*spapr_rtas_fn)(struct kvm_cpu *vcpu, uint32_t token,
-- 
2.5.5

[PATCH 2/5] Add basic infrastructure to run tasks on vCPUs

[Balbir Singh]

This patch adds kvm_cpu__run_on_all_cpus() to run a task on each vCPU.
This infrastructure uses signals to signal the vCPU to allow a task
to be added to each vCPU's task. The vCPU executes any pending tasks
in the cpu run loop

Signed-off-by: Balbir Singh <bsingharora@gmail.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
---
 arm/include/arm-common/kvm-cpu-arch.h |  1 +
 include/kvm/kvm-cpu.h                 |  6 +++
 include/kvm/kvm.h                     |  1 +
 kvm-cpu.c                             | 73 +++++++++++++++++++++++++++++++++++
 mips/include/kvm/kvm-cpu-arch.h       |  1 +
 powerpc/include/kvm/kvm-cpu-arch.h    |  1 +
 x86/include/kvm/kvm-cpu-arch.h        |  1 +
 7 files changed, 84 insertions(+)

diff --git a/arm/include/arm-common/kvm-cpu-arch.h b/arm/include/arm-common/kvm-cpu-arch.h
index 329979a..8a6a6e7 100644
--- a/arm/include/arm-common/kvm-cpu-arch.h
+++ b/arm/include/arm-common/kvm-cpu-arch.h
@@ -17,6 +17,7 @@ struct kvm_cpu {
 	struct kvm	*kvm;
 	int		vcpu_fd;
 	struct kvm_run	*kvm_run;
+	struct kvm_cpu_task	*task;
 
 	u8		is_running;
 	u8		paused;
diff --git a/include/kvm/kvm-cpu.h b/include/kvm/kvm-cpu.h
index aa0cb54..4481096 100644
--- a/include/kvm/kvm-cpu.h
+++ b/include/kvm/kvm-cpu.h
@@ -4,6 +4,11 @@
 #include "kvm/kvm-cpu-arch.h"
 #include <stdbool.h>
 
+struct kvm_cpu_task {
+	void (*func)(struct kvm_cpu *vcpu, void *data);
+	void *data;
+};
+
 int kvm_cpu__init(struct kvm *kvm);
 int kvm_cpu__exit(struct kvm *kvm);
 struct kvm_cpu *kvm_cpu__arch_init(struct kvm *kvm, unsigned long cpu_id);
@@ -23,5 +28,6 @@ void kvm_cpu__show_code(struct kvm_cpu *vcpu);
 void kvm_cpu__show_registers(struct kvm_cpu *vcpu);
 void kvm_cpu__show_page_tables(struct kvm_cpu *vcpu);
 void kvm_cpu__arch_nmi(struct kvm_cpu *cpu);
+void kvm_cpu__run_on_all_cpus(struct kvm *kvm, struct kvm_cpu_task *task);
 
 #endif /* KVM__KVM_CPU_H */
diff --git a/include/kvm/kvm.h b/include/kvm/kvm.h
index 37155db..731abee 100644
--- a/include/kvm/kvm.h
+++ b/include/kvm/kvm.h
@@ -15,6 +15,7 @@
 
 #define SIGKVMEXIT		(SIGRTMIN + 0)
 #define SIGKVMPAUSE		(SIGRTMIN + 1)
+#define SIGKVMTASK		(SIGRTMIN + 2)
 
 #define KVM_PID_FILE_PATH	"/.lkvm/"
 #define HOME_DIR		getenv("HOME")
diff --git a/kvm-cpu.c b/kvm-cpu.c
index ad4441b..4b202f8 100644
--- a/kvm-cpu.c
+++ b/kvm-cpu.c
@@ -4,9 +4,12 @@
 #include "kvm/util.h"
 #include "kvm/kvm.h"
 #include "kvm/virtio.h"
+#include "kvm/mutex.h"
+#include "kvm/barrier.h"
 
 #include <sys/ioctl.h>
 #include <sys/mman.h>
+#include <sys/eventfd.h>
 #include <signal.h>
 #include <stdlib.h>
 #include <string.h>
@@ -52,6 +55,8 @@ static void kvm_cpu_signal_handler(int signum)
 	} else if (signum == SIGKVMPAUSE) {
 		current_kvm_cpu->paused = 1;
 	}
+
+	/* For SIGKVMTASK cpu->task is already set */
 }
 
 static void kvm_cpu__handle_coalesced_mmio(struct kvm_cpu *cpu)
@@ -83,6 +88,62 @@ void kvm_cpu__reboot(struct kvm *kvm)
 	}
 }
 
+static DEFINE_MUTEX(task_lock);
+static int task_eventfd;
+
+static void kvm_cpu__run_task(struct kvm_cpu *cpu)
+{
+	u64 inc = 1;
+
+	pr_debug("Running task %p on cpu %lu", cpu->task, cpu->cpu_id);
+
+	/* Make sure we see the store to cpu->task */
+	rmb();
+	cpu->task->func(cpu, cpu->task->data);
+
+	/* Clear task before we signal completion */
+	cpu->task = NULL;
+	wmb();
+
+	if (write(task_eventfd, &inc, sizeof(inc)) < 0)
+		die("Failed notifying of completed task.");
+}
+
+void kvm_cpu__run_on_all_cpus(struct kvm *kvm, struct kvm_cpu_task *task)
+{
+	int i, done = 0;
+
+	pr_debug("Running task %p on all cpus", task);
+
+	mutex_lock(&task_lock);
+
+	for (i = 0; i < kvm->nrcpus; i++) {
+		if (kvm->cpus[i]->task) {
+			/* Should never happen */
+			die("CPU %d already has a task pending!", i);
+		}
+
+		kvm->cpus[i]->task = task;
+		wmb();
+
+		if (kvm->cpus[i] == current_kvm_cpu)
+			kvm_cpu__run_task(current_kvm_cpu);
+		else
+			pthread_kill(kvm->cpus[i]->thread, SIGKVMTASK);
+	}
+
+	while (done < kvm->nrcpus) {
+		u64 count;
+
+		if (read(task_eventfd, &count, sizeof(count)) < 0)
+			die("Failed reading task eventfd");
+
+		done += count;
+	}
+
+	mutex_unlock(&task_lock);
+}
+
 int kvm_cpu__start(struct kvm_cpu *cpu)
 {
 	sigset_t sigset;
@@ -94,6 +155,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)
 
 	signal(SIGKVMEXIT, kvm_cpu_signal_handler);
 	signal(SIGKVMPAUSE, kvm_cpu_signal_handler);
+	signal(SIGKVMTASK, kvm_cpu_signal_handler);
 
 	kvm_cpu__reset_vcpu(cpu);
 
@@ -111,6 +173,9 @@ int kvm_cpu__start(struct kvm_cpu *cpu)
 			cpu->needs_nmi = 0;
 		}
 
+		if (cpu->task)
+			kvm_cpu__run_task(cpu);
+
 		kvm_cpu__run(cpu);
 
 		switch (cpu->kvm_run->exit_reason) {
@@ -217,6 +282,12 @@ int kvm_cpu__init(struct kvm *kvm)
 
 	kvm->nrcpus = kvm->cfg.nrcpus;
 
+	task_eventfd = eventfd(0, 0);
+	if (task_eventfd < 0) {
+		pr_warning("Couldn't create task_eventfd");
+		return task_eventfd;
+	}
+
 	/* Alloc one pointer too many, so array ends up 0-terminated */
 	kvm->cpus = calloc(kvm->nrcpus + 1, sizeof(void *));
 	if (!kvm->cpus) {
@@ -264,6 +335,8 @@ int kvm_cpu__exit(struct kvm *kvm)
 
 	kvm->nrcpus = 0;
 
+	close(task_eventfd);
+
 	return r;
 }
 core_exit(kvm_cpu__exit);
diff --git a/mips/include/kvm/kvm-cpu-arch.h b/mips/include/kvm/kvm-cpu-arch.h
index 3db7f2b..45e69f6 100644
--- a/mips/include/kvm/kvm-cpu-arch.h
+++ b/mips/include/kvm/kvm-cpu-arch.h
@@ -15,6 +15,7 @@ struct kvm_cpu {
 	struct kvm	*kvm;		/* parent KVM */
 	int		vcpu_fd;	/* For VCPU ioctls() */
 	struct kvm_run	*kvm_run;
+	struct kvm_cpu_task	*task;
 
 	struct kvm_regs	regs;
 
diff --git a/powerpc/include/kvm/kvm-cpu-arch.h b/powerpc/include/kvm/kvm-cpu-arch.h
index 01eafdf..f2bfbb5 100644
--- a/powerpc/include/kvm/kvm-cpu-arch.h
+++ b/powerpc/include/kvm/kvm-cpu-arch.h
@@ -48,6 +48,7 @@ struct kvm_cpu {
 	struct kvm		*kvm;		/* parent KVM */
 	int			vcpu_fd;	/* For VCPU ioctls() */
 	struct kvm_run		*kvm_run;
+	struct kvm_cpu_task	*task;
 
 	struct kvm_regs		regs;
 	struct kvm_sregs	sregs;
diff --git a/x86/include/kvm/kvm-cpu-arch.h b/x86/include/kvm/kvm-cpu-arch.h
index 89c8059..05e5bb6 100644
--- a/x86/include/kvm/kvm-cpu-arch.h
+++ b/x86/include/kvm/kvm-cpu-arch.h
@@ -18,6 +18,7 @@ struct kvm_cpu {
 	struct kvm		*kvm;		/* parent KVM */
 	int			vcpu_fd;	/* For VCPU ioctls() */
 	struct kvm_run		*kvm_run;
+	struct kvm_cpu_task	*task;
 
 	struct kvm_regs		regs;
 	struct kvm_sregs	sregs;
-- 
2.5.5

Re: [PATCH 2/5] Add basic infrastructure to run tasks on vCPUs

[Michael Ellerman]

On Thu, 2016-03-31 at 22:28 +1100, Balbir Singh wrote:

> This patch adds kvm_cpu__run_on_all_cpus() to run a task on each vCPU.
> This infrastructure uses signals to signal the vCPU to allow a task
> to be added to each vCPU's task. The vCPU executes any pending tasks
> in the cpu run loop
> 
> Signed-off-by: Balbir Singh <bsingharora@gmail.com>
> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>

This was based on your code, but I think I pretty much rewrote it. So this
should probably be From me. So I get the blame for the inevitable bugs :)

cheers

Re: [PATCH 2/5] Add basic infrastructure to run tasks on vCPUs

[Balbir Singh]

Agreed

Will should I resend this bit with a new From?

Balbir

On Fri, Apr 1, 2016 at 9:42 PM, Michael Ellerman <mpe@ellerman.id.au> wrote:
> On Thu, 2016-03-31 at 22:28 +1100, Balbir Singh wrote:
>
>> This patch adds kvm_cpu__run_on_all_cpus() to run a task on each vCPU.
>> This infrastructure uses signals to signal the vCPU to allow a task
>> to be added to each vCPU's task. The vCPU executes any pending tasks
>> in the cpu run loop
>>
>> Signed-off-by: Balbir Singh <bsingharora@gmail.com>
>> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
>
> This was based on your code, but I think I pretty much rewrote it. So this
> should probably be From me. So I get the blame for the inevitable bugs :)
>
> cheers
>

Re: [PATCH 2/5] Add basic infrastructure to run tasks on vCPUs

[Will Deacon]

On Fri, Apr 01, 2016 at 10:29:27PM +1100, Balbir Singh wrote:
> Agreed
> 
> Will should I resend this bit with a new From?

Sure, feel free to repost the whole series with Michael's acks too.

Will

Re: [PATCH 2/5] Add basic infrastructure to run tasks on vCPUs

[Balbir Singh]

On Fri, Apr 1, 2016 at 11:27 PM, Will Deacon <will.deacon@arm.com> wrote:
> On Fri, Apr 01, 2016 at 10:29:27PM +1100, Balbir Singh wrote:
>> Agreed
>>
>> Will should I resend this bit with a new From?
>
> Sure, feel free to repost the whole series with Michael's acks too.
>

Done a few days back

Balbir

[PATCH 3/5] Implement H_SET_MODE for ppc64le

[Balbir Singh]

Use the infrastructure for queuing a task to a specific vCPU and
sett ILE (Little Endian Interrupt Handling) on power via h_set_mode
hypercall

Signed-off-by: Balbir Singh <bsingharora@gmail.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
---
 powerpc/include/kvm/kvm-cpu-arch.h |  2 ++
 powerpc/kvm.c                      |  2 +-
 powerpc/spapr.h                    | 15 +++++++--
 powerpc/spapr_hcall.c              | 66 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 82 insertions(+), 3 deletions(-)

diff --git a/powerpc/include/kvm/kvm-cpu-arch.h b/powerpc/include/kvm/kvm-cpu-arch.h
index f2bfbb5..a69e0cc 100644
--- a/powerpc/include/kvm/kvm-cpu-arch.h
+++ b/powerpc/include/kvm/kvm-cpu-arch.h
@@ -38,6 +38,8 @@
 
 #define POWER7_EXT_IRQ	0
 
+#define LPCR_ILE	(1 << (63-38))
+
 struct kvm;
 
 struct kvm_cpu {
diff --git a/powerpc/kvm.c b/powerpc/kvm.c
index d147e0c..2dbd0fe 100644
--- a/powerpc/kvm.c
+++ b/powerpc/kvm.c
@@ -286,7 +286,7 @@ static int setup_fdt(struct kvm *kvm)
 	uint32_t	int_server_ranges_prop[] = {0, cpu_to_be32(smp_cpus)};
 	char 		hypertas_prop_kvm[] = "hcall-pft\0hcall-term\0"
 		"hcall-dabr\0hcall-interrupt\0hcall-tce\0hcall-vio\0"
-		"hcall-splpar\0hcall-bulk";
+		"hcall-splpar\0hcall-bulk\0hcall-set-mode";
 	int 		i, j;
 	char 		cpu_name[30];
 	u8		staging_fdt[FDT_MAX_SIZE];
diff --git a/powerpc/spapr.h b/powerpc/spapr.h
index 8b294d1..f851f4a 100644
--- a/powerpc/spapr.h
+++ b/powerpc/spapr.h
@@ -27,7 +27,7 @@ typedef uintptr_t target_phys_addr_t;
 #define H_HARDWARE	-1	/* Hardware error */
 #define H_FUNCTION	-2	/* Function not supported */
 #define H_PARAMETER	-4	/* Parameter invalid, out-of-range or conflicting */
-
+#define H_P2		-55
 #define H_SET_DABR		0x28
 #define H_LOGICAL_CI_LOAD	0x3c
 #define H_LOGICAL_CI_STORE	0x40
@@ -41,7 +41,18 @@ typedef uintptr_t target_phys_addr_t;
 #define H_EOI			0x64
 #define H_IPI			0x6c
 #define H_XIRR			0x74
-#define MAX_HCALL_OPCODE	H_XIRR
+#define H_SET_MODE		0x31C
+#define MAX_HCALL_OPCODE	H_SET_MODE
+
+/* Values for 2nd argument to H_SET_MODE */
+#define H_SET_MODE_RESOURCE_SET_CIABR		1
+#define H_SET_MODE_RESOURCE_SET_DAWR		2
+#define H_SET_MODE_RESOURCE_ADDR_TRANS_MODE	3
+#define H_SET_MODE_RESOURCE_LE			4
+
+/* Flags for H_SET_MODE_RESOURCE_LE */
+#define H_SET_MODE_ENDIAN_BIG		0
+#define H_SET_MODE_ENDIAN_LITTLE	1
 
 /*
  * The hcalls above are standardized in PAPR and implemented by pHyp
diff --git a/powerpc/spapr_hcall.c b/powerpc/spapr_hcall.c
index ff1d63a..25bec82 100644
--- a/powerpc/spapr_hcall.c
+++ b/powerpc/spapr_hcall.c
@@ -18,6 +18,7 @@
 
 #include <stdio.h>
 #include <assert.h>
+#include <sys/eventfd.h>
 
 static spapr_hcall_fn papr_hypercall_table[(MAX_HCALL_OPCODE / 4) + 1];
 static spapr_hcall_fn kvmppc_hypercall_table[KVMPPC_HCALL_MAX -
@@ -74,6 +75,70 @@ static target_ulong h_logical_dcbf(struct kvm_cpu *vcpu, target_ulong opcode, ta
 	return H_SUCCESS;
 }
 
+struct lpcr_data {
+	struct kvm_cpu	*cpu;
+	int		mode;
+};
+
+static void get_cpu_lpcr(struct kvm_cpu *vcpu, target_ulong *lpcr)
+{
+	struct kvm_one_reg reg = {
+		.id = KVM_REG_PPC_LPCR_64,
+		.addr = (__u64)lpcr
+	};
+
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg))
+		die("Couldn't read vcpu reg?!");
+}
+
+static void set_cpu_lpcr(struct kvm_cpu *vcpu, target_ulong *lpcr)
+{
+	struct kvm_one_reg reg = {
+		.id = KVM_REG_PPC_LPCR_64,
+		.addr = (__u64)lpcr
+	};
+
+	if (ioctl(vcpu->vcpu_fd, KVM_SET_ONE_REG, &reg))
+		die("Couldn't write vcpu reg?!");
+}
+
+static void set_endian_task(struct kvm_cpu *vcpu, void *data)
+{
+	target_ulong mflags = (target_ulong)data;
+	target_ulong lpcr;
+
+	get_cpu_lpcr(vcpu, &lpcr);
+
+	if (mflags == H_SET_MODE_ENDIAN_BIG)
+		lpcr &= ~LPCR_ILE;
+	else
+		lpcr |= LPCR_ILE;
+
+	set_cpu_lpcr(vcpu, &lpcr);
+}
+
+static target_ulong h_set_mode(struct kvm_cpu *vcpu, target_ulong opcode, target_ulong *args)
+{
+	int ret;
+
+	switch (args[1]) {
+	case H_SET_MODE_RESOURCE_LE: {
+		struct kvm_cpu_task task;
+		task.func = set_endian_task;
+		task.data = (void *)args[0];
+		kvm_cpu__run_on_all_cpus(vcpu->kvm, &task);
+		ret = H_SUCCESS;
+		break;
+	}
+	default:
+		ret = H_FUNCTION;
+		break;
+	}
+
+	return ret;
+}
+
+
 void spapr_register_hypercall(target_ulong opcode, spapr_hcall_fn fn)
 {
 	spapr_hcall_fn *slot;
@@ -128,6 +193,7 @@ void hypercall_init(void)
 	spapr_register_hypercall(H_LOGICAL_CACHE_STORE, h_logical_store);
 	spapr_register_hypercall(H_LOGICAL_ICBI, h_logical_icbi);
 	spapr_register_hypercall(H_LOGICAL_DCBF, h_logical_dcbf);
+	spapr_register_hypercall(H_SET_MODE, h_set_mode);
 
 	/* KVM-PPC specific hcalls */
 	spapr_register_hypercall(KVMPPC_H_RTAS, h_rtas);
-- 
2.5.5

[PATCH 4/5] Fix a race during exit processing

[Balbir Singh]

Fix a race, described below

	lkvm stop ...	handle_stop
			kvm_cpu__reboot
			kvm_cmd_run_exit
			vcpus exit
			...
			dev_exit
			...
			ioport__unregister
			..serial...
			kvm__pause --> br_write_lock
			pthread_kill

But the thread is already dead above.

We mark the cpus as dying so that kvm_pause does nothing.
This should not break any semantics

Signed-off-by: Balbir Singh <bsingharora@gmail.com>
---
 builtin-run.c | 3 +++
 kvm.c         | 5 +++++
 2 files changed, 8 insertions(+)

diff --git a/builtin-run.c b/builtin-run.c
index 17b1428..cdc7158 100644
--- a/builtin-run.c
+++ b/builtin-run.c
@@ -58,6 +58,7 @@ __thread struct kvm_cpu *current_kvm_cpu;
 static int  kvm_run_wrapper;
 
 bool do_debug_print = false;
+int kvm_cmd_exit;
 
 static const char * const run_usage[] = {
 	"lkvm run [<options>] [<kernel image>]",
@@ -648,6 +649,7 @@ static void kvm_cmd_run_exit(struct kvm *kvm, int guest_ret)
 {
 	compat__print_all_messages();
 
+	kvm_cmd_exit = 1;
 	init_list__exit(kvm);
 
 	if (guest_ret == 0 && do_debug_print)
@@ -659,6 +661,7 @@ int kvm_cmd_run(int argc, const char **argv, const char *prefix)
 	int ret = -EFAULT;
 	struct kvm *kvm;
 
+	kvm_cmd_exit = 0;
 	kvm = kvm_cmd_run_init(argc, argv);
 	if (IS_ERR(kvm))
 		return PTR_ERR(kvm);
diff --git a/kvm.c b/kvm.c
index 1081072..53cf0e2 100644
--- a/kvm.c
+++ b/kvm.c
@@ -33,6 +33,8 @@
 
 #define DEFINE_KVM_EXIT_REASON(reason) [reason] = #reason
 
+extern int kvm_cmd_exit;
+
 const char *kvm_exit_reasons[] = {
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_UNKNOWN),
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_EXCEPTION),
@@ -435,6 +437,9 @@ void kvm__pause(struct kvm *kvm)
 	if (!kvm->cpus[0] || kvm->cpus[0]->thread == 0)
 		return;
 
+	if (kvm_cmd_exit)
+		return;
+
 	mutex_lock(&pause_lock);
 
 	pause_event = eventfd(0, 0);
-- 
2.5.5

Re: [PATCH 4/5] Fix a race during exit processing

[Will Deacon]

On Thu, Mar 31, 2016 at 10:28:26PM +1100, Balbir Singh wrote:
> Fix a race, described below
> 
> 	lkvm stop ...	handle_stop
> 			kvm_cpu__reboot
> 			kvm_cmd_run_exit
> 			vcpus exit
> 			...
> 			dev_exit
> 			...
> 			ioport__unregister
> 			..serial...
> 			kvm__pause --> br_write_lock
> 			pthread_kill
> 
> But the thread is already dead above.
> 
> We mark the cpus as dying so that kvm_pause does nothing.
> This should not break any semantics

I'm not convinced that this solves the whole problem. The fact of the
matter is that we're tearing down the VM whilst there may still be active
vcpus, so I'd much rather delay the teardown until we know that the vcpus
are all dead.

Patch incoming.

Will

[PATCH 5/5] Implement spapr pci for little endian systems.

[Balbir Singh]

Port the spapr_pci implementation for ppc64le.
Based on suggestions by Alexey Kardashevskiy <aik@ozlabs.ru>
We should have always used phys_hi and 64 bit addr and size.

Cc: Alexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: Balbir Singh <bsingharora@gmail.com>
---
 powerpc/spapr_pci.c | 25 ++++++++++++++-----------
 1 file changed, 14 insertions(+), 11 deletions(-)

diff --git a/powerpc/spapr_pci.c b/powerpc/spapr_pci.c
index 768e3f2..a15f7d8 100644
--- a/powerpc/spapr_pci.c
+++ b/powerpc/spapr_pci.c
@@ -234,8 +234,11 @@ int spapr_populate_pci_devices(struct kvm *kvm,
 	int bus_off, node_off = 0, devid, fn, i, n, devices;
 	struct device_header *dev_hdr;
 	char nodename[256];
-	struct of_pci_unit_address reg[PCI_NUM_REGIONS + 1],
-				   assigned_addresses[PCI_NUM_REGIONS];
+	struct of_pci_unit64_address {
+		u32 phys_hi;
+		u64 addr;
+		u64 size;
+	} __attribute((packed)) reg[PCI_NUM_REGIONS + 1], assigned_addresses[PCI_NUM_REGIONS];
 	uint32_t bus_range[] = { cpu_to_be32(0), cpu_to_be32(0xff) };
 	struct of_pci_ranges_entry ranges[] = {
 		{
@@ -339,7 +342,7 @@ int spapr_populate_pci_devices(struct kvm *kvm,
 				      le16_to_cpu(hdr->subsys_vendor_id)));
 
 		/* Config space region comes first */
-		reg[0].hi = cpu_to_be32(
+		reg[0].phys_hi = cpu_to_be32(
 			of_pci_b_n(0) |
 			of_pci_b_p(0) |
 			of_pci_b_t(0) |
@@ -347,8 +350,8 @@ int spapr_populate_pci_devices(struct kvm *kvm,
 			of_pci_b_bbbbbbbb(0) |
 			of_pci_b_ddddd(devid) |
 			of_pci_b_fff(fn));
-		reg[0].mid = 0;
-		reg[0].lo = 0;
+		reg[0].addr = 0;
+		reg[0].size = 0;
 
 		n = 0;
 		/* Six BARs, no ROM supported, addresses are 32bit */
@@ -357,7 +360,7 @@ int spapr_populate_pci_devices(struct kvm *kvm,
 				continue;
 			}
 
-			reg[n+1].hi = cpu_to_be32(
+			reg[n+1].phys_hi = cpu_to_be32(
 				of_pci_b_n(0) |
 				of_pci_b_p(0) |
 				of_pci_b_t(0) |
@@ -366,10 +369,10 @@ int spapr_populate_pci_devices(struct kvm *kvm,
 				of_pci_b_ddddd(devid) |
 				of_pci_b_fff(fn) |
 				of_pci_b_rrrrrrrr(bars[i]));
-			reg[n+1].mid = 0;
-			reg[n+1].lo = cpu_to_be64(hdr->bar_size[i]);
+			reg[n+1].size = cpu_to_be64(hdr->bar_size[i]);
+			reg[n+1].addr = 0;
 
-			assigned_addresses[n].hi = cpu_to_be32(
+			assigned_addresses[n].phys_hi = cpu_to_be32(
 				of_pci_b_n(1) |
 				of_pci_b_p(0) |
 				of_pci_b_t(0) |
@@ -383,8 +386,8 @@ int spapr_populate_pci_devices(struct kvm *kvm,
 			 * Writing zeroes to assigned_addresses causes the guest kernel to
 			 * reassign BARs
 			 */
-			assigned_addresses[n].mid = cpu_to_be64(bar_to_addr(le32_to_cpu(hdr->bar[i])));
-			assigned_addresses[n].lo = reg[n+1].lo;
+			assigned_addresses[n].addr = cpu_to_be64(bar_to_addr(le32_to_cpu(hdr->bar[i])));
+			assigned_addresses[n].size = reg[n+1].size;
 
 			++n;
 		}
-- 
2.5.5

Re: [PATCH 0/5] Add support for ppc64le in kvmtool

[Laurent Vivier]

Hi,


could you add "kvmtool" in the subject, next time, please? :)
And it could be a good idea to send to kvm-ppc ML too.

On 31/03/2016 13:28, Balbir Singh wrote:
> Changelog v2: Add support for running a task on all cpus by
>               Michael Ellerman. Large improvements to patch 2/4
> 
> This patchset adds support for ppc64le. As a part of the support 1/4 converts
> key data structures in the fdt to big endian. 2/4 introduces h_set_mode call
> to support little endian interrupt processing. This requires support to execute
> and queue commands to a particular vcpu and hence a generic infrastructure
> is added in patch 2/4. Patch 3/4 fixes a race condition found during exit.
> Patch 4/4 adds support for fixing spapr_pci to support little endian guests
> so that virtio-pci can be detected and virtio can work
> 
> This patchset was tested on x64 (on my laptop) and on a ppc64le system.
> 
> Balbir Singh (5):
>   Add basic little endian support.
>   Add basic infrastructure to run tasks on vCPUs
>   Implement H_SET_MODE for ppc64le
>   Fix a race during exit processing
>   Implement spapr pci for little endian systems.

I was trying to use kvmtool on ppc64, some days ago, and I was not able
to run anything (yes, I'm stupid), So could you also update the README
with some ppc64/ppc64le stuff explaining how to use kvmtool on POWER?

Or give me a link to the information, and I'll do.

Thanks,
Laurent

Re: [PATCH 0/5] Add support for ppc64le in kvmtool

[Balbir Singh]

On 01/04/16 00:38, Laurent Vivier wrote:
> Hi,
>
>
> could you add "kvmtool" in the subject, next time, please? :)
> And it could be a good idea to send to kvm-ppc ML too.
You mean like in [kvmtool]? The cover letter does have kvmtool, I think
adding [kvmtool] tag helps. I'll keep it in mind
> On 31/03/2016 13:28, Balbir Singh wrote:
>> Changelog v2: Add support for running a task on all cpus by
>>               Michael Ellerman. Large improvements to patch 2/4
>>
>> This patchset adds support for ppc64le. As a part of the support 1/4 converts
>> key data structures in the fdt to big endian. 2/4 introduces h_set_mode call
>> to support little endian interrupt processing. This requires support to execute
>> and queue commands to a particular vcpu and hence a generic infrastructure
>> is added in patch 2/4. Patch 3/4 fixes a race condition found during exit.
>> Patch 4/4 adds support for fixing spapr_pci to support little endian guests
>> so that virtio-pci can be detected and virtio can work
>>
>> This patchset was tested on x64 (on my laptop) and on a ppc64le system.
>>
>> Balbir Singh (5):
>>   Add basic little endian support.
>>   Add basic infrastructure to run tasks on vCPUs
>>   Implement H_SET_MODE for ppc64le
>>   Fix a race during exit processing
>>   Implement spapr pci for little endian systems.
> I was trying to use kvmtool on ppc64, some days ago, and I was not able
> to run anything (yes, I'm stupid), So could you also update the README
> with some ppc64/ppc64le stuff explaining how to use kvmtool on POWER?
No.. I don't think anyone is stupid, may be it just doesn't work. For ppc64le
here are the steps. I'll write a README after this series is committed

1. Make sure you have this patchset
2. Compile a ppc64le_defconfig kernel
3. Strip the elf header from vmlinux (yes, we don't yet have an elf parser - TODO)
4. lkvm run -c <cpus> -m <mem{constrained to 4G}> -k <kernel from step 3> -i <initrd> -d <diskimage> -p <args> --console hv

There are two limitations that need to be fixed
1. memory is limited to 4G because of the way RTAS is currently setup in kvmtool, I have a brute force
   fix for it, I'll post it after this series is merged
2. HPT size is limited to 16MB, will post a patch after this series is merged

TODOs
1. Implement an elf parser so that we don't have to do step 3


> Or give me a link to the information, and I'll do.
I hope this helps, I can do some documentation once we get the limitations and TODOs
resolved
> Thanks,
> Laurent
Thanks for looking into this
Balbir

Re: [PATCH 0/5] Add support for ppc64le in kvmtool

[Balbir Singh]

On Fri, Apr 1, 2016 at 11:12 AM, Balbir Singh <bsingharora@gmail.com> wrote:
>
>
> On 01/04/16 00:38, Laurent Vivier wrote:
>> Hi,
>>
>>
>> could you add "kvmtool" in the subject, next time, please? :)
>> And it could be a good idea to send to kvm-ppc ML too.
> You mean like in [kvmtool]? The cover letter does have kvmtool, I think
> adding [kvmtool] tag helps. I'll keep it in mind
>> On 31/03/2016 13:28, Balbir Singh wrote:
>>> Changelog v2: Add support for running a task on all cpus by
>>>               Michael Ellerman. Large improvements to patch 2/4
>>>
>>> This patchset adds support for ppc64le. As a part of the support 1/4 converts
>>> key data structures in the fdt to big endian. 2/4 introduces h_set_mode call
>>> to support little endian interrupt processing. This requires support to execute
>>> and queue commands to a particular vcpu and hence a generic infrastructure
>>> is added in patch 2/4. Patch 3/4 fixes a race condition found during exit.
>>> Patch 4/4 adds support for fixing spapr_pci to support little endian guests
>>> so that virtio-pci can be detected and virtio can work
>>>
>>> This patchset was tested on x64 (on my laptop) and on a ppc64le system.
>>>
>>> Balbir Singh (5):
>>>   Add basic little endian support.
>>>   Add basic infrastructure to run tasks on vCPUs
>>>   Implement H_SET_MODE for ppc64le
>>>   Fix a race during exit processing
>>>   Implement spapr pci for little endian systems.
>> I was trying to use kvmtool on ppc64, some days ago, and I was not able
>> to run anything (yes, I'm stupid), So could you also update the README
>> with some ppc64/ppc64le stuff explaining how to use kvmtool on POWER?
> No.. I don't think anyone is stupid, may be it just doesn't work. For ppc64le
> here are the steps. I'll write a README after this series is committed
>
> 1. Make sure you have this patchset
> 2. Compile a ppc64le_defconfig kernel
> 3. Strip the elf header from vmlinux (yes, we don't yet have an elf parser - TODO)
> 4. lkvm run -c <cpus> -m <mem{constrained to 4G}> -k <kernel from step 3> -i <initrd> -d <diskimage> -p <args> --console hv
>
> There are two limitations that need to be fixed
> 1. memory is limited to 4G because of the way RTAS is currently setup in kvmtool, I have a brute force
>    fix for it, I'll post it after this series is merged
> 2. HPT size is limited to 16MB, will post a patch after this series is merged
>
> TODOs
> 1. Implement an elf parser so that we don't have to do step 3
>
>
>> Or give me a link to the information, and I'll do.
> I hope this helps, I can do some documentation once we get the limitations and TODOs
> resolved
>> Thanks,
>> Laurent
> Thanks for looking into this
> Balbir

Hi,

Did the instructions help? Any luck with running kvmtool?

Balbir

Re: [PATCH 0/5] Add support for ppc64le in kvmtool

[Laurent Vivier]

On 04/04/2016 10:34, Balbir Singh wrote:
> On Fri, Apr 1, 2016 at 11:12 AM, Balbir Singh <bsingharora@gmail.com> wrote:
>>
>>
>> On 01/04/16 00:38, Laurent Vivier wrote:
>>> Hi,
>>>
>>>
>>> could you add "kvmtool" in the subject, next time, please? :)
>>> And it could be a good idea to send to kvm-ppc ML too.
>> You mean like in [kvmtool]? The cover letter does have kvmtool, I think
>> adding [kvmtool] tag helps. I'll keep it in mind
>>> On 31/03/2016 13:28, Balbir Singh wrote:
>>>> Changelog v2: Add support for running a task on all cpus by
>>>>               Michael Ellerman. Large improvements to patch 2/4
>>>>
>>>> This patchset adds support for ppc64le. As a part of the support 1/4 converts
>>>> key data structures in the fdt to big endian. 2/4 introduces h_set_mode call
>>>> to support little endian interrupt processing. This requires support to execute
>>>> and queue commands to a particular vcpu and hence a generic infrastructure
>>>> is added in patch 2/4. Patch 3/4 fixes a race condition found during exit.
>>>> Patch 4/4 adds support for fixing spapr_pci to support little endian guests
>>>> so that virtio-pci can be detected and virtio can work
>>>>
>>>> This patchset was tested on x64 (on my laptop) and on a ppc64le system.
>>>>
>>>> Balbir Singh (5):
>>>>   Add basic little endian support.
>>>>   Add basic infrastructure to run tasks on vCPUs
>>>>   Implement H_SET_MODE for ppc64le
>>>>   Fix a race during exit processing
>>>>   Implement spapr pci for little endian systems.
>>> I was trying to use kvmtool on ppc64, some days ago, and I was not able
>>> to run anything (yes, I'm stupid), So could you also update the README
>>> with some ppc64/ppc64le stuff explaining how to use kvmtool on POWER?
>> No.. I don't think anyone is stupid, may be it just doesn't work. For ppc64le
>> here are the steps. I'll write a README after this series is committed
>>
>> 1. Make sure you have this patchset
>> 2. Compile a ppc64le_defconfig kernel
>> 3. Strip the elf header from vmlinux (yes, we don't yet have an elf parser - TODO)
>> 4. lkvm run -c <cpus> -m <mem{constrained to 4G}> -k <kernel from step 3> -i <initrd> -d <diskimage> -p <args> --console hv
>>
>> There are two limitations that need to be fixed
>> 1. memory is limited to 4G because of the way RTAS is currently setup in kvmtool, I have a brute force
>>    fix for it, I'll post it after this series is merged
>> 2. HPT size is limited to 16MB, will post a patch after this series is merged

Do you know David Gibson is working on a patch series for QEMU/KVM to
change the size of the HPT while the guest is running?

http://patchwork.ozlabs.org/patch/599965/

>>
>> TODOs
>> 1. Implement an elf parser so that we don't have to do step 3
>>
>>
>>> Or give me a link to the information, and I'll do.
>> I hope this helps, I can do some documentation once we get the limitations and TODOs
>> resolved
>>> Thanks,
>>> Laurent
>> Thanks for looking into this
>> Balbir
> 
> Hi,

Hi,

> Did the instructions help? Any luck with running kvmtool?
> 

This is helpful but I didn't have the time to try again.

Just one more question: how do you strip the ELF header (step 3)?

Thanks,
Laurent

Re: [PATCH 0/5] Add support for ppc64le in kvmtool

[Balbir Singh]

>>
>
> This is helpful but I didn't have the time to try again.
>
> Just one more question: how do you strip the ELF header (step 3)?

I use

objcopy -O binary vmlinux vmlinux.bin

Balbir

Re: [PATCH 0/5] Add support for ppc64le in kvmtool

[Laurent Vivier]

On 04/04/2016 12:24, Balbir Singh wrote:
>>>
>>
>> This is helpful but I didn't have the time to try again.
>>
>> Just one more question: how do you strip the ELF header (step 3)?
> 
> I use
> 
> objcopy -O binary vmlinux vmlinux.bin

Many thanks!

It was the missing part :)

I'm able to boot a big-endian kernel now, I'll test your patches as soon
as I have access to a little-endian capable host.

Laurent

Re: [PATCH 0/5] Add support for ppc64le in kvmtool

[Michael Ellerman]

On Thu, 2016-03-31 at 22:28 +1100, Balbir Singh wrote:

> Changelog v2: Add support for running a task on all cpus by
>               Michael Ellerman. Large improvements to patch 2/4
> 
> This patchset adds support for ppc64le. As a part of the support 1/4 converts
> key data structures in the fdt to big endian. 2/4 introduces h_set_mode call
> to support little endian interrupt processing. This requires support to execute
> and queue commands to a particular vcpu and hence a generic infrastructure
> is added in patch 2/4. Patch 3/4 fixes a race condition found during exit.
> Patch 4/4 adds support for fixing spapr_pci to support little endian guests
> so that virtio-pci can be detected and virtio can work
> 
> This patchset was tested on x64 (on my laptop) and on a ppc64le system.

Hi Will,

I'm pretty happy with this series. We still have plenty of warts in the ppc code
but this is a definite improvement.

Acked-by: Michael Ellerman <mpe@ellerman.id.au>

cheers

Re: [PATCH 0/5] Add support for ppc64le in kvmtool

[Will Deacon]

On Fri, Apr 01, 2016 at 09:47:01PM +1100, Michael Ellerman wrote:
> On Thu, 2016-03-31 at 22:28 +1100, Balbir Singh wrote:
> 
> > Changelog v2: Add support for running a task on all cpus by
> >               Michael Ellerman. Large improvements to patch 2/4
> > 
> > This patchset adds support for ppc64le. As a part of the support 1/4 converts
> > key data structures in the fdt to big endian. 2/4 introduces h_set_mode call
> > to support little endian interrupt processing. This requires support to execute
> > and queue commands to a particular vcpu and hence a generic infrastructure
> > is added in patch 2/4. Patch 3/4 fixes a race condition found during exit.
> > Patch 4/4 adds support for fixing spapr_pci to support little endian guests
> > so that virtio-pci can be detected and virtio can work
> > 
> > This patchset was tested on x64 (on my laptop) and on a ppc64le system.
> 
> Hi Will,
> 
> I'm pretty happy with this series. We still have plenty of warts in the ppc code
> but this is a definite improvement.
> 
> Acked-by: Michael Ellerman <mpe@ellerman.id.au>

Brill, thanks Michael. I need to take a closer look at the race condition
fix, because it doesn't sit right with me, but I'll consider the PPC bits
as good for merging.

Will