[PATCH 0/3][RFC] kvmppc: paravirtualization interface - guest part v3

[PATCH 0/3][RFC] kvmppc: paravirtualization interface - guest part v3

[ehrhardt]

From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>

Version 3 updates:
- guest hypercall infrastructure is now generic (in epapr_hcalls.h)
  while the kvm specific functions stay in kvm_para.h
- the hypercalls now use beat style ABI
- dropped the guest coop patch changing wrteei to wrtee (now mfmsr is
  rewritten avoiding side effects and a lot of corner cases. Additionally this
  does not need any guest cooperation to be effective)

This patch series implements a paravirtualization interface using:
- the device tree mechanism to pass hypervisor informations to the guest
- hypercalls for guest->host calls
- an example exploiter of that interface (magic page)

The device tree format used here (=base for the discussions on
embedded-hypervisor) is the following.
- A node "hypervisor" to show the general availability of some hypervisor data
- flags for features like the example "feature,pv-magicpage"
  setting 1 = available, everything else = unavailable
- Some features might need to pass more data and can use an entry in the
  device tree like the example of "data,pv-magicpage-size"

The host side of these patches can be found on kvm-ppc@vger.kernel.org

I hope that eventually this guest patch series (that is modifying the ppc boot
process and adding e.g. new ppc fixmaps could go upstream (when discussed
and agreed somewhen) via linuxppc-dev, while the kvm host part will go via
kvm (Avi Kivity).

[patches in series]
[PATCH 1/3] kvmppc: read device tree hypervisor node infrastructure
[PATCH 2/3] kvmppc: add hypercall infrastructure - guest part
[PATCH 3/3] kvmppc: magic page paravirtualization - guest part

---
[diffstat]
 arch/powerpc/kernel/kvm.c            |   53 +++++++++++++++++++++++++++++++
 b/arch/powerpc/kernel/Makefile       |    2 +
 b/arch/powerpc/kernel/kvm.c          |   30 +++++++++++++++++
 b/arch/powerpc/kernel/setup_32.c     |    3 +
 b/arch/powerpc/platforms/44x/Kconfig |    7 ++++
 b/include/asm-powerpc/epapr_hcalls.h |   59 +++++++++++++++++++++++++++++++++++
 b/include/asm-powerpc/fixmap.h       |   10 +++++
 b/include/asm-powerpc/kvm_para.h     |   43 +++++++++++++++++++++++--
 include/asm-powerpc/kvm_para.h       |   26 +++++++++++++++
 9 files changed, 229 insertions(+), 4 deletions(-)

[PATCH 1/3] kvmppc: read device tree hypervisor node infrastructure

[ehrhardt]

From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>

This patch adds the guest portion of the device tree based host->guest
communication. Using the device tree infrastructure this patch implements
kvm_para_available and kvm_arch_para_features (in this patch just the
infrastructure, no specific feature registered).

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---

[diffstat]
 arch/powerpc/kernel/Makefile       |    2 +
 arch/powerpc/kernel/kvm.c          |   30 +++++++++++++++++++++++++
 arch/powerpc/kernel/setup_32.c     |    3 ++
 arch/powerpc/platforms/44x/Kconfig |    7 ++++++
 include/asm-powerpc/kvm_para.h     |   43 ++++++++++++++++++++++++++++++++++---
 5 files changed, 82 insertions(+), 3 deletions(-)
[diff]

diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile
--- a/arch/powerpc/kernel/Makefile
+++ b/arch/powerpc/kernel/Makefile
@@ -80,6 +80,8 @@
 
 obj-$(CONFIG_8XX_MINIMAL_FPEMU) += softemu8xx.o
 
+obj-$(CONFIG_KVM_GUEST)		+= kvm.o
+
 ifneq ($(CONFIG_PPC_INDIRECT_IO),y)
 obj-y				+= iomap.o
 endif
diff --git a/arch/powerpc/kernel/kvm.c b/arch/powerpc/kernel/kvm.c
new file mode 100644
--- /dev/null
+++ b/arch/powerpc/kernel/kvm.c
@@ -0,0 +1,30 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+ *
+ * Copyright IBM Corp. 2008
+ *
+ * Authors:
+ * 	Hollis Blanchard <hollisb@us.ibm.com>
+ * 	Christian Ehrhardt <ehrhardt@de.ibm.com>
+ */
+
+#include <linux/percpu.h>
+#include <linux/mm.h>
+#include <linux/kvm_para.h>
+
+void __init kvm_guest_init(void)
+{
+	if (!kvm_para_available())
+		return;
+}
diff --git a/arch/powerpc/kernel/setup_32.c b/arch/powerpc/kernel/setup_32.c
--- a/arch/powerpc/kernel/setup_32.c
+++ b/arch/powerpc/kernel/setup_32.c
@@ -17,6 +17,7 @@
 #include <linux/cpu.h>
 #include <linux/console.h>
 #include <linux/lmb.h>
+#include <linux/kvm_para.h>
 
 #include <asm/io.h>
 #include <asm/prom.h>
@@ -319,5 +320,7 @@
 		ppc_md.setup_arch();
 	if ( ppc_md.progress ) ppc_md.progress("arch: exit", 0x3eab);
 
+	kvm_guest_init();
+
 	paging_init();
 }
diff --git a/arch/powerpc/platforms/44x/Kconfig b/arch/powerpc/platforms/44x/Kconfig
--- a/arch/powerpc/platforms/44x/Kconfig
+++ b/arch/powerpc/platforms/44x/Kconfig
@@ -152,3 +152,10 @@
 # 44x errata/workaround config symbols, selected by the CPU models above
 config IBM440EP_ERR42
 	bool
+
+config KVM_GUEST
+	bool "KVM Guest support"
+	depends on EXPERIMENTAL
+	help
+	This option enables various optimizations for running under the KVM
+	hypervisor.
diff --git a/include/asm-powerpc/kvm_para.h b/include/asm-powerpc/kvm_para.h
--- a/include/asm-powerpc/kvm_para.h
+++ b/include/asm-powerpc/kvm_para.h
@@ -14,7 +14,9 @@
  *
  * Copyright IBM Corp. 2008
  *
- * Authors: Hollis Blanchard <hollisb@us.ibm.com>
+ * Authors:
+ * 	Hollis Blanchard <hollisb@us.ibm.com>
+ * 	Christian Ehrhardt <ehrhardt@de.ibm.com>
  */
 
 #ifndef __POWERPC_KVM_PARA_H__
@@ -22,15 +24,50 @@
 
 #ifdef __KERNEL__
 
+#include <linux/of.h>
+
+static struct kvmppc_para_features {
+	char *dtcell;
+	int feature;
+} para_features[] = {
+};
+
 static inline int kvm_para_available(void)
 {
-	return 0;
+	struct device_node *dn;
+	int ret;
+
+	dn = of_find_node_by_path("/hypervisor");
+	ret = !!dn;
+
+	of_node_put(dn);
+
+	return ret;
 }
 
 static inline unsigned int kvm_arch_para_features(void)
 {
-	return 0;
+	struct device_node *dn;
+	const int *dtval;
+	unsigned int features = 0;
+	int i;
+
+	dn = of_find_node_by_path("/hypervisor");
+	if (!dn)
+		return 0;
+
+	for (i = 0; i < ARRAY_SIZE(para_features); i++) {
+		dtval = of_get_property(dn, para_features[i].dtcell, NULL);
+		if (dtval && *dtval == 1)
+			features |= (1 << para_features[i].feature);
+	}
+
+	of_node_put(dn);
+
+	return features;
 }
+
+void kvm_guest_init(void);
 
 #endif /* __KERNEL__ */
 

[PATCH 2/3] kvmppc: add hypercall infrastructure - guest part v3

[ehrhardt]

From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>

This adds the guest portion of the hypercall infrastructure.

Version 3 now follows the beat ABI, but proposes a new implementation style as
static inline asm functions instead of pure assembler code. That should allow
the compiler to be more flexible and therefore a better optimization.

If people agree on that new implementation style we might merge this code.
The current implementation of beat style hypercalls can be found in
arch/powerpc/platforms/cell/beat_hvCall.S

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---

[diffstat]
 epapr_hcalls.h |   59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 59 insertions(+)

[diff]

diff --git a/include/asm-powerpc/epapr_hcalls.h b/include/asm-powerpc/epapr_hcalls.h
new file mode 100644
--- /dev/null
+++ b/include/asm-powerpc/epapr_hcalls.h
@@ -0,0 +1,59 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+ *
+ * Copyright IBM Corp. 2008
+ *
+ * Authors:
+ * 	Christian Ehrhardt <ehrhardt@de.ibm.com>
+ */
+
+#ifndef __POWERPC_EPAPR_HCALLS_H__
+#define __POWERPC_EPAPR_HCALLS_H__
+
+#ifdef __KERNEL__
+
+/* Hypercalls use the beat ABI */
+#define KVM_HYPERCALL_BIN 0x44000022
+
+static inline long epapr_hypercall_1in_1out(unsigned int nr, unsigned long p1)
+{
+	register unsigned long hcall asm ("r11") = nr;
+	register unsigned long arg1_ret asm ("r3") = p1;
+
+	asm volatile(".long %1"
+			: "+r"(arg1_ret)
+			: "i"(KVM_HYPERCALL_BIN), "r"(hcall)
+			: "r4", "r5", "r6", "r7", "r8",
+			  "r9", "r10", "r12", "cc");
+	return arg1_ret;
+}
+
+static inline long epapr_hypercall_2in_1out(unsigned int nr,
+					unsigned long p1, unsigned long p2)
+{
+	register unsigned long hcall asm ("r11") = nr;
+	register unsigned long arg1_ret asm ("r3") = p1;
+	register unsigned long arg2 asm ("r4") = p2;
+
+	asm volatile(".long %1"
+			: "+r"(arg1_ret)
+			: "i"(KVM_HYPERCALL_BIN), "r"(hcall), "r"(arg2)
+			: "r5", "r6", "r7", "r8",
+			  "r9", "r10", "r12", "cc");
+	return arg1_ret;
+}
+
+#endif /* __KERNEL__ */
+
+#endif /* __POWERPC_EPAPR_HCALLS_H__ */

[PATCH 3/3] kvmppc: magic page paravirtualization - guest part

[ehrhardt]

From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>

This patch adds the guest handling for the magic page mechanism. A Hypervisor
can modify the device tree passed to the guest. Using that already existing
interface a guest can simply detect available hypervisor features and agree
on the supported ones using hypercalls.
In this example it is checked for the feature switch "feature,pv-magicpage"
in the hypervisor node and additional data which represents the size the
hypervisor requests in "data,pv-magicpage-size".
When the guest reads that data and wants to support it the memory is allocated
and passed to the hypervisor using the KVM_HCALL_RESERVE_MAGICPAGE hypercall.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---

[diffstat]
 arch/powerpc/kernel/kvm.c      |   53 +++++++++++++++++++++++++++++++++++++++++
 include/asm-powerpc/fixmap.h   |   10 ++++++-
 include/asm-powerpc/kvm_para.h |   26 ++++++++++++++++++++
 3 files changed, 88 insertions(+), 1 deletion(-)
[diff]

diff --git a/arch/powerpc/kernel/kvm.c b/arch/powerpc/kernel/kvm.c
--- a/arch/powerpc/kernel/kvm.c
+++ b/arch/powerpc/kernel/kvm.c
@@ -22,9 +22,62 @@
 #include <linux/percpu.h>
 #include <linux/mm.h>
 #include <linux/kvm_para.h>
+#include <linux/bootmem.h>
+#include <asm/fixmap.h>
+#include <asm/epapr_hcalls.h>
+
+/*
+ * this is guest memory granted to the hypervisor;
+ * the hypervisor can place data in this area and rewrite
+ * privileged instructions to read from this area without
+ * trapping.
+ * Only the Hypervisor needs to be aware of the structure layout
+ * which makes the guest more felxible - the guest only guarantees
+ * the size which is requested by the hypervisor and read from a
+ * device tree entry.
+ */
+static void *kvm_magicpage;
+
+static void __init kvmppc_register_magic_page(void)
+{
+	unsigned long gvaddr;
+	unsigned long gpaddr;
+	int size;
+	long err;
+
+	size = kvmppc_pv_read_data(KVM_PVDATA_MAGICPAGE_SIZE);
+	if (size < 0) {
+		printk(KERN_ERR "%s: couldn't read size for kvmppc style "
+			"paravirtualization support (got %d)\n",
+			__func__, size);
+		return;
+	}
+
+	/* FIXME Guest SMP needs that percpu which */
+	kvm_magicpage = alloc_bootmem(size);
+	if (!kvm_magicpage) {
+		printk(KERN_ERR "%s - failed to allocate %d bytes\n",
+			 __func__, size);
+		return;
+	}
+	gpaddr = (unsigned long)__pa(kvm_magicpage);
+	gvaddr = fix_to_virt(FIX_KVM_PV);
+
+	err = epapr_hypercall_2in_1out(KVM_HCALL_RESERVE_MAGICPAGE,
+					gvaddr, gpaddr);
+	if (err)
+		printk(KERN_ERR "%s: couldn't register pv mem\n", __func__);
+	else
+		printk(KERN_NOTICE "%s: registered %d bytes for pv mem support"
+			" (gvaddr 0x%08lx gpaddr 0x%08lx)\n",
+			 __func__, size, gvaddr, gpaddr);
+}
 
 void __init kvm_guest_init(void)
 {
 	if (!kvm_para_available())
 		return;
+
+	if (kvm_para_has_feature(KVM_FEATURE_PPCPV_MAGICPAGE))
+		kvmppc_register_magic_page();
 }
diff --git a/include/asm-powerpc/fixmap.h b/include/asm-powerpc/fixmap.h
--- a/include/asm-powerpc/fixmap.h
+++ b/include/asm-powerpc/fixmap.h
@@ -36,7 +36,7 @@
  *
  * these 'compile-time allocated' memory buffers are
  * fixed-size 4k pages. (or larger if used with an increment
- * highger than 1) use fixmap_set(idx,phys) to associate
+ * higher than 1) use fixmap_set(idx,phys) to associate
  * physical memory with fixmap indices.
  *
  * TLB entries of such buffers will not be flushed across
@@ -44,6 +44,14 @@
  */
 enum fixed_addresses {
 	FIX_HOLE,
+#ifdef CONFIG_KVM_GUEST
+	/*
+	 * reserved virtual address space for paravirtualization - needs to be
+	 *  <=32k away from base address 0 to be able to reach it with
+	 * immediate addressing using base 0 instead of needing a register.
+	 */
+	FIX_KVM_PV,
+#endif
 #ifdef CONFIG_HIGHMEM
 	FIX_KMAP_BEGIN,	/* reserved pte's for temporary kernel mappings */
 	FIX_KMAP_END = FIX_KMAP_BEGIN+(KM_TYPE_NR*NR_CPUS)-1,
diff --git a/include/asm-powerpc/kvm_para.h b/include/asm-powerpc/kvm_para.h
--- a/include/asm-powerpc/kvm_para.h
+++ b/include/asm-powerpc/kvm_para.h
@@ -26,10 +26,18 @@
 
 #include <linux/of.h>
 
+#define KVM_HCALL_RESERVE_MAGICPAGE	0
+
+#define KVM_PVDATA_MAGICPAGE_SIZE	"data,pv-magicpage-size"
+
+/* List of PV features supported, returned as a bitfield */
+#define KVM_FEATURE_PPCPV_MAGICPAGE	0
+
 static struct kvmppc_para_features {
 	char *dtcell;
 	int feature;
 } para_features[] = {
+	{ "feature,pv-magicpage", KVM_FEATURE_PPCPV_MAGICPAGE }
 };
 
 static inline int kvm_para_available(void)
@@ -67,6 +75,24 @@
 	return features;
 }
 
+/* reads the specified data field out of the hypervisor node */
+static inline int kvmppc_pv_read_data(char *dtcell)
+{
+	struct device_node *dn;
+	const int *dtval;
+
+	dn = of_find_node_by_path("/hypervisor");
+	if (!dn)
+		return -EINVAL;
+
+	dtval = of_get_property(dn, dtcell, NULL);
+	of_node_put(dn);
+	if (dtval)
+		return *dtval;
+	else
+		return -EINVAL;
+}
+
 void kvm_guest_init(void);
 
 #endif /* __KERNEL__ */

Re: [PATCH 2/3] kvmppc: add hypercall infrastructure - guest part v3

[Kumar Gala]

On Sep 16, 2008, at 1:27 AM, ehrhardt@linux.vnet.ibm.com wrote:

> From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
>
> This adds the guest portion of the hypercall infrastructure.
>
> Version 3 now follows the beat ABI, but proposes a new  
> implementation style as
> static inline asm functions instead of pure assembler code. That  
> should allow
> the compiler to be more flexible and therefore a better optimization.
>
> If people agree on that new implementation style we might merge this  
> code.
> The current implementation of beat style hypercalls can be found in
> arch/powerpc/platforms/cell/beat_hvCall.S
>
> Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
> ---
>
> [diffstat]
> epapr_hcalls.h |   59 +++++++++++++++++++++++++++++++++++++++++++++++ 
> ++++++++++
> 1 file changed, 59 insertions(+)
>
> [diff]
>
> diff --git a/include/asm-powerpc/epapr_hcalls.h b/include/asm- 
> powerpc/epapr_hcalls.h
> new file mode 100644
> --- /dev/null
> +++ b/include/asm-powerpc/epapr_hcalls.h
> @@ -0,0 +1,59 @@
> +/*
> + * This program is free software; you can redistribute it and/or  
> modify
> + * it under the terms of the GNU General Public License, version 2,  
> as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, write to the Free Software
> + * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA   
> 02110-1301, USA.
> + *
> + * Copyright IBM Corp. 2008
> + *
> + * Authors:
> + * 	Christian Ehrhardt <ehrhardt@de.ibm.com>
> + */
> +
> +#ifndef __POWERPC_EPAPR_HCALLS_H__
> +#define __POWERPC_EPAPR_HCALLS_H__
> +
> +#ifdef __KERNEL__
> +
> +/* Hypercalls use the beat ABI */
> +#define KVM_HYPERCALL_BIN 0x44000022

Any reason this isn't 'sc' ?

Also, can we make this "sc 1" so its works when we have HW hypervisor  
support?

- k

Re: [PATCH 2/3] kvmppc: add hypercall infrastructure - guest part v3

[Hollis Blanchard]

On Mon, 2008-10-13 at 11:42 -0500, Kumar Gala wrote:
> > +
> > +/* Hypercalls use the beat ABI */
> > +#define KVM_HYPERCALL_BIN 0x44000022
> 
> Any reason this isn't 'sc' ?
> 
> Also, can we make this "sc 1" so its works when we have HW hypervisor  
> support?

Actually, it is "sc 1".

Many versions of as (including 2.16.1) don't recognize that instruction.
This is the standard workaround for situations like that.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: [PATCH 2/3] kvmppc: add hypercall infrastructure - guest part v3

[Kumar Gala]

On Oct 13, 2008, at 12:29 PM, Hollis Blanchard wrote:

> On Mon, 2008-10-13 at 11:42 -0500, Kumar Gala wrote:
>>> +
>>> +/* Hypercalls use the beat ABI */
>>> +#define KVM_HYPERCALL_BIN 0x44000022
>>
>> Any reason this isn't 'sc' ?
>>
>> Also, can we make this "sc 1" so its works when we have HW hypervisor
>> support?
>
> Actually, it is "sc 1".
>
> Many versions of as (including 2.16.1) don't recognize that  
> instruction.
> This is the standard workaround for situations like that.

Ahh, we'll than can we add that to the comment, and rename  
'KVM_HYPERCALL_BIN' to something like 'EPAPR_HYPERCALL_INSTR'

- k