[PATCH 0/4][RFC] kvmppc: paravirtualization interface - guest part v2

[PATCH 0/4][RFC] kvmppc: paravirtualization interface - guest part v2

[ehrhardt]

From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>

This patch series implements a paravirtualization interface using:
- the device tree mechanism to pass hypervisor informations to the guest
- hypercalls for guest->host calls
- an example exploiter of that interface (magic page)

Version 2 includes changes to the feedback of my last submission and is now
tested against the implemented and working host part. The host part discussion
can be found on kvm-ppc@vger.kernel.org.

The used hypercall ABI was already discussed on the embedded-hypervisor mailing
list and is available at http://kvm.qumranet.com/kvmwiki/PowerPC_Hypercall_ABI

The device tree format used here (=base for the discussions on
embedded-hypervisor) is the following.
- A node "hypervisor" to show the general availability of some hypervisor data
- flags for features like the example "feature,pv-magicpage"
  setting 1 = available, everything else = unavailable
- Some features might need to pass more data and can use an entry in the
  device tree like the example of "data,pv-magicpage-size"

I hope that eventually this guest patch series (that is modifying the boot
process and adding e.g. new ppc fixmaps could go upstream (when discussed
and agreed somewhen) via linuxppc-dev, while the kvm host part will go via
kvm (Avi Kivity).

[patches in series]
[PATCH 1/4] kvmppc: read device tree hypervisor node infrastructure
[PATCH 2/4] kvmppc: add hypercall infrastructure - guest part
[PATCH 3/4] kvmppc: magic page paravirtualization - guest part
[PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

---
[diffstat]
 arch/powerpc/kernel/kvm.c            |   51 ++++++++++++++++++++++++++++++
 b/arch/powerpc/kernel/Makefile       |    2 +
 b/arch/powerpc/kernel/kvm.c          |   30 +++++++++++++++++
 b/arch/powerpc/kernel/setup_32.c     |    3 +
 b/arch/powerpc/platforms/44x/Kconfig |    7 ++++
 b/include/asm-powerpc/fixmap.h       |   10 +++++
 b/include/asm-powerpc/hw_irq.h       |   12 +++++++
 b/include/asm-powerpc/kvm_para.h     |   43 +++++++++++++++++++++++--
 b/mm/page_alloc.c                    |    1
 include/asm-powerpc/kvm_para.h       |   59 +++++++++++++++++++++++++++++++++++
 10 files changed, 214 insertions(+), 4 deletions(-)

[PATCH 1/4] kvmppc: read device tree hypervisor node infrastructure

[ehrhardt]

From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>

This patch adds the guest portion of the device tree based host->guest
communication. Using the device tree infrastructure this patch implements
kvm_para_available and kvm_arch_para_features (in this patch just the
infrastructure, no specific feature registered).

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---

[diffstat]
 arch/powerpc/kernel/Makefile       |    2 ++
 arch/powerpc/kernel/kvm.c          |   30 ++++++++++++++++++++++++++++++
 arch/powerpc/kernel/setup_32.c     |    3 +++
 arch/powerpc/platforms/44x/Kconfig |    7 +++++++
 include/asm-powerpc/kvm_para.h     |   37 ++++++++++++++++++++++++++++++++++---
 5 files changed, 76 insertions(+), 3 deletions(-)

[diff]

diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile
--- a/arch/powerpc/kernel/Makefile
+++ b/arch/powerpc/kernel/Makefile
@@ -80,6 +80,8 @@
 
 obj-$(CONFIG_8XX_MINIMAL_FPEMU) += softemu8xx.o
 
+obj-$(CONFIG_KVM_GUEST)		+= kvm.o
+
 ifneq ($(CONFIG_PPC_INDIRECT_IO),y)
 obj-y				+= iomap.o
 endif
diff --git a/arch/powerpc/kernel/kvm.c b/arch/powerpc/kernel/kvm.c
new file mode 100644
--- /dev/null
+++ b/arch/powerpc/kernel/kvm.c
@@ -0,0 +1,30 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+ *
+ * Copyright IBM Corp. 2008
+ *
+ * Authors:
+ * 	Hollis Blanchard <hollisb@us.ibm.com>
+ * 	Christian Ehrhardt <ehrhardt@de.ibm.com>
+ */
+
+#include <linux/percpu.h>
+#include <linux/mm.h>
+#include <linux/kvm_para.h>
+
+void __init kvm_guest_init(void)
+{
+	if (!kvm_para_available())
+		return;
+}
diff --git a/arch/powerpc/kernel/setup_32.c b/arch/powerpc/kernel/setup_32.c
--- a/arch/powerpc/kernel/setup_32.c
+++ b/arch/powerpc/kernel/setup_32.c
@@ -17,6 +17,7 @@
 #include <linux/cpu.h>
 #include <linux/console.h>
 #include <linux/lmb.h>
+#include <linux/kvm_para.h>
 
 #include <asm/io.h>
 #include <asm/prom.h>
@@ -319,5 +320,7 @@
 		ppc_md.setup_arch();
 	if ( ppc_md.progress ) ppc_md.progress("arch: exit", 0x3eab);
 
+	kvm_guest_init();
+
 	paging_init();
 }
diff --git a/arch/powerpc/platforms/44x/Kconfig b/arch/powerpc/platforms/44x/Kconfig
--- a/arch/powerpc/platforms/44x/Kconfig
+++ b/arch/powerpc/platforms/44x/Kconfig
@@ -152,3 +152,10 @@
 # 44x errata/workaround config symbols, selected by the CPU models above
 config IBM440EP_ERR42
 	bool
+
+config KVM_GUEST
+	bool "KVM Guest support"
+	depends on EXPERIMENTAL
+	help
+	This option enables various optimizations for running under the KVM
+	hypervisor.
diff --git a/include/asm-powerpc/kvm_para.h b/include/asm-powerpc/kvm_para.h
--- a/include/asm-powerpc/kvm_para.h
+++ b/include/asm-powerpc/kvm_para.h
@@ -14,7 +14,9 @@
  *
  * Copyright IBM Corp. 2008
  *
- * Authors: Hollis Blanchard <hollisb@us.ibm.com>
+ * Authors:
+ * 	Hollis Blanchard <hollisb@us.ibm.com>
+ * 	Christian Ehrhardt <ehrhardt@de.ibm.com>
  */
 
 #ifndef __POWERPC_KVM_PARA_H__
@@ -22,15 +24,50 @@
 
 #ifdef __KERNEL__
 
+#include <linux/of.h>
+
+static struct kvmppc_para_features {
+	char *dtcell;
+	int feature;
+} para_features[] = {
+};
+
 static inline int kvm_para_available(void)
 {
-	return 0;
+	struct device_node *dn;
+	int ret;
+
+	dn = of_find_node_by_path("/hypervisor");
+	ret = !!dn;
+
+	of_node_put(dn);
+
+	return ret;
 }
 
 static inline unsigned int kvm_arch_para_features(void)
 {
-	return 0;
+	struct device_node *dn;
+	const int *dtval;
+	unsigned int features = 0;
+	int i;
+
+	dn = of_find_node_by_path("/hypervisor");
+	if (!dn)
+		return 0;
+
+	for (i = 0; i < ARRAY_SIZE(para_features); i++) {
+		dtval = of_get_property(dn, para_features[i].dtcell, NULL);
+		if (dtval && *dtval == 1)
+			features |= (1 << para_features[i].feature);
+	}
+
+	of_node_put(dn);
+
+	return features;
 }
+
+void kvm_guest_init(void);
 
 #endif /* __KERNEL__ */
 

[PATCH 2/4] kvmppc: add hypercall infrastructure - guest part

[ehrhardt]

From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>

This adds the guest portion of the hypercall infrastructure, basically an
illegal instruction with a defined layout.
See http://kvm.qumranet.com/kvmwiki/PowerPC_Hypercall_ABI for more detail
on the hypercall ABI for powerpc.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---

[diffstat]
 kvm_para.h |   33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

[diff]

diff --git a/include/asm-powerpc/kvm_para.h b/include/asm-powerpc/kvm_para.h
--- a/include/asm-powerpc/kvm_para.h
+++ b/include/asm-powerpc/kvm_para.h
@@ -25,6 +25,8 @@
 #ifdef __KERNEL__
 
 #include <linux/of.h>
+
+#define KVM_HYPERCALL_BIN 0x03ffffff
 
 static struct kvmppc_para_features {
 	char *dtcell;
@@ -69,6 +71,37 @@
 
 void kvm_guest_init(void);
 
+static inline long kvm_hypercall1(unsigned int nr, unsigned long p1)
+{
+	register unsigned long hcall asm ("r0") = nr;
+	register unsigned long arg1 asm ("r3") = p1;
+	register long ret asm ("r11");
+
+	asm volatile(".long %1"
+			: "=r"(ret)
+			: "i"(KVM_HYPERCALL_BIN), "r"(hcall), "r"(arg1)
+			: "r4", "r5", "r6", "r7", "r8",
+			  "r9", "r10", "r12", "cc");
+	return ret;
+}
+
+static inline long kvm_hypercall2(unsigned int nr,
+					unsigned long p1, unsigned long p2)
+{
+	register unsigned long hcall asm ("r0") = nr;
+	register unsigned long arg1 asm ("r3") = p1;
+	register unsigned long arg2 asm ("r4") = p2;
+	register long ret asm ("r11");
+
+	asm volatile(".long %1"
+			: "=r"(ret)
+			: "i"(KVM_HYPERCALL_BIN), "r"(hcall),
+				"r"(arg1), "r"(arg2)
+			: "r5", "r6", "r7", "r8",
+			  "r9", "r10", "r12", "cc");
+	return ret;
+}
+
 #endif /* __KERNEL__ */
 
 #endif /* __POWERPC_KVM_PARA_H__ */

[PATCH 3/4] kvmppc: magic page paravirtualization - guest part

[ehrhardt]

From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>

This patch adds the guest handling for the magic page mechanism. A Hypervisor
can modify the device tree passed to the guest. Using that already existing
interface a guest can simply detect available hypervisor features and agree
on the supported ones using hypercalls.
In this example it is checked for the feature switch "feature,pv-magicpage"
in the hypervisor node and additional data which represents the size the
hypervisor requests in "data,pv-magicpage-size".
When the guest reads that data and wants to support it the memory is allocated
and passed to the hypervisor using the KVM_HCALL_RESERVE_MAGICPAGE hypercall.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---

[diffstat]
 arch/powerpc/kernel/kvm.c      |   51 +++++++++++++++++++++++++++++++++++++++++
 include/asm-powerpc/fixmap.h   |   10 +++++++-
 include/asm-powerpc/kvm_para.h |   26 ++++++++++++++++++++
 mm/page_alloc.c                |    1
 4 files changed, 87 insertions(+), 1 deletion(-)

[diff]

diff --git a/arch/powerpc/kernel/kvm.c b/arch/powerpc/kernel/kvm.c
--- a/arch/powerpc/kernel/kvm.c
+++ b/arch/powerpc/kernel/kvm.c
@@ -22,9 +22,60 @@
 #include <linux/percpu.h>
 #include <linux/mm.h>
 #include <linux/kvm_para.h>
+#include <linux/bootmem.h>
+#include <asm/fixmap.h>
+
+/*
+ * this is guest memory granted to the hypervisor;
+ * the hypervisor can place data in this area and rewrite
+ * privileged instructions to read from this area without
+ * trapping.
+ * Only the Hypervisor needs to be aware of the structure layout
+ * which makes the guest more felxible - the guest only guarantees
+ * the size which is requested by the hypervisor and read from a
+ * device tree entry.
+ */
+static void *kvm_magicpage;
+
+static void __init kvmppc_register_magic_page(void)
+{
+	unsigned long gvaddr;
+	unsigned long gpaddr;
+	int size;
+	long err;
+
+	size = kvmppc_pv_read_data(KVM_PVDATA_MAGICPAGE_SIZE);
+	if (size < 0) {
+		printk(KERN_ERR "%s: couldn't read size for kvmppc style "
+			"paravirtualization support (got %d)\n",
+			__func__, size);
+		return;
+	}
+
+	/* FIXME Guest SMP needs that percpu which */
+	kvm_magicpage = alloc_bootmem(size);
+	if (!kvm_magicpage) {
+		printk(KERN_ERR "%s - failed to allocate %d bytes\n",
+			 __func__, size);
+		return;
+	}
+	gpaddr = (unsigned long)__pa(kvm_magicpage);
+	gvaddr = fix_to_virt(FIX_KVM_PV);
+
+	err = kvm_hypercall2(KVM_HCALL_RESERVE_MAGICPAGE, gvaddr, gpaddr);
+	if (err)
+		printk(KERN_ERR "%s: couldn't register pv mem\n", __func__);
+	else
+		printk(KERN_NOTICE "%s: registered %d bytes for pv mem support"
+			" (gvaddr 0x%08lx gpaddr 0x%08lx)\n",
+			 __func__, size, gvaddr, gpaddr);
+}
 
 void __init kvm_guest_init(void)
 {
 	if (!kvm_para_available())
 		return;
+
+	if (kvm_para_has_feature(KVM_FEATURE_PPCPV_MAGICPAGE))
+		kvmppc_register_magic_page();
 }
diff --git a/include/asm-powerpc/fixmap.h b/include/asm-powerpc/fixmap.h
--- a/include/asm-powerpc/fixmap.h
+++ b/include/asm-powerpc/fixmap.h
@@ -36,7 +36,7 @@
  *
  * these 'compile-time allocated' memory buffers are
  * fixed-size 4k pages. (or larger if used with an increment
- * highger than 1) use fixmap_set(idx,phys) to associate
+ * higher than 1) use fixmap_set(idx,phys) to associate
  * physical memory with fixmap indices.
  *
  * TLB entries of such buffers will not be flushed across
@@ -44,6 +44,14 @@
  */
 enum fixed_addresses {
 	FIX_HOLE,
+#ifdef CONFIG_KVM_GUEST
+	/*
+	 * reserved virtual address space for paravirtualization - needs to be
+	 *  <=32k away from base address 0 to be able to reach it with
+	 * immediate addressing using base 0 instead of needing a register.
+	 */
+	FIX_KVM_PV,
+#endif
 #ifdef CONFIG_HIGHMEM
 	FIX_KMAP_BEGIN,	/* reserved pte's for temporary kernel mappings */
 	FIX_KMAP_END = FIX_KMAP_BEGIN+(KM_TYPE_NR*NR_CPUS)-1,
diff --git a/include/asm-powerpc/kvm_para.h b/include/asm-powerpc/kvm_para.h
--- a/include/asm-powerpc/kvm_para.h
+++ b/include/asm-powerpc/kvm_para.h
@@ -28,10 +28,18 @@
 
 #define KVM_HYPERCALL_BIN 0x03ffffff
 
+#define KVM_HCALL_RESERVE_MAGICPAGE	0
+
+#define KVM_PVDATA_MAGICPAGE_SIZE	"data,pv-magicpage-size"
+
+/* List of PV features supported, returned as a bitfield */
+#define KVM_FEATURE_PPCPV_MAGICPAGE	0
+
 static struct kvmppc_para_features {
 	char *dtcell;
 	int feature;
 } para_features[] = {
+	{ "feature,pv-magicpage", KVM_FEATURE_PPCPV_MAGICPAGE }
 };
 
 static inline int kvm_para_available(void)
@@ -67,6 +75,24 @@
 	of_node_put(dn);
 
 	return features;
+}
+
+/* reads the specified data field out of the hypervisor node */
+static inline int kvmppc_pv_read_data(char *dtcell)
+{
+	struct device_node *dn;
+	const int *dtval;
+
+	dn = of_find_node_by_path("/hypervisor");
+	if (!dn)
+		return -EINVAL;
+
+	dtval = of_get_property(dn, dtcell, NULL);
+	of_node_put(dn);
+	if (dtval)
+		return *dtval;
+	else
+		return -EINVAL;
 }
 
 void kvm_guest_init(void);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -550,6 +550,7 @@
 				prefetchw(p + 1);
 			__ClearPageReserved(p);
 			set_page_count(p, 0);
+
 		}
 
 		set_page_refcounted(page);

[PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[ehrhardt]

From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>

Dependent on the already existing CONFIG_KVM_GUEST config option this patch
changes wrteei to wrtee allowing the hypervisor to rewrite those to nontrapping
instructions. Maybe we should split the kvm guest otpimizations in two parts
one for the overhead free optimizations and on for the rest that might add
some complexity for non virtualized execution (like this one).

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---

[diffstat]
 hw_irq.h |   12 ++++++++++++
 1 file changed, 12 insertions(+)

[diff]
diff --git a/include/asm-powerpc/hw_irq.h b/include/asm-powerpc/hw_irq.h
--- a/include/asm-powerpc/hw_irq.h
+++ b/include/asm-powerpc/hw_irq.h
@@ -72,7 +72,11 @@
 static inline void local_irq_disable(void)
 {
 #ifdef CONFIG_BOOKE
+#ifdef CONFIG_KVM_GUEST
+	__asm__ __volatile__("wrtee %0": : "r"(0) :"memory");
+#else
 	__asm__ __volatile__("wrteei 0": : :"memory");
+#endif
 #else
 	unsigned long msr;
 	__asm__ __volatile__("": : :"memory");
@@ -84,7 +88,11 @@
 static inline void local_irq_enable(void)
 {
 #ifdef CONFIG_BOOKE
+#ifdef CONFIG_KVM_GUEST
+	__asm__ __volatile__("wrtee %0": : "r"(MSR_EE) :"memory");
+#else
 	__asm__ __volatile__("wrteei 1": : :"memory");
+#endif
 #else
 	unsigned long msr;
 	__asm__ __volatile__("": : :"memory");
@@ -99,7 +107,11 @@
 	msr = mfmsr();
 	*flags = msr;
 #ifdef CONFIG_BOOKE
+#ifdef CONFIG_KVM_GUEST
+	__asm__ __volatile__("wrtee %0": : "r"(0) :"memory");
+#else
 	__asm__ __volatile__("wrteei 0": : :"memory");
+#endif
 #else
 	SET_MSR_EE(msr & ~MSR_EE);
 #endif

Re: [PATCH 2/4] kvmppc: add hypercall infrastructure - guest part

[Arnd Bergmann]

T24gVHVlc2RheSAxOSBBdWd1c3QgMjAwOCwgZWhyaGFyZHRAbGludXgudm5ldC5pYm0uY29tIHdy
b3RlOgo+ICtzdGF0aWMgaW5saW5lIGxvbmcga3ZtX2h5cGVyY2FsbDEodW5zaWduZWQgaW50IG5y
LCB1bnNpZ25lZCBsb25nIHAxKQo+ICt7Cj4gK6CgoKCgoKByZWdpc3RlciB1bnNpZ25lZCBsb25n
IGhjYWxsIGFzbSAoInIwIikgPSBucjsKPiAroKCgoKCgoHJlZ2lzdGVyIHVuc2lnbmVkIGxvbmcg
YXJnMSBhc20gKCJyMyIpID0gcDE7Cj4gK6CgoKCgoKByZWdpc3RlciBsb25nIHJldCBhc20gKCJy
MTEiKTsKPiArCj4gK6CgoKCgoKBhc20gdm9sYXRpbGUoIi5sb25nICUxIgo+ICugoKCgoKCgoKCg
oKCgoKCgoKCgoKCgoDogIj1yIihyZXQpCj4gK6CgoKCgoKCgoKCgoKCgoKCgoKCgoKCgOiAiaSIo
S1ZNX0hZUEVSQ0FMTF9CSU4pLCAiciIoaGNhbGwpLCAiciIoYXJnMSkKPiAroKCgoKCgoKCgoKCg
oKCgoKCgoKCgoKA6ICJyNCIsICJyNSIsICJyNiIsICJyNyIsICJyOCIsCj4gK6CgoKCgoKCgoKCg
oKCgoKCgoKCgoKCgIKAicjkiLCAicjEwIiwgInIxMiIsICJjYyIpOwo+ICugoKCgoKCgcmV0dXJu
IHJldDsKPiArfQoKV2hhdCBpcyB0aGUgcmVhc29uaW5nIGZvciBtYWtpbmcgdGhlIGNhbGxpbmcg
Y29udmVudGlvbiBkaWZmZXJlbnQgZnJvbQphbGwgdGhlIGV4aXN0aW5nIGhjYWxsIGludGVyZmFj
ZXMgaGVyZT8KCnBzZXJpZXMgdXNlcyByMyBmb3IgdGhlIGhjYWxsIG51bWJlciwgbHYxIGFuZCBi
ZWF0IHVzZSByMTEsIHNvIHVzaW5nCnIwIGp1c3QgZm9yIHRoZSBzYWtlIG9mIGJlaW5nIGRpZmZl
cmVudCBzZWVtcyBjb3VudGVyaW50dWl0aXZlLgoKCUFybmQgPD48Cg==

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Arnd Bergmann]

On Tuesday 19 August 2008, ehrhardt@linux.vnet.ibm.com wrote:
> Dependent on the already existing CONFIG_KVM_GUEST config option this patch
> changes wrteei to wrtee allowing the hypervisor to rewrite those to nontrapping
> instructions. Maybe we should split the kvm guest otpimizations in two parts
> one for the overhead free optimizations and on for the rest that might add
> some complexity for non virtualized execution (like this one).
> 
> Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>

How significant is the performance impact of this change for non-virtualized
systems? If it's very low, maybe you should not bother with the #ifdef, and
if it's noticable, you might be better off using dynamic patching for this.

	Arnd <><

Re: [PATCH 1/4] kvmppc: read device tree hypervisor node infrastructure

[Josh Boyer]

On Tue, 2008-08-19 at 12:36 +0200, ehrhardt@linux.vnet.ibm.com wrote:
> diff --git a/arch/powerpc/kernel/kvm.c b/arch/powerpc/kernel/kvm.c
> new file mode 100644
> --- /dev/null
> +++ b/arch/powerpc/kernel/kvm.c
> @@ -0,0 +1,30 @@

> +#include <linux/percpu.h>
> +#include <linux/mm.h>
> +#include <linux/kvm_para.h>
> +
> +void __init kvm_guest_init(void)
> +{
> +	if (!kvm_para_available())
> +		return;
> +}

This looks really odd.  You have a void function that checks the return
value of another function and returns if not true or.. returns if true.
Why bother with the if at all?

One could ask the same about the entire function, but it does look
cleaner to call kvm_guest_init that kvm_para_available directly from
other code.

josh

Re: [PATCH 1/4] kvmppc: read device tree hypervisor node infrastructure

[Josh Boyer]

On Tue, 2008-08-19 at 07:52 -0400, Josh Boyer wrote:
> On Tue, 2008-08-19 at 12:36 +0200, ehrhardt@linux.vnet.ibm.com wrote:
> > diff --git a/arch/powerpc/kernel/kvm.c b/arch/powerpc/kernel/kvm.c
> > new file mode 100644
> > --- /dev/null
> > +++ b/arch/powerpc/kernel/kvm.c
> > @@ -0,0 +1,30 @@
> 
> > +#include <linux/percpu.h>
> > +#include <linux/mm.h>
> > +#include <linux/kvm_para.h>
> > +
> > +void __init kvm_guest_init(void)
> > +{
> > +	if (!kvm_para_available())
> > +		return;
> > +}
> 
> This looks really odd.  You have a void function that checks the return
> value of another function and returns if not true or.. returns if true.
> Why bother with the if at all?

Nevermind.  I see you add more code below this in patch 3.  Still looks
odd by itself, but makes more sense when the whole series is applied.

josh

Re: [PATCH 3/4] kvmppc: magic page paravirtualization - guest part

[Tony Breeds]

Hi Christian,
	One very minor nit.

On Tue, Aug 19, 2008 at 12:36:43PM +0200, ehrhardt@linux.vnet.ibm.com wrote:
 
> [diffstat]
>  mm/page_alloc.c                |    1

<snip>

>  void kvm_guest_init(void);
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -550,6 +550,7 @@
>  				prefetchw(p + 1);
>  			__ClearPageReserved(p);
>  			set_page_count(p, 0);
> +
>  		}
>  
>  		set_page_refcounted(page);

I guess this is a leftover from some debugging but still should be
killed :)

Yours Tony

  linux.conf.au    http://www.marchsouth.org/
  Jan 19 - 24 2009 The Australian Linux Technical Conference!

Re: [PATCH 2/4] kvmppc: add hypercall infrastructure - guest part

[Christian Ehrhardt]

Arnd Bergmann wrote:
> On Tuesday 19 August 2008, ehrhardt@linux.vnet.ibm.com wrote:
>   
>> +static inline long kvm_hypercall1(unsigned int nr, unsigned long p1)
>> +{
>> +       register unsigned long hcall asm ("r0") = nr;
>> +       register unsigned long arg1 asm ("r3") = p1;
>> +       register long ret asm ("r11");
>> +
>> +       asm volatile(".long %1"
>> +                       : "=r"(ret)
>> +                       : "i"(KVM_HYPERCALL_BIN), "r"(hcall), "r"(arg1)
>> +                       : "r4", "r5", "r6", "r7", "r8",
>> +                         "r9", "r10", "r12", "cc");
>> +       return ret;
>> +}
>>     
>
> What is the reasoning for making the calling convention different from
> all the existing hcall interfaces here?
>
> pseries uses r3 for the hcall number, lv1 and beat use r11, so using
> r0 just for the sake of being different seems counterintuitive.
>
> 	Arnd <><
>   
Some documentation is here 
http://kvm.qumranet.com/kvmwiki/PowerPC_Hypercall_ABI
As far as I remember it was oriented on system calls, from my point we 
can still change it atm.
When we discussed about that I was too new to the power architecture to 
really get all the details, but I assume Hollis and Jimi can answer you 
that.


-- 

Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Christian Ehrhardt]

Arnd Bergmann wrote:
> On Tuesday 19 August 2008, ehrhardt@linux.vnet.ibm.com wrote:
>   
>> Dependent on the already existing CONFIG_KVM_GUEST config option this patch
>> changes wrteei to wrtee allowing the hypervisor to rewrite those to nontrapping
>> instructions. Maybe we should split the kvm guest otpimizations in two parts
>> one for the overhead free optimizations and on for the rest that might add
>> some complexity for non virtualized execution (like this one).
>>
>> Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
>>     
>
> How significant is the performance impact of this change for non-virtualized
> systems? If it's very low, maybe you should not bother with the #ifdef, and
> if it's noticable, you might be better off using dynamic patching for this.
>
> 	Arnd <><
>   
To be honest I unfortunately don't know how big the impact for 
non-virtualized systems is. I would like to test it, but without 
hardware performance counters on the core I have I'm not sure (yet) how 
to measure that in a good way - any suggestion welcome.
I'm really sure that any jumping around style dynamic patching in the 
guest like function pointers etc will be slower than just let the load 
be there. Unfortunately I can not rewrite it from the hypervisor because 
for "wrteei" I would need a "stwi" to rewrite it in one instruction.
The patch as it is today let you choose between 10% benefit for 
virtualized guest and an unkown but surely very small overhead on native 
hardware.

-- 

Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Hollis Blanchard]

On Wed, 2008-08-20 at 14:53 +0200, Christian Ehrhardt wrote:
> 
> Arnd Bergmann wrote:
> > On Tuesday 19 August 2008, ehrhardt@linux.vnet.ibm.com wrote:
> >   
> >> Dependent on the already existing CONFIG_KVM_GUEST config option
> this patch
> >> changes wrteei to wrtee allowing the hypervisor to rewrite those to
> nontrapping
> >> instructions. Maybe we should split the kvm guest otpimizations in
> two parts
> >> one for the overhead free optimizations and on for the rest that
> might add
> >> some complexity for non virtualized execution (like this one).
> >>
> >> Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
> >>     
> >
> > How significant is the performance impact of this change for
> non-virtualized
> > systems? If it's very low, maybe you should not bother with the
> #ifdef, and
> > if it's noticable, you might be better off using dynamic patching
> for this.
> >
> >       Arnd <><
> >   
> To be honest I unfortunately don't know how big the impact for 
> non-virtualized systems is. I would like to test it, but without 
> hardware performance counters on the core I have I'm not sure (yet)
> how 
> to measure that in a good way - any suggestion welcome.

I don't see why we need performance counters. Can't we just compare any
bare metal benchmark results with the patch both applied and not?

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Josh Boyer]

On Wed, 2008-08-20 at 13:30 -0500, Hollis Blanchard wrote:
> On Wed, 2008-08-20 at 14:53 +0200, Christian Ehrhardt wrote:
> > 
> > Arnd Bergmann wrote:
> > > On Tuesday 19 August 2008, ehrhardt@linux.vnet.ibm.com wrote:
> > >   
> > >> Dependent on the already existing CONFIG_KVM_GUEST config option
> > this patch
> > >> changes wrteei to wrtee allowing the hypervisor to rewrite those to
> > nontrapping
> > >> instructions. Maybe we should split the kvm guest otpimizations in
> > two parts
> > >> one for the overhead free optimizations and on for the rest that
> > might add
> > >> some complexity for non virtualized execution (like this one).
> > >>
> > >> Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
> > >>     
> > >
> > > How significant is the performance impact of this change for
> > non-virtualized
> > > systems? If it's very low, maybe you should not bother with the
> > #ifdef, and
> > > if it's noticable, you might be better off using dynamic patching
> > for this.
> > >
> > >       Arnd <><
> > >   
> > To be honest I unfortunately don't know how big the impact for 
> > non-virtualized systems is. I would like to test it, but without 
> > hardware performance counters on the core I have I'm not sure (yet)
> > how 
> > to measure that in a good way - any suggestion welcome.
> 
> I don't see why we need performance counters. Can't we just compare any
> bare metal benchmark results with the patch both applied and not?

Do you know of one that causes a large amount of
local_irq_{disable,enable}s to be called?

josh

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Hollis Blanchard]

On Wed, 2008-08-20 at 14:52 -0400, Josh Boyer wrote:
> On Wed, 2008-08-20 at 13:30 -0500, Hollis Blanchard wrote:
> > On Wed, 2008-08-20 at 14:53 +0200, Christian Ehrhardt wrote:
> > > 
> > > Arnd Bergmann wrote:
> > > > On Tuesday 19 August 2008, ehrhardt@linux.vnet.ibm.com wrote:
> > > >   
> > > >> Dependent on the already existing CONFIG_KVM_GUEST config option
> > > this patch
> > > >> changes wrteei to wrtee allowing the hypervisor to rewrite those to
> > > nontrapping
> > > >> instructions. Maybe we should split the kvm guest otpimizations in
> > > two parts
> > > >> one for the overhead free optimizations and on for the rest that
> > > might add
> > > >> some complexity for non virtualized execution (like this one).
> > > >>
> > > >> Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
> > > >>     
> > > >
> > > > How significant is the performance impact of this change for
> > > non-virtualized
> > > > systems? If it's very low, maybe you should not bother with the
> > > #ifdef, and
> > > > if it's noticable, you might be better off using dynamic patching
> > > for this.
> > > >
> > > >       Arnd <><
> > > >   
> > > To be honest I unfortunately don't know how big the impact for 
> > > non-virtualized systems is. I would like to test it, but without 
> > > hardware performance counters on the core I have I'm not sure (yet)
> > > how 
> > > to measure that in a good way - any suggestion welcome.
> > 
> > I don't see why we need performance counters. Can't we just compare any
> > bare metal benchmark results with the patch both applied and not?
> 
> Do you know of one that causes a large amount of
> local_irq_{disable,enable}s to be called?

I think *every* workload causes a large number of
local_irq_{disable,enable} calls... :)

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Josh Boyer]

On Wed, 20 Aug 2008 14:06:51 -0500
Hollis Blanchard <hollisb@us.ibm.com> wrote:  
> > > > To be honest I unfortunately don't know how big the impact for 
> > > > non-virtualized systems is. I would like to test it, but without 
> > > > hardware performance counters on the core I have I'm not sure (yet)
> > > > how 
> > > > to measure that in a good way - any suggestion welcome.
> > > 
> > > I don't see why we need performance counters. Can't we just compare any
> > > bare metal benchmark results with the patch both applied and not?
> > 
> > Do you know of one that causes a large amount of
> > local_irq_{disable,enable}s to be called?
> 
> I think *every* workload causes a large number of
> local_irq_{disable,enable} calls... :)

Well, sure.  I was just going for "test the change as specifically as
possible."  One could write a module that did X number of
disable/enable pairs and reported the timebase at start and end to
compare.  X could even be a module parameter.  Just to try and
eliminate noise or whatever from the testing.

/me shrugs.

josh

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Christian Ehrhardt]

Josh Boyer wrote:
> On Wed, 20 Aug 2008 14:06:51 -0500
> Hollis Blanchard <hollisb@us.ibm.com> wrote:  
>   
>>>>> To be honest I unfortunately don't know how big the impact for 
>>>>> non-virtualized systems is. I would like to test it, but without 
>>>>> hardware performance counters on the core I have I'm not sure (yet)
>>>>> how 
>>>>> to measure that in a good way - any suggestion welcome.
>>>>>           
>>>> I don't see why we need performance counters. Can't we just compare any
>>>> bare metal benchmark results with the patch both applied and not?
>>>>         
>>> Do you know of one that causes a large amount of
>>> local_irq_{disable,enable}s to be called?
>>>       
>> I think *every* workload causes a large number of
>> local_irq_{disable,enable} calls... :)
>>     
>
> Well, sure.  I was just going for "test the change as specifically as
> possible."  One could write a module that did X number of
> disable/enable pairs and reported the timebase at start and end to
> compare.  X could even be a module parameter.  Just to try and
> eliminate noise or whatever from the testing.
>
> /me shrugs.
>
> josh
>   
yeah I thought of something like that too, because I expect the 
difference to be very small.
Instead of a module I wanted to put this somewhere prior to the kernel 
mounting root-fs to avoid interferences from whatever userspace is doing 
(e.g. causing  thousands of interrupts come back while the module 
perform that test.).
Eventually we need a synthetic benchmark like that AND a check how it 
affects a common system to be sure.


-- 

Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Kumar Gala]

On Aug 19, 2008, at 5:36 AM, ehrhardt@linux.vnet.ibm.com wrote:

> From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
>
> Dependent on the already existing CONFIG_KVM_GUEST config option  
> this patch
> changes wrteei to wrtee allowing the hypervisor to rewrite those to  
> nontrapping
> instructions. Maybe we should split the kvm guest otpimizations in  
> two parts
> one for the overhead free optimizations and on for the rest that  
> might add
> some complexity for non virtualized execution (like this one).
>
> Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
> ---

So this commit message doesnt explain why 'wrtee' facilities doing  
whatever optimization and 'wrteei' doesnt.

- k

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Christian Ehrhardt]

Kumar Gala wrote:
>
> On Aug 19, 2008, at 5:36 AM, ehrhardt@linux.vnet.ibm.com wrote:
>
>> From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
>>
>> Dependent on the already existing CONFIG_KVM_GUEST config option this 
>> patch
>> changes wrteei to wrtee allowing the hypervisor to rewrite those to 
>> nontrapping
>> instructions. Maybe we should split the kvm guest otpimizations in 
>> two parts
>> one for the overhead free optimizations and on for the rest that 
>> might add
>> some complexity for non virtualized execution (like this one).
>>
>> Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
>> ---
>
> So this commit message doesnt explain why 'wrtee' facilities doing 
> whatever optimization and 'wrteei' doesnt.
>
yep I only explained it elsewhere.
I'll add it here too in the next version - thanks Kumar
(and fix the word otpimizations)

-- 

Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Kumar Gala]

On Aug 21, 2008, at 9:13 AM, Christian Ehrhardt wrote:

> Kumar Gala wrote:
>>
>> On Aug 19, 2008, at 5:36 AM, ehrhardt@linux.vnet.ibm.com wrote:
>>
>>> From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
>>>
>>> Dependent on the already existing CONFIG_KVM_GUEST config option  
>>> this patch
>>> changes wrteei to wrtee allowing the hypervisor to rewrite those  
>>> to nontrapping
>>> instructions. Maybe we should split the kvm guest otpimizations in  
>>> two parts
>>> one for the overhead free optimizations and on for the rest that  
>>> might add
>>> some complexity for non virtualized execution (like this one).
>>>
>>> Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
>>> ---
>>
>> So this commit message doesnt explain why 'wrtee' facilities doing  
>> whatever optimization and 'wrteei' doesnt.
>>
> yep I only explained it elsewhere.
> I'll add it here too in the next version - thanks Kumar
> (and fix the word otpimizations)

Where is the other discussion?  I'd like to understand what's going on  
here.. (especially since I added the wrtee[i] changes to kernel way  
back when).

- k

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Scott Wood]

On Thu, Aug 21, 2008 at 09:21:39AM -0500, Kumar Gala wrote:
> Where is the other discussion?  I'd like to understand what's going on  
> here.. (especially since I added the wrtee[i] changes to kernel way  
> back when).

Presumably, they want to be able to replace wrtee with a store to a
hypervisor/guest shared memory area, and there's no store-immediate
instruction.

-Scott

Re: [PATCH 2/4] kvmppc: add hypercall infrastructure - guest part

[Hollis Blanchard]

On Tue, 2008-08-19 at 13:28 +0200, Arnd Bergmann wrote:
> On Tuesday 19 August 2008, ehrhardt@linux.vnet.ibm.com wrote:
> > +static inline long kvm_hypercall1(unsigned int nr, unsigned long p1)
> > +{
> > +       register unsigned long hcall asm ("r0") = nr;
> > +       register unsigned long arg1 asm ("r3") = p1;
> > +       register long ret asm ("r11");
> > +
> > +       asm volatile(".long %1"
> > +                       : "=r"(ret)
> > +                       : "i"(KVM_HYPERCALL_BIN), "r"(hcall), "r"(arg1)
> > +                       : "r4", "r5", "r6", "r7", "r8",
> > +                         "r9", "r10", "r12", "cc");
> > +       return ret;
> > +}
> 
> What is the reasoning for making the calling convention different from
> all the existing hcall interfaces here?
> 
> pseries uses r3 for the hcall number, lv1 and beat use r11, so using
> r0 just for the sake of being different seems counterintuitive.

There was a really good reason at the time, but I can't seem to remember
it now. ;) We're thinking about it.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Christian Ehrhardt]

Scott Wood wrote:
> On Thu, Aug 21, 2008 at 09:21:39AM -0500, Kumar Gala wrote:
>   
>> Where is the other discussion?  I'd like to understand what's going on  
>> here.. (especially since I added the wrtee[i] changes to kernel way  
>> back when).
>>     
>
> Presumably, they want to be able to replace wrtee with a store to a
> hypervisor/guest shared memory area, and there's no store-immediate
> instruction.
>
> -Scott
>   
Exactly Scott

And for your question Kumar, in the last submission I was asked to split 
host and guest patches.
So the host discussion lives on kvm-ppc@vger.kernel.org as I mentioned 
(maybe a bit too hidden)
in the [0/4] mail of this series.


-- 

Grüsse / regards, 
Christian Ehrhardt
IBM Linux Technology Center, Open Virtualization

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Kumar Gala]

On Aug 22, 2008, at 3:08 AM, Christian Ehrhardt wrote:

> Scott Wood wrote:
>> On Thu, Aug 21, 2008 at 09:21:39AM -0500, Kumar Gala wrote:
>>
>>> Where is the other discussion?  I'd like to understand what's  
>>> going on  here.. (especially since I added the wrtee[i] changes to  
>>> kernel way  back when).
>>>
>>
>> Presumably, they want to be able to replace wrtee with a store to a
>> hypervisor/guest shared memory area, and there's no store-immediate
>> instruction.
>>
>> -Scott
>>
> Exactly Scott
>
> And for your question Kumar, in the last submission I was asked to  
> split host and guest patches.
> So the host discussion lives on kvm-ppc@vger.kernel.org as I  
> mentioned (maybe a bit too hidden)
> in the [0/4] mail of this series.

Maybe I'm missing something but it seems like just changing the code  
to wrtee isn't the best way to ensure you have enough space to put the  
set of instructions you guys want.

- k

Re: [PATCH 2/4] kvmppc: add hypercall infrastructure - guest part

[Kumar Gala]

On Aug 19, 2008, at 5:36 AM, ehrhardt@linux.vnet.ibm.com wrote:

> From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
>
> This adds the guest portion of the hypercall infrastructure,  
> basically an
> illegal instruction with a defined layout.
> See http://kvm.qumranet.com/kvmwiki/PowerPC_Hypercall_ABI for more  
> detail
> on the hypercall ABI for powerpc.
>
> Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
> ---
>
> [diffstat]
> kvm_para.h |   33 +++++++++++++++++++++++++++++++++
> 1 file changed, 33 insertions(+)
>
> [diff]

Can we make this more generic (naming) and location.  I believe there  
is some interesting in using the same ABI on the Freescale HV so it  
seems like have the wrappers be a bit more generic saves us.

- k

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Jimi Xenidis]

On Aug 22, 2008, at 4:17 AM, Kumar Gala wrote:

>
> On Aug 22, 2008, at 3:08 AM, Christian Ehrhardt wrote:
>
>> Scott Wood wrote:
>>> On Thu, Aug 21, 2008 at 09:21:39AM -0500, Kumar Gala wrote:
>>>
>>>> Where is the other discussion?  I'd like to understand what's  
>>>> going on  here.. (especially since I added the wrtee[i] changes  
>>>> to kernel way  back when).
>>>>
>>>
>>> Presumably, they want to be able to replace wrtee with a store to a
>>> hypervisor/guest shared memory area, and there's no store-immediate
>>> instruction.
>>>
>>> -Scott
>>>
>> Exactly Scott
>>
>> And for your question Kumar, in the last submission I was asked to  
>> split host and guest patches.
>> So the host discussion lives on kvm-ppc@vger.kernel.org as I  
>> mentioned (maybe a bit too hidden)
>> in the [0/4] mail of this series.
>
> Maybe I'm missing something but it seems like just changing the code  
> to wrtee isn't the best way to ensure you have enough space to put  
> the set of instructions you guys want.
>

Sorry, I have not really been following this, I assume you are  
replacing SPR accesses with load/stores that do not trap.
I believe thats fine for reading any SPR, but not true for modifying  
some SPRs, particularly MSR.
IMHO you must trap on any instruction that modifies the MSR.
So you should prolly not rewrite mtmsr or wrtee* but let them trap.

BTW: actually moving problem SPRs (like MSRs) to a read only page  
makes decoding far more efficient, but it does burn another TLB entry.

-JX


> - k
> --
> To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/4] kvmppc: add hypercall infrastructure - guest part

[Hollis Blanchard]

On Friday 22 August 2008 05:38:54 Kumar Gala wrote:
> 
> On Aug 19, 2008, at 5:36 AM, ehrhardt@linux.vnet.ibm.com wrote:
> 
> > From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
> >
> > This adds the guest portion of the hypercall infrastructure,  
> > basically an
> > illegal instruction with a defined layout.
> > See http://kvm.qumranet.com/kvmwiki/PowerPC_Hypercall_ABI for more  
> > detail
> > on the hypercall ABI for powerpc.
> >
> > Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
> > ---
> >
> > [diffstat]
> > kvm_para.h |   33 +++++++++++++++++++++++++++++++++
> > 1 file changed, 33 insertions(+)
> >
> > [diff]
> 
> Can we make this more generic (naming) and location.  I believe there  
> is some interesting in using the same ABI on the Freescale HV so it  
> seems like have the wrappers be a bit more generic saves us.

Agreed. Maybe something like epapr_hcalls.h .

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: [PATCH 4/4] kvmppc: convert wrteei to wrtee as kvm guest optimization

[Scott Wood]

On Fri, Aug 22, 2008 at 09:56:16AM -0400, Jimi Xenidis wrote:
> Sorry, I have not really been following this, I assume you are  
> replacing SPR accesses with load/stores that do not trap.
> I believe thats fine for reading any SPR, but not true for modifying  
> some SPRs, particularly MSR.
> IMHO you must trap on any instruction that modifies the MSR.
> So you should prolly not rewrite mtmsr or wrtee* but let them trap.

You only need to trap when changing MSR[EE] if there's a pending
interrupt.  The hypervisor can change the mapping of the shared page when
it is unable to deliver an interrupt due to the virtual EE value being
zero.

-Scott