[PATCH 0 of 12] PV on HVM Xen

xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed

* [PATCH 0 of 12] PV on HVM Xen
@ 2010-05-24 18:25 Stefano Stabellini
  2010-05-24 18:27 ` [PATCH 01/11] Add support for hvm_op Stefano Stabellini
                   ` (14 more replies)
  0 siblings, 15 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:25 UTC (permalink / raw)
  To: linux-kernel@vger.kernel.org
  Cc: Stefano Stabellini, Jeremy Fitzhardinge,
	xen-devel@lists.xensource.com, Don Dutile, Sheng Yang

Hi all,
this is another update of the PV on HVM Xen series that addresses
Jeremy's comments.
The platform_pci hooks have been removed, suspend/resume for HVM
domains is now much more similar to the PV case and shares the same
do_suspend function.
Alloc_xen_mmio_hook has been removed has well, now the memory allocation for
the grant table is done by the xen platform pci driver directly.
The per_cpu xen_vcpu variable is set by a cpu_notifier function so that
secondary vcpus have the variable set correctly no matter what the xen
features are on the host.
The kernel command line option xen_unplug has been renamed to
xen_emul_unplug and the code that makes use of it has been moved to a
separate file (arch/x86/xen/platform-pci-unplug.c).
Xen_unplug_emulated_devices is now able to detect if blkfront, netfront
and the Xen platform PCI driver have been compiled, and set the default
value of xen_emul_unplug accordingly.
The patch "Initialize xenbus device structs with ENODEV as
default" has been removed from the series and it will be sent
separately.
Finally the comments on most of the patches have been improved.

The series is based on 2.6.34 and supports Xen PV frontends running
in a HVM domain, including netfront, blkfront and the VIRQ_TIMER.

In order to be able to use VIRQ_TIMER and to improve performances you
need a patch to Xen to implement the vector callback mechanism
for event channel delivery.

A git tree is also available here:

git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git

branch name 2.6.34-pvhvm-v2.

Cheers,

Stefano

^ permalink raw reply	[flat|nested] 39+ messages in thread

* [PATCH 01/11] Add support for hvm_op
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
@ 2010-05-24 18:27 ` Stefano Stabellini
  2010-05-24 18:27 ` [PATCH 02/11] early PV on HVM Stefano Stabellini
                   ` (13 subsequent siblings)
  14 siblings, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:27 UTC (permalink / raw)
  To: linux-kernel
  Cc: xen-devel, Don Dutile, Stefano Stabellini, Jeremy Fitzhardinge,
	Jeremy Fitzhardinge, Sheng Yang, Stefano Stabellini

From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Sheng Yang <sheng@linux.intel.com>
Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 arch/x86/include/asm/xen/hypercall.h |    6 ++
 include/xen/hvm.h                    |   24 +++++++++
 include/xen/interface/hvm/hvm_op.h   |   35 ++++++++++++
 include/xen/interface/hvm/params.h   |   95 ++++++++++++++++++++++++++++++++++
 4 files changed, 160 insertions(+), 0 deletions(-)
 create mode 100644 include/xen/hvm.h
 create mode 100644 include/xen/interface/hvm/hvm_op.h
 create mode 100644 include/xen/interface/hvm/params.h

diff --git a/arch/x86/include/asm/xen/hypercall.h b/arch/x86/include/asm/xen/hypercall.h
index 9c371e4..7fda040 100644
--- a/arch/x86/include/asm/xen/hypercall.h
+++ b/arch/x86/include/asm/xen/hypercall.h
@@ -417,6 +417,12 @@ HYPERVISOR_nmi_op(unsigned long op, unsigned long arg)
 	return _hypercall2(int, nmi_op, op, arg);
 }
 
+static inline unsigned long __must_check
+HYPERVISOR_hvm_op(int op, void *arg)
+{
+       return _hypercall2(unsigned long, hvm_op, op, arg);
+}
+
 static inline void
 MULTI_fpu_taskswitch(struct multicall_entry *mcl, int set)
 {
diff --git a/include/xen/hvm.h b/include/xen/hvm.h
new file mode 100644
index 0000000..6b0d418
--- /dev/null
+++ b/include/xen/hvm.h
@@ -0,0 +1,24 @@
+/* Simple wrappers around HVM functions */
+#ifndef XEN_HVM_H__
+#define XEN_HVM_H__
+
+#include <xen/interface/hvm/params.h>
+
+static inline int hvm_get_parameter(int idx, uint64_t *value)
+{
+       struct xen_hvm_param xhv;
+       int r;
+
+       xhv.domid = DOMID_SELF;
+       xhv.index = idx;
+       r = HYPERVISOR_hvm_op(HVMOP_get_param, &xhv);
+       if (r < 0) {
+               printk(KERN_ERR "cannot get hvm parameter %d: %d.\n",
+                      idx, r);
+               return r;
+       }
+       *value = xhv.value;
+       return r;
+}
+
+#endif /* XEN_HVM_H__ */
diff --git a/include/xen/interface/hvm/hvm_op.h b/include/xen/interface/hvm/hvm_op.h
new file mode 100644
index 0000000..73c8c7e
--- /dev/null
+++ b/include/xen/interface/hvm/hvm_op.h
@@ -0,0 +1,35 @@
+/*
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef __XEN_PUBLIC_HVM_HVM_OP_H__
+#define __XEN_PUBLIC_HVM_HVM_OP_H__
+
+/* Get/set subcommands: the second argument of the hypercall is a
+ * pointer to a xen_hvm_param struct. */
+#define HVMOP_set_param           0
+#define HVMOP_get_param           1
+struct xen_hvm_param {
+    domid_t  domid;    /* IN */
+    uint32_t index;    /* IN */
+    uint64_t value;    /* IN/OUT */
+};
+DEFINE_GUEST_HANDLE_STRUCT(xen_hvm_param);
+
+#endif /* __XEN_PUBLIC_HVM_HVM_OP_H__ */
diff --git a/include/xen/interface/hvm/params.h b/include/xen/interface/hvm/params.h
new file mode 100644
index 0000000..1888d8c
--- /dev/null
+++ b/include/xen/interface/hvm/params.h
@@ -0,0 +1,95 @@
+/*
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef __XEN_PUBLIC_HVM_PARAMS_H__
+#define __XEN_PUBLIC_HVM_PARAMS_H__
+
+#include "hvm_op.h"
+
+/*
+ * Parameter space for HVMOP_{set,get}_param.
+ */
+
+/*
+ * How should CPU0 event-channel notifications be delivered?
+ * val[63:56] == 0: val[55:0] is a delivery GSI (Global System Interrupt).
+ * val[63:56] == 1: val[55:0] is a delivery PCI INTx line, as follows:
+ *                  Domain = val[47:32], Bus  = val[31:16],
+ *                  DevFn  = val[15: 8], IntX = val[ 1: 0]
+ * val[63:56] == 2: val[7:0] is a vector number.
+ * If val == 0 then CPU0 event-channel notifications are not delivered.
+ */
+#define HVM_PARAM_CALLBACK_IRQ 0
+
+#define HVM_PARAM_STORE_PFN    1
+#define HVM_PARAM_STORE_EVTCHN 2
+
+#define HVM_PARAM_PAE_ENABLED  4
+
+#define HVM_PARAM_IOREQ_PFN    5
+
+#define HVM_PARAM_BUFIOREQ_PFN 6
+
+/*
+ * Set mode for virtual timers (currently x86 only):
+ *  delay_for_missed_ticks (default):
+ *   Do not advance a vcpu's time beyond the correct delivery time for
+ *   interrupts that have been missed due to preemption. Deliver missed
+ *   interrupts when the vcpu is rescheduled and advance the vcpu's virtual
+ *   time stepwise for each one.
+ *  no_delay_for_missed_ticks:
+ *   As above, missed interrupts are delivered, but guest time always tracks
+ *   wallclock (i.e., real) time while doing so.
+ *  no_missed_ticks_pending:
+ *   No missed interrupts are held pending. Instead, to ensure ticks are
+ *   delivered at some non-zero rate, if we detect missed ticks then the
+ *   internal tick alarm is not disabled if the VCPU is preempted during the
+ *   next tick period.
+ *  one_missed_tick_pending:
+ *   Missed interrupts are collapsed together and delivered as one 'late tick'.
+ *   Guest time always tracks wallclock (i.e., real) time.
+ */
+#define HVM_PARAM_TIMER_MODE   10
+#define HVMPTM_delay_for_missed_ticks    0
+#define HVMPTM_no_delay_for_missed_ticks 1
+#define HVMPTM_no_missed_ticks_pending   2
+#define HVMPTM_one_missed_tick_pending   3
+
+/* Boolean: Enable virtual HPET (high-precision event timer)? (x86-only) */
+#define HVM_PARAM_HPET_ENABLED 11
+
+/* Identity-map page directory used by Intel EPT when CR0.PG=0. */
+#define HVM_PARAM_IDENT_PT     12
+
+/* Device Model domain, defaults to 0. */
+#define HVM_PARAM_DM_DOMAIN    13
+
+/* ACPI S state: currently support S0 and S3 on x86. */
+#define HVM_PARAM_ACPI_S_STATE 14
+
+/* TSS used on Intel when CR0.PE=0. */
+#define HVM_PARAM_VM86_TSS     15
+
+/* Boolean: Enable aligning all periodic vpts to reduce interrupts */
+#define HVM_PARAM_VPT_ALIGN    16
+
+#define HVM_NR_PARAMS          17
+
+#endif /* __XEN_PUBLIC_HVM_PARAMS_H__ */
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* [PATCH 02/11] early PV on HVM
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
  2010-05-24 18:27 ` [PATCH 01/11] Add support for hvm_op Stefano Stabellini
@ 2010-05-24 18:27 ` Stefano Stabellini
  2010-05-24 18:27 ` [PATCH 03/11] evtchn delivery " Stefano Stabellini
                   ` (12 subsequent siblings)
  14 siblings, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:27 UTC (permalink / raw)
  To: linux-kernel
  Cc: xen-devel, Don Dutile, Stefano Stabellini, Jeremy Fitzhardinge,
	Sheng Yang, Stefano Stabellini, Yaozu (Eddie) Dong

From: Sheng Yang <sheng@linux.intel.com>

Initialize basic pv on hvm features in xen_guest_init.

The hook in arch/x86/kernel/setup.c can easily be removed using the new
generic hypervisor independent initialization infrastructure present in
the linux-2.6-tip tree.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Signed-off-by: Sheng Yang <sheng@linux.intel.com>
Signed-off-by: Yaozu (Eddie) Dong <eddie.dong@intel.com>
---
 arch/x86/kernel/setup.c           |    2 +
 arch/x86/xen/enlighten.c          |   86 +++++++++++++++++++++++++++++++++++++
 drivers/input/xen-kbdfront.c      |    2 +-
 drivers/video/xen-fbfront.c       |    2 +-
 drivers/xen/xenbus/xenbus_probe.c |   21 ++++++++-
 include/xen/xen.h                 |    2 +
 6 files changed, 110 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index c4851ef..ae9b6cb 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -69,6 +69,7 @@
 #include <linux/tboot.h>
 
 #include <video/edid.h>
+#include <xen/xen.h>
 
 #include <asm/mtrr.h>
 #include <asm/apic.h>
@@ -1032,6 +1033,7 @@ void __init setup_arch(char **cmdline_p)
 	probe_nr_irqs_gsi();
 
 	kvm_guest_init();
+	xen_guest_init();
 
 	e820_reserve_resources();
 	e820_mark_nosave_regions(max_low_pfn);
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 65d8d79..c1f6545 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -35,6 +35,7 @@
 #include <xen/interface/version.h>
 #include <xen/interface/physdev.h>
 #include <xen/interface/vcpu.h>
+#include <xen/interface/memory.h>
 #include <xen/features.h>
 #include <xen/page.h>
 #include <xen/hvc-console.h>
@@ -56,6 +57,7 @@
 #include <asm/tlbflush.h>
 #include <asm/reboot.h>
 #include <asm/stackprotector.h>
+#include <asm/hypervisor.h>
 
 #include "xen-ops.h"
 #include "mmu.h"
@@ -1206,3 +1208,87 @@ asmlinkage void __init xen_start_kernel(void)
 	x86_64_start_reservations((char *)__pa_symbol(&boot_params));
 #endif
 }
+
+static uint32_t xen_cpuid_base(void)
+{
+	uint32_t base, eax, ebx, ecx, edx;
+	char signature[13];
+
+	for (base = 0x40000000; base < 0x40010000; base += 0x100) {
+		cpuid(base, &eax, &ebx, &ecx, &edx);
+		*(uint32_t*)(signature + 0) = ebx;
+		*(uint32_t*)(signature + 4) = ecx;
+		*(uint32_t*)(signature + 8) = edx;
+		signature[12] = 0;
+
+		if (!strcmp("XenVMMXenVMM", signature) && ((eax - base) >= 2))
+			return base;
+	}
+
+	return 0;
+}
+
+static int init_hvm_pv_info(int *major, int *minor)
+{
+	uint32_t eax, ebx, ecx, edx, pages, msr, base;
+	u64 pfn;
+
+	base = xen_cpuid_base();
+	if (!base)
+		return -EINVAL;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	*major = eax >> 16;
+	*minor = eax & 0xffff;
+	printk(KERN_INFO "Xen version %d.%d.\n", *major, *minor);
+
+	cpuid(base + 2, &pages, &msr, &ecx, &edx);
+
+	pfn = __pa(hypercall_page);
+	wrmsr_safe(msr, (u32)pfn, (u32)(pfn >> 32));
+
+	xen_setup_features();
+
+	pv_info = xen_info;
+	pv_info.kernel_rpl = 0;
+
+	xen_domain_type = XEN_HVM_DOMAIN;
+
+	return 0;
+}
+
+static void __init init_shared_info(void)
+{
+	struct xen_add_to_physmap xatp;
+	struct shared_info *shared_info_page;
+
+	shared_info_page = (struct shared_info *) alloc_bootmem_pages(PAGE_SIZE);
+	xatp.domid = DOMID_SELF;
+	xatp.idx = 0;
+	xatp.space = XENMAPSPACE_shared_info;
+	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
+		BUG();
+
+	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+
+	/* Don't do the full vcpu_info placement stuff until we have a
+	   possible map and a non-dummy shared_info. */
+	per_cpu(xen_vcpu, 0) = &HYPERVISOR_shared_info->vcpu_info[0];
+}
+
+void __init xen_guest_init(void)
+{
+	int r;
+	int major, minor;
+
+	if (xen_pv_domain())
+		return;
+
+	r = init_hvm_pv_info(&major, &minor);
+	if (r < 0)
+		return;
+
+	init_shared_info();
+}
diff --git a/drivers/input/xen-kbdfront.c b/drivers/input/xen-kbdfront.c
index e140816..7451d78 100644
--- a/drivers/input/xen-kbdfront.c
+++ b/drivers/input/xen-kbdfront.c
@@ -339,7 +339,7 @@ static struct xenbus_driver xenkbd_driver = {
 
 static int __init xenkbd_init(void)
 {
-	if (!xen_domain())
+	if (!xen_domain() || xen_hvm_domain())
 		return -ENODEV;
 
 	/* Nothing to do if running in dom0. */
diff --git a/drivers/video/xen-fbfront.c b/drivers/video/xen-fbfront.c
index fa97d3e..a105a19 100644
--- a/drivers/video/xen-fbfront.c
+++ b/drivers/video/xen-fbfront.c
@@ -684,7 +684,7 @@ static struct xenbus_driver xenfb_driver = {
 
 static int __init xenfb_init(void)
 {
-	if (!xen_domain())
+	if (!xen_domain() || xen_hvm_domain())
 		return -ENODEV;
 
 	/* Nothing to do if running in dom0. */
diff --git a/drivers/xen/xenbus/xenbus_probe.c b/drivers/xen/xenbus/xenbus_probe.c
index 3479332..0b05b62 100644
--- a/drivers/xen/xenbus/xenbus_probe.c
+++ b/drivers/xen/xenbus/xenbus_probe.c
@@ -56,6 +56,8 @@
 #include <xen/events.h>
 #include <xen/page.h>
 
+#include <xen/hvm.h>
+
 #include "xenbus_comms.h"
 #include "xenbus_probe.h"
 
@@ -806,10 +808,23 @@ static int __init xenbus_probe_init(void)
 		/* dom0 not yet supported */
 	} else {
 		xenstored_ready = 1;
-		xen_store_evtchn = xen_start_info->store_evtchn;
-		xen_store_mfn = xen_start_info->store_mfn;
+		if (xen_hvm_domain()) {
+			uint64_t v = 0;
+			err = hvm_get_parameter(HVM_PARAM_STORE_EVTCHN, &v);
+			if (err)
+				goto out_error;
+			xen_store_evtchn = (int)v;
+			err = hvm_get_parameter(HVM_PARAM_STORE_PFN, &v);
+			if (err)
+				goto out_error;
+			xen_store_mfn = (unsigned long)v;
+			xen_store_interface = ioremap(xen_store_mfn << PAGE_SHIFT, PAGE_SIZE);
+		} else {
+			xen_store_evtchn = xen_start_info->store_evtchn;
+			xen_store_mfn = xen_start_info->store_mfn;
+			xen_store_interface = mfn_to_virt(xen_store_mfn);
+		}
 	}
-	xen_store_interface = mfn_to_virt(xen_store_mfn);
 
 	/* Initialize the interface to xenstore. */
 	err = xs_init();
diff --git a/include/xen/xen.h b/include/xen/xen.h
index a164024..cb8c48b 100644
--- a/include/xen/xen.h
+++ b/include/xen/xen.h
@@ -9,8 +9,10 @@ enum xen_domain_type {
 
 #ifdef CONFIG_XEN
 extern enum xen_domain_type xen_domain_type;
+extern void xen_guest_init(void);
 #else
 #define xen_domain_type		XEN_NATIVE
+#define xen_guest_init() do { } while (0)
 #endif
 
 #define xen_domain()		(xen_domain_type != XEN_NATIVE)
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* [PATCH 03/11] evtchn delivery on HVM
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
  2010-05-24 18:27 ` [PATCH 01/11] Add support for hvm_op Stefano Stabellini
  2010-05-24 18:27 ` [PATCH 02/11] early PV on HVM Stefano Stabellini
@ 2010-05-24 18:27 ` Stefano Stabellini
  2010-05-24 18:27 ` [PATCH 04/11] Fix find_unbound_irq in presence of ioapic irqs Stefano Stabellini
                   ` (11 subsequent siblings)
  14 siblings, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:27 UTC (permalink / raw)
  To: linux-kernel
  Cc: xen-devel, Don Dutile, Stefano Stabellini, Jeremy Fitzhardinge,
	Sheng Yang, Stefano Stabellini

From: Sheng Yang <sheng@linux.intel.com>

Set the callback to receive evtchns from Xen, using the
callback vector delivery mechanism.

The traditional way for receiving event channel notifications from Xen
is interrupts from the platform PCI device.
The callback vector is a newer alternative that allow us to receive
notifications on any vcpu and doesn't need any PCI support.
Besides it is possible to avoid any interaction with the emulated lapic
even though at the moment we are still going through the IPI vector.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Signed-off-by: Sheng Yang <sheng@linux.intel.com>
---
 arch/x86/xen/enlighten.c         |   29 ++++++++++++++++
 arch/x86/xen/xen-ops.h           |    2 +
 drivers/xen/events.c             |   69 ++++++++++++++++++++++++++++++++++----
 include/xen/events.h             |    5 +++
 include/xen/hvm.h                |    6 +++
 include/xen/interface/features.h |    3 ++
 6 files changed, 107 insertions(+), 7 deletions(-)

diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index c1f6545..ed34b30 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -11,6 +11,7 @@
  * Jeremy Fitzhardinge <jeremy@xensource.com>, XenSource Inc, 2007
  */
 
+#include <linux/cpu.h>
 #include <linux/kernel.h>
 #include <linux/init.h>
 #include <linux/smp.h>
@@ -38,6 +39,7 @@
 #include <xen/interface/memory.h>
 #include <xen/features.h>
 #include <xen/page.h>
+#include <xen/hvm.h>
 #include <xen/hvc-console.h>
 
 #include <asm/paravirt.h>
@@ -78,6 +80,9 @@ struct shared_info xen_dummy_shared_info;
 
 void *xen_initial_gdt;
 
+__read_mostly int xen_have_vector_callback;
+EXPORT_SYMBOL_GPL(xen_have_vector_callback);
+
 /*
  * Point at some empty memory to start with. We map the real shared_info
  * page as soon as fixmap is up and running.
@@ -1278,6 +1283,24 @@ static void __init init_shared_info(void)
 	per_cpu(xen_vcpu, 0) = &HYPERVISOR_shared_info->vcpu_info[0];
 }
 
+static int __cpuinit xen_hvm_cpu_notify(struct notifier_block *self,
+				    unsigned long action, void *hcpu)
+{
+	int cpu = (long)hcpu;
+	switch (action) {
+	case CPU_UP_PREPARE:
+		per_cpu(xen_vcpu, cpu) = &HYPERVISOR_shared_info->vcpu_info[cpu];
+		break;
+	default:
+		break;
+	}
+	return NOTIFY_OK;
+}
+
+static struct notifier_block __cpuinitdata xen_hvm_cpu_notifier = {
+	.notifier_call	= xen_hvm_cpu_notify,
+};
+
 void __init xen_guest_init(void)
 {
 	int r;
@@ -1291,4 +1314,10 @@ void __init xen_guest_init(void)
 		return;
 
 	init_shared_info();
+
+	if (xen_feature(XENFEAT_hvm_callback_vector))
+		xen_have_vector_callback = 1;
+	register_cpu_notifier(&xen_hvm_cpu_notifier);
+	have_vcpu_info_placement = 0;
+	x86_init.irqs.intr_init = xen_init_IRQ;
 }
diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
index f9153a3..0d0e0e6 100644
--- a/arch/x86/xen/xen-ops.h
+++ b/arch/x86/xen/xen-ops.h
@@ -38,6 +38,8 @@ void xen_enable_sysenter(void);
 void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
+void xen_callback_vector(void);
+
 void __init xen_build_dynamic_phys_to_machine(void);
 
 void xen_init_irq_ops(void);
diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index db8f506..8b68fc4 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -36,10 +36,14 @@
 #include <asm/xen/hypercall.h>
 #include <asm/xen/hypervisor.h>
 
+#include <xen/xen.h>
+#include <xen/hvm.h>
 #include <xen/xen-ops.h>
 #include <xen/events.h>
 #include <xen/interface/xen.h>
 #include <xen/interface/event_channel.h>
+#include <xen/interface/hvm/hvm_op.h>
+#include <xen/interface/hvm/params.h>
 
 /*
  * This lock protects updates to the following mapping and reference-count
@@ -617,17 +621,13 @@ static DEFINE_PER_CPU(unsigned, xed_nesting_count);
  * a bitset of words which contain pending event bits.  The second
  * level is a bitset of pending events themselves.
  */
-void xen_evtchn_do_upcall(struct pt_regs *regs)
+static void __xen_evtchn_do_upcall(void)
 {
 	int cpu = get_cpu();
-	struct pt_regs *old_regs = set_irq_regs(regs);
 	struct shared_info *s = HYPERVISOR_shared_info;
 	struct vcpu_info *vcpu_info = __get_cpu_var(xen_vcpu);
  	unsigned count;
 
-	exit_idle();
-	irq_enter();
-
 	do {
 		unsigned long pending_words;
 
@@ -667,10 +667,26 @@ void xen_evtchn_do_upcall(struct pt_regs *regs)
 	} while(count != 1);
 
 out:
+
+	put_cpu();
+}
+
+void xen_evtchn_do_upcall(struct pt_regs *regs)
+{
+	struct pt_regs *old_regs = set_irq_regs(regs);
+
+	exit_idle();
+	irq_enter();
+
+	__xen_evtchn_do_upcall();
+
 	irq_exit();
 	set_irq_regs(old_regs);
+}
 
-	put_cpu();
+void xen_hvm_evtchn_do_upcall(struct pt_regs *regs)
+{
+	__xen_evtchn_do_upcall();
 }
 
 /* Rebind a new event channel to an existing irq. */
@@ -933,6 +949,40 @@ static struct irq_chip xen_dynamic_chip __read_mostly = {
 	.retrigger	= retrigger_dynirq,
 };
 
+int xen_set_callback_via(uint64_t via)
+{
+	struct xen_hvm_param a;
+	a.domid = DOMID_SELF;
+	a.index = HVM_PARAM_CALLBACK_IRQ;
+	a.value = via;
+	return HYPERVISOR_hvm_op(HVMOP_set_param, &a);
+}
+EXPORT_SYMBOL_GPL(xen_set_callback_via);
+
+void do_hvm_pv_evtchn_intr(void)
+{
+	xen_hvm_evtchn_do_upcall(get_irq_regs());
+}
+
+/* the callback vector mechanism is a newer alternative way of receiving
+ * event channel notifications from Xen: we can receive vector callbacks
+ * on any vcpus and we don't need any PCI or IO APIC support */
+void xen_callback_vector(void)
+{
+	int rc;
+	uint64_t callback_via;
+	if (xen_have_vector_callback) {
+		callback_via = HVM_CALLBACK_VECTOR(X86_PLATFORM_IPI_VECTOR);
+		rc = xen_set_callback_via(callback_via);
+		if (rc) {
+			printk(KERN_ERR "request for callback vector failed\n");
+			xen_have_vector_callback = 0;
+			return;
+		}
+		x86_platform_ipi_callback = do_hvm_pv_evtchn_intr;
+	}
+}
+
 void __init xen_init_IRQ(void)
 {
 	int i;
@@ -947,5 +997,10 @@ void __init xen_init_IRQ(void)
 	for (i = 0; i < NR_EVENT_CHANNELS; i++)
 		mask_evtchn(i);
 
-	irq_ctx_init(smp_processor_id());
+	if (xen_hvm_domain()) {
+		xen_callback_vector();
+		native_init_IRQ();
+	} else {
+		irq_ctx_init(smp_processor_id());
+	}
 }
diff --git a/include/xen/events.h b/include/xen/events.h
index e68d59a..a3375f7 100644
--- a/include/xen/events.h
+++ b/include/xen/events.h
@@ -56,4 +56,9 @@ void xen_poll_irq(int irq);
 /* Determine the IRQ which is bound to an event channel */
 unsigned irq_from_evtchn(unsigned int evtchn);
 
+extern int xen_have_vector_callback;
+int xen_set_callback_via(uint64_t via);
+void xen_evtchn_do_upcall(struct pt_regs *regs);
+void xen_hvm_evtchn_do_upcall(struct pt_regs *regs);
+
 #endif	/* _XEN_EVENTS_H */
diff --git a/include/xen/hvm.h b/include/xen/hvm.h
index 6b0d418..d0cf7a6 100644
--- a/include/xen/hvm.h
+++ b/include/xen/hvm.h
@@ -3,6 +3,7 @@
 #define XEN_HVM_H__
 
 #include <xen/interface/hvm/params.h>
+#include <asm/xen/hypercall.h>
 
 static inline int hvm_get_parameter(int idx, uint64_t *value)
 {
@@ -21,4 +22,9 @@ static inline int hvm_get_parameter(int idx, uint64_t *value)
        return r;
 }
 
+#define HVM_CALLBACK_VIA_TYPE_VECTOR 0x2
+#define HVM_CALLBACK_VIA_TYPE_SHIFT 56
+#define HVM_CALLBACK_VECTOR(x) (((uint64_t)HVM_CALLBACK_VIA_TYPE_VECTOR)<<\
+                               HVM_CALLBACK_VIA_TYPE_SHIFT | (x))
+
 #endif /* XEN_HVM_H__ */
diff --git a/include/xen/interface/features.h b/include/xen/interface/features.h
index f51b641..8ab08b9 100644
--- a/include/xen/interface/features.h
+++ b/include/xen/interface/features.h
@@ -41,6 +41,9 @@
 /* x86: Does this Xen host support the MMU_PT_UPDATE_PRESERVE_AD hypercall? */
 #define XENFEAT_mmu_pt_update_preserve_ad  5
 
+/* x86: Does this Xen host support the HVM callback vector type? */
+#define XENFEAT_hvm_callback_vector        8
+
 #define XENFEAT_NR_SUBMAPS 1
 
 #endif /* __XEN_PUBLIC_FEATURES_H__ */
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* [PATCH 04/11] Fix find_unbound_irq in presence of ioapic irqs.
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (2 preceding siblings ...)
  2010-05-24 18:27 ` [PATCH 03/11] evtchn delivery " Stefano Stabellini
@ 2010-05-24 18:27 ` Stefano Stabellini
  2010-05-26 18:26   ` [Xen-devel] " Konrad Rzeszutek Wilk
  2010-05-24 18:27 ` [PATCH 05/11] Xen PCI platform device driver Stefano Stabellini
                   ` (10 subsequent siblings)
  14 siblings, 1 reply; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:27 UTC (permalink / raw)
  To: linux-kernel
  Cc: xen-devel, Don Dutile, Stefano Stabellini, Jeremy Fitzhardinge,
	Stefano Stabellini

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 drivers/xen/events.c |   11 ++++++++++-
 1 files changed, 10 insertions(+), 1 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 8b68fc4..1356fa2 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -339,9 +339,18 @@ static int find_unbound_irq(void)
 	int irq;
 	struct irq_desc *desc;
 
-	for (irq = 0; irq < nr_irqs; irq++)
+	for (irq = 0; irq < nr_irqs; irq++) {
+		desc = irq_to_desc(irq);
+		/* only 0->15 have init'd desc; handle irq > 16 */
+		if (desc == NULL)
+			break;
+		if (desc->chip == &no_irq_chip)
+			break;
+		if (desc->chip != &xen_dynamic_chip)
+			continue;
 		if (irq_info[irq].type == IRQT_UNBOUND)
 			break;
+	}
 
 	if (irq == nr_irqs)
 		panic("No available IRQ to bind to: increase nr_irqs!\n");
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* Re: [Xen-devel] [PATCH 04/11] Fix find_unbound_irq in presence of ioapic irqs.
  2010-05-24 18:27 ` [PATCH 04/11] Fix find_unbound_irq in presence of ioapic irqs Stefano Stabellini
@ 2010-05-26 18:26   ` Konrad Rzeszutek Wilk
  2010-05-27 14:49     ` Stefano Stabellini
  0 siblings, 1 reply; 39+ messages in thread
From: Konrad Rzeszutek Wilk @ 2010-05-26 18:26 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: linux-kernel, Jeremy Fitzhardinge, xen-devel, Don Dutile

On Mon, May 24, 2010 at 07:27:30PM +0100, Stefano Stabellini wrote:
> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

Can you provide a more detailed description of why this is needed?

> ---
>  drivers/xen/events.c |   11 ++++++++++-
>  1 files changed, 10 insertions(+), 1 deletions(-)
> 
> diff --git a/drivers/xen/events.c b/drivers/xen/events.c
> index 8b68fc4..1356fa2 100644
> --- a/drivers/xen/events.c
> +++ b/drivers/xen/events.c
> @@ -339,9 +339,18 @@ static int find_unbound_irq(void)
>  	int irq;
>  	struct irq_desc *desc;
>  
> -	for (irq = 0; irq < nr_irqs; irq++)
> +	for (irq = 0; irq < nr_irqs; irq++) {
> +		desc = irq_to_desc(irq);
> +		/* only 0->15 have init'd desc; handle irq > 16 */
> +		if (desc == NULL)
> +			break;
> +		if (desc->chip == &no_irq_chip)
> +			break;
> +		if (desc->chip != &xen_dynamic_chip)
> +			continue;
>  		if (irq_info[irq].type == IRQT_UNBOUND)
>  			break;
> +	}
>  
>  	if (irq == nr_irqs)
>  		panic("No available IRQ to bind to: increase nr_irqs!\n");
> -- 
> 1.5.4.3
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [Xen-devel] [PATCH 04/11] Fix find_unbound_irq in presence of ioapic irqs.
  2010-05-26 18:26   ` [Xen-devel] " Konrad Rzeszutek Wilk
@ 2010-05-27 14:49     ` Stefano Stabellini
  2010-05-27 15:36       ` Konrad Rzeszutek Wilk
  0 siblings, 1 reply; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-27 14:49 UTC (permalink / raw)
  To: Konrad Rzeszutek Wilk
  Cc: Stefano Stabellini, linux-kernel@vger.kernel.org,
	Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Don Dutile

On Wed, 26 May 2010, Konrad Rzeszutek Wilk wrote:
> On Mon, May 24, 2010 at 07:27:30PM +0100, Stefano Stabellini wrote:
> > Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> 
> Can you provide a more detailed description of why this is needed?
> 

We need it because we don't want to break the assumption that the first
16 irqs are ISA irqs, and among the other irqs we actually want to be
sure they are available before using them.


> > ---
> >  drivers/xen/events.c |   11 ++++++++++-
> >  1 files changed, 10 insertions(+), 1 deletions(-)
> > 
> > diff --git a/drivers/xen/events.c b/drivers/xen/events.c
> > index 8b68fc4..1356fa2 100644
> > --- a/drivers/xen/events.c
> > +++ b/drivers/xen/events.c
> > @@ -339,9 +339,18 @@ static int find_unbound_irq(void)
> >  	int irq;
> >  	struct irq_desc *desc;
> >  
> > -	for (irq = 0; irq < nr_irqs; irq++)
> > +	for (irq = 0; irq < nr_irqs; irq++) {
> > +		desc = irq_to_desc(irq);
> > +		/* only 0->15 have init'd desc; handle irq > 16 */
> > +		if (desc == NULL)
> > +			break;
> > +		if (desc->chip == &no_irq_chip)
> > +			break;
> > +		if (desc->chip != &xen_dynamic_chip)
> > +			continue;
> >  		if (irq_info[irq].type == IRQT_UNBOUND)
> >  			break;
> > +	}
> >  
> >  	if (irq == nr_irqs)
> >  		panic("No available IRQ to bind to: increase nr_irqs!\n");
> > -- 
> > 1.5.4.3
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 04/11] Fix find_unbound_irq in presence of ioapic irqs.
  2010-05-27 14:49     ` Stefano Stabellini
@ 2010-05-27 15:36       ` Konrad Rzeszutek Wilk
  0 siblings, 0 replies; 39+ messages in thread
From: Konrad Rzeszutek Wilk @ 2010-05-27 15:36 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Don Dutile,
	linux-kernel@vger.kernel.org

On Thu, May 27, 2010 at 03:49:12PM +0100, Stefano Stabellini wrote:
> On Wed, 26 May 2010, Konrad Rzeszutek Wilk wrote:
> > On Mon, May 24, 2010 at 07:27:30PM +0100, Stefano Stabellini wrote:
> > > Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> > 
> > Can you provide a more detailed description of why this is needed?
> > 
> 
> We need it because we don't want to break the assumption that the first
> 16 irqs are ISA irqs, and among the other irqs we actually want to be
> sure they are available before using them.

Interesting. So with this commit: 482839e7 and bfbc226d
where we try our best to allocate IRQs for Xen starting from high and
then going down (so that when we are running in PV context with Xen PCI
passthrough we can allocate IRQs for physical devices above IRQ 16 and
not collide with Xen event channels), this would still work.

But we actually wouldn't test the availability of the first sixteen IRQ
for a long time (or never).

> 
> 
> > > ---
> > >  drivers/xen/events.c |   11 ++++++++++-
> > >  1 files changed, 10 insertions(+), 1 deletions(-)
> > > 
> > > diff --git a/drivers/xen/events.c b/drivers/xen/events.c
> > > index 8b68fc4..1356fa2 100644
> > > --- a/drivers/xen/events.c
> > > +++ b/drivers/xen/events.c
> > > @@ -339,9 +339,18 @@ static int find_unbound_irq(void)
> > >  	int irq;
> > >  	struct irq_desc *desc;
> > >  
> > > -	for (irq = 0; irq < nr_irqs; irq++)
> > > +	for (irq = 0; irq < nr_irqs; irq++) {
> > > +		desc = irq_to_desc(irq);
> > > +		/* only 0->15 have init'd desc; handle irq > 16 */
> > > +		if (desc == NULL)
> > > +			break;
> > > +		if (desc->chip == &no_irq_chip)
> > > +			break;
> > > +		if (desc->chip != &xen_dynamic_chip)
> > > +			continue;
> > >  		if (irq_info[irq].type == IRQT_UNBOUND)
> > >  			break;
> > > +	}
> > >  
> > >  	if (irq == nr_irqs)
> > >  		panic("No available IRQ to bind to: increase nr_irqs!\n");
> > > -- 
> > > 1.5.4.3
> > > 
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > 

^ permalink raw reply	[flat|nested] 39+ messages in thread

* [PATCH 05/11] Xen PCI platform device driver
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (3 preceding siblings ...)
  2010-05-24 18:27 ` [PATCH 04/11] Fix find_unbound_irq in presence of ioapic irqs Stefano Stabellini
@ 2010-05-24 18:27 ` Stefano Stabellini
  2010-05-28  8:55   ` [Xen-devel] " Zhigang Wang
  2010-05-24 18:27 ` [PATCH 06/11] Add suspend\resume support for PV on HVM guests Stefano Stabellini
                   ` (9 subsequent siblings)
  14 siblings, 1 reply; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:27 UTC (permalink / raw)
  To: linux-kernel
  Cc: xen-devel, Don Dutile, Stefano Stabellini, Jeremy Fitzhardinge,
	Stefano Stabellini, Sheng Yang

Add the xen pci platform device driver that is responsible
for initializing the grant table and xenbus in PV on HVM mode.
Few changes to xenbus and grant table are necessary to allow the delayed
initialization in HVM mode.
Grant table needs few additional modifications to work in HVM mode.

The Xen PCI platform device raises an irq every time an event has been
delivered to us. However these interrupts are only delivered to vcpu 0.
The Xen PCI platform interrupt handler calls xen_hvm_evtchn_do_upcall
that is a little wrapper around __xen_evtchn_do_upcall, the traditional
Xen upcall handler, the very same used with traditional PV guests.

When running on HVM the event channel upcall is never called while in
progress because it is a normal Linux irq handler, therefore we cannot
be sure that evtchn_upcall_pending is 0 when returning.
For this reason if evtchn_upcall_pending is set by Xen we need to loop
again on the event channels set pending otherwise we might loose some
event channel deliveries.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Signed-off-by: Sheng Yang <sheng@linux.intel.com>
---
 drivers/xen/Kconfig                 |    8 ++
 drivers/xen/Makefile                |    3 +-
 drivers/xen/events.c                |    7 +-
 drivers/xen/grant-table.c           |   69 ++++++++++++--
 drivers/xen/platform-pci.c          |  186 +++++++++++++++++++++++++++++++++++
 drivers/xen/xenbus/xenbus_probe.c   |   20 +++-
 include/linux/pci_ids.h             |    3 +
 include/xen/grant_table.h           |    1 +
 include/xen/interface/grant_table.h |    1 +
 include/xen/xenbus.h                |    1 +
 10 files changed, 283 insertions(+), 16 deletions(-)
 create mode 100644 drivers/xen/platform-pci.c

diff --git a/drivers/xen/Kconfig b/drivers/xen/Kconfig
index fad3df2..da312e2 100644
--- a/drivers/xen/Kconfig
+++ b/drivers/xen/Kconfig
@@ -62,4 +62,12 @@ config XEN_SYS_HYPERVISOR
 	 virtual environment, /sys/hypervisor will still be present,
 	 but will have no xen contents.
 
+config XEN_PLATFORM_PCI
+	tristate "xen platform pci device driver"
+	depends on XEN
+	help
+	  Driver for the Xen PCI Platform device: it is responsible for
+	  initializing xenbus and grant_table when running in a Xen HVM
+	  domain. As a consequence this driver is required to run any Xen PV
+	  frontend on Xen HVM.
 endmenu
diff --git a/drivers/xen/Makefile b/drivers/xen/Makefile
index 7c28434..e392fb7 100644
--- a/drivers/xen/Makefile
+++ b/drivers/xen/Makefile
@@ -9,4 +9,5 @@ obj-$(CONFIG_XEN_XENCOMM)	+= xencomm.o
 obj-$(CONFIG_XEN_BALLOON)	+= balloon.o
 obj-$(CONFIG_XEN_DEV_EVTCHN)	+= evtchn.o
 obj-$(CONFIG_XENFS)		+= xenfs/
-obj-$(CONFIG_XEN_SYS_HYPERVISOR)	+= sys-hypervisor.o
\ No newline at end of file
+obj-$(CONFIG_XEN_SYS_HYPERVISOR)	+= sys-hypervisor.o
+obj-$(CONFIG_XEN_PLATFORM_PCI)	+= platform-pci.o
diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 1356fa2..c6f561d 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -673,7 +673,7 @@ static void __xen_evtchn_do_upcall(void)
 
 		count = __get_cpu_var(xed_nesting_count);
 		__get_cpu_var(xed_nesting_count) = 0;
-	} while(count != 1);
+	} while(count != 1 || vcpu_info->evtchn_upcall_pending);
 
 out:
 
@@ -733,7 +733,10 @@ static int rebind_irq_to_cpu(unsigned irq, unsigned tcpu)
 	struct evtchn_bind_vcpu bind_vcpu;
 	int evtchn = evtchn_from_irq(irq);
 
-	if (!VALID_EVTCHN(evtchn))
+	/* events delivered via platform PCI interrupts are always
+	 * routed to vcpu 0 */
+	if (!VALID_EVTCHN(evtchn) ||
+		(xen_hvm_domain() && !xen_have_vector_callback))
 		return -1;
 
 	/* Send future instances of this interrupt to other vcpu. */
diff --git a/drivers/xen/grant-table.c b/drivers/xen/grant-table.c
index f66db3b..4c959a5 100644
--- a/drivers/xen/grant-table.c
+++ b/drivers/xen/grant-table.c
@@ -37,11 +37,13 @@
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 #include <linux/uaccess.h>
+#include <linux/io.h>
 
 #include <xen/xen.h>
 #include <xen/interface/xen.h>
 #include <xen/page.h>
 #include <xen/grant_table.h>
+#include <xen/interface/memory.h>
 #include <asm/xen/hypercall.h>
 
 #include <asm/pgtable.h>
@@ -59,6 +61,7 @@ static unsigned int boot_max_nr_grant_frames;
 static int gnttab_free_count;
 static grant_ref_t gnttab_free_head;
 static DEFINE_SPINLOCK(gnttab_list_lock);
+static unsigned long hvm_pv_resume_frames;
 
 static struct grant_entry *shared;
 
@@ -449,6 +452,30 @@ static int gnttab_map(unsigned int start_idx, unsigned int end_idx)
 	unsigned int nr_gframes = end_idx + 1;
 	int rc;
 
+	if (xen_hvm_domain()) {
+		struct xen_add_to_physmap xatp;
+		unsigned int i = end_idx;
+		rc = 0;
+		/*
+		 * Loop backwards, so that the first hypercall has the largest
+		 * index, ensuring that the table will grow only once.
+		 */
+		do {
+			xatp.domid = DOMID_SELF;
+			xatp.idx = i;
+			xatp.space = XENMAPSPACE_grant_table;
+			xatp.gpfn = (hvm_pv_resume_frames >> PAGE_SHIFT) + i;
+			rc = HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp);
+			if (rc != 0) {
+				printk(KERN_WARNING
+						"grant table add_to_physmap failed, err=%d\n", rc);
+				break;
+			}
+		} while (i-- > start_idx);
+
+		return rc;
+	}
+
 	frames = kmalloc(nr_gframes * sizeof(unsigned long), GFP_ATOMIC);
 	if (!frames)
 		return -ENOMEM;
@@ -476,9 +503,28 @@ static int gnttab_map(unsigned int start_idx, unsigned int end_idx)
 
 int gnttab_resume(void)
 {
-	if (max_nr_grant_frames() < nr_grant_frames)
+	unsigned int max_nr_gframes;
+
+	max_nr_gframes = max_nr_grant_frames();
+	if (max_nr_gframes < nr_grant_frames)
 		return -ENOSYS;
-	return gnttab_map(0, nr_grant_frames - 1);
+
+	if (xen_pv_domain())
+		return gnttab_map(0, nr_grant_frames - 1);
+
+	if (!hvm_pv_resume_frames) {
+		hvm_pv_resume_frames = alloc_xen_mmio(PAGE_SIZE * max_nr_gframes);
+		shared = ioremap(hvm_pv_resume_frames, PAGE_SIZE * max_nr_gframes);
+		if (shared == NULL) {
+			printk(KERN_WARNING
+					"Fail to ioremap gnttab share frames\n");
+			return -ENOMEM;
+		}
+	}
+
+	gnttab_map(0, nr_grant_frames - 1);
+
+	return 0;
 }
 
 int gnttab_suspend(void)
@@ -505,15 +551,12 @@ static int gnttab_expand(unsigned int req_entries)
 	return rc;
 }
 
-static int __devinit gnttab_init(void)
+int gnttab_init(void)
 {
 	int i;
 	unsigned int max_nr_glist_frames, nr_glist_frames;
 	unsigned int nr_init_grefs;
 
-	if (!xen_domain())
-		return -ENODEV;
-
 	nr_grant_frames = 1;
 	boot_max_nr_grant_frames = __max_nr_grant_frames();
 
@@ -557,4 +600,16 @@ static int __devinit gnttab_init(void)
 	return -ENOMEM;
 }
 
-core_initcall(gnttab_init);
+static int __devinit __gnttab_init(void)
+{
+	/* Delay grant-table initialization in the PV on HVM case */
+	if (xen_hvm_domain())
+		return 0;
+
+	if (!xen_pv_domain())
+		return -ENODEV;
+
+	return gnttab_init();
+}
+
+core_initcall(__gnttab_init);
diff --git a/drivers/xen/platform-pci.c b/drivers/xen/platform-pci.c
new file mode 100644
index 0000000..0f78318
--- /dev/null
+++ b/drivers/xen/platform-pci.c
@@ -0,0 +1,186 @@
+/******************************************************************************
+ * platform-pci.c
+ *
+ * Xen platform PCI device driver
+ * Copyright (c) 2005, Intel Corporation.
+ * Copyright (c) 2007, XenSource Inc.
+ * Copyright (c) 2010, Citrix
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
+ * Place - Suite 330, Boston, MA 02111-1307 USA.
+ *
+ */
+
+#include <asm/io.h>
+
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+
+#include <xen/grant_table.h>
+#include <xen/xenbus.h>
+#include <xen/events.h>
+#include <xen/hvm.h>
+
+#define DRV_NAME    "xen-platform-pci"
+
+MODULE_AUTHOR("ssmith@xensource.com and stefano.stabellini@eu.citrix.com");
+MODULE_DESCRIPTION("Xen platform PCI device");
+MODULE_LICENSE("GPL");
+
+static unsigned long platform_mmio;
+static unsigned long platform_mmio_alloc;
+static unsigned long platform_mmiolen;
+
+unsigned long alloc_xen_mmio(unsigned long len)
+{
+	unsigned long addr;
+
+	addr = platform_mmio + platform_mmio_alloc;
+	platform_mmio_alloc += len;
+	BUG_ON(platform_mmio_alloc > platform_mmiolen);
+
+	return addr;
+}
+
+static uint64_t get_callback_via(struct pci_dev *pdev)
+{
+	u8 pin;
+	int irq;
+
+	irq = pdev->irq;
+	if (irq < 16)
+		return irq; /* ISA IRQ */
+
+	pin = pdev->pin;
+
+	/* We don't know the GSI. Specify the PCI INTx line instead. */
+	return ((uint64_t)0x01 << 56) | /* PCI INTx identifier */
+		((uint64_t)pci_domain_nr(pdev->bus) << 32) |
+		((uint64_t)pdev->bus->number << 16) |
+		((uint64_t)(pdev->devfn & 0xff) << 8) |
+		((uint64_t)(pin - 1) & 3);
+}
+
+static irqreturn_t do_hvm_evtchn_intr(int irq, void *dev_id)
+{
+	xen_hvm_evtchn_do_upcall(get_irq_regs());
+	return IRQ_HANDLED;
+}
+
+static int xen_allocate_irq(struct pci_dev *pdev)
+{
+	return request_irq(pdev->irq, do_hvm_evtchn_intr,
+			IRQF_DISABLED | IRQF_NOBALANCING | IRQF_TRIGGER_RISING,
+			"xen-platform-pci", pdev);
+}
+
+static int __devinit platform_pci_init(struct pci_dev *pdev,
+				       const struct pci_device_id *ent)
+{
+	int i, ret;
+	long ioaddr, iolen;
+	long mmio_addr, mmio_len;
+	uint64_t callback_via;
+
+	i = pci_enable_device(pdev);
+	if (i)
+		return i;
+
+	ioaddr = pci_resource_start(pdev, 0);
+	iolen = pci_resource_len(pdev, 0);
+
+	mmio_addr = pci_resource_start(pdev, 1);
+	mmio_len = pci_resource_len(pdev, 1);
+
+	if (mmio_addr == 0 || ioaddr == 0) {
+		dev_err(&pdev->dev, "no resources found\n");
+		ret = -ENOENT;
+	}
+
+	if (request_mem_region(mmio_addr, mmio_len, DRV_NAME) == NULL) {
+		dev_err(&pdev->dev, "MEM I/O resource 0x%lx @ 0x%lx busy\n",
+		       mmio_addr, mmio_len);
+		ret = -EBUSY;
+	}
+
+	if (request_region(ioaddr, iolen, DRV_NAME) == NULL) {
+		dev_err(&pdev->dev, "I/O resource 0x%lx @ 0x%lx busy\n",
+		       iolen, ioaddr);
+		ret = -EBUSY;
+		goto out;
+	}
+
+	platform_mmio = mmio_addr;
+	platform_mmiolen = mmio_len;
+
+	if (!xen_have_vector_callback) {
+		ret = xen_allocate_irq(pdev);
+		if (ret) {
+			printk(KERN_WARNING "request_irq failed err=%d\n", ret);
+			goto out;
+		}
+		callback_via = get_callback_via(pdev);
+		ret = xen_set_callback_via(callback_via);
+		if (ret) {
+			printk(KERN_WARNING
+					"Unable to set the evtchn callback err=%d\n", ret);
+			goto out;
+		}
+	}
+
+	ret = gnttab_init();
+	if (ret)
+		goto out;
+	ret = xenbus_probe_init();
+	if (ret)
+		goto out;
+
+out:
+	if (ret) {
+		release_mem_region(mmio_addr, mmio_len);
+		release_region(ioaddr, iolen);
+		pci_disable_device(pdev);
+	}
+
+	return ret;
+}
+
+static struct pci_device_id platform_pci_tbl[] __devinitdata = {
+	{PCI_VENDOR_ID_XEN, PCI_DEVICE_ID_XEN_PLATFORM,
+		PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0},
+	{0,}
+};
+
+MODULE_DEVICE_TABLE(pci, platform_pci_tbl);
+
+static struct pci_driver platform_driver = {
+	.name =           DRV_NAME,
+	.probe =          platform_pci_init,
+	.id_table =       platform_pci_tbl,
+};
+
+static int __init platform_pci_module_init(void)
+{
+	int rc;
+
+	rc = pci_register_driver(&platform_driver);
+	if (rc) {
+		printk(KERN_INFO DRV_NAME
+		       ": No platform pci device model found\n");
+		return rc;
+	}
+	return 0;
+}
+
+module_init(platform_pci_module_init);
diff --git a/drivers/xen/xenbus/xenbus_probe.c b/drivers/xen/xenbus/xenbus_probe.c
index 0b05b62..dc6ed06 100644
--- a/drivers/xen/xenbus/xenbus_probe.c
+++ b/drivers/xen/xenbus/xenbus_probe.c
@@ -782,16 +782,24 @@ void xenbus_probe(struct work_struct *unused)
 	blocking_notifier_call_chain(&xenstore_chain, 0, NULL);
 }
 
-static int __init xenbus_probe_init(void)
+static int __init __xenbus_probe_init(void)
+{
+	/* Delay initialization in the PV on HVM case */
+	if (xen_hvm_domain())
+		return 0;
+
+	if (!xen_pv_domain())
+		return -ENODEV;
+
+	return xenbus_probe_init();
+}
+
+int xenbus_probe_init(void)
 {
 	int err = 0;
 
 	DPRINTK("");
 
-	err = -ENODEV;
-	if (!xen_domain())
-		goto out_error;
-
 	/* Register ourselves with the kernel bus subsystem */
 	err = bus_register(&xenbus_frontend.bus);
 	if (err)
@@ -857,7 +865,7 @@ static int __init xenbus_probe_init(void)
 	return err;
 }
 
-postcore_initcall(xenbus_probe_init);
+postcore_initcall(__xenbus_probe_init);
 
 MODULE_LICENSE("GPL");
 
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index 9f688d2..64b528d 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2718,3 +2718,6 @@
 #define PCI_DEVICE_ID_RME_DIGI32	0x9896
 #define PCI_DEVICE_ID_RME_DIGI32_PRO	0x9897
 #define PCI_DEVICE_ID_RME_DIGI32_8	0x9898
+
+#define PCI_VENDOR_ID_XEN	0x5853
+#define PCI_DEVICE_ID_XEN_PLATFORM	0x0001
diff --git a/include/xen/grant_table.h b/include/xen/grant_table.h
index a40f1cd..811cda5 100644
--- a/include/xen/grant_table.h
+++ b/include/xen/grant_table.h
@@ -51,6 +51,7 @@ struct gnttab_free_callback {
 	u16 count;
 };
 
+int gnttab_init(void);
 int gnttab_suspend(void);
 int gnttab_resume(void);
 
diff --git a/include/xen/interface/grant_table.h b/include/xen/interface/grant_table.h
index 39da93c..39e5717 100644
--- a/include/xen/interface/grant_table.h
+++ b/include/xen/interface/grant_table.h
@@ -28,6 +28,7 @@
 #ifndef __XEN_PUBLIC_GRANT_TABLE_H__
 #define __XEN_PUBLIC_GRANT_TABLE_H__
 
+#include <xen/interface/xen.h>
 
 /***********************************
  * GRANT TABLE REPRESENTATION
diff --git a/include/xen/xenbus.h b/include/xen/xenbus.h
index 43e2d7d..ffa97de 100644
--- a/include/xen/xenbus.h
+++ b/include/xen/xenbus.h
@@ -174,6 +174,7 @@ void unregister_xenbus_watch(struct xenbus_watch *watch);
 void xs_suspend(void);
 void xs_resume(void);
 void xs_suspend_cancel(void);
+int xenbus_probe_init(void);
 
 /* Used by xenbus_dev to borrow kernel's store connection. */
 void *xenbus_dev_request_and_reply(struct xsd_sockmsg *msg);
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* Re: [Xen-devel] [PATCH 05/11] Xen PCI platform device driver
  2010-05-24 18:27 ` [PATCH 05/11] Xen PCI platform device driver Stefano Stabellini
@ 2010-05-28  8:55   ` Zhigang Wang
  2010-05-28 14:18     ` Stefano Stabellini
  0 siblings, 1 reply; 39+ messages in thread
From: Zhigang Wang @ 2010-05-28  8:55 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: linux-kernel, Jeremy Fitzhardinge, xen-devel, Don Dutile,
	Sheng Yang

On 05/25/2010 02:27 AM, Stefano Stabellini wrote:
> Add the xen pci platform device driver that is responsible
> for initializing the grant table and xenbus in PV on HVM mode.
> Few changes to xenbus and grant table are necessary to allow the delayed
> initialization in HVM mode.
> Grant table needs few additional modifications to work in HVM mode.
> 
> The Xen PCI platform device raises an irq every time an event has been
> delivered to us. However these interrupts are only delivered to vcpu 0.
> The Xen PCI platform interrupt handler calls xen_hvm_evtchn_do_upcall
> that is a little wrapper around __xen_evtchn_do_upcall, the traditional
> Xen upcall handler, the very same used with traditional PV guests.
> 
> When running on HVM the event channel upcall is never called while in
> progress because it is a normal Linux irq handler, therefore we cannot
> be sure that evtchn_upcall_pending is 0 when returning.
> For this reason if evtchn_upcall_pending is set by Xen we need to loop
> again on the event channels set pending otherwise we might loose some
> event channel deliveries.
> 
> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> Signed-off-by: Sheng Yang <sheng@linux.intel.com>
> ---
>  drivers/xen/Kconfig                 |    8 ++
>  drivers/xen/Makefile                |    3 +-
>  drivers/xen/events.c                |    7 +-
>  drivers/xen/grant-table.c           |   69 ++++++++++++--
>  drivers/xen/platform-pci.c          |  186 +++++++++++++++++++++++++++++++++++
>  drivers/xen/xenbus/xenbus_probe.c   |   20 +++-
>  include/linux/pci_ids.h             |    3 +
>  include/xen/grant_table.h           |    1 +
>  include/xen/interface/grant_table.h |    1 +
>  include/xen/xenbus.h                |    1 +
>  10 files changed, 283 insertions(+), 16 deletions(-)
>  create mode 100644 drivers/xen/platform-pci.c
> 
> diff --git a/drivers/xen/Kconfig b/drivers/xen/Kconfig
> index fad3df2..da312e2 100644
> --- a/drivers/xen/Kconfig
> +++ b/drivers/xen/Kconfig
> @@ -62,4 +62,12 @@ config XEN_SYS_HYPERVISOR
>  	 virtual environment, /sys/hypervisor will still be present,
>  	 but will have no xen contents.
>  
> +config XEN_PLATFORM_PCI
> +	tristate "xen platform pci device driver"
> +	depends on XEN
        default m

?
> +	help
> +	  Driver for the Xen PCI Platform device: it is responsible for
> +	  initializing xenbus and grant_table when running in a Xen HVM
> +	  domain. As a consequence this driver is required to run any Xen PV
> +	  frontend on Xen HVM.
>  endmenu
> diff --git a/drivers/xen/Makefile b/drivers/xen/Makefile
> index 7c28434..e392fb7 100644
> --- a/drivers/xen/Makefile
> +++ b/drivers/xen/Makefile
> @@ -9,4 +9,5 @@ obj-$(CONFIG_XEN_XENCOMM)	+= xencomm.o
>  obj-$(CONFIG_XEN_BALLOON)	+= balloon.o
>  obj-$(CONFIG_XEN_DEV_EVTCHN)	+= evtchn.o
>  obj-$(CONFIG_XENFS)		+= xenfs/
> -obj-$(CONFIG_XEN_SYS_HYPERVISOR)	+= sys-hypervisor.o
> \ No newline at end of file
> +obj-$(CONFIG_XEN_SYS_HYPERVISOR)	+= sys-hypervisor.o
> +obj-$(CONFIG_XEN_PLATFORM_PCI)	+= platform-pci.o
> diff --git a/drivers/xen/events.c b/drivers/xen/events.c
> index 1356fa2..c6f561d 100644
> --- a/drivers/xen/events.c
> +++ b/drivers/xen/events.c
> @@ -673,7 +673,7 @@ static void __xen_evtchn_do_upcall(void)
>  
>  		count = __get_cpu_var(xed_nesting_count);
>  		__get_cpu_var(xed_nesting_count) = 0;
> -	} while(count != 1);
> +	} while(count != 1 || vcpu_info->evtchn_upcall_pending);
>  
>  out:
>  
> @@ -733,7 +733,10 @@ static int rebind_irq_to_cpu(unsigned irq, unsigned tcpu)
>  	struct evtchn_bind_vcpu bind_vcpu;
>  	int evtchn = evtchn_from_irq(irq);
>  
> -	if (!VALID_EVTCHN(evtchn))
> +	/* events delivered via platform PCI interrupts are always
> +	 * routed to vcpu 0 */
> +	if (!VALID_EVTCHN(evtchn) ||
> +		(xen_hvm_domain() && !xen_have_vector_callback))
>  		return -1;
>  
>  	/* Send future instances of this interrupt to other vcpu. */
> diff --git a/drivers/xen/grant-table.c b/drivers/xen/grant-table.c
> index f66db3b..4c959a5 100644
> --- a/drivers/xen/grant-table.c
> +++ b/drivers/xen/grant-table.c
> @@ -37,11 +37,13 @@
>  #include <linux/slab.h>
>  #include <linux/vmalloc.h>
>  #include <linux/uaccess.h>
> +#include <linux/io.h>
>  
>  #include <xen/xen.h>
>  #include <xen/interface/xen.h>
>  #include <xen/page.h>
>  #include <xen/grant_table.h>
> +#include <xen/interface/memory.h>
>  #include <asm/xen/hypercall.h>
>  
>  #include <asm/pgtable.h>
> @@ -59,6 +61,7 @@ static unsigned int boot_max_nr_grant_frames;
>  static int gnttab_free_count;
>  static grant_ref_t gnttab_free_head;
>  static DEFINE_SPINLOCK(gnttab_list_lock);
> +static unsigned long hvm_pv_resume_frames;
>  
>  static struct grant_entry *shared;
>  
> @@ -449,6 +452,30 @@ static int gnttab_map(unsigned int start_idx, unsigned int end_idx)
>  	unsigned int nr_gframes = end_idx + 1;
>  	int rc;
>  
> +	if (xen_hvm_domain()) {
> +		struct xen_add_to_physmap xatp;
> +		unsigned int i = end_idx;
> +		rc = 0;
> +		/*
> +		 * Loop backwards, so that the first hypercall has the largest
> +		 * index, ensuring that the table will grow only once.
> +		 */
> +		do {
> +			xatp.domid = DOMID_SELF;
> +			xatp.idx = i;
> +			xatp.space = XENMAPSPACE_grant_table;
> +			xatp.gpfn = (hvm_pv_resume_frames >> PAGE_SHIFT) + i;
> +			rc = HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp);
> +			if (rc != 0) {
> +				printk(KERN_WARNING
> +						"grant table add_to_physmap failed, err=%d\n", rc);
> +				break;
> +			}
> +		} while (i-- > start_idx);
> +
> +		return rc;
> +	}
> +
>  	frames = kmalloc(nr_gframes * sizeof(unsigned long), GFP_ATOMIC);
>  	if (!frames)
>  		return -ENOMEM;
> @@ -476,9 +503,28 @@ static int gnttab_map(unsigned int start_idx, unsigned int end_idx)
>  
>  int gnttab_resume(void)
>  {
> -	if (max_nr_grant_frames() < nr_grant_frames)
> +	unsigned int max_nr_gframes;
> +
> +	max_nr_gframes = max_nr_grant_frames();
> +	if (max_nr_gframes < nr_grant_frames)
>  		return -ENOSYS;
> -	return gnttab_map(0, nr_grant_frames - 1);
> +
> +	if (xen_pv_domain())
> +		return gnttab_map(0, nr_grant_frames - 1);
> +
> +	if (!hvm_pv_resume_frames) {
> +		hvm_pv_resume_frames = alloc_xen_mmio(PAGE_SIZE * max_nr_gframes);
> +		shared = ioremap(hvm_pv_resume_frames, PAGE_SIZE * max_nr_gframes);
> +		if (shared == NULL) {
> +			printk(KERN_WARNING
> +					"Fail to ioremap gnttab share frames\n");
> +			return -ENOMEM;
> +		}
> +	}
> +
> +	gnttab_map(0, nr_grant_frames - 1);
> +
> +	return 0;
>  }
>  
>  int gnttab_suspend(void)
> @@ -505,15 +551,12 @@ static int gnttab_expand(unsigned int req_entries)
>  	return rc;
>  }
>  
> -static int __devinit gnttab_init(void)
> +int gnttab_init(void)
>  {
>  	int i;
>  	unsigned int max_nr_glist_frames, nr_glist_frames;
>  	unsigned int nr_init_grefs;
>  
> -	if (!xen_domain())
> -		return -ENODEV;
> -
>  	nr_grant_frames = 1;
>  	boot_max_nr_grant_frames = __max_nr_grant_frames();
>  
> @@ -557,4 +600,16 @@ static int __devinit gnttab_init(void)
>  	return -ENOMEM;
>  }
>  
> -core_initcall(gnttab_init);
> +static int __devinit __gnttab_init(void)
> +{
> +	/* Delay grant-table initialization in the PV on HVM case */
> +	if (xen_hvm_domain())
> +		return 0;
> +
> +	if (!xen_pv_domain())
> +		return -ENODEV;
> +
> +	return gnttab_init();
> +}
> +
> +core_initcall(__gnttab_init);
> diff --git a/drivers/xen/platform-pci.c b/drivers/xen/platform-pci.c
> new file mode 100644
> index 0000000..0f78318
> --- /dev/null
> +++ b/drivers/xen/platform-pci.c
> @@ -0,0 +1,186 @@
> +/******************************************************************************
> + * platform-pci.c
> + *
> + * Xen platform PCI device driver
> + * Copyright (c) 2005, Intel Corporation.
> + * Copyright (c) 2007, XenSource Inc.
> + * Copyright (c) 2010, Citrix
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
> + * Place - Suite 330, Boston, MA 02111-1307 USA.
> + *
> + */
> +
> +#include <asm/io.h>
> +
> +#include <linux/interrupt.h>
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +
> +#include <xen/grant_table.h>
> +#include <xen/xenbus.h>
> +#include <xen/events.h>
> +#include <xen/hvm.h>
> +
> +#define DRV_NAME    "xen-platform-pci"
> +
> +MODULE_AUTHOR("ssmith@xensource.com and stefano.stabellini@eu.citrix.com");
> +MODULE_DESCRIPTION("Xen platform PCI device");
> +MODULE_LICENSE("GPL");
> +
> +static unsigned long platform_mmio;
> +static unsigned long platform_mmio_alloc;
> +static unsigned long platform_mmiolen;
> +
> +unsigned long alloc_xen_mmio(unsigned long len)
> +{
> +	unsigned long addr;
> +
> +	addr = platform_mmio + platform_mmio_alloc;
> +	platform_mmio_alloc += len;
> +	BUG_ON(platform_mmio_alloc > platform_mmiolen);
> +
> +	return addr;
> +}
> +
> +static uint64_t get_callback_via(struct pci_dev *pdev)
> +{
> +	u8 pin;
> +	int irq;
> +
> +	irq = pdev->irq;
> +	if (irq < 16)
> +		return irq; /* ISA IRQ */
> +
> +	pin = pdev->pin;
> +
> +	/* We don't know the GSI. Specify the PCI INTx line instead. */
> +	return ((uint64_t)0x01 << 56) | /* PCI INTx identifier */
> +		((uint64_t)pci_domain_nr(pdev->bus) << 32) |
> +		((uint64_t)pdev->bus->number << 16) |
> +		((uint64_t)(pdev->devfn & 0xff) << 8) |
> +		((uint64_t)(pin - 1) & 3);
> +}
> +
> +static irqreturn_t do_hvm_evtchn_intr(int irq, void *dev_id)
> +{
> +	xen_hvm_evtchn_do_upcall(get_irq_regs());
> +	return IRQ_HANDLED;
> +}
> +
> +static int xen_allocate_irq(struct pci_dev *pdev)
> +{
> +	return request_irq(pdev->irq, do_hvm_evtchn_intr,
> +			IRQF_DISABLED | IRQF_NOBALANCING | IRQF_TRIGGER_RISING,
> +			"xen-platform-pci", pdev);
> +}
> +
> +static int __devinit platform_pci_init(struct pci_dev *pdev,
> +				       const struct pci_device_id *ent)
> +{
> +	int i, ret;
> +	long ioaddr, iolen;
> +	long mmio_addr, mmio_len;
> +	uint64_t callback_via;
> +
> +	i = pci_enable_device(pdev);
> +	if (i)
> +		return i;
> +
> +	ioaddr = pci_resource_start(pdev, 0);
> +	iolen = pci_resource_len(pdev, 0);
> +
> +	mmio_addr = pci_resource_start(pdev, 1);
> +	mmio_len = pci_resource_len(pdev, 1);
> +
> +	if (mmio_addr == 0 || ioaddr == 0) {
> +		dev_err(&pdev->dev, "no resources found\n");
> +		ret = -ENOENT;
> +	}
> +
> +	if (request_mem_region(mmio_addr, mmio_len, DRV_NAME) == NULL) {
> +		dev_err(&pdev->dev, "MEM I/O resource 0x%lx @ 0x%lx busy\n",
> +		       mmio_addr, mmio_len);
> +		ret = -EBUSY;
> +	}
> +
> +	if (request_region(ioaddr, iolen, DRV_NAME) == NULL) {
> +		dev_err(&pdev->dev, "I/O resource 0x%lx @ 0x%lx busy\n",
> +		       iolen, ioaddr);
> +		ret = -EBUSY;
> +		goto out;
> +	}
> +
> +	platform_mmio = mmio_addr;
> +	platform_mmiolen = mmio_len;
> +
> +	if (!xen_have_vector_callback) {
> +		ret = xen_allocate_irq(pdev);
> +		if (ret) {
> +			printk(KERN_WARNING "request_irq failed err=%d\n", ret);
> +			goto out;
> +		}
> +		callback_via = get_callback_via(pdev);
> +		ret = xen_set_callback_via(callback_via);
> +		if (ret) {
> +			printk(KERN_WARNING
> +					"Unable to set the evtchn callback err=%d\n", ret);
> +			goto out;
> +		}
> +	}
> +
> +	ret = gnttab_init();
> +	if (ret)
> +		goto out;
> +	ret = xenbus_probe_init();
> +	if (ret)
> +		goto out;
> +
> +out:
> +	if (ret) {
> +		release_mem_region(mmio_addr, mmio_len);
> +		release_region(ioaddr, iolen);
> +		pci_disable_device(pdev);
> +	}
> +
> +	return ret;
> +}
> +
> +static struct pci_device_id platform_pci_tbl[] __devinitdata = {
> +	{PCI_VENDOR_ID_XEN, PCI_DEVICE_ID_XEN_PLATFORM,
> +		PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0},
> +	{0,}
> +};
> +
> +MODULE_DEVICE_TABLE(pci, platform_pci_tbl);
> +
> +static struct pci_driver platform_driver = {
> +	.name =           DRV_NAME,
> +	.probe =          platform_pci_init,
> +	.id_table =       platform_pci_tbl,
> +};
> +
> +static int __init platform_pci_module_init(void)
> +{
> +	int rc;
> +
> +	rc = pci_register_driver(&platform_driver);
> +	if (rc) {
> +		printk(KERN_INFO DRV_NAME
> +		       ": No platform pci device model found\n");
> +		return rc;
> +	}
> +	return 0;
> +}
> +
> +module_init(platform_pci_module_init);
> diff --git a/drivers/xen/xenbus/xenbus_probe.c b/drivers/xen/xenbus/xenbus_probe.c
> index 0b05b62..dc6ed06 100644
> --- a/drivers/xen/xenbus/xenbus_probe.c
> +++ b/drivers/xen/xenbus/xenbus_probe.c
> @@ -782,16 +782,24 @@ void xenbus_probe(struct work_struct *unused)
>  	blocking_notifier_call_chain(&xenstore_chain, 0, NULL);
>  }
>  
> -static int __init xenbus_probe_init(void)
> +static int __init __xenbus_probe_init(void)
> +{
> +	/* Delay initialization in the PV on HVM case */
> +	if (xen_hvm_domain())
> +		return 0;
> +
> +	if (!xen_pv_domain())
> +		return -ENODEV;
> +
> +	return xenbus_probe_init();
> +}
> +
> +int xenbus_probe_init(void)
>  {
>  	int err = 0;
>  
>  	DPRINTK("");
>  
> -	err = -ENODEV;
> -	if (!xen_domain())
> -		goto out_error;
> -
>  	/* Register ourselves with the kernel bus subsystem */
>  	err = bus_register(&xenbus_frontend.bus);
>  	if (err)
> @@ -857,7 +865,7 @@ static int __init xenbus_probe_init(void)
>  	return err;
>  }
>  
> -postcore_initcall(xenbus_probe_init);
> +postcore_initcall(__xenbus_probe_init);
>  
>  MODULE_LICENSE("GPL");
>  
> diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> index 9f688d2..64b528d 100644
> --- a/include/linux/pci_ids.h
> +++ b/include/linux/pci_ids.h
> @@ -2718,3 +2718,6 @@
>  #define PCI_DEVICE_ID_RME_DIGI32	0x9896
>  #define PCI_DEVICE_ID_RME_DIGI32_PRO	0x9897
>  #define PCI_DEVICE_ID_RME_DIGI32_8	0x9898
> +
> +#define PCI_VENDOR_ID_XEN	0x5853
> +#define PCI_DEVICE_ID_XEN_PLATFORM	0x0001
> diff --git a/include/xen/grant_table.h b/include/xen/grant_table.h
> index a40f1cd..811cda5 100644
> --- a/include/xen/grant_table.h
> +++ b/include/xen/grant_table.h
> @@ -51,6 +51,7 @@ struct gnttab_free_callback {
>  	u16 count;
>  };
>  
> +int gnttab_init(void);
>  int gnttab_suspend(void);
>  int gnttab_resume(void);
>  
> diff --git a/include/xen/interface/grant_table.h b/include/xen/interface/grant_table.h
> index 39da93c..39e5717 100644
> --- a/include/xen/interface/grant_table.h
> +++ b/include/xen/interface/grant_table.h
> @@ -28,6 +28,7 @@
>  #ifndef __XEN_PUBLIC_GRANT_TABLE_H__
>  #define __XEN_PUBLIC_GRANT_TABLE_H__
>  
> +#include <xen/interface/xen.h>
>  
>  /***********************************
>   * GRANT TABLE REPRESENTATION
> diff --git a/include/xen/xenbus.h b/include/xen/xenbus.h
> index 43e2d7d..ffa97de 100644
> --- a/include/xen/xenbus.h
> +++ b/include/xen/xenbus.h
> @@ -174,6 +174,7 @@ void unregister_xenbus_watch(struct xenbus_watch *watch);
>  void xs_suspend(void);
>  void xs_resume(void);
>  void xs_suspend_cancel(void);
> +int xenbus_probe_init(void);
>  
>  /* Used by xenbus_dev to borrow kernel's store connection. */
>  void *xenbus_dev_request_and_reply(struct xsd_sockmsg *msg);

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 05/11] Xen PCI platform device driver
  2010-05-28  8:55   ` [Xen-devel] " Zhigang Wang
@ 2010-05-28 14:18     ` Stefano Stabellini
  0 siblings, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-28 14:18 UTC (permalink / raw)
  To: Zhigang Wang
  Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com,
	Stefano Stabellini, linux-kernel@vger.kernel.org, Don Dutile,
	Sheng Yang

On Fri, 28 May 2010, Zhigang Wang wrote:
> > +config XEN_PLATFORM_PCI
> > +     tristate "xen platform pci device driver"
> > +     depends on XEN
>         default m
> 
> ?

given the size of the platform pci driver and the fact that blkfront and
netfront are both 'default y' I am going to go with 'default y'  here
too

^ permalink raw reply	[flat|nested] 39+ messages in thread

* [PATCH 06/11] Add suspend\resume support for PV on HVM guests.
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (4 preceding siblings ...)
  2010-05-24 18:27 ` [PATCH 05/11] Xen PCI platform device driver Stefano Stabellini
@ 2010-05-24 18:27 ` Stefano Stabellini
  2010-05-25 20:20   ` Jeremy Fitzhardinge
  2010-05-24 18:27 ` [PATCH 07/11] Allow xen platform pci device to be compiled as a module Stefano Stabellini
                   ` (8 subsequent siblings)
  14 siblings, 1 reply; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:27 UTC (permalink / raw)
  To: linux-kernel
  Cc: xen-devel, Don Dutile, Stefano Stabellini, Jeremy Fitzhardinge,
	Stefano Stabellini

Suspend\resume requires few different thing on HVM: the suspend
hypercall is different, we don't need to save\restore any memory related
setting, but we need to reinitialize the shared info page and the
callback mechanism.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 arch/x86/xen/enlighten.c   |   14 +++++++++---
 arch/x86/xen/suspend.c     |    6 +++++
 arch/x86/xen/xen-ops.h     |    1 +
 drivers/xen/manage.c       |   45 ++++++++++++++++++++++++++++++++++++++++---
 drivers/xen/platform-pci.c |   23 +++++++++++++++++++++-
 include/xen/xen-ops.h      |    3 ++
 6 files changed, 83 insertions(+), 9 deletions(-)

diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index ed34b30..cb316b5 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -1263,12 +1263,14 @@ static int init_hvm_pv_info(int *major, int *minor)
 	return 0;
 }
 
-static void __init init_shared_info(void)
+void init_shared_info(void)
 {
+	int cpu;
 	struct xen_add_to_physmap xatp;
-	struct shared_info *shared_info_page;
+	static struct shared_info *shared_info_page = 0;
 
-	shared_info_page = (struct shared_info *) alloc_bootmem_pages(PAGE_SIZE);
+	if (!shared_info_page)
+		shared_info_page = (struct shared_info *) alloc_bootmem_pages(PAGE_SIZE);
 	xatp.domid = DOMID_SELF;
 	xatp.idx = 0;
 	xatp.space = XENMAPSPACE_shared_info;
@@ -1280,7 +1282,11 @@ static void __init init_shared_info(void)
 
 	/* Don't do the full vcpu_info placement stuff until we have a
 	   possible map and a non-dummy shared_info. */
-	per_cpu(xen_vcpu, 0) = &HYPERVISOR_shared_info->vcpu_info[0];
+	/* init_shared_info is run at resume time too, in that case multiple
+	 * vcpus might be online */
+ 	for_each_online_cpu(cpu) {
+		per_cpu(xen_vcpu, cpu) = &HYPERVISOR_shared_info->vcpu_info[cpu];
+	}
 }
 
 static int __cpuinit xen_hvm_cpu_notify(struct notifier_block *self,
diff --git a/arch/x86/xen/suspend.c b/arch/x86/xen/suspend.c
index 987267f..ef517ee 100644
--- a/arch/x86/xen/suspend.c
+++ b/arch/x86/xen/suspend.c
@@ -26,6 +26,12 @@ void xen_pre_suspend(void)
 		BUG();
 }
 
+void xen_hvm_post_suspend(int suspend_cancelled)
+{
+	init_shared_info();
+	xen_callback_vector();
+}
+
 void xen_post_suspend(int suspend_cancelled)
 {
 	xen_build_mfn_list_list();
diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
index 0d0e0e6..caf89ee 100644
--- a/arch/x86/xen/xen-ops.h
+++ b/arch/x86/xen/xen-ops.h
@@ -39,6 +39,7 @@ void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
+void init_shared_info(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
 
diff --git a/drivers/xen/manage.c b/drivers/xen/manage.c
index 2ac4440..0716ba6 100644
--- a/drivers/xen/manage.c
+++ b/drivers/xen/manage.c
@@ -9,6 +9,7 @@
 #include <linux/stop_machine.h>
 #include <linux/freezer.h>
 
+#include <xen/xen.h>
 #include <xen/xenbus.h>
 #include <xen/grant_table.h>
 #include <xen/events.h>
@@ -17,6 +18,7 @@
 
 #include <asm/xen/hypercall.h>
 #include <asm/xen/page.h>
+#include <asm/xen/hypervisor.h>
 
 enum shutdown_state {
 	SHUTDOWN_INVALID = -1,
@@ -33,10 +35,30 @@ enum shutdown_state {
 static enum shutdown_state shutting_down = SHUTDOWN_INVALID;
 
 #ifdef CONFIG_PM_SLEEP
-static int xen_suspend(void *data)
+static int xen_hvm_suspend(void *data)
 {
+	struct sched_shutdown r = { .reason = SHUTDOWN_suspend };
 	int *cancelled = data;
+
+	BUG_ON(!irqs_disabled());
+
+	*cancelled = HYPERVISOR_sched_op(SCHEDOP_shutdown, &r);
+
+	xen_hvm_post_suspend(*cancelled);
+	gnttab_resume();
+
+	if (!*cancelled) {
+		xen_irq_resume();
+		xen_timer_resume();
+	}
+
+	return 0;
+}
+
+static int xen_suspend(void *data)
+{
 	int err;
+	int *cancelled = data;
 
 	BUG_ON(!irqs_disabled());
 
@@ -112,7 +134,10 @@ static void do_suspend(void)
 		goto out_resume;
 	}
 
-	err = stop_machine(xen_suspend, &cancelled, cpumask_of(0));
+	if (xen_hvm_domain())
+		err = stop_machine(xen_hvm_suspend, &cancelled, cpumask_of(0));
+	else
+		err = stop_machine(xen_suspend, &cancelled, cpumask_of(0));
 
 	dpm_resume_noirq(PMSG_RESUME);
 
@@ -261,7 +286,19 @@ static int shutdown_event(struct notifier_block *notifier,
 	return NOTIFY_DONE;
 }
 
-static int __init setup_shutdown_event(void)
+static int __init __setup_shutdown_event(void)
+{
+	/* Delay initialization in the PV on HVM case */
+	if (xen_hvm_domain())
+		return 0;
+
+	if (!xen_pv_domain())
+		return -ENODEV;
+
+	return xen_setup_shutdown_event();
+}
+
+int xen_setup_shutdown_event(void)
 {
 	static struct notifier_block xenstore_notifier = {
 		.notifier_call = shutdown_event
@@ -271,4 +308,4 @@ static int __init setup_shutdown_event(void)
 	return 0;
 }
 
-subsys_initcall(setup_shutdown_event);
+subsys_initcall(__setup_shutdown_event);
diff --git a/drivers/xen/platform-pci.c b/drivers/xen/platform-pci.c
index 0f78318..70395c9 100644
--- a/drivers/xen/platform-pci.c
+++ b/drivers/xen/platform-pci.c
@@ -31,6 +31,7 @@
 #include <xen/xenbus.h>
 #include <xen/events.h>
 #include <xen/hvm.h>
+#include <xen/xen-ops.h>
 
 #define DRV_NAME    "xen-platform-pci"
 
@@ -41,6 +42,7 @@ MODULE_LICENSE("GPL");
 static unsigned long platform_mmio;
 static unsigned long platform_mmio_alloc;
 static unsigned long platform_mmiolen;
+static uint64_t callback_via;
 
 unsigned long alloc_xen_mmio(unsigned long len)
 {
@@ -85,13 +87,25 @@ static int xen_allocate_irq(struct pci_dev *pdev)
 			"xen-platform-pci", pdev);
 }
 
+static int platform_pci_resume(struct pci_dev *pdev)
+{
+	int err;
+	if (xen_have_vector_callback)
+		return 0;
+	err = xen_set_callback_via(callback_via);
+	if (err) {
+		printk("platform_pci_resume failure!\n");
+		return err;
+	}
+	return 0;
+}
+
 static int __devinit platform_pci_init(struct pci_dev *pdev,
 				       const struct pci_device_id *ent)
 {
 	int i, ret;
 	long ioaddr, iolen;
 	long mmio_addr, mmio_len;
-	uint64_t callback_via;
 
 	i = pci_enable_device(pdev);
 	if (i)
@@ -145,6 +159,10 @@ static int __devinit platform_pci_init(struct pci_dev *pdev,
 	ret = xenbus_probe_init();
 	if (ret)
 		goto out;
+	ret = xen_setup_shutdown_event();
+	if (ret)
+		goto out;
+
 
 out:
 	if (ret) {
@@ -168,6 +186,9 @@ static struct pci_driver platform_driver = {
 	.name =           DRV_NAME,
 	.probe =          platform_pci_init,
 	.id_table =       platform_pci_tbl,
+#ifdef CONFIG_PM
+	.resume_early =   platform_pci_resume,
+#endif
 };
 
 static int __init platform_pci_module_init(void)
diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h
index 883a21b..46bc81e 100644
--- a/include/xen/xen-ops.h
+++ b/include/xen/xen-ops.h
@@ -7,6 +7,7 @@ DECLARE_PER_CPU(struct vcpu_info *, xen_vcpu);
 
 void xen_pre_suspend(void);
 void xen_post_suspend(int suspend_cancelled);
+void xen_hvm_post_suspend(int suspend_cancelled);
 
 void xen_mm_pin_all(void);
 void xen_mm_unpin_all(void);
@@ -14,4 +15,6 @@ void xen_mm_unpin_all(void);
 void xen_timer_resume(void);
 void xen_arch_resume(void);
 
+int xen_setup_shutdown_event(void);
+
 #endif /* INCLUDE_XEN_OPS_H */
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* Re: [PATCH 06/11] Add suspend\resume support for PV on HVM guests.
  2010-05-24 18:27 ` [PATCH 06/11] Add suspend\resume support for PV on HVM guests Stefano Stabellini
@ 2010-05-25 20:20   ` Jeremy Fitzhardinge
  0 siblings, 0 replies; 39+ messages in thread
From: Jeremy Fitzhardinge @ 2010-05-25 20:20 UTC (permalink / raw)
  To: Stefano Stabellini; +Cc: xen-devel, Don Dutile, linux-kernel

On 05/24/2010 11:27 AM, Stefano Stabellini wrote:
> Suspend\resume requires few different thing on HVM: the suspend
> hypercall is different, we don't need to save\restore any memory related
> setting, but we need to reinitialize the shared info page and the
> callback mechanism.
>
> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> ---
>  arch/x86/xen/enlighten.c   |   14 +++++++++---
>  arch/x86/xen/suspend.c     |    6 +++++
>  arch/x86/xen/xen-ops.h     |    1 +
>  drivers/xen/manage.c       |   45 ++++++++++++++++++++++++++++++++++++++++---
>  drivers/xen/platform-pci.c |   23 +++++++++++++++++++++-
>  include/xen/xen-ops.h      |    3 ++
>  6 files changed, 83 insertions(+), 9 deletions(-)
>
> diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
> index ed34b30..cb316b5 100644
> --- a/arch/x86/xen/enlighten.c
> +++ b/arch/x86/xen/enlighten.c
> @@ -1263,12 +1263,14 @@ static int init_hvm_pv_info(int *major, int *minor)
>  	return 0;
>  }
>  
> -static void __init init_shared_info(void)
> +void init_shared_info(void)
>   

This needs a xen_ prefix.

    J

>  {
> +	int cpu;
>  	struct xen_add_to_physmap xatp;
> -	struct shared_info *shared_info_page;
> +	static struct shared_info *shared_info_page = 0;
>  
> -	shared_info_page = (struct shared_info *) alloc_bootmem_pages(PAGE_SIZE);
> +	if (!shared_info_page)
> +		shared_info_page = (struct shared_info *) alloc_bootmem_pages(PAGE_SIZE);
>  	xatp.domid = DOMID_SELF;
>  	xatp.idx = 0;
>  	xatp.space = XENMAPSPACE_shared_info;
> @@ -1280,7 +1282,11 @@ static void __init init_shared_info(void)
>  
>  	/* Don't do the full vcpu_info placement stuff until we have a
>  	   possible map and a non-dummy shared_info. */
> -	per_cpu(xen_vcpu, 0) = &HYPERVISOR_shared_info->vcpu_info[0];
> +	/* init_shared_info is run at resume time too, in that case multiple
> +	 * vcpus might be online */
> + 	for_each_online_cpu(cpu) {
> +		per_cpu(xen_vcpu, cpu) = &HYPERVISOR_shared_info->vcpu_info[cpu];
> +	}
>  }
>  
>  static int __cpuinit xen_hvm_cpu_notify(struct notifier_block *self,
> diff --git a/arch/x86/xen/suspend.c b/arch/x86/xen/suspend.c
> index 987267f..ef517ee 100644
> --- a/arch/x86/xen/suspend.c
> +++ b/arch/x86/xen/suspend.c
> @@ -26,6 +26,12 @@ void xen_pre_suspend(void)
>  		BUG();
>  }
>  
> +void xen_hvm_post_suspend(int suspend_cancelled)
> +{
> +	init_shared_info();
> +	xen_callback_vector();
> +}
> +
>  void xen_post_suspend(int suspend_cancelled)
>  {
>  	xen_build_mfn_list_list();
> diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
> index 0d0e0e6..caf89ee 100644
> --- a/arch/x86/xen/xen-ops.h
> +++ b/arch/x86/xen/xen-ops.h
> @@ -39,6 +39,7 @@ void xen_enable_syscall(void);
>  void xen_vcpu_restore(void);
>  
>  void xen_callback_vector(void);
> +void init_shared_info(void);
>  
>  void __init xen_build_dynamic_phys_to_machine(void);
>  
> diff --git a/drivers/xen/manage.c b/drivers/xen/manage.c
> index 2ac4440..0716ba6 100644
> --- a/drivers/xen/manage.c
> +++ b/drivers/xen/manage.c
> @@ -9,6 +9,7 @@
>  #include <linux/stop_machine.h>
>  #include <linux/freezer.h>
>  
> +#include <xen/xen.h>
>  #include <xen/xenbus.h>
>  #include <xen/grant_table.h>
>  #include <xen/events.h>
> @@ -17,6 +18,7 @@
>  
>  #include <asm/xen/hypercall.h>
>  #include <asm/xen/page.h>
> +#include <asm/xen/hypervisor.h>
>  
>  enum shutdown_state {
>  	SHUTDOWN_INVALID = -1,
> @@ -33,10 +35,30 @@ enum shutdown_state {
>  static enum shutdown_state shutting_down = SHUTDOWN_INVALID;
>  
>  #ifdef CONFIG_PM_SLEEP
> -static int xen_suspend(void *data)
> +static int xen_hvm_suspend(void *data)
>  {
> +	struct sched_shutdown r = { .reason = SHUTDOWN_suspend };
>  	int *cancelled = data;
> +
> +	BUG_ON(!irqs_disabled());
> +
> +	*cancelled = HYPERVISOR_sched_op(SCHEDOP_shutdown, &r);
> +
> +	xen_hvm_post_suspend(*cancelled);
> +	gnttab_resume();
> +
> +	if (!*cancelled) {
> +		xen_irq_resume();
> +		xen_timer_resume();
> +	}
> +
> +	return 0;
> +}
> +
> +static int xen_suspend(void *data)
> +{
>  	int err;
> +	int *cancelled = data;
>  
>  	BUG_ON(!irqs_disabled());
>  
> @@ -112,7 +134,10 @@ static void do_suspend(void)
>  		goto out_resume;
>  	}
>  
> -	err = stop_machine(xen_suspend, &cancelled, cpumask_of(0));
> +	if (xen_hvm_domain())
> +		err = stop_machine(xen_hvm_suspend, &cancelled, cpumask_of(0));
> +	else
> +		err = stop_machine(xen_suspend, &cancelled, cpumask_of(0));
>  
>  	dpm_resume_noirq(PMSG_RESUME);
>  
> @@ -261,7 +286,19 @@ static int shutdown_event(struct notifier_block *notifier,
>  	return NOTIFY_DONE;
>  }
>  
> -static int __init setup_shutdown_event(void)
> +static int __init __setup_shutdown_event(void)
> +{
> +	/* Delay initialization in the PV on HVM case */
> +	if (xen_hvm_domain())
> +		return 0;
> +
> +	if (!xen_pv_domain())
> +		return -ENODEV;
> +
> +	return xen_setup_shutdown_event();
> +}
> +
> +int xen_setup_shutdown_event(void)
>  {
>  	static struct notifier_block xenstore_notifier = {
>  		.notifier_call = shutdown_event
> @@ -271,4 +308,4 @@ static int __init setup_shutdown_event(void)
>  	return 0;
>  }
>  
> -subsys_initcall(setup_shutdown_event);
> +subsys_initcall(__setup_shutdown_event);
> diff --git a/drivers/xen/platform-pci.c b/drivers/xen/platform-pci.c
> index 0f78318..70395c9 100644
> --- a/drivers/xen/platform-pci.c
> +++ b/drivers/xen/platform-pci.c
> @@ -31,6 +31,7 @@
>  #include <xen/xenbus.h>
>  #include <xen/events.h>
>  #include <xen/hvm.h>
> +#include <xen/xen-ops.h>
>  
>  #define DRV_NAME    "xen-platform-pci"
>  
> @@ -41,6 +42,7 @@ MODULE_LICENSE("GPL");
>  static unsigned long platform_mmio;
>  static unsigned long platform_mmio_alloc;
>  static unsigned long platform_mmiolen;
> +static uint64_t callback_via;
>  
>  unsigned long alloc_xen_mmio(unsigned long len)
>  {
> @@ -85,13 +87,25 @@ static int xen_allocate_irq(struct pci_dev *pdev)
>  			"xen-platform-pci", pdev);
>  }
>  
> +static int platform_pci_resume(struct pci_dev *pdev)
> +{
> +	int err;
> +	if (xen_have_vector_callback)
> +		return 0;
> +	err = xen_set_callback_via(callback_via);
> +	if (err) {
> +		printk("platform_pci_resume failure!\n");
> +		return err;
> +	}
> +	return 0;
> +}
> +
>  static int __devinit platform_pci_init(struct pci_dev *pdev,
>  				       const struct pci_device_id *ent)
>  {
>  	int i, ret;
>  	long ioaddr, iolen;
>  	long mmio_addr, mmio_len;
> -	uint64_t callback_via;
>  
>  	i = pci_enable_device(pdev);
>  	if (i)
> @@ -145,6 +159,10 @@ static int __devinit platform_pci_init(struct pci_dev *pdev,
>  	ret = xenbus_probe_init();
>  	if (ret)
>  		goto out;
> +	ret = xen_setup_shutdown_event();
> +	if (ret)
> +		goto out;
> +
>  
>  out:
>  	if (ret) {
> @@ -168,6 +186,9 @@ static struct pci_driver platform_driver = {
>  	.name =           DRV_NAME,
>  	.probe =          platform_pci_init,
>  	.id_table =       platform_pci_tbl,
> +#ifdef CONFIG_PM
> +	.resume_early =   platform_pci_resume,
> +#endif
>  };
>  
>  static int __init platform_pci_module_init(void)
> diff --git a/include/xen/xen-ops.h b/include/xen/xen-ops.h
> index 883a21b..46bc81e 100644
> --- a/include/xen/xen-ops.h
> +++ b/include/xen/xen-ops.h
> @@ -7,6 +7,7 @@ DECLARE_PER_CPU(struct vcpu_info *, xen_vcpu);
>  
>  void xen_pre_suspend(void);
>  void xen_post_suspend(int suspend_cancelled);
> +void xen_hvm_post_suspend(int suspend_cancelled);
>  
>  void xen_mm_pin_all(void);
>  void xen_mm_unpin_all(void);
> @@ -14,4 +15,6 @@ void xen_mm_unpin_all(void);
>  void xen_timer_resume(void);
>  void xen_arch_resume(void);
>  
> +int xen_setup_shutdown_event(void);
> +
>  #endif /* INCLUDE_XEN_OPS_H */
>   

^ permalink raw reply	[flat|nested] 39+ messages in thread

* [PATCH 07/11] Allow xen platform pci device to be compiled as a module
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (5 preceding siblings ...)
  2010-05-24 18:27 ` [PATCH 06/11] Add suspend\resume support for PV on HVM guests Stefano Stabellini
@ 2010-05-24 18:27 ` Stefano Stabellini
  2010-05-24 18:27 ` [PATCH 08/11] Fix possible NULL pointer dereference in print_IO_APIC Stefano Stabellini
                   ` (7 subsequent siblings)
  14 siblings, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:27 UTC (permalink / raw)
  To: linux-kernel
  Cc: xen-devel, Don Dutile, Stefano Stabellini, Jeremy Fitzhardinge,
	Stefano Stabellini

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 drivers/xen/events.c              |    1 +
 drivers/xen/grant-table.c         |   20 +++++++++++---------
 drivers/xen/manage.c              |    1 +
 drivers/xen/platform-pci.c        |    3 +++
 drivers/xen/xenbus/xenbus_probe.c |    1 +
 include/xen/grant_table.h         |    3 +++
 6 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index c6f561d..e6dd8ba 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -697,6 +697,7 @@ void xen_hvm_evtchn_do_upcall(struct pt_regs *regs)
 {
 	__xen_evtchn_do_upcall();
 }
+EXPORT_SYMBOL_GPL(xen_hvm_evtchn_do_upcall);
 
 /* Rebind a new event channel to an existing irq. */
 void rebind_evtchn_irq(int evtchn, int irq)
diff --git a/drivers/xen/grant-table.c b/drivers/xen/grant-table.c
index 4c959a5..64bcad1 100644
--- a/drivers/xen/grant-table.c
+++ b/drivers/xen/grant-table.c
@@ -61,7 +61,8 @@ static unsigned int boot_max_nr_grant_frames;
 static int gnttab_free_count;
 static grant_ref_t gnttab_free_head;
 static DEFINE_SPINLOCK(gnttab_list_lock);
-static unsigned long hvm_pv_resume_frames;
+unsigned long xen_hvm_resume_frames;
+EXPORT_SYMBOL_GPL(xen_hvm_resume_frames);
 
 static struct grant_entry *shared;
 
@@ -436,7 +437,7 @@ static unsigned int __max_nr_grant_frames(void)
 	return query.max_nr_frames;
 }
 
-static inline unsigned int max_nr_grant_frames(void)
+unsigned int gnttab_max_nr_grant_frames(void)
 {
 	unsigned int xen_max = __max_nr_grant_frames();
 
@@ -444,6 +445,7 @@ static inline unsigned int max_nr_grant_frames(void)
 		return boot_max_nr_grant_frames;
 	return xen_max;
 }
+EXPORT_SYMBOL_GPL(gnttab_max_nr_grant_frames);
 
 static int gnttab_map(unsigned int start_idx, unsigned int end_idx)
 {
@@ -464,7 +466,7 @@ static int gnttab_map(unsigned int start_idx, unsigned int end_idx)
 			xatp.domid = DOMID_SELF;
 			xatp.idx = i;
 			xatp.space = XENMAPSPACE_grant_table;
-			xatp.gpfn = (hvm_pv_resume_frames >> PAGE_SHIFT) + i;
+			xatp.gpfn = (xen_hvm_resume_frames >> PAGE_SHIFT) + i;
 			rc = HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp);
 			if (rc != 0) {
 				printk(KERN_WARNING
@@ -492,7 +494,7 @@ static int gnttab_map(unsigned int start_idx, unsigned int end_idx)
 
 	BUG_ON(rc || setup.status);
 
-	rc = arch_gnttab_map_shared(frames, nr_gframes, max_nr_grant_frames(),
+	rc = arch_gnttab_map_shared(frames, nr_gframes, gnttab_max_nr_grant_frames(),
 				    &shared);
 	BUG_ON(rc);
 
@@ -505,16 +507,15 @@ int gnttab_resume(void)
 {
 	unsigned int max_nr_gframes;
 
-	max_nr_gframes = max_nr_grant_frames();
+	max_nr_gframes = gnttab_max_nr_grant_frames();
 	if (max_nr_gframes < nr_grant_frames)
 		return -ENOSYS;
 
 	if (xen_pv_domain())
 		return gnttab_map(0, nr_grant_frames - 1);
 
-	if (!hvm_pv_resume_frames) {
-		hvm_pv_resume_frames = alloc_xen_mmio(PAGE_SIZE * max_nr_gframes);
-		shared = ioremap(hvm_pv_resume_frames, PAGE_SIZE * max_nr_gframes);
+	if (!shared) {
+		shared = ioremap(xen_hvm_resume_frames, PAGE_SIZE * max_nr_gframes);
 		if (shared == NULL) {
 			printk(KERN_WARNING
 					"Fail to ioremap gnttab share frames\n");
@@ -541,7 +542,7 @@ static int gnttab_expand(unsigned int req_entries)
 	cur = nr_grant_frames;
 	extra = ((req_entries + (GREFS_PER_GRANT_FRAME-1)) /
 		 GREFS_PER_GRANT_FRAME);
-	if (cur + extra > max_nr_grant_frames())
+	if (cur + extra > gnttab_max_nr_grant_frames())
 		return -ENOSPC;
 
 	rc = gnttab_map(cur, cur + extra - 1);
@@ -599,6 +600,7 @@ int gnttab_init(void)
 	kfree(gnttab_list);
 	return -ENOMEM;
 }
+EXPORT_SYMBOL_GPL(gnttab_init);
 
 static int __devinit __gnttab_init(void)
 {
diff --git a/drivers/xen/manage.c b/drivers/xen/manage.c
index 0716ba6..f5162e4 100644
--- a/drivers/xen/manage.c
+++ b/drivers/xen/manage.c
@@ -307,5 +307,6 @@ int xen_setup_shutdown_event(void)
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(xen_setup_shutdown_event);
 
 subsys_initcall(__setup_shutdown_event);
diff --git a/drivers/xen/platform-pci.c b/drivers/xen/platform-pci.c
index 70395c9..07c821c 100644
--- a/drivers/xen/platform-pci.c
+++ b/drivers/xen/platform-pci.c
@@ -106,6 +106,7 @@ static int __devinit platform_pci_init(struct pci_dev *pdev,
 	int i, ret;
 	long ioaddr, iolen;
 	long mmio_addr, mmio_len;
+	unsigned int max_nr_gframes;
 
 	i = pci_enable_device(pdev);
 	if (i)
@@ -153,6 +154,8 @@ static int __devinit platform_pci_init(struct pci_dev *pdev,
 		}
 	}
 
+	max_nr_gframes = gnttab_max_nr_grant_frames();
+	xen_hvm_resume_frames = alloc_xen_mmio(PAGE_SIZE * max_nr_gframes);
 	ret = gnttab_init();
 	if (ret)
 		goto out;
diff --git a/drivers/xen/xenbus/xenbus_probe.c b/drivers/xen/xenbus/xenbus_probe.c
index dc6ed06..cf971da 100644
--- a/drivers/xen/xenbus/xenbus_probe.c
+++ b/drivers/xen/xenbus/xenbus_probe.c
@@ -864,6 +864,7 @@ int xenbus_probe_init(void)
   out_error:
 	return err;
 }
+EXPORT_SYMBOL_GPL(xenbus_probe_init);
 
 postcore_initcall(__xenbus_probe_init);
 
diff --git a/include/xen/grant_table.h b/include/xen/grant_table.h
index 811cda5..2a58ee0 100644
--- a/include/xen/grant_table.h
+++ b/include/xen/grant_table.h
@@ -113,6 +113,9 @@ int arch_gnttab_map_shared(unsigned long *frames, unsigned long nr_gframes,
 void arch_gnttab_unmap_shared(struct grant_entry *shared,
 			      unsigned long nr_gframes);
 
+extern unsigned long xen_hvm_resume_frames;
+unsigned int gnttab_max_nr_grant_frames(void);
+
 #define gnttab_map_vaddr(map) ((void *)(map.host_virt_addr))
 
 #endif /* __ASM_GNTTAB_H__ */
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* [PATCH 08/11] Fix possible NULL pointer dereference in print_IO_APIC
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (6 preceding siblings ...)
  2010-05-24 18:27 ` [PATCH 07/11] Allow xen platform pci device to be compiled as a module Stefano Stabellini
@ 2010-05-24 18:27 ` Stefano Stabellini
  2010-05-24 18:27 ` [PATCH 09/11] __setup_vector_irq: handle NULL chip_data Stefano Stabellini
                   ` (6 subsequent siblings)
  14 siblings, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:27 UTC (permalink / raw)
  To: linux-kernel
  Cc: xen-devel, Don Dutile, Stefano Stabellini, Jeremy Fitzhardinge,
	Stefano Stabellini

Make sure chip_data is not NULL before accessing it (the VIRQ_TIMER
handler and virq handlers in general don't have any chip_data).

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 arch/x86/kernel/apic/io_apic.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index eb2789c..c64499c 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -1732,6 +1732,8 @@ __apicdebuginit(void) print_IO_APIC(void)
 		struct irq_pin_list *entry;
 
 		cfg = desc->chip_data;
+		if (!cfg)
+			continue;
 		entry = cfg->irq_2_pin;
 		if (!entry)
 			continue;
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* [PATCH 09/11] __setup_vector_irq: handle NULL chip_data
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (7 preceding siblings ...)
  2010-05-24 18:27 ` [PATCH 08/11] Fix possible NULL pointer dereference in print_IO_APIC Stefano Stabellini
@ 2010-05-24 18:27 ` Stefano Stabellini
  2010-05-24 18:27 ` [PATCH 10/11] Support VIRQ_TIMER and pvclock on HVM Stefano Stabellini
                   ` (5 subsequent siblings)
  14 siblings, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:27 UTC (permalink / raw)
  To: linux-kernel
  Cc: xen-devel, Don Dutile, Stefano Stabellini, Jeremy Fitzhardinge,
	Stefano Stabellini

the VIRQ_TIMER handler and virq handlers in general don't have any
chip_data.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 arch/x86/kernel/apic/io_apic.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index c64499c..4d3d391 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -1269,6 +1269,8 @@ void __setup_vector_irq(int cpu)
 	/* Mark the inuse vectors */
 	for_each_irq_desc(irq, desc) {
 		cfg = desc->chip_data;
+		if (!cfg)
+			continue;
 
 		/*
 		 * If it is a legacy IRQ handled by the legacy PIC, this cpu
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* [PATCH 10/11] Support VIRQ_TIMER and pvclock on HVM
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (8 preceding siblings ...)
  2010-05-24 18:27 ` [PATCH 09/11] __setup_vector_irq: handle NULL chip_data Stefano Stabellini
@ 2010-05-24 18:27 ` Stefano Stabellini
  2010-05-25 20:24   ` Jeremy Fitzhardinge
  2010-05-24 18:27 ` [PATCH 11/11] Unplug emulated disks and nics Stefano Stabellini
                   ` (4 subsequent siblings)
  14 siblings, 1 reply; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:27 UTC (permalink / raw)
  To: linux-kernel
  Cc: xen-devel, Don Dutile, Stefano Stabellini, Jeremy Fitzhardinge,
	Stefano Stabellini

Use the paravirtualized timer (VIRQ_TIMER) as timer and the
paravirtualized clock (pvclock) as wallclock on HVM domains too.
Any interaction with emulated hardware is not very fast, so using the
paravirtualized timer instead of the emulated hpet is defenitely a
performance improvement.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 arch/x86/xen/enlighten.c         |   31 +++++++++++++++++++++++++++++++
 arch/x86/xen/suspend.c           |    4 ++++
 include/xen/interface/features.h |    3 +++
 3 files changed, 38 insertions(+), 0 deletions(-)

diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index cb316b5..88e8a09 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -1307,6 +1307,36 @@ static struct notifier_block __cpuinitdata xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+static void xen_hvm_setup_cpu_clockevents(void)
+{
+	int cpu = smp_processor_id();
+	xen_setup_runstate_info(cpu);
+	xen_setup_timer(cpu);
+	xen_setup_cpu_clockevents();
+}
+
+static void init_hvm_time(void)
+{
+	/* vector callback is needed otherwise we cannot receive interrupts
+	 * on cpu > 0 */
+	if (!xen_have_vector_callback && num_present_cpus() > 1)
+		return;
+	if (!xen_feature(XENFEAT_hvm_safe_pvclock)) {
+		printk(KERN_INFO "Xen doesn't support pvclock on HVM,"
+				"disable pv timer\n");
+		return;
+	}
+
+	pv_time_ops = xen_time_ops;
+	x86_init.timers.timer_init = xen_time_init;
+	x86_init.timers.setup_percpu_clockev = x86_init_noop;
+	x86_cpuinit.setup_percpu_clockev = xen_hvm_setup_cpu_clockevents;
+
+	x86_platform.calibrate_tsc = xen_tsc_khz;
+	x86_platform.get_wallclock = xen_get_wallclock;
+	x86_platform.set_wallclock = xen_set_wallclock;
+}
+
 void __init xen_guest_init(void)
 {
 	int r;
@@ -1326,4 +1356,5 @@ void __init xen_guest_init(void)
 	register_cpu_notifier(&xen_hvm_cpu_notifier);
 	have_vcpu_info_placement = 0;
 	x86_init.irqs.intr_init = xen_init_IRQ;
+	init_hvm_time();
 }
diff --git a/arch/x86/xen/suspend.c b/arch/x86/xen/suspend.c
index ef517ee..02c3253 100644
--- a/arch/x86/xen/suspend.c
+++ b/arch/x86/xen/suspend.c
@@ -28,8 +28,12 @@ void xen_pre_suspend(void)
 
 void xen_hvm_post_suspend(int suspend_cancelled)
 {
+	int cpu;
 	init_shared_info();
 	xen_callback_vector();
+	for_each_online_cpu(cpu) {
+		xen_setup_runstate_info(cpu);
+	}
 }
 
 void xen_post_suspend(int suspend_cancelled)
diff --git a/include/xen/interface/features.h b/include/xen/interface/features.h
index 8ab08b9..70d2563 100644
--- a/include/xen/interface/features.h
+++ b/include/xen/interface/features.h
@@ -44,6 +44,9 @@
 /* x86: Does this Xen host support the HVM callback vector type? */
 #define XENFEAT_hvm_callback_vector        8
 
+/* x86: pvclock algorithm is safe to use on HVM */
+#define XENFEAT_hvm_safe_pvclock           9
+
 #define XENFEAT_NR_SUBMAPS 1
 
 #endif /* __XEN_PUBLIC_FEATURES_H__ */
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* Re: [PATCH 10/11] Support VIRQ_TIMER and pvclock on HVM
  2010-05-24 18:27 ` [PATCH 10/11] Support VIRQ_TIMER and pvclock on HVM Stefano Stabellini
@ 2010-05-25 20:24   ` Jeremy Fitzhardinge
  2010-05-26 13:08     ` Stefano Stabellini
  0 siblings, 1 reply; 39+ messages in thread
From: Jeremy Fitzhardinge @ 2010-05-25 20:24 UTC (permalink / raw)
  To: Stefano Stabellini; +Cc: xen-devel, Don Dutile, linux-kernel

On 05/24/2010 11:27 AM, Stefano Stabellini wrote:
> Use the paravirtualized timer (VIRQ_TIMER) as timer and the
> paravirtualized clock (pvclock) as wallclock on HVM domains too.
>   

That's not quite correct.  There's the Xen clockevent driver, the Xen
clocksource, and a distinct-but-related Xen notion of wallclock time. 
In general, clocksources and wallclock don't have much direct
relationship to each other.

> Any interaction with emulated hardware is not very fast, so using the
> paravirtualized timer instead of the emulated hpet is defenitely a
>   
"definitely"

> performance improvement.
>   

Are there accuracy/stability/precision improvements too?

    J

> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> ---
>  arch/x86/xen/enlighten.c         |   31 +++++++++++++++++++++++++++++++
>  arch/x86/xen/suspend.c           |    4 ++++
>  include/xen/interface/features.h |    3 +++
>  3 files changed, 38 insertions(+), 0 deletions(-)
>
> diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
> index cb316b5..88e8a09 100644
> --- a/arch/x86/xen/enlighten.c
> +++ b/arch/x86/xen/enlighten.c
> @@ -1307,6 +1307,36 @@ static struct notifier_block __cpuinitdata xen_hvm_cpu_notifier = {
>  	.notifier_call	= xen_hvm_cpu_notify,
>  };
>  
> +static void xen_hvm_setup_cpu_clockevents(void)
> +{
> +	int cpu = smp_processor_id();
> +	xen_setup_runstate_info(cpu);
> +	xen_setup_timer(cpu);
> +	xen_setup_cpu_clockevents();
> +}
> +
> +static void init_hvm_time(void)
> +{
> +	/* vector callback is needed otherwise we cannot receive interrupts
> +	 * on cpu > 0 */
> +	if (!xen_have_vector_callback && num_present_cpus() > 1)
> +		return;
> +	if (!xen_feature(XENFEAT_hvm_safe_pvclock)) {
> +		printk(KERN_INFO "Xen doesn't support pvclock on HVM,"
> +				"disable pv timer\n");
> +		return;
> +	}
> +
> +	pv_time_ops = xen_time_ops;
> +	x86_init.timers.timer_init = xen_time_init;
> +	x86_init.timers.setup_percpu_clockev = x86_init_noop;
> +	x86_cpuinit.setup_percpu_clockev = xen_hvm_setup_cpu_clockevents;
> +
> +	x86_platform.calibrate_tsc = xen_tsc_khz;
> +	x86_platform.get_wallclock = xen_get_wallclock;
> +	x86_platform.set_wallclock = xen_set_wallclock;
> +}
> +
>  void __init xen_guest_init(void)
>  {
>  	int r;
> @@ -1326,4 +1356,5 @@ void __init xen_guest_init(void)
>  	register_cpu_notifier(&xen_hvm_cpu_notifier);
>  	have_vcpu_info_placement = 0;
>  	x86_init.irqs.intr_init = xen_init_IRQ;
> +	init_hvm_time();
>  }
> diff --git a/arch/x86/xen/suspend.c b/arch/x86/xen/suspend.c
> index ef517ee..02c3253 100644
> --- a/arch/x86/xen/suspend.c
> +++ b/arch/x86/xen/suspend.c
> @@ -28,8 +28,12 @@ void xen_pre_suspend(void)
>  
>  void xen_hvm_post_suspend(int suspend_cancelled)
>  {
> +	int cpu;
>  	init_shared_info();
>  	xen_callback_vector();
> +	for_each_online_cpu(cpu) {
> +		xen_setup_runstate_info(cpu);
> +	}
>  }
>  
>  void xen_post_suspend(int suspend_cancelled)
> diff --git a/include/xen/interface/features.h b/include/xen/interface/features.h
> index 8ab08b9..70d2563 100644
> --- a/include/xen/interface/features.h
> +++ b/include/xen/interface/features.h
> @@ -44,6 +44,9 @@
>  /* x86: Does this Xen host support the HVM callback vector type? */
>  #define XENFEAT_hvm_callback_vector        8
>  
> +/* x86: pvclock algorithm is safe to use on HVM */
> +#define XENFEAT_hvm_safe_pvclock           9
> +
>  #define XENFEAT_NR_SUBMAPS 1
>  
>  #endif /* __XEN_PUBLIC_FEATURES_H__ */
>   

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 10/11] Support VIRQ_TIMER and pvclock on HVM
  2010-05-25 20:24   ` Jeremy Fitzhardinge
@ 2010-05-26 13:08     ` Stefano Stabellini
  0 siblings, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-26 13:08 UTC (permalink / raw)
  To: Jeremy Fitzhardinge
  Cc: Stefano Stabellini, linux-kernel@vger.kernel.org,
	xen-devel@lists.xensource.com, Don Dutile

On Tue, 25 May 2010, Jeremy Fitzhardinge wrote:
> On 05/24/2010 11:27 AM, Stefano Stabellini wrote:
> > Use the paravirtualized timer (VIRQ_TIMER) as timer and the
> > paravirtualized clock (pvclock) as wallclock on HVM domains too.
> >   
> 
> That's not quite correct.  There's the Xen clockevent driver, the Xen
> clocksource, and a distinct-but-related Xen notion of wallclock time. 
> In general, clocksources and wallclock don't have much direct
> relationship to each other.

Let me rephrase it:

use xen_vcpuop_clockevent instead of hpet and APIC timers as main
clockevent device on all vcpus, use the xen wallclock time as wallclock
instead of rtc and use xen_clocksource as clocksource.
The pv clock algorithm needs to work correctly for the xen_clocksource
and xen wallclock to be usable, only modern Xen versions offer a
reliable pv clock in HVM guests (XENFEAT_hvm_safe_pvclock).

> 
> > Any interaction with emulated hardware is not very fast, so using the
> > paravirtualized timer instead of the emulated hpet is defenitely a
> >   
> "definitely"
> 
> > performance improvement.
> >   
> 
> Are there accuracy/stability/precision improvements too?
> 

Using the hpet as clocksource means a VMEXIT every time we read/write to
the hpet mmio addresses, pvclock give us a better rating without
VMEXITs. Same goes for the xen wallclock and xen_vcpuop_clockevent.

^ permalink raw reply	[flat|nested] 39+ messages in thread

* [PATCH 11/11] Unplug emulated disks and nics
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (9 preceding siblings ...)
  2010-05-24 18:27 ` [PATCH 10/11] Support VIRQ_TIMER and pvclock on HVM Stefano Stabellini
@ 2010-05-24 18:27 ` Stefano Stabellini
  2010-05-25 20:31   ` Jeremy Fitzhardinge
  2010-05-24 18:29 ` [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (3 subsequent siblings)
  14 siblings, 1 reply; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:27 UTC (permalink / raw)
  To: linux-kernel
  Cc: xen-devel, Don Dutile, Stefano Stabellini, Jeremy Fitzhardinge,
	Stefano Stabellini

Add a xen_emul_unplug command line option to the kernel to unplug
xen emulated disks and nics.

Set the default value of xen_emul_unplug depending on whether or
not the Xen PV frontends and the Xen platform PCI driver have
been compiled for this kernel (modules or built-in are both OK).

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 Documentation/kernel-parameters.txt  |   11 +++
 arch/x86/xen/Makefile                |    2 +-
 arch/x86/xen/enlighten.c             |    1 +
 arch/x86/xen/platform-pci-unplug.c   |  131 ++++++++++++++++++++++++++++++++++
 arch/x86/xen/xen-ops.h               |    1 +
 drivers/xen/platform-pci.c           |    4 +
 include/xen/interface/platform_pci.h |   46 ++++++++++++
 include/xen/platform_pci.h           |   34 +++++++++
 8 files changed, 229 insertions(+), 1 deletions(-)
 create mode 100644 arch/x86/xen/platform-pci-unplug.c
 create mode 100644 include/xen/interface/platform_pci.h
 create mode 100644 include/xen/platform_pci.h

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 839b21b..716eea8 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -113,6 +113,7 @@ parameter is applicable:
 			More X86-64 boot options can be found in
 			Documentation/x86/x86_64/boot-options.txt .
 	X86	Either 32bit or 64bit x86 (same as X86-32+X86-64)
+	XEN	Xen support is enabled
 
 In addition, the following text indicates that the option:
 
@@ -2834,6 +2835,16 @@ and is between 256 and 4096 characters. It is defined in the file
 	xd=		[HW,XT] Original XT pre-IDE (RLL encoded) disks.
 	xd_geo=		See header of drivers/block/xd.c.
 
+	xen_emul_unplug=		[HW,X86,XEN]
+			Unplug Xen emulated devices
+			Format: [unplug0,][unplug1]
+			ide-disks -- unplug primary master IDE devices
+			aux-ide-disks -- unplug non-primary-master IDE devices
+			nics -- unplug network devices
+			all -- unplug all emulated devices (NICs and IDE disks)
+			ignore -- continue loading the Xen platform PCI driver even
+				if the version check failed
+
 	xirc2ps_cs=	[NET,PCMCIA]
 			Format:
 			<irq>,<irq_mask>,<io>,<full_duplex>,<do_sound>,<lockup_hack>[,<irq2>[,<irq3>[,<irq4>]]]
diff --git a/arch/x86/xen/Makefile b/arch/x86/xen/Makefile
index 3bb4fc2..9309546 100644
--- a/arch/x86/xen/Makefile
+++ b/arch/x86/xen/Makefile
@@ -12,7 +12,7 @@ CFLAGS_mmu.o			:= $(nostackp)
 
 obj-y		:= enlighten.o setup.o multicalls.o mmu.o irq.o \
 			time.o xen-asm.o xen-asm_$(BITS).o \
-			grant-table.o suspend.o
+			grant-table.o suspend.o platform-pci-unplug.o
 
 obj-$(CONFIG_SMP)		+= smp.o
 obj-$(CONFIG_PARAVIRT_SPINLOCKS)+= spinlock.o
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 88e8a09..fbf7c43 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -1354,6 +1354,7 @@ void __init xen_guest_init(void)
 	if (xen_feature(XENFEAT_hvm_callback_vector))
 		xen_have_vector_callback = 1;
 	register_cpu_notifier(&xen_hvm_cpu_notifier);
+	xen_unplug_emulated_devices();
 	have_vcpu_info_placement = 0;
 	x86_init.irqs.intr_init = xen_init_IRQ;
 	init_hvm_time();
diff --git a/arch/x86/xen/platform-pci-unplug.c b/arch/x86/xen/platform-pci-unplug.c
new file mode 100644
index 0000000..4094f29
--- /dev/null
+++ b/arch/x86/xen/platform-pci-unplug.c
@@ -0,0 +1,131 @@
+/******************************************************************************
+ * platform-pci-unplug.c
+ *
+ * Xen platform PCI device driver
+ * Copyright (c) 2010, Citrix
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
+ * Place - Suite 330, Boston, MA 02111-1307 USA.
+ *
+ */
+
+#include <asm/io.h>
+
+#include <linux/init.h>
+#include <linux/module.h>
+
+#include <xen/platform_pci.h>
+#include <xen/interface/platform_pci.h>
+
+int xen_platform_pci;
+EXPORT_SYMBOL_GPL(xen_platform_pci);
+static int xen_emul_unplug;
+
+static int __init check_platform_magic(void)
+{
+	short magic;
+	char protocol;
+
+	magic = inw(XEN_IOPORT_MAGIC);
+	if (magic != XEN_IOPORT_MAGIC_VAL) {
+		printk(KERN_ERR "Xen Platform PCI: unrecognised magic value\n");
+		return -1;
+	}
+
+	protocol = inb(XEN_IOPORT_PROTOVER);
+
+	printk(KERN_DEBUG "Xen Platform PCI: I/O protocol version %d\n",
+			protocol);
+
+	switch (protocol) {
+	case 1:
+		outw(XEN_IOPORT_LINUX_PRODNUM, XEN_IOPORT_PRODNUM);
+		outl(XEN_IOPORT_LINUX_DRVVER, XEN_IOPORT_DRVVER);
+		if (inw(XEN_IOPORT_MAGIC) != XEN_IOPORT_MAGIC_VAL) {
+			printk(KERN_ERR "Xen Platform: blacklisted by host\n");
+			return -3;
+		}
+		break;
+	default:
+		printk(KERN_WARNING "Xen Platform PCI: unknown I/O protocol version");
+		return -2;
+	}
+
+	return 0;
+}
+
+void __init xen_unplug_emulated_devices(void)
+{
+	int r;
+
+	/* check the version of the xen platform PCI device */
+	r = check_platform_magic();
+	/* If the version matches enable the Xen platform PCI driver.
+	 * Also enable the Xen platform PCI driver if the version is really old
+	 * and the user told us to ignore it. */
+	if (!r || (r == -1 && (xen_emul_unplug & UNPLUG_IGNORE)))
+		xen_platform_pci = 1;
+	/* Set the default value of xen_emul_unplug depending on whether or
+	 * not the Xen PV frontends and the Xen platform PCI driver have
+	 * been compiled for this kernel (modules or built-in are both OK). */
+	if (xen_platform_pci && !xen_emul_unplug) {
+#if (defined(CONFIG_XEN_NETDEV_FRONTEND) || \
+		defined(CONFIG_XEN_NETDEV_FRONTEND_MODULE)) && \
+		(defined(CONFIG_XEN_PLATFORM_PCI) || \
+		 defined(CONFIG_XEN_PLATFORM_PCI_MODULE))
+		printk(KERN_INFO "Netfront and the Xen platform PCI driver have "
+				"been compiled for this kernel: unplug emulated NICs.\n");
+		xen_emul_unplug |= UNPLUG_ALL_NICS;
+#endif
+#if (defined(CONFIG_XEN_BLKDEV_FRONTEND) || \
+		defined(CONFIG_XEN_BLKDEV_FRONTEND_MODULE)) && \
+		(defined(CONFIG_XEN_PLATFORM_PCI) || \
+		 defined(CONFIG_XEN_PLATFORM_PCI_MODULE))
+		printk(KERN_INFO "Blkfront and the Xen platform PCI driver have "
+				"been compiled for this kernel: unplug emulated disks.\n"
+				"You might have to change the root device\n"
+				"from /dev/hd[a-d] to /dev/xvd[a-d]\n"
+				"in your root= kernel command line option\n");
+		xen_emul_unplug |= UNPLUG_ALL_IDE_DISKS;
+#endif
+	}
+	/* Now unplug the emulated devices */
+	if (xen_platform_pci && !(xen_emul_unplug & UNPLUG_IGNORE))
+		outw(xen_emul_unplug, XEN_IOPORT_UNPLUG);
+}
+
+static int __init parse_xen_emul_unplug(char *arg)
+{
+	char *p, *q;
+
+	for (p = arg; p; p = q) {
+		q = strchr(arg, ',');
+		if (q)
+			*q++ = '\0';
+		if (!strcmp(p, "all"))
+			xen_emul_unplug |= UNPLUG_ALL;
+		else if (!strcmp(p, "ide-disks"))
+			xen_emul_unplug |= UNPLUG_ALL_IDE_DISKS;
+		else if (!strcmp(p, "aux-ide-disks"))
+			xen_emul_unplug |= UNPLUG_AUX_IDE_DISKS;
+		else if (!strcmp(p, "nics"))
+			xen_emul_unplug |= UNPLUG_ALL_NICS;
+		else if (!strcmp(p, "ignore"))
+			xen_emul_unplug |= UNPLUG_IGNORE;
+		else
+			printk(KERN_WARNING "unrecognised option '%s' "
+				 "in module parameter 'dev_unplug'\n", p);
+	}
+	return 0;
+}
+early_param("xen_emul_unplug", parse_xen_emul_unplug);
diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
index caf89ee..e146f5f 100644
--- a/arch/x86/xen/xen-ops.h
+++ b/arch/x86/xen/xen-ops.h
@@ -40,6 +40,7 @@ void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
 void init_shared_info(void);
+void __init xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
 
diff --git a/drivers/xen/platform-pci.c b/drivers/xen/platform-pci.c
index 07c821c..de3ce1e 100644
--- a/drivers/xen/platform-pci.c
+++ b/drivers/xen/platform-pci.c
@@ -27,6 +27,7 @@
 #include <linux/module.h>
 #include <linux/pci.h>
 
+#include <xen/platform_pci.h>
 #include <xen/grant_table.h>
 #include <xen/xenbus.h>
 #include <xen/events.h>
@@ -198,6 +199,9 @@ static int __init platform_pci_module_init(void)
 {
 	int rc;
 
+	if (!xen_platform_pci)
+		return -ENODEV;
+
 	rc = pci_register_driver(&platform_driver);
 	if (rc) {
 		printk(KERN_INFO DRV_NAME
diff --git a/include/xen/interface/platform_pci.h b/include/xen/interface/platform_pci.h
new file mode 100644
index 0000000..720eaf5
--- /dev/null
+++ b/include/xen/interface/platform_pci.h
@@ -0,0 +1,46 @@
+/******************************************************************************
+ * platform_pci.h
+ *
+ * Interface for granting foreign access to page frames, and receiving
+ * page-ownership transfers.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef __XEN_PUBLIC_PLATFORM_PCI_H__
+#define __XEN_PUBLIC_PLATFORM_PCI_H__
+
+#define XEN_IOPORT_BASE 0x10
+
+#define XEN_IOPORT_PLATFLAGS	(XEN_IOPORT_BASE + 0) /* 1 byte access (R/W) */
+#define XEN_IOPORT_MAGIC	(XEN_IOPORT_BASE + 0) /* 2 byte access (R) */
+#define XEN_IOPORT_UNPLUG	(XEN_IOPORT_BASE + 0) /* 2 byte access (W) */
+#define XEN_IOPORT_DRVVER	(XEN_IOPORT_BASE + 0) /* 4 byte access (W) */
+
+#define XEN_IOPORT_SYSLOG	(XEN_IOPORT_BASE + 2) /* 1 byte access (W) */
+#define XEN_IOPORT_PROTOVER	(XEN_IOPORT_BASE + 2) /* 1 byte access (R) */
+#define XEN_IOPORT_PRODNUM	(XEN_IOPORT_BASE + 2) /* 2 byte access (W) */
+
+#define UNPLUG_ALL_IDE_DISKS 1
+#define UNPLUG_ALL_NICS 2
+#define UNPLUG_AUX_IDE_DISKS 4
+#define UNPLUG_ALL 7
+#define UNPLUG_IGNORE 8
+
+#endif /* __XEN_PUBLIC_PLATFORM_PCI_H__ */
diff --git a/include/xen/platform_pci.h b/include/xen/platform_pci.h
new file mode 100644
index 0000000..ae04d8a
--- /dev/null
+++ b/include/xen/platform_pci.h
@@ -0,0 +1,34 @@
+/******************************************************************************
+ * platform-pci.h
+ *
+ * Xen platform PCI device driver
+ * Copyright (c) 2004, Intel Corporation. <xiaofeng.ling@intel.com>
+ * Copyright (c) 2007, XenSource Inc.
+ * Copyright (c) 2010, Citrix
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
+ * Place - Suite 330, Boston, MA 02111-1307 USA.
+ */
+
+#ifndef _XEN_PLATFORM_PCI_H
+#define _XEN_PLATFORM_PCI_H
+
+#include <linux/version.h>
+
+#define XEN_IOPORT_MAGIC_VAL 0x49d2
+#define XEN_IOPORT_LINUX_PRODNUM 0xffff
+#define XEN_IOPORT_LINUX_DRVVER  ((LINUX_VERSION_CODE << 8) + 0x0)
+
+extern int xen_platform_pci;
+
+#endif /* _XEN_PLATFORM_PCI_H */
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

* Re: [PATCH 11/11] Unplug emulated disks and nics
  2010-05-24 18:27 ` [PATCH 11/11] Unplug emulated disks and nics Stefano Stabellini
@ 2010-05-25 20:31   ` Jeremy Fitzhardinge
  2010-05-26 12:27     ` Stefano Stabellini
  0 siblings, 1 reply; 39+ messages in thread
From: Jeremy Fitzhardinge @ 2010-05-25 20:31 UTC (permalink / raw)
  To: Stefano Stabellini; +Cc: linux-kernel, xen-devel, Don Dutile

On 05/24/2010 11:27 AM, Stefano Stabellini wrote:
> Add a xen_emul_unplug command line option to the kernel to unplug
> xen emulated disks and nics.
>
> Set the default value of xen_emul_unplug depending on whether or
> not the Xen PV frontends and the Xen platform PCI driver have
> been compiled for this kernel (modules or built-in are both OK).
>
> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> ---
>  Documentation/kernel-parameters.txt  |   11 +++
>  arch/x86/xen/Makefile                |    2 +-
>  arch/x86/xen/enlighten.c             |    1 +
>  arch/x86/xen/platform-pci-unplug.c   |  131 ++++++++++++++++++++++++++++++++++
>  arch/x86/xen/xen-ops.h               |    1 +
>  drivers/xen/platform-pci.c           |    4 +
>  include/xen/interface/platform_pci.h |   46 ++++++++++++
>  include/xen/platform_pci.h           |   34 +++++++++
>  8 files changed, 229 insertions(+), 1 deletions(-)
>  create mode 100644 arch/x86/xen/platform-pci-unplug.c
>  create mode 100644 include/xen/interface/platform_pci.h
>  create mode 100644 include/xen/platform_pci.h
>
> diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
> index 839b21b..716eea8 100644
> --- a/Documentation/kernel-parameters.txt
> +++ b/Documentation/kernel-parameters.txt
> @@ -113,6 +113,7 @@ parameter is applicable:
>  			More X86-64 boot options can be found in
>  			Documentation/x86/x86_64/boot-options.txt .
>  	X86	Either 32bit or 64bit x86 (same as X86-32+X86-64)
> +	XEN	Xen support is enabled
>  
>  In addition, the following text indicates that the option:
>  
> @@ -2834,6 +2835,16 @@ and is between 256 and 4096 characters. It is defined in the file
>  	xd=		[HW,XT] Original XT pre-IDE (RLL encoded) disks.
>  	xd_geo=		See header of drivers/block/xd.c.
>  
> +	xen_emul_unplug=		[HW,X86,XEN]
> +			Unplug Xen emulated devices
> +			Format: [unplug0,][unplug1]
> +			ide-disks -- unplug primary master IDE devices
> +			aux-ide-disks -- unplug non-primary-master IDE devices
> +			nics -- unplug network devices
> +			all -- unplug all emulated devices (NICs and IDE disks)
> +			ignore -- continue loading the Xen platform PCI driver even
> +				if the version check failed
> +
>  	xirc2ps_cs=	[NET,PCMCIA]
>  			Format:
>  			<irq>,<irq_mask>,<io>,<full_duplex>,<do_sound>,<lockup_hack>[,<irq2>[,<irq3>[,<irq4>]]]
> diff --git a/arch/x86/xen/Makefile b/arch/x86/xen/Makefile
> index 3bb4fc2..9309546 100644
> --- a/arch/x86/xen/Makefile
> +++ b/arch/x86/xen/Makefile
> @@ -12,7 +12,7 @@ CFLAGS_mmu.o			:= $(nostackp)
>  
>  obj-y		:= enlighten.o setup.o multicalls.o mmu.o irq.o \
>  			time.o xen-asm.o xen-asm_$(BITS).o \
> -			grant-table.o suspend.o
> +			grant-table.o suspend.o platform-pci-unplug.o
>  
>  obj-$(CONFIG_SMP)		+= smp.o
>  obj-$(CONFIG_PARAVIRT_SPINLOCKS)+= spinlock.o
> diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
> index 88e8a09..fbf7c43 100644
> --- a/arch/x86/xen/enlighten.c
> +++ b/arch/x86/xen/enlighten.c
> @@ -1354,6 +1354,7 @@ void __init xen_guest_init(void)
>  	if (xen_feature(XENFEAT_hvm_callback_vector))
>  		xen_have_vector_callback = 1;
>  	register_cpu_notifier(&xen_hvm_cpu_notifier);
> +	xen_unplug_emulated_devices();
>  	have_vcpu_info_placement = 0;
>  	x86_init.irqs.intr_init = xen_init_IRQ;
>  	init_hvm_time();
> diff --git a/arch/x86/xen/platform-pci-unplug.c b/arch/x86/xen/platform-pci-unplug.c
> new file mode 100644
> index 0000000..4094f29
> --- /dev/null
> +++ b/arch/x86/xen/platform-pci-unplug.c
> @@ -0,0 +1,131 @@
> +/******************************************************************************
> + * platform-pci-unplug.c
> + *
> + * Xen platform PCI device driver
> + * Copyright (c) 2010, Citrix
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
> + * Place - Suite 330, Boston, MA 02111-1307 USA.
> + *
> + */
> +
> +#include <asm/io.h>
> +
> +#include <linux/init.h>
> +#include <linux/module.h>
> +
> +#include <xen/platform_pci.h>
> +#include <xen/interface/platform_pci.h>
> +
> +int xen_platform_pci;
> +EXPORT_SYMBOL_GPL(xen_platform_pci);
>   

Could you give this a bit more of a descriptive name, type and/or
comment?  What is it indicating about the xen_platform_pci driver?  That
it has been detected?  That it is in use?  That it can be used?  Is it
boolean, or can it have any value?

> +static int xen_emul_unplug;
> +
> +static int __init check_platform_magic(void)
> +{
> +	short magic;
> +	char protocol;
> +
> +	magic = inw(XEN_IOPORT_MAGIC);
> +	if (magic != XEN_IOPORT_MAGIC_VAL) {
> +		printk(KERN_ERR "Xen Platform PCI: unrecognised magic value\n");
> +		return -1;
> +	}
> +
> +	protocol = inb(XEN_IOPORT_PROTOVER);
> +
> +	printk(KERN_DEBUG "Xen Platform PCI: I/O protocol version %d\n",
> +			protocol);
> +
> +	switch (protocol) {
> +	case 1:
> +		outw(XEN_IOPORT_LINUX_PRODNUM, XEN_IOPORT_PRODNUM);
> +		outl(XEN_IOPORT_LINUX_DRVVER, XEN_IOPORT_DRVVER);
> +		if (inw(XEN_IOPORT_MAGIC) != XEN_IOPORT_MAGIC_VAL) {
> +			printk(KERN_ERR "Xen Platform: blacklisted by host\n");
> +			return -3;
> +		}
> +		break;
> +	default:
> +		printk(KERN_WARNING "Xen Platform PCI: unknown I/O protocol version");
> +		return -2;
> +	}
> +
> +	return 0;
> +}
> +
> +void __init xen_unplug_emulated_devices(void)
> +{
> +	int r;
> +
> +	/* check the version of the xen platform PCI device */
> +	r = check_platform_magic();
> +	/* If the version matches enable the Xen platform PCI driver.
> +	 * Also enable the Xen platform PCI driver if the version is really old
> +	 * and the user told us to ignore it. */
> +	if (!r || (r == -1 && (xen_emul_unplug & UNPLUG_IGNORE)))
> +		xen_platform_pci = 1;
> +	/* Set the default value of xen_emul_unplug depending on whether or
> +	 * not the Xen PV frontends and the Xen platform PCI driver have
> +	 * been compiled for this kernel (modules or built-in are both OK). */
> +	if (xen_platform_pci && !xen_emul_unplug) {
> +#if (defined(CONFIG_XEN_NETDEV_FRONTEND) || \
> +		defined(CONFIG_XEN_NETDEV_FRONTEND_MODULE)) && \
> +		(defined(CONFIG_XEN_PLATFORM_PCI) || \
> +		 defined(CONFIG_XEN_PLATFORM_PCI_MODULE))
> +		printk(KERN_INFO "Netfront and the Xen platform PCI driver have "
> +				"been compiled for this kernel: unplug emulated NICs.\n");
> +		xen_emul_unplug |= UNPLUG_ALL_NICS;
> +#endif
> +#if (defined(CONFIG_XEN_BLKDEV_FRONTEND) || \
> +		defined(CONFIG_XEN_BLKDEV_FRONTEND_MODULE)) && \
> +		(defined(CONFIG_XEN_PLATFORM_PCI) || \
> +		 defined(CONFIG_XEN_PLATFORM_PCI_MODULE))
> +		printk(KERN_INFO "Blkfront and the Xen platform PCI driver have "
> +				"been compiled for this kernel: unplug emulated disks.\n"
> +				"You might have to change the root device\n"
> +				"from /dev/hd[a-d] to /dev/xvd[a-d]\n"
> +				"in your root= kernel command line option\n");
> +		xen_emul_unplug |= UNPLUG_ALL_IDE_DISKS;
> +#endif
> +	}
> +	/* Now unplug the emulated devices */
> +	if (xen_platform_pci && !(xen_emul_unplug & UNPLUG_IGNORE))
> +		outw(xen_emul_unplug, XEN_IOPORT_UNPLUG);
> +}
> +
> +static int __init parse_xen_emul_unplug(char *arg)
> +{
> +	char *p, *q;
> +
> +	for (p = arg; p; p = q) {
> +		q = strchr(arg, ',');
> +		if (q)
> +			*q++ = '\0';
> +		if (!strcmp(p, "all"))
> +			xen_emul_unplug |= UNPLUG_ALL;
> +		else if (!strcmp(p, "ide-disks"))
> +			xen_emul_unplug |= UNPLUG_ALL_IDE_DISKS;
> +		else if (!strcmp(p, "aux-ide-disks"))
> +			xen_emul_unplug |= UNPLUG_AUX_IDE_DISKS;
> +		else if (!strcmp(p, "nics"))
> +			xen_emul_unplug |= UNPLUG_ALL_NICS;
> +		else if (!strcmp(p, "ignore"))
> +			xen_emul_unplug |= UNPLUG_IGNORE;
> +		else
> +			printk(KERN_WARNING "unrecognised option '%s' "
> +				 "in module parameter 'dev_unplug'\n", p);
> +	}
> +	return 0;
> +}
> +early_param("xen_emul_unplug", parse_xen_emul_unplug);
> diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
> index caf89ee..e146f5f 100644
> --- a/arch/x86/xen/xen-ops.h
> +++ b/arch/x86/xen/xen-ops.h
> @@ -40,6 +40,7 @@ void xen_vcpu_restore(void);
>  
>  void xen_callback_vector(void);
>  void init_shared_info(void);
> +void __init xen_unplug_emulated_devices(void);
>  
>  void __init xen_build_dynamic_phys_to_machine(void);
>  
> diff --git a/drivers/xen/platform-pci.c b/drivers/xen/platform-pci.c
> index 07c821c..de3ce1e 100644
> --- a/drivers/xen/platform-pci.c
> +++ b/drivers/xen/platform-pci.c
> @@ -27,6 +27,7 @@
>  #include <linux/module.h>
>  #include <linux/pci.h>
>  
> +#include <xen/platform_pci.h>
>  #include <xen/grant_table.h>
>  #include <xen/xenbus.h>
>  #include <xen/events.h>
> @@ -198,6 +199,9 @@ static int __init platform_pci_module_init(void)
>  {
>  	int rc;
>  
> +	if (!xen_platform_pci)
> +		return -ENODEV;
> +
>  	rc = pci_register_driver(&platform_driver);
>  	if (rc) {
>  		printk(KERN_INFO DRV_NAME
> diff --git a/include/xen/interface/platform_pci.h b/include/xen/interface/platform_pci.h
> new file mode 100644
> index 0000000..720eaf5
> --- /dev/null
> +++ b/include/xen/interface/platform_pci.h
> @@ -0,0 +1,46 @@
> +/******************************************************************************
> + * platform_pci.h
> + *
> + * Interface for granting foreign access to page frames, and receiving
> + * page-ownership transfers.
>   

That doesn't look very accurate.

> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a copy
> + * of this software and associated documentation files (the "Software"), to
> + * deal in the Software without restriction, including without limitation the
> + * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
> + * sell copies of the Software, and to permit persons to whom the Software is
> + * furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
> + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
> + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
> + * DEALINGS IN THE SOFTWARE.
>   

Drop all this copyright goo; these kinds of definitions can't be
copyright in the US anyway.

> + */
> +
> +#ifndef __XEN_PUBLIC_PLATFORM_PCI_H__
> +#define __XEN_PUBLIC_PLATFORM_PCI_H__
> +
> +#define XEN_IOPORT_BASE 0x10
> +
> +#define XEN_IOPORT_PLATFLAGS	(XEN_IOPORT_BASE + 0) /* 1 byte access (R/W) */
> +#define XEN_IOPORT_MAGIC	(XEN_IOPORT_BASE + 0) /* 2 byte access (R) */
> +#define XEN_IOPORT_UNPLUG	(XEN_IOPORT_BASE + 0) /* 2 byte access (W) */
> +#define XEN_IOPORT_DRVVER	(XEN_IOPORT_BASE + 0) /* 4 byte access (W) */
> +
> +#define XEN_IOPORT_SYSLOG	(XEN_IOPORT_BASE + 2) /* 1 byte access (W) */
> +#define XEN_IOPORT_PROTOVER	(XEN_IOPORT_BASE + 2) /* 1 byte access (R) */
> +#define XEN_IOPORT_PRODNUM	(XEN_IOPORT_BASE + 2) /* 2 byte access (W) */
> +
> +#define UNPLUG_ALL_IDE_DISKS 1
> +#define UNPLUG_ALL_NICS 2
> +#define UNPLUG_AUX_IDE_DISKS 4
> +#define UNPLUG_ALL 7
> +#define UNPLUG_IGNORE 8
>   

These need XEN_ prefixes, otherwise they look too generic.

> +
> +#endif /* __XEN_PUBLIC_PLATFORM_PCI_H__ */
> diff --git a/include/xen/platform_pci.h b/include/xen/platform_pci.h
> new file mode 100644
> index 0000000..ae04d8a
> --- /dev/null
> +++ b/include/xen/platform_pci.h
> @@ -0,0 +1,34 @@
> +/******************************************************************************
> + * platform-pci.h
> + *
> + * Xen platform PCI device driver
> + * Copyright (c) 2004, Intel Corporation. <xiaofeng.ling@intel.com>
> + * Copyright (c) 2007, XenSource Inc.
> + * Copyright (c) 2010, Citrix
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
> + * Place - Suite 330, Boston, MA 02111-1307 USA.
>   

Ditto; there's no copyrightable content in this file.

> + */
> +
> +#ifndef _XEN_PLATFORM_PCI_H
> +#define _XEN_PLATFORM_PCI_H
> +
> +#include <linux/version.h>
>   
Why?

> +
> +#define XEN_IOPORT_MAGIC_VAL 0x49d2
> +#define XEN_IOPORT_LINUX_PRODNUM 0xffff
> +#define XEN_IOPORT_LINUX_DRVVER  ((LINUX_VERSION_CODE << 8) + 0x0)
> +
> +extern int xen_platform_pci;
>   

I still think these headers should be folded together.  There's no point
in having them separate.  They describe an ABI, so there's no need to
keep the files in sync with the master Xen sources (since the ABI will
never change).

    J

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 11/11] Unplug emulated disks and nics
  2010-05-25 20:31   ` Jeremy Fitzhardinge
@ 2010-05-26 12:27     ` Stefano Stabellini
  2010-05-26 20:59       ` Jeremy Fitzhardinge
  0 siblings, 1 reply; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-26 12:27 UTC (permalink / raw)
  To: Jeremy Fitzhardinge
  Cc: Stefano Stabellini, linux-kernel@vger.kernel.org,
	xen-devel@lists.xensource.com, Don Dutile

On Tue, 25 May 2010, Jeremy Fitzhardinge wrote:
> Could you give this a bit more of a descriptive name, type and/or
> comment?  What is it indicating about the xen_platform_pci driver?  That
> it has been detected?  That it is in use?  That it can be used?  Is it
> boolean, or can it have any value?
> 
> 

It is boolean and it signals that the xen platform pci device can be
used. I'll add a comment and rename it to xen_platform_pci_enabled.


> > + */
> > +
> > +#ifndef _XEN_PLATFORM_PCI_H
> > +#define _XEN_PLATFORM_PCI_H
> > +
> > +#include <linux/version.h>
> >
> Why?
> 

It is needed for LINUX_VERSION_CODE, see below:

> > +
> > +#define XEN_IOPORT_MAGIC_VAL 0x49d2
> > +#define XEN_IOPORT_LINUX_PRODNUM 0xffff
> > +#define XEN_IOPORT_LINUX_DRVVER  ((LINUX_VERSION_CODE << 8) + 0x0)
> > +
> > +extern int xen_platform_pci;
> >
> 
> I still think these headers should be folded together.  There's no point
> in having them separate.  They describe an ABI, so there's no need to
> keep the files in sync with the master Xen sources (since the ABI will
> never change).
 
All right, I also agree with the other comments.

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 11/11] Unplug emulated disks and nics
  2010-05-26 12:27     ` Stefano Stabellini
@ 2010-05-26 20:59       ` Jeremy Fitzhardinge
  2010-05-27 12:29         ` Stefano Stabellini
       [not found]         ` <m2n.s.1OHcBK-0018DA@chiark.greenend.org.uk>
  0 siblings, 2 replies; 39+ messages in thread
From: Jeremy Fitzhardinge @ 2010-05-26 20:59 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: linux-kernel@vger.kernel.org, xen-devel@lists.xensource.com,
	Don Dutile

On 05/26/2010 05:27 AM, Stefano Stabellini wrote:
> It is needed for LINUX_VERSION_CODE, see below:
>
>   
>>> +
>>> +#define XEN_IOPORT_MAGIC_VAL 0x49d2
>>> +#define XEN_IOPORT_LINUX_PRODNUM 0xffff
>>> +#define XEN_IOPORT_LINUX_DRVVER  ((LINUX_VERSION_CODE << 8) + 0x0)
>>>       

Ah.  What's that for?  Does qemu change its behaviour depending on the
kernel version?  Ah, something to do with blacklisting.  But is the
Linux kernel version the most useful value here?  Shouldn't there be
some kind of separate version number?  And what's the blacklisting stuff
for anyway?

Wow, this interface is perverse.  It reuses the same IO port but changes
function depending on the size of the IO?  Again, wow.

    J

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 11/11] Unplug emulated disks and nics
  2010-05-26 20:59       ` Jeremy Fitzhardinge
@ 2010-05-27 12:29         ` Stefano Stabellini
       [not found]         ` <m2n.s.1OHcBK-0018DA@chiark.greenend.org.uk>
  1 sibling, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-27 12:29 UTC (permalink / raw)
  To: Jeremy Fitzhardinge
  Cc: xen-devel@lists.xensource.com, Don Dutile,
	linux-kernel@vger.kernel.org, Stefano Stabellini

On Wed, 26 May 2010, Jeremy Fitzhardinge wrote:
> On 05/26/2010 05:27 AM, Stefano Stabellini wrote:
> > It is needed for LINUX_VERSION_CODE, see below:
> >
> >   
> >>> +
> >>> +#define XEN_IOPORT_MAGIC_VAL 0x49d2
> >>> +#define XEN_IOPORT_LINUX_PRODNUM 0xffff
> >>> +#define XEN_IOPORT_LINUX_DRVVER  ((LINUX_VERSION_CODE << 8) + 0x0)
> >>>       
> 
> Ah.  What's that for?  Does qemu change its behaviour depending on the
> kernel version?  Ah, something to do with blacklisting.  But is the
> Linux kernel version the most useful value here?  Shouldn't there be
> some kind of separate version number?  And what's the blacklisting stuff
> for anyway?
> 

Well at the moment qemu doesn't actually change behaviour but as part of
the protocol negotiation the pv drivers are supposed to communicate
their version number to qemu. Also it is possible to blacklist a
particular pv driver version in qemu even though I don't think it is
actually used at the moment.
Regarding what version number should we give to qemu, I guess we could
have our own separate version number here, independent fron the linux
version. I'll start from 1 then.


> Wow, this interface is perverse.  It reuses the same IO port but changes
> function depending on the size of the IO?  Again, wow.
 
Yeah, before you ask, I didn't write it :)

^ permalink raw reply	[flat|nested] 39+ messages in thread

[parent not found: <m2n.s.1OHcBK-0018DA@chiark.greenend.org.uk>]

* Re: [PATCH 11/11] Unplug emulated disks and nics
       [not found]         ` <m2n.s.1OHcBK-0018DA@chiark.greenend.org.uk>
@ 2010-05-27 14:49           ` Ian Jackson
  2010-05-27 17:25             ` [Xen-devel] " Stefano Stabellini
  2010-05-27 17:51             ` Jeremy Fitzhardinge
  0 siblings, 2 replies; 39+ messages in thread
From: Ian Jackson @ 2010-05-27 14:49 UTC (permalink / raw)
  To: Jeremy Fitzhardinge
  Cc: xen-devel@lists.xensource.com, Don Dutile,
	linux-kernel@vger.kernel.org, Stefano Stabellini

Stefano Stabellini writes ("[Xen-devel] Re: [PATCH 11/11] Unplug emulated disks and nics"):
> On Wed, 26 May 2010, Jeremy Fitzhardinge wrote:
> > Wow, this interface is perverse.  It reuses the same IO port but changes
> > function depending on the size of the IO?  Again, wow.
>  
> Yeah, before you ask, I didn't write it :)

Yes, neither did I :-).  However, I did document it and now I also
maintain the "product number" registry.  Did you find the interface
spec ?  Enclosed below in case not.

I hereby allocate you ("pvops PV-on-HVM Linux, upstream") product
number 3.  Does the kernel have a way to distinguish between upstream
and other versions ?  Eg, there's the kernel version name suffix
thingy if I remember rightly.  Perhaps we should allocate a different
number for "some pvops pv-on-hvm Linux with a nonempty kernel version
name suffix".  Please advise.

You are welcome to use whatever you like for the "build number".
Perhaps the best thing would a two-byte encoding of the kernel version
number if that is possible.  As the purpose is logging and
blacklisting, it's not that critical although it's better to reuse the
same number for excessively similar builds than to use a random scheme
which might generate accidental clashes between unrelated versions.

Ian.

MAGIC IOPORT 0x10 PROTOCOL

The protocol covers three basic things:

-- Disconnecting emulated devices.
-- Getting log messages out of the drivers and into dom0.
-- Allowing dom0 to block the loading of specific drivers.  This is
   intended as a backwards-compatibility thing: if we discover a bug
   in some old version of the drivers, then rather than working around
   it in Xen, we have the option of just making those drivers fall
   back to emulated mode.

The current protocol works like this (from the point of view of
drivers):

1) When the drivers first come up, they check whether the unplug logic
   is available by reading a two-byte magic number from IO port 0x10.
   These should be 0x49d2.  If the magic number doesn't match, the
   drivers don't do anything.

2) The drivers read a one-byte protocol version from IO port 0x12.  If
   this is 0, skip to 6.

3) The drivers write a two-byte product number to IO port 0x12.  At
   the moment, the only drivers using this protocol are our
   closed-source ones, which use product number 1.

4) The drivers write a four-byte build number to IO port 0x10.

5) The drivers check the magic number by reading two bytes from 0x10
   again.  If it's changed from 0x49d2 to 0xd249, the drivers are
   blacklisted and should not load.

6) The drivers write a two-byte bitmask of devices to unplug to IO
   port 0x10.  The defined fields are:

   1 -- All IDE disks (not including CD drives)
   2 -- All emulated NICs
   4 -- All IDE disks except for the primary master (not including CD
	drives)

   The relevant emulated devices then disappear from the relevant
   buses.  For most guest operating systems, you want to do this
   before device enumeration happens.

...) Once the drivers have checked the magic number, they can send log
     messages to qemu which will be logged to wherever qemu's logs go
     (/var/log/xen/qemu-dm.log on normal Xen, dom0 syslog on
     XenServer).  These messages are written to IO port 0x12 a byte at
     a time, and are terminated by newlines.  There's a fairly
     aggressive rate limiter on these messages, so they shouldn't be
     used for anything even vaguely high-volume, but they're rather
     useful for debugging and support.

     It is still permitted for a driver to use this logging feature if
     it is blacklisted, but ONLY if it has checked the magic number
     and found it to be 0x49d2 or 0xd249.

This isn't exactly a pretty protocol, but it does solve the problem.

The blacklist is, from qemu's point of view, handled mostly through
xenstore.  A driver version is considered to be blacklisted if
/mh/driver-blacklist/{product_name}/{build_number} exists and is
readable, where {build_number} is the build number from step 4 as a
decimal number.  {product_name} is a string corresponding to the
product number in step 3.

The master registry of product names and numbers is in
qemu-xen-unstable's xenstore.c.

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [Xen-devel] Re: [PATCH 11/11] Unplug emulated disks and nics
  2010-05-27 14:49           ` Ian Jackson
@ 2010-05-27 17:25             ` Stefano Stabellini
  2010-05-27 17:51             ` Jeremy Fitzhardinge
  1 sibling, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-27 17:25 UTC (permalink / raw)
  To: Ian Jackson
  Cc: Jeremy Fitzhardinge, Stefano Stabellini,
	xen-devel@lists.xensource.com, Don Dutile,
	linux-kernel@vger.kernel.org

On Thu, 27 May 2010, Ian Jackson wrote:
> Stefano Stabellini writes ("[Xen-devel] Re: [PATCH 11/11] Unplug emulated disks and nics"):
> > On Wed, 26 May 2010, Jeremy Fitzhardinge wrote:
> > > Wow, this interface is perverse.  It reuses the same IO port but changes
> > > function depending on the size of the IO?  Again, wow.
> >  
> > Yeah, before you ask, I didn't write it :)
> 
> Yes, neither did I :-).  However, I did document it and now I also
> maintain the "product number" registry.  Did you find the interface
> spec ?  Enclosed below in case not.
> 
> I hereby allocate you ("pvops PV-on-HVM Linux, upstream") product
> number 3.  Does the kernel have a way to distinguish between upstream
> and other versions ?  Eg, there's the kernel version name suffix
> thingy if I remember rightly.  Perhaps we should allocate a different
> number for "some pvops pv-on-hvm Linux with a nonempty kernel version
> name suffix".  Please advise.
> 
> You are welcome to use whatever you like for the "build number".
> Perhaps the best thing would a two-byte encoding of the kernel version
> number if that is possible.  As the purpose is logging and
> blacklisting, it's not that critical although it's better to reuse the
> same number for excessively similar builds than to use a random scheme
> which might generate accidental clashes between unrelated versions.
> 

It seems reasonable to me.
Jeremy, what do you think?

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: Re: [PATCH 11/11] Unplug emulated disks and nics
  2010-05-27 14:49           ` Ian Jackson
  2010-05-27 17:25             ` [Xen-devel] " Stefano Stabellini
@ 2010-05-27 17:51             ` Jeremy Fitzhardinge
  2010-05-27 18:10               ` Stefano Stabellini
  1 sibling, 1 reply; 39+ messages in thread
From: Jeremy Fitzhardinge @ 2010-05-27 17:51 UTC (permalink / raw)
  To: Ian Jackson
  Cc: xen-devel@lists.xensource.com, Don Dutile,
	linux-kernel@vger.kernel.org, Stefano Stabellini

On 05/27/2010 07:49 AM, Ian Jackson wrote:
> Stefano Stabellini writes ("[Xen-devel] Re: [PATCH 11/11] Unplug emulated disks and nics"):
>   
>> On Wed, 26 May 2010, Jeremy Fitzhardinge wrote:
>>     
>>> Wow, this interface is perverse.  It reuses the same IO port but changes
>>> function depending on the size of the IO?  Again, wow.
>>>       
>>  
>> Yeah, before you ask, I didn't write it :)
>>     
> Yes, neither did I :-).  However, I did document it and now I also
> maintain the "product number" registry.  Did you find the interface
> spec ?  Enclosed below in case not.
>   

Thanks.  We should probably start a Documentation/xen/ and put this in
there as part of the patch.

> I hereby allocate you ("pvops PV-on-HVM Linux, upstream") product
> number 3.  Does the kernel have a way to distinguish between upstream
> and other versions ?  Eg, there's the kernel version name suffix
> thingy if I remember rightly.  Perhaps we should allocate a different
> number for "some pvops pv-on-hvm Linux with a nonempty kernel version
> name suffix".  Please advise.
>
> You are welcome to use whatever you like for the "build number".
> Perhaps the best thing would a two-byte encoding of the kernel version
> number if that is possible.  As the purpose is logging and
> blacklisting, it's not that critical although it's better to reuse the
> same number for excessively similar builds than to use a random scheme
> which might generate accidental clashes between unrelated versions.
>   

We could include 2 bytes of the HEAD changeset or something, with some
risk of collision.  Or just choose a constant and stick with it until
some interesting qualitative driver change makes it worthwhile bumping
the version.

I guess I can see some value in this info for recording in a log to do
some diagnostics, but the whole blacklist concept seems highly dubious
to me.  Some kind of feature negotiation makes a lot more sense to me...

    J

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: Re: [PATCH 11/11] Unplug emulated disks and nics
  2010-05-27 17:51             ` Jeremy Fitzhardinge
@ 2010-05-27 18:10               ` Stefano Stabellini
  0 siblings, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-27 18:10 UTC (permalink / raw)
  To: Jeremy Fitzhardinge
  Cc: Don Dutile, xen-devel@lists.xensource.com, Ian Jackson,
	linux-kernel@vger.kernel.org, Stefano Stabellini

On Thu, 27 May 2010, Jeremy Fitzhardinge wrote:
> We could include 2 bytes of the HEAD changeset or something, with some
> risk of collision.  Or just choose a constant and stick with it until
> some interesting qualitative driver change makes it worthwhile bumping
> the version.
> 

the constant would be more useful than a CS I think

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 0 of 12] PV on HVM Xen
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (10 preceding siblings ...)
  2010-05-24 18:27 ` [PATCH 11/11] Unplug emulated disks and nics Stefano Stabellini
@ 2010-05-24 18:29 ` Stefano Stabellini
  2010-05-24 18:30 ` Boris Derzhavets
                   ` (2 subsequent siblings)
  14 siblings, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 18:29 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: linux-kernel@vger.kernel.org, Jeremy Fitzhardinge,
	xen-devel@lists.xensource.com, Don Dutile, Sheng Yang

The subject should be [PATCH 0 of 11] PV on HVM Xen, sorry about that.

On Mon, 24 May 2010, Stefano Stabellini wrote:
> Hi all,
> this is another update of the PV on HVM Xen series that addresses
> Jeremy's comments.
> The platform_pci hooks have been removed, suspend/resume for HVM
> domains is now much more similar to the PV case and shares the same
> do_suspend function.
> Alloc_xen_mmio_hook has been removed has well, now the memory allocation for
> the grant table is done by the xen platform pci driver directly.
> The per_cpu xen_vcpu variable is set by a cpu_notifier function so that
> secondary vcpus have the variable set correctly no matter what the xen
> features are on the host.
> The kernel command line option xen_unplug has been renamed to
> xen_emul_unplug and the code that makes use of it has been moved to a
> separate file (arch/x86/xen/platform-pci-unplug.c).
> Xen_unplug_emulated_devices is now able to detect if blkfront, netfront
> and the Xen platform PCI driver have been compiled, and set the default
> value of xen_emul_unplug accordingly.
> The patch "Initialize xenbus device structs with ENODEV as
> default" has been removed from the series and it will be sent
> separately.
> Finally the comments on most of the patches have been improved.
> 
> The series is based on 2.6.34 and supports Xen PV frontends running
> in a HVM domain, including netfront, blkfront and the VIRQ_TIMER.
> 
> In order to be able to use VIRQ_TIMER and to improve performances you
> need a patch to Xen to implement the vector callback mechanism
> for event channel delivery.
> 
> A git tree is also available here:
> 
> git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git
> 
> branch name 2.6.34-pvhvm-v2.
> 
> Cheers,
> 
> Stefano
> 

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 0 of 12] PV on HVM Xen
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (11 preceding siblings ...)
  2010-05-24 18:29 ` [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
@ 2010-05-24 18:30 ` Boris Derzhavets
  2010-05-24 19:06   ` [Xen-devel] " Stefano Stabellini
  2010-05-24 18:36 ` Boris Derzhavets
  2010-05-28 10:25 ` Boris Derzhavets
  14 siblings, 1 reply; 39+ messages in thread
From: Boris Derzhavets @ 2010-05-24 18:30 UTC (permalink / raw)
  To: linux-kernel@vger.kernel.org
  Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Don Dutile,
	Sheng Yang, Stefano Stabellini

[-- Attachment #1.1: Type: text/plain, Size: 2442 bytes --]

> In order to be able to use VIRQ_TIMER and to improve performances you
> need
 a patch to Xen to implement the vector callback mechanism
> for event 
channel delivery.

Where to get it ?

Boris.

--- On Mon, 5/24/10, Stefano Stabellini <Stefano.Stabellini@eu.citrix.com> wrote:

From: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com>
Subject: [Xen-devel] [PATCH 0 of 12] PV on HVM Xen
To: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Cc: "Stefano Stabellini" <Stefano.Stabellini@eu.citrix.com>, "Jeremy Fitzhardinge" <jeremy@goop.org>, "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>, "Don Dutile" <ddutile@redhat.com>, "Sheng Yang" <sheng@linux.intel.com>
Date: Monday, May 24, 2010, 2:25 PM

Hi all,
this is another update of the PV on HVM Xen series that addresses
Jeremy's comments.
The platform_pci hooks have been removed, suspend/resume for HVM
domains is now much more similar to the PV case and shares the same
do_suspend function.
Alloc_xen_mmio_hook has been removed has well, now the memory allocation for
the grant table is done by the xen platform pci driver directly.
The per_cpu xen_vcpu variable is set by a cpu_notifier function so that
secondary vcpus have the variable set correctly no matter what the xen
features are on the host.
The kernel command line option xen_unplug has been renamed to
xen_emul_unplug and the code that makes use of it has been moved to a
separate file (arch/x86/xen/platform-pci-unplug.c).
Xen_unplug_emulated_devices is now able to detect if blkfront, netfront
and the Xen platform PCI driver have been compiled, and set the default
value of xen_emul_unplug accordingly.
The patch "Initialize xenbus device structs with ENODEV as
default" has been removed from the series and it will be sent
separately.
Finally the comments on most of the patches have been improved.

The series is based on 2.6.34 and supports Xen PV frontends running
in a HVM domain, including netfront, blkfront and the VIRQ_TIMER.

In order to be able to use VIRQ_TIMER and to improve performances you
need a patch to Xen to implement the vector callback mechanism
for event channel delivery.

A git tree is also available here:

git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git

branch name 2.6.34-pvhvm-v2.

Cheers,

Stefano

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[-- Attachment #1.2: Type: text/html, Size: 3130 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [Xen-devel] [PATCH 0 of 12] PV on HVM Xen
  2010-05-24 18:30 ` Boris Derzhavets
@ 2010-05-24 19:06   ` Stefano Stabellini
  2010-05-25  6:14     ` Boris Derzhavets
  0 siblings, 1 reply; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-24 19:06 UTC (permalink / raw)
  To: Boris Derzhavets
  Cc: linux-kernel@vger.kernel.org, Stefano Stabellini,
	Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Don Dutile,
	Sheng Yang

On Mon, 24 May 2010, Boris Derzhavets wrote:
> > In order to be able to use VIRQ_TIMER and to improve performances you
> > need a patch to Xen to implement the vector callback mechanism
> > for event channel delivery.
> 
> Where to get it ?
> 

It is this one:

http://lists.xensource.com/archives/html/xen-devel/2010-05/msg00875.html

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 0 of 12] PV on HVM Xen
  2010-05-24 19:06   ` [Xen-devel] " Stefano Stabellini
@ 2010-05-25  6:14     ` Boris Derzhavets
  2010-05-25  9:55       ` [Xen-devel] " Stefano Stabellini
  0 siblings, 1 reply; 39+ messages in thread
From: Boris Derzhavets @ 2010-05-25  6:14 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Stabellini,
	linux-kernel@vger.kernel.org, Stefano, Don Dutile, Sheng Yang


[-- Attachment #1.1: Type: text/plain, Size: 1164 bytes --]

What Xen Version this patch is supposed to be applied ?
Seems like not 4.0.

Boris.

--- On Mon, 5/24/10, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:

From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Subject: Re: [Xen-devel] [PATCH 0 of 12] PV on HVM Xen
To: "Boris Derzhavets" <bderzhavets@yahoo.com>
Cc: "Jeremy Fitzhardinge" <jeremy@goop.org>, "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>, "Stabellini" <Stefano.Stabellini@eu.citrix.com>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, Stefano@yahoo.com, "Don Dutile" <ddutile@redhat.com>, "Sheng Yang" <sheng@linux.intel.com>
Date: Monday, May 24, 2010, 3:06 PM

On Mon, 24 May 2010, Boris Derzhavets wrote:
> > In order to be able to use VIRQ_TIMER and to improve performances you
> > need a patch to Xen to implement the vector callback mechanism
> > for event channel delivery.
> 
> Where to get it ?
> 

It is this one:

http://lists.xensource.com/archives/html/xen-devel/2010-05/msg00875.html


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 1886 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [Xen-devel] [PATCH 0 of 12] PV on HVM Xen
  2010-05-25  6:14     ` Boris Derzhavets
@ 2010-05-25  9:55       ` Stefano Stabellini
  2010-05-25 11:15         ` Boris Derzhavets
  0 siblings, 1 reply; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-25  9:55 UTC (permalink / raw)
  To: Boris Derzhavets
  Cc: Stefano Stabellini, Jeremy Fitzhardinge,
	xen-devel@lists.xensource.com, linux-kernel@vger.kernel.org,
	Stefano@yahoo.com, Don Dutile, Sheng Yang

On Tue, 25 May 2010, Boris Derzhavets wrote:
> What Xen Version this patch is supposed to be applied ?
> Seems like not 4.0.
> 

xen-unstable, but it shouldn't be difficult to port to 4.0.
BTW I just sent an updated version of the patch to the list.

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 0 of 12] PV on HVM Xen
  2010-05-25  9:55       ` [Xen-devel] " Stefano Stabellini
@ 2010-05-25 11:15         ` Boris Derzhavets
  0 siblings, 0 replies; 39+ messages in thread
From: Boris Derzhavets @ 2010-05-25 11:15 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com,
	Stefano Stabellini, Stefano@yahoo.com, Don Dutile, Sheng Yang


[-- Attachment #1.1: Type: text/plain, Size: 943 bytes --]

Could you,please, resubmit it as raw attachment.

Boris.

--- On Tue, 5/25/10, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:

From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Subject: Re: [Xen-devel] [PATCH 0 of 12] PV on HVM Xen
To: "Boris Derzhavets" <bderzhavets@yahoo.com>
Cc: "Stefano Stabellini" <Stefano.Stabellini@eu.citrix.com>, "Jeremy Fitzhardinge" <jeremy@goop.org>, "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "Stefano@yahoo.com" <Stefano@yahoo.com>, "Don Dutile" <ddutile@redhat.com>, "Sheng Yang" <sheng@linux.intel.com>
Date: Tuesday, May 25, 2010, 5:55 AM

On Tue, 25 May 2010, Boris Derzhavets wrote:
> What Xen Version this patch is supposed to be applied ?
> Seems like not 4.0.
> 

xen-unstable, but it shouldn't be difficult to port to 4.0.
BTW I just sent an updated version of the patch to the list.



      

[-- Attachment #1.2: Type: text/html, Size: 1342 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 0 of 12] PV on HVM Xen
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (12 preceding siblings ...)
  2010-05-24 18:30 ` Boris Derzhavets
@ 2010-05-24 18:36 ` Boris Derzhavets
  2010-05-28 10:25 ` Boris Derzhavets
  14 siblings, 0 replies; 39+ messages in thread
From: Boris Derzhavets @ 2010-05-24 18:36 UTC (permalink / raw)
  Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Don Dutile,
	Sheng Yang, Stefano Stabellini

[-- Attachment #1.1: Type: text/plain, Size: 2443 bytes --]

> In order to be able to use VIRQ_TIMER and to improve performances 
you
> need
 a patch to Xen to implement the vector callback mechanism
> for 
event 
channel delivery.

Where to get it ?

Boris.

--- On Mon, 5/24/10, Stefano Stabellini <Stefano.Stabellini@eu.citrix.com> wrote:

From: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com>
Subject: [Xen-devel] [PATCH 0 of 12] PV on HVM Xen
To: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Cc: "Stefano Stabellini" <Stefano.Stabellini@eu.citrix.com>, "Jeremy Fitzhardinge" <jeremy@goop.org>, "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>, "Don Dutile" <ddutile@redhat.com>, "Sheng Yang" <sheng@linux.intel.com>
Date: Monday, May 24, 2010, 2:25 PM

Hi all,
this is another update of the PV on HVM Xen series that addresses
Jeremy's comments.
The platform_pci hooks have been removed, suspend/resume for HVM
domains is now much more similar to the PV case and shares the same
do_suspend function.
Alloc_xen_mmio_hook has been removed has well, now the memory allocation for
the grant table is done by the xen platform pci driver directly.
The per_cpu xen_vcpu variable is set by a cpu_notifier function so that
secondary vcpus have the variable set correctly no matter what the xen
features are on the host.
The kernel command line option xen_unplug has been renamed to
xen_emul_unplug and the code that makes use of it has been moved to a
separate file (arch/x86/xen/platform-pci-unplug.c).
Xen_unplug_emulated_devices is now able to detect if blkfront, netfront
and the Xen platform PCI driver have been compiled, and set the default
value of xen_emul_unplug accordingly.
The patch "Initialize xenbus device structs with ENODEV as
default" has been removed from the series and it will be sent
separately.
Finally the comments on most of the patches have been improved.

The series is based on 2.6.34 and supports Xen PV frontends running
in a HVM domain, including netfront, blkfront and the VIRQ_TIMER.

In order to be able to use VIRQ_TIMER and to improve performances you
need a patch to Xen to implement the vector callback mechanism
for event channel delivery.

A git tree is also available here:

git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git

branch name 2.6.34-pvhvm-v2.

Cheers,

Stefano

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[-- Attachment #1.2: Type: text/html, Size: 3128 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 0 of 12] PV on HVM Xen
  2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
                   ` (13 preceding siblings ...)
  2010-05-24 18:36 ` Boris Derzhavets
@ 2010-05-28 10:25 ` Boris Derzhavets
  2010-05-28 10:45   ` Pasi Kärkkäinen
  2010-05-28 11:06   ` Stefano Stabellini
  14 siblings, 2 replies; 39+ messages in thread
From: Boris Derzhavets @ 2010-05-28 10:25 UTC (permalink / raw)
  Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Don Dutile,
	Sheng Yang, Stefano Stabellini

[-- Attachment #1.1: Type: text/plain, Size: 2507 bytes --]

What is an advantage of PV on HVM ?
Kernel 2.6.34 with Stefano's patches may be built i believe only on Linux HVM DomU.
At the same time any recent Linux ( >=24 or >=26) supports PV guest install ( it's in
mainline for a while).
What i am missing here ?

Boris.

--- On Mon, 5/24/10, Stefano Stabellini <Stefano.Stabellini@eu.citrix.com> wrote:

From: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com>
Subject: [Xen-devel] [PATCH 0 of 12] PV on HVM Xen
To: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Cc: "Stefano Stabellini" <Stefano.Stabellini@eu.citrix.com>, "Jeremy Fitzhardinge" <jeremy@goop.org>, "xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>, "Don Dutile" <ddutile@redhat.com>, "Sheng Yang" <sheng@linux.intel.com>
Date: Monday, May 24, 2010, 2:25 PM

Hi all,
this is another update of the PV on HVM Xen series that addresses
Jeremy's comments.
The platform_pci hooks have been removed, suspend/resume for HVM
domains is now much more similar to the PV case and shares the same
do_suspend function.
Alloc_xen_mmio_hook has been removed has well, now the memory allocation for
the grant table is done by the xen platform pci driver directly.
The per_cpu xen_vcpu variable is set by a cpu_notifier function so that
secondary vcpus have the variable set correctly no matter what the xen
features are on the host.
The kernel command line option xen_unplug has been renamed to
xen_emul_unplug and the code that makes use of it has been moved to a
separate file (arch/x86/xen/platform-pci-unplug.c).
Xen_unplug_emulated_devices is now able to detect if blkfront, netfront
and the Xen platform PCI driver have been compiled, and set the default
value of xen_emul_unplug accordingly.
The patch "Initialize xenbus device structs with ENODEV as
default" has been removed from the series and it will be sent
separately.
Finally the comments on most of the patches have been improved.

The series is based on 2.6.34 and supports Xen PV frontends running
in a HVM domain, including netfront, blkfront and the VIRQ_TIMER.

In order to be able to use VIRQ_TIMER and to improve performances you
need a patch to Xen to implement the vector callback mechanism
for event channel delivery.

A git tree is also available here:

git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git

branch name 2.6.34-pvhvm-v2.

Cheers,

Stefano

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[-- Attachment #1.2: Type: text/html, Size: 3190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 0 of 12] PV on HVM Xen
  2010-05-28 10:25 ` Boris Derzhavets
@ 2010-05-28 10:45   ` Pasi Kärkkäinen
  2010-05-28 11:06   ` Stefano Stabellini
  1 sibling, 0 replies; 39+ messages in thread
From: Pasi Kärkkäinen @ 2010-05-28 10:45 UTC (permalink / raw)
  To: Boris Derzhavets
  Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Don Dutile,
	Sheng Yang, Stefano Stabellini

On Fri, May 28, 2010 at 03:25:34AM -0700, Boris Derzhavets wrote:
>    What is an advantage of PV on HVM ?
>

Pure HVM guests using the Qemu emulated disk/network devices are slow. 
PV-on-HVM drivers make disk- and network IO fast for HVM guests.

>    Kernel 2.6.34 with Stefano's patches may be built i believe only on Linux
>    HVM DomU.
>

Exactly. They're meant for an upstream kernel, running as Xen HVM guest.

>    At the same time any recent Linux ( >=24 or >=26) supports PV guest
>    install ( it's in
>    mainline for a while).
>    What i am missing here ?
> 

HVM guests might be faster for some workloads compared to PV guests.
Kernel compilation could be one example..ie. workloads spawning a lot 
of new processes all the time.

-- Pasi

>    Boris.
> 
>    --- On Mon, 5/24/10, Stefano Stabellini <Stefano.Stabellini@eu.citrix.com>
>    wrote:
> 
>      From: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com>
>      Subject: [Xen-devel] [PATCH 0 of 12] PV on HVM Xen
>      To: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
>      Cc: "Stefano Stabellini" <Stefano.Stabellini@eu.citrix.com>, "Jeremy
>      Fitzhardinge" <jeremy@goop.org>, "xen-devel@lists.xensource.com"
>      <xen-devel@lists.xensource.com>, "Don Dutile" <ddutile@redhat.com>,
>      "Sheng Yang" <sheng@linux.intel.com>
>      Date: Monday, May 24, 2010, 2:25 PM
> 
>      Hi all,
>      this is another update of the PV on HVM Xen series that addresses
>      Jeremy's comments.
>      The platform_pci hooks have been removed, suspend/resume for HVM
>      domains is now much more similar to the PV case and shares the same
>      do_suspend function.
>      Alloc_xen_mmio_hook has been removed has well, now the memory allocation
>      for
>      the grant table is done by the xen platform pci driver directly.
>      The per_cpu xen_vcpu variable is set by a cpu_notifier function so that
>      secondary vcpus have the variable set correctly no matter what the xen
>      features are on the host.
>      The kernel command line option xen_unplug has been renamed to
>      xen_emul_unplug and the code that makes use of it has been moved to a
>      separate file (arch/x86/xen/platform-pci-unplug.c).
>      Xen_unplug_emulated_devices is now able to detect if blkfront, netfront
>      and the Xen platform PCI driver have been compiled, and set the default
>      value of xen_emul_unplug accordingly.
>      The patch "Initialize xenbus device structs with ENODEV as
>      default" has been removed from the series and it will be sent
>      separately.
>      Finally the comments on most of the patches have been improved.
> 
>      The series is based on 2.6.34 and supports Xen PV frontends running
>      in a HVM domain, including netfront, blkfront and the VIRQ_TIMER.
> 
>      In order to be able to use VIRQ_TIMER and to improve performances you
>      need a patch to Xen to implement the vector callback mechanism
>      for event channel delivery.
> 
>      A git tree is also available here:
> 
>      git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git
> 
>      branch name 2.6.34-pvhvm-v2.
> 
>      Cheers,
> 
>      Stefano
> 
>      _______________________________________________
>      Xen-devel mailing list
>      [1]Xen-devel@lists.xensource.com
>      [2]http://lists.xensource.com/xen-devel
> 
> References
> 
>    Visible links
>    1. file:///mc/compose?to=Xen-devel@lists.xensource.com
>    2. http://lists.xensource.com/xen-devel

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 39+ messages in thread

* Re: [PATCH 0 of 12] PV on HVM Xen
  2010-05-28 10:25 ` Boris Derzhavets
  2010-05-28 10:45   ` Pasi Kärkkäinen
@ 2010-05-28 11:06   ` Stefano Stabellini
  1 sibling, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-28 11:06 UTC (permalink / raw)
  To: Boris Derzhavets
  Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Don Dutile,
	Sheng Yang, Stefano Stabellini

On Fri, 28 May 2010, Boris Derzhavets wrote:
> What is an advantage of PV on HVM ?
> Kernel 2.6.34 with Stefano's patches may be built i believe only on Linux HVM DomU.
> At the same time any recent Linux ( >=24 or >=26) supports PV guest install ( it's in
> mainline for a while).
> What i am missing here ?
> 

A PV on HVM kernel doesn't need to be a special kernel: your pvops
kernel that can be used as native kernel or PV kernel can also be used
as PV on HVM kernel just adding CONFIG_XEN_PLATFORM_PCI.

PV on HVM support is good mainly for performances of 64 bit guests and
simplicity of installation.

^ permalink raw reply	[flat|nested] 39+ messages in thread

* [PATCH 04/11] Fix find_unbound_irq in presence of ioapic irqs.
@ 2010-05-10 14:20 Stefano Stabellini
  0 siblings, 0 replies; 39+ messages in thread
From: Stefano Stabellini @ 2010-05-10 14:20 UTC (permalink / raw)
  To: xen-devel; +Cc: Stefano Stabellini

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 drivers/xen/events.c |   11 ++++++++++-
 1 files changed, 10 insertions(+), 1 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index bb04824..cd609f4 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -335,9 +335,18 @@ static int find_unbound_irq(void)
 	int irq;
 	struct irq_desc *desc;
 
-	for (irq = 0; irq < nr_irqs; irq++)
+	for (irq = 0; irq < nr_irqs; irq++) {
+		desc = irq_to_desc(irq);
+		/* only 0->15 have init'd desc; handle irq > 16 */
+		if (desc == NULL)
+			break;
+		if (desc->chip == &no_irq_chip)
+			break;
+		if (desc->chip != &xen_dynamic_chip)
+			continue;
 		if (irq_info[irq].type == IRQT_UNBOUND)
 			break;
+	}
 
 	if (irq == nr_irqs)
 		panic("No available IRQ to bind to: increase nr_irqs!\n");
-- 
1.5.4.3

^ permalink raw reply related	[flat|nested] 39+ messages in thread

end of thread, other threads:[~2010-05-28 14:18 UTC | newest]

Thread overview: 39+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-05-24 18:25 [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
2010-05-24 18:27 ` [PATCH 01/11] Add support for hvm_op Stefano Stabellini
2010-05-24 18:27 ` [PATCH 02/11] early PV on HVM Stefano Stabellini
2010-05-24 18:27 ` [PATCH 03/11] evtchn delivery " Stefano Stabellini
2010-05-24 18:27 ` [PATCH 04/11] Fix find_unbound_irq in presence of ioapic irqs Stefano Stabellini
2010-05-26 18:26   ` [Xen-devel] " Konrad Rzeszutek Wilk
2010-05-27 14:49     ` Stefano Stabellini
2010-05-27 15:36       ` Konrad Rzeszutek Wilk
2010-05-24 18:27 ` [PATCH 05/11] Xen PCI platform device driver Stefano Stabellini
2010-05-28  8:55   ` [Xen-devel] " Zhigang Wang
2010-05-28 14:18     ` Stefano Stabellini
2010-05-24 18:27 ` [PATCH 06/11] Add suspend\resume support for PV on HVM guests Stefano Stabellini
2010-05-25 20:20   ` Jeremy Fitzhardinge
2010-05-24 18:27 ` [PATCH 07/11] Allow xen platform pci device to be compiled as a module Stefano Stabellini
2010-05-24 18:27 ` [PATCH 08/11] Fix possible NULL pointer dereference in print_IO_APIC Stefano Stabellini
2010-05-24 18:27 ` [PATCH 09/11] __setup_vector_irq: handle NULL chip_data Stefano Stabellini
2010-05-24 18:27 ` [PATCH 10/11] Support VIRQ_TIMER and pvclock on HVM Stefano Stabellini
2010-05-25 20:24   ` Jeremy Fitzhardinge
2010-05-26 13:08     ` Stefano Stabellini
2010-05-24 18:27 ` [PATCH 11/11] Unplug emulated disks and nics Stefano Stabellini
2010-05-25 20:31   ` Jeremy Fitzhardinge
2010-05-26 12:27     ` Stefano Stabellini
2010-05-26 20:59       ` Jeremy Fitzhardinge
2010-05-27 12:29         ` Stefano Stabellini
     [not found]         ` <m2n.s.1OHcBK-0018DA@chiark.greenend.org.uk>
2010-05-27 14:49           ` Ian Jackson
2010-05-27 17:25             ` [Xen-devel] " Stefano Stabellini
2010-05-27 17:51             ` Jeremy Fitzhardinge
2010-05-27 18:10               ` Stefano Stabellini
2010-05-24 18:29 ` [PATCH 0 of 12] PV on HVM Xen Stefano Stabellini
2010-05-24 18:30 ` Boris Derzhavets
2010-05-24 19:06   ` [Xen-devel] " Stefano Stabellini
2010-05-25  6:14     ` Boris Derzhavets
2010-05-25  9:55       ` [Xen-devel] " Stefano Stabellini
2010-05-25 11:15         ` Boris Derzhavets
2010-05-24 18:36 ` Boris Derzhavets
2010-05-28 10:25 ` Boris Derzhavets
2010-05-28 10:45   ` Pasi Kärkkäinen
2010-05-28 11:06   ` Stefano Stabellini
  -- strict thread matches above, loose matches on Subject: below --
2010-05-10 14:20 [PATCH 04/11] Fix find_unbound_irq in presence of ioapic irqs Stefano Stabellini

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).