[PATCH PV_OPS PCIFRONT]

[PATCH PV_OPS PCIFRONT]

[Konrad Rzeszutek Wilk]

The following patches provide pcifront support for the Linux pv-ops kernel.

The first two lay some groundwork, neccessary for the driver to function
in a PV non-privileged domain capacity:

 [PATCH 01/31] [xen-core] Provide a variant of xen_poll_irq with timeout.
 [PATCH 02/31] Enable Xen-SWIOTLB if running in [non-]privileged and disable the Xen-IOMMU if an IOMMU is detected.

The next fourteen ones that follow it, are simple fixes to shape the
driver into compiling on the kernel.
 [PATCH 03/31] Initial copy from linux-2.6.18.hg of the pcifront driver.
 [PATCH 04/31] Fix include header name change.
 [PATCH 05/31] Fix compile warning: ignoring return value of 'pci_bus_add_device', declared with attribute warn_unused_result
 [PATCH 06/31] Fix compile warning: passing argument 2 of 'pci_walk_bus' from incompatible pointer type
 [PATCH 07/31] Fix compile error. The  bind_to_irq_handler has different arguments.
 [PATCH 08/31] Fix compile error: implicit declaration of function 'virt_to_mfn'
 [PATCH 09/31] Fix compile error: implicit declaration of function 'clear_evtchn'
 [PATCH 10/31] Fix compile error: implicit declaration of function 'gnttab_end_foreign_access'
 [PATCH 11/31] Fix compile error: too few arguments to function 'gnttab_end_foreign_access'
 [PATCH 12/31] Remove function declerations (CONFIG_PCI_DOMAIN) that exist in recent kernels.
 [PATCH 13/31] Fix uage of INIT_WORK.
 [PATCH 14/31] Add proper check to see if running under Xen.
 [PATCH 15/31] Improper assumption that event channel == IRQ number.
 [PATCH 16/31] Replace HYPERVISOR_poll with 'xen_poll_irq_timout' function.

The next eight are to squish the driver from its set of various files,
in a xen-pcifront.c driver. No new functionality is added - just squishing
and removing pieces that don't make sense:
 [PATCH 17/31] Coalesce pci.c functions in xenbus.c.
 [PATCH 18/31] Coalesce xen/pcifront.h in drivers/xen/pcifront/pcifront.h
 [PATCH 19/31] Remove ia64 from pcifront.c support.
 [PATCH 20/31] Remove unused pci_bus_sem extern, as we don't use it.
 [PATCH 21/31] Coalesce pcifront.h in xenbus.c.
 [PATCH 22/31] Coalesce pci_op.c in xenbus.c.
 [PATCH 23/31] Remove unnecessary function declerations.
 [PATCH 24/31] Rename the drivers/xen/pcifront/* driver to drivers/pci/xen-pcifront.c.

At this point, we have a driver that compiles, but does not
actuall work. The next one makes it bootable:
 [PATCH 25/31] Change the boot-order of initialising the PCI frontend.

And at this point, I've decided to clean up the driver. Running
it through the checkpatch showed a wealth of warning which I've
rolled up in one patch:
 [PATCH 26/31] Fix warnings/errors reported by checkpatch.pl on xen-pcifront.c

The next five are neccessary to make the Xen core functionality provide
an IRQ for the INTx and MSI devices:
 [PATCH 27/31] Find an unbound irq number in reverse order (high to low).
 [PATCH 28/31] For non-privileged domains, implement a pcibios_enable_irq (xen_pcifront_enable_irq) function.
 [PATCH 29/31] xen_destroy_irq + xen_allocate_pirq in PV non-priv mode should not make certain Xen-HYPERCALLs.
 [PATCH 30/31] Add pci_frontend_[enable|disable]_[msi|msix] function decleration and EXPORT_SYMBOL_GPL.
 [PATCH 31/31] To enable MSI devices in a non-privileged PV domain use pci_frontend_enable_msi.

That is it for right now. The driver works with INTx and MSI cards. I've tested
with USB and network (Broadcom) succesfully. There is still some more work to do:
 - MSI disable is not yet in,
 - no MSI-X enable/disable functionality.

[PATCH 01/31] [xen-core] Provide a variant of xen_poll_irq with timeout.

[Konrad Rzeszutek Wilk]

The 'xen_poll_irq_timeout' provides a method to pass in
the poll timeout for IRQs if requested.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/events.c |   12 +++++++++---
 include/xen/events.h |    4 ++++
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 41772d5..e9f7551 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -1302,9 +1302,9 @@ bool xen_test_irq_pending(int irq)
 	return ret;
 }
 
-/* Poll waiting for an irq to become pending.  In the usual case, the
+/* Poll waiting for an irq to become pending with timeout.  In the usual case, the
    irq will be disabled so it won't deliver an interrupt. */
-void xen_poll_irq(int irq)
+void xen_poll_irq_timeout(int irq, u64 timeout)
 {
 	evtchn_port_t evtchn = evtchn_from_irq(irq);
 
@@ -1312,13 +1312,19 @@ void xen_poll_irq(int irq)
 		struct sched_poll poll;
 
 		poll.nr_ports = 1;
-		poll.timeout = 0;
+		poll.timeout = timeout;
 		set_xen_guest_handle(poll.ports, &evtchn);
 
 		if (HYPERVISOR_sched_op(SCHEDOP_poll, &poll) != 0)
 			BUG();
 	}
 }
+/* Poll waiting for an irq to become pending.  In the usual case, the
+   irq will be disabled so it won't deliver an interrupt. */
+void xen_poll_irq(int irq)
+{
+	xen_poll_irq_timeout(irq, 0 /* no timeout */);
+}
 
 void xen_irq_resume(void)
 {
diff --git a/include/xen/events.h b/include/xen/events.h
index bb654f2..ba02c54 100644
--- a/include/xen/events.h
+++ b/include/xen/events.h
@@ -62,6 +62,10 @@ bool xen_test_irq_pending(int irq);
    irq will be disabled so it won't deliver an interrupt. */
 void xen_poll_irq(int irq);
 
+/* Poll waiting for an irq to become pending with a timeout.  In the usual case, the
+   irq will be disabled so it won't deliver an interrupt. */
+void xen_poll_irq_timeout(int irq, u64 timeout);
+
 /* Determine the IRQ which is bound to an event channel */
 unsigned irq_from_evtchn(unsigned int evtchn);
 
-- 
1.6.2.5

[PATCH 02/31] Enable Xen-SWIOTLB if running in [non-]privileged and disable the Xen-IOMMU if an IOMMU is detected.

[Konrad Rzeszutek Wilk]

From: root <root@localhost.localdomain>

For PCI passthrough to work correctly, we need the Xen-SWIOTLB.
Otherwise PCI devices in the non-privileged domains might not be
able to do DMA.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/xen/pci-swiotlb.c |    2 +-
 drivers/pci/xen-iommu.c    |    3 +++
 2 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/arch/x86/xen/pci-swiotlb.c b/arch/x86/xen/pci-swiotlb.c
index 00f2260..ecdbfe2 100644
--- a/arch/x86/xen/pci-swiotlb.c
+++ b/arch/x86/xen/pci-swiotlb.c
@@ -984,7 +984,7 @@ static struct dma_map_ops xen_swiotlb_dma_ops = {
 
 void __init xen_swiotlb_init(void)
 {
-	if (xen_initial_domain()) {
+	if (xen_domain()) {
 		printk(KERN_INFO "PCI-DMA: Using Xen software bounce buffering for IO (Xen-SWIOTLB)\n");
 		xen_swiotlb_init_with_default_size(64 * (1<<20));	/* default to 64MB */
 		dma_ops = &xen_swiotlb_dma_ops;
diff --git a/drivers/pci/xen-iommu.c b/drivers/pci/xen-iommu.c
index 9ba63b1..c9a2af5 100644
--- a/drivers/pci/xen-iommu.c
+++ b/drivers/pci/xen-iommu.c
@@ -263,6 +263,9 @@ void __init xen_iommu_init(void)
 	if (xen_initial_domain()) /* For dom0, the IOMMU is handled by arch/x86/xen/pci-swiotlb.c. */
 		return;
 
+	if (iommu_detected)
+		return;
+
 	printk(KERN_INFO "Xen: Initializing Xen DMA ops\n");
 
 	force_iommu = 0;
-- 
1.6.2.5

[PATCH 03/31] Initial copy from linux-2.6.18.hg of the pcifront driver.

[Konrad Rzeszutek Wilk]

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/Kconfig             |   18 +
 drivers/xen/Makefile            |    1 +
 drivers/xen/pcifront/Makefile   |    7 +
 drivers/xen/pcifront/pci.c      |   46 +++
 drivers/xen/pcifront/pci_op.c   |  666 +++++++++++++++++++++++++++++++++++++++
 drivers/xen/pcifront/pcifront.h |   55 ++++
 drivers/xen/pcifront/xenbus.c   |  468 +++++++++++++++++++++++++++
 include/xen/pcifront.h          |   83 +++++
 8 files changed, 1344 insertions(+), 0 deletions(-)
 create mode 100644 drivers/xen/pcifront/Makefile
 create mode 100644 drivers/xen/pcifront/pci.c
 create mode 100644 drivers/xen/pcifront/pci_op.c
 create mode 100644 drivers/xen/pcifront/pcifront.h
 create mode 100644 drivers/xen/pcifront/xenbus.c
 create mode 100644 include/xen/pcifront.h

diff --git a/drivers/xen/Kconfig b/drivers/xen/Kconfig
index bc6c3f4..d4a33b6 100644
--- a/drivers/xen/Kconfig
+++ b/drivers/xen/Kconfig
@@ -173,3 +173,21 @@ config ACPI_PROCESSOR_XEN
 	bool
 	depends on XEN_DOM0 && ACPI_PROCESSOR && CPU_FREQ
 	default y
+
+config XEN_PCIDEV_FRONTEND
+        bool "Xen PCI Frontend"
+        depends on PCI && X86_64
+        select HOTPLUG
+        default y
+        help
+          The PCI device frontend driver allows the kernel to import arbitrary
+          PCI devices from a PCI backend to support PCI driver domains.
+
+config XEN_PCIDEV_FE_DEBUG
+        bool "Xen PCI Frontend Debugging"
+        depends on XEN_PCIDEV_FRONTEND
+        default n
+        help
+          Enables some debug statements within the PCI Frontend.
+
+
diff --git a/drivers/xen/Makefile b/drivers/xen/Makefile
index cddfffb..0bba5f2 100644
--- a/drivers/xen/Makefile
+++ b/drivers/xen/Makefile
@@ -13,6 +13,7 @@ obj-$(CONFIG_XEN_GNTDEV)		+= gntdev.o
 obj-$(CONFIG_XEN_BLKDEV_BACKEND)	+= blkback/
 obj-$(CONFIG_XEN_NETDEV_BACKEND)	+= netback/
 obj-$(CONFIG_XEN_PCIDEV_BACKEND)	+= pciback/
+obj-$(CONFIG_XEN_PCIDEV_FRONTEND)       += pcifront/
 obj-$(CONFIG_XENFS)			+= xenfs/
 obj-$(CONFIG_XEN_SYS_HYPERVISOR)	+= sys-hypervisor.o
 obj-$(CONFIG_XEN_S3)			+= acpi.o
diff --git a/drivers/xen/pcifront/Makefile b/drivers/xen/pcifront/Makefile
new file mode 100644
index 0000000..621e988
--- /dev/null
+++ b/drivers/xen/pcifront/Makefile
@@ -0,0 +1,7 @@
+obj-y += pcifront.o
+
+pcifront-y := pci_op.o xenbus.o pci.o
+
+ifeq ($(CONFIG_XEN_PCIDEV_FE_DEBUG),y)
+EXTRA_CFLAGS += -DDEBUG
+endif
diff --git a/drivers/xen/pcifront/pci.c b/drivers/xen/pcifront/pci.c
new file mode 100644
index 0000000..4239f00
--- /dev/null
+++ b/drivers/xen/pcifront/pci.c
@@ -0,0 +1,46 @@
+/*
+ * PCI Frontend Operations - ensure only one PCI frontend runs at a time
+ *
+ *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
+ */
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/pci.h>
+#include <linux/spinlock.h>
+#include "pcifront.h"
+
+DEFINE_SPINLOCK(pcifront_dev_lock);
+static struct pcifront_device *pcifront_dev = NULL;
+
+int pcifront_connect(struct pcifront_device *pdev)
+{
+	int err = 0;
+
+	spin_lock(&pcifront_dev_lock);
+
+	if (!pcifront_dev) {
+		dev_info(&pdev->xdev->dev, "Installing PCI frontend\n");
+		pcifront_dev = pdev;
+	}
+	else {
+		dev_err(&pdev->xdev->dev, "PCI frontend already installed!\n");
+		err = -EEXIST;
+	}
+
+	spin_unlock(&pcifront_dev_lock);
+
+	return err;
+}
+
+void pcifront_disconnect(struct pcifront_device *pdev)
+{
+	spin_lock(&pcifront_dev_lock);
+
+	if (pdev == pcifront_dev) {
+		dev_info(&pdev->xdev->dev,
+			 "Disconnecting PCI Frontend Buses\n");
+		pcifront_dev = NULL;
+	}
+
+	spin_unlock(&pcifront_dev_lock);
+}
diff --git a/drivers/xen/pcifront/pci_op.c b/drivers/xen/pcifront/pci_op.c
new file mode 100644
index 0000000..2034d20
--- /dev/null
+++ b/drivers/xen/pcifront/pci_op.c
@@ -0,0 +1,666 @@
+/*
+ * PCI Frontend Operations - Communicates with frontend
+ *
+ *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
+ */
+#include <linux/module.h>
+#include <linux/version.h>
+#include <linux/init.h>
+#include <linux/pci.h>
+#include <linux/spinlock.h>
+#include <asm/bitops.h>
+#include <linux/time.h>
+#include <xen/evtchn.h>
+#include "pcifront.h"
+
+static int verbose_request = 0;
+module_param(verbose_request, int, 0644);
+
+#ifdef __ia64__
+static void pcifront_init_sd(struct pcifront_sd *sd,
+			     unsigned int domain, unsigned int bus,
+			     struct pcifront_device *pdev)
+{
+	int err, i, j, k, len, root_num, res_count;
+	struct acpi_resource res;
+	unsigned int d, b, byte;
+	unsigned long magic;
+	char str[64], tmp[3];
+	unsigned char *buf, *bufp;
+	u8 *ptr;
+
+	memset(sd, 0, sizeof(*sd));
+
+	sd->segment = domain;
+	sd->node = -1;	/* Revisit for NUMA */
+	sd->platform_data = pdev;
+
+	/* Look for resources for this controller in xenbus. */
+	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, "root_num",
+			   "%d", &root_num);
+	if (err != 1)
+		return;
+
+	for (i = 0; i < root_num; i++) {
+		len = snprintf(str, sizeof(str), "root-%d", i);
+		if (unlikely(len >= (sizeof(str) - 1)))
+			return;
+
+		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
+				   str, "%x:%x", &d, &b);
+		if (err != 2)
+			return;
+
+		if (d == domain && b == bus)
+			break;
+	}
+
+	if (i == root_num)
+		return;
+
+	len = snprintf(str, sizeof(str), "root-resource-magic");
+
+	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
+			   str, "%lx", &magic);
+
+	if (err != 1)
+		return; /* No resources, nothing to do */
+
+	if (magic != (sizeof(res) * 2) + 1) {
+		printk(KERN_WARNING "pcifront: resource magic mismatch\n");
+		return;
+	}
+
+	len = snprintf(str, sizeof(str), "root-%d-resources", i);
+	if (unlikely(len >= (sizeof(str) - 1)))
+		return;
+
+	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
+			   str, "%d", &res_count);
+
+	if (err != 1)
+		return; /* No resources, nothing to do */
+
+	sd->window = kzalloc(sizeof(*sd->window) * res_count, GFP_KERNEL);
+	if (!sd->window)
+		return;
+
+	/* magic is also the size of the byte stream in xenbus */
+	buf = kmalloc(magic, GFP_KERNEL);
+	if (!buf) {
+		kfree(sd->window);
+		sd->window = NULL;
+		return;
+	}
+
+	/* Read the resources out of xenbus */
+	for (j = 0; j < res_count; j++) {
+		memset(&res, 0, sizeof(res));
+		memset(buf, 0, magic);
+
+		len = snprintf(str, sizeof(str), "root-%d-resource-%d", i, j);
+		if (unlikely(len >= (sizeof(str) - 1)))
+			return;
+
+		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
+				   "%s", buf);
+		if (err != 1) {
+			printk(KERN_WARNING "pcifront: error reading "
+			       "resource %d on bus %04x:%02x\n",
+			       j, domain, bus);
+			continue;
+		}
+
+		bufp = buf;
+		ptr = (u8 *)&res;
+		memset(tmp, 0, sizeof(tmp));
+
+		/* Copy ASCII byte stream into structure */
+		for (k = 0; k < magic - 1; k += 2) {
+			memcpy(tmp, bufp, 2);
+			bufp += 2;
+
+			sscanf(tmp, "%02x", &byte);
+			*ptr = byte;
+			ptr++;
+		}
+
+		xen_add_resource(sd, domain, bus, &res);
+		sd->windows++;
+	}
+	kfree(buf);
+}
+#endif
+
+static int errno_to_pcibios_err(int errno)
+{
+	switch (errno) {
+	case XEN_PCI_ERR_success:
+		return PCIBIOS_SUCCESSFUL;
+
+	case XEN_PCI_ERR_dev_not_found:
+		return PCIBIOS_DEVICE_NOT_FOUND;
+
+	case XEN_PCI_ERR_invalid_offset:
+	case XEN_PCI_ERR_op_failed:
+		return PCIBIOS_BAD_REGISTER_NUMBER;
+
+	case XEN_PCI_ERR_not_implemented:
+		return PCIBIOS_FUNC_NOT_SUPPORTED;
+
+	case XEN_PCI_ERR_access_denied:
+		return PCIBIOS_SET_FAILED;
+	}
+	return errno;
+}
+
+static inline void schedule_pcifront_aer_op(struct pcifront_device *pdev)
+{
+	if (test_bit(_XEN_PCIB_active, (unsigned long *)&pdev->sh_info->flags)
+		&& !test_and_set_bit(_PDEVB_op_active, &pdev->flags)) {
+		dev_dbg(&pdev->xdev->dev, "schedule aer frontend job\n");
+		schedule_work(&pdev->op_work);
+	}
+}
+
+static int do_pci_op(struct pcifront_device *pdev, struct xen_pci_op *op)
+{
+	int err = 0;
+	struct xen_pci_op *active_op = &pdev->sh_info->op;
+	unsigned long irq_flags;
+	evtchn_port_t port = pdev->evtchn;
+	s64 ns, ns_timeout;
+	struct timeval tv;
+
+	spin_lock_irqsave(&pdev->sh_info_lock, irq_flags);
+
+	memcpy(active_op, op, sizeof(struct xen_pci_op));
+
+	/* Go */
+	wmb();
+	set_bit(_XEN_PCIF_active, (unsigned long *)&pdev->sh_info->flags);
+	notify_remote_via_evtchn(port);
+
+	/*
+	 * We set a poll timeout of 3 seconds but give up on return after
+	 * 2 seconds. It is better to time out too late rather than too early
+	 * (in the latter case we end up continually re-executing poll() with a
+	 * timeout in the past). 1s difference gives plenty of slack for error.
+	 */
+	do_gettimeofday(&tv);
+	ns_timeout = timeval_to_ns(&tv) + 2 * (s64)NSEC_PER_SEC;
+
+	clear_evtchn(port);
+
+	while (test_bit(_XEN_PCIF_active,
+			(unsigned long *)&pdev->sh_info->flags)) {
+		if (HYPERVISOR_poll(&port, 1, jiffies + 3*HZ))
+			BUG();
+		clear_evtchn(port);
+		do_gettimeofday(&tv);
+		ns = timeval_to_ns(&tv);
+		if (ns > ns_timeout) {
+			dev_err(&pdev->xdev->dev,
+				"pciback not responding!!!\n");
+			clear_bit(_XEN_PCIF_active,
+				  (unsigned long *)&pdev->sh_info->flags);
+			err = XEN_PCI_ERR_dev_not_found;
+			goto out;
+		}
+	}
+
+	/*
+	* We might lose backend service request since we 
+	* reuse same evtchn with pci_conf backend response. So re-schedule
+	* aer pcifront service.
+	*/
+	if (test_bit(_XEN_PCIB_active, 
+			(unsigned long*)&pdev->sh_info->flags)) {
+		dev_err(&pdev->xdev->dev, 
+			"schedule aer pcifront service\n");
+		schedule_pcifront_aer_op(pdev);
+	}
+
+	memcpy(op, active_op, sizeof(struct xen_pci_op));
+
+	err = op->err;
+      out:
+	spin_unlock_irqrestore(&pdev->sh_info_lock, irq_flags);
+	return err;
+}
+
+/* Access to this function is spinlocked in drivers/pci/access.c */
+static int pcifront_bus_read(struct pci_bus *bus, unsigned int devfn,
+			     int where, int size, u32 * val)
+{
+	int err = 0;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_conf_read,
+		.domain = pci_domain_nr(bus),
+		.bus    = bus->number,
+		.devfn  = devfn,
+		.offset = where,
+		.size   = size,
+	};
+	struct pcifront_sd *sd = bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	if (verbose_request)
+		dev_info(&pdev->xdev->dev,
+			 "read dev=%04x:%02x:%02x.%01x - offset %x size %d\n",
+			 pci_domain_nr(bus), bus->number, PCI_SLOT(devfn),
+			 PCI_FUNC(devfn), where, size);
+
+	err = do_pci_op(pdev, &op);
+
+	if (likely(!err)) {
+		if (verbose_request)
+			dev_info(&pdev->xdev->dev, "read got back value %x\n",
+				 op.value);
+
+		*val = op.value;
+	} else if (err == -ENODEV) {
+		/* No device here, pretend that it just returned 0 */
+		err = 0;
+		*val = 0;
+	}
+
+	return errno_to_pcibios_err(err);
+}
+
+/* Access to this function is spinlocked in drivers/pci/access.c */
+static int pcifront_bus_write(struct pci_bus *bus, unsigned int devfn,
+			      int where, int size, u32 val)
+{
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_conf_write,
+		.domain = pci_domain_nr(bus),
+		.bus    = bus->number,
+		.devfn  = devfn,
+		.offset = where,
+		.size   = size,
+		.value  = val,
+	};
+	struct pcifront_sd *sd = bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	if (verbose_request)
+		dev_info(&pdev->xdev->dev,
+			 "write dev=%04x:%02x:%02x.%01x - "
+			 "offset %x size %d val %x\n",
+			 pci_domain_nr(bus), bus->number,
+			 PCI_SLOT(devfn), PCI_FUNC(devfn), where, size, val);
+
+	return errno_to_pcibios_err(do_pci_op(pdev, &op));
+}
+
+struct pci_ops pcifront_bus_ops = {
+	.read = pcifront_bus_read,
+	.write = pcifront_bus_write,
+};
+
+#ifdef CONFIG_PCI_MSI
+int pci_frontend_enable_msix(struct pci_dev *dev,
+		struct msix_entry *entries,
+		int nvec)
+{
+	int err;
+	int i;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_enable_msix,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+		.value = nvec,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	if (nvec > SH_INFO_MAX_VEC) {
+		printk("too much vector for pci frontend%x\n", nvec);
+		return -EINVAL;
+	}
+
+	for (i = 0; i < nvec; i++) {
+		op.msix_entries[i].entry = entries[i].entry;
+		op.msix_entries[i].vector = entries[i].vector;
+	}
+
+	err = do_pci_op(pdev, &op);
+
+	if (!err) {
+		if (!op.value) {
+			/* we get the result */
+			for ( i = 0; i < nvec; i++)
+				entries[i].vector = op.msix_entries[i].vector;
+			return 0;
+		}
+		else {
+            printk("enable msix get value %x\n", op.value);
+			return op.value;
+		}
+	}
+	else {
+        printk("enable msix get err %x\n", err);
+		return err;
+	}
+}
+
+void pci_frontend_disable_msix(struct pci_dev* dev)
+{
+	int err;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_disable_msix,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	err = do_pci_op(pdev, &op);
+
+	/* What should do for error ? */
+	if (err)
+		printk("pci_disable_msix get err %x\n", err);
+}
+
+int pci_frontend_enable_msi(struct pci_dev *dev)
+{
+	int err;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_enable_msi,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	err = do_pci_op(pdev, &op);
+	if (likely(!err)) {
+		dev->irq = op.value;
+	}
+	else {
+		printk("pci frontend enable msi failed for dev %x:%x \n",
+				op.bus, op.devfn);
+		err = -EINVAL;
+	}
+	return err;
+}
+
+void pci_frontend_disable_msi(struct pci_dev* dev)
+{
+	int err;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_disable_msi,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	err = do_pci_op(pdev, &op);
+	if (err == XEN_PCI_ERR_dev_not_found) {
+		/* XXX No response from backend, what shall we do? */
+		printk("get no response from backend for disable MSI\n");
+		return;
+	}
+	if (likely(!err))
+		dev->irq = op.value;
+	else
+		/* how can pciback notify us fail? */
+		printk("get fake response frombackend \n");
+}
+#endif /* CONFIG_PCI_MSI */
+
+/* Claim resources for the PCI frontend as-is, backend won't allow changes */
+static void pcifront_claim_resource(struct pci_dev *dev, void *data)
+{
+	struct pcifront_device *pdev = data;
+	int i;
+	struct resource *r;
+
+	for (i = 0; i < PCI_NUM_RESOURCES; i++) {
+		r = &dev->resource[i];
+
+		if (!r->parent && r->start && r->flags) {
+			dev_dbg(&pdev->xdev->dev, "claiming resource %s/%d\n",
+				pci_name(dev), i);
+			pci_claim_resource(dev, i);
+		}
+	}
+}
+
+int __devinit pcifront_scan_root(struct pcifront_device *pdev,
+				 unsigned int domain, unsigned int bus)
+{
+	struct pci_bus *b;
+	struct pcifront_sd *sd = NULL;
+	struct pci_bus_entry *bus_entry = NULL;
+	int err = 0;
+
+#ifndef CONFIG_PCI_DOMAINS
+	if (domain != 0) {
+		dev_err(&pdev->xdev->dev,
+			"PCI Root in non-zero PCI Domain! domain=%d\n", domain);
+		dev_err(&pdev->xdev->dev,
+			"Please compile with CONFIG_PCI_DOMAINS\n");
+		err = -EINVAL;
+		goto err_out;
+	}
+#endif
+
+	dev_info(&pdev->xdev->dev, "Creating PCI Frontend Bus %04x:%02x\n",
+		 domain, bus);
+
+	bus_entry = kmalloc(sizeof(*bus_entry), GFP_KERNEL);
+	sd = kmalloc(sizeof(*sd), GFP_KERNEL);
+	if (!bus_entry || !sd) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	pcifront_init_sd(sd, domain, bus, pdev);
+
+	b = pci_scan_bus_parented(&pdev->xdev->dev, bus,
+				  &pcifront_bus_ops, sd);
+	if (!b) {
+		dev_err(&pdev->xdev->dev,
+			"Error creating PCI Frontend Bus!\n");
+		err = -ENOMEM;
+		goto err_out;
+	}
+
+	pcifront_setup_root_resources(b, sd);
+	bus_entry->bus = b;
+
+	list_add(&bus_entry->list, &pdev->root_buses);
+
+	/* Claim resources before going "live" with our devices */
+	pci_walk_bus(b, pcifront_claim_resource, pdev);
+
+	pci_bus_add_devices(b);
+
+	return 0;
+
+      err_out:
+	kfree(bus_entry);
+	kfree(sd);
+
+	return err;
+}
+
+int __devinit pcifront_rescan_root(struct pcifront_device *pdev,
+				   unsigned int domain, unsigned int bus)
+{
+	struct pci_bus *b;
+	struct pci_dev *d;
+	unsigned int devfn;
+
+#ifndef CONFIG_PCI_DOMAINS
+	if (domain != 0) {
+		dev_err(&pdev->xdev->dev,
+			"PCI Root in non-zero PCI Domain! domain=%d\n", domain);
+		dev_err(&pdev->xdev->dev,
+			"Please compile with CONFIG_PCI_DOMAINS\n");
+		return -EINVAL;
+	}
+#endif
+
+	dev_info(&pdev->xdev->dev, "Rescanning PCI Frontend Bus %04x:%02x\n",
+		 domain, bus);
+
+	b = pci_find_bus(domain, bus);
+	if(!b)
+		/* If the bus is unknown, create it. */
+		return pcifront_scan_root(pdev, domain, bus);
+
+	/* Rescan the bus for newly attached functions and add.
+	 * We omit handling of PCI bridge attachment because pciback prevents
+	 * bridges from being exported.
+	 */ 
+	for (devfn = 0; devfn < 0x100; devfn++) {
+		d = pci_get_slot(b, devfn);
+		if(d) {
+			/* Device is already known. */
+			pci_dev_put(d);
+			continue;
+		}
+
+		d = pci_scan_single_device(b, devfn);
+		if (d) {
+			dev_info(&pdev->xdev->dev, "New device on "
+				 "%04x:%02x:%02x.%02x found.\n", domain, bus,
+				 PCI_SLOT(devfn), PCI_FUNC(devfn));
+			pci_bus_add_device(d);
+		}
+	}
+
+	return 0;
+}
+
+static void free_root_bus_devs(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+
+	while (!list_empty(&bus->devices)) {
+		dev = container_of(bus->devices.next, struct pci_dev,
+				   bus_list);
+		dev_dbg(&dev->dev, "removing device\n");
+		pci_remove_bus_device(dev);
+	}
+}
+
+void pcifront_free_roots(struct pcifront_device *pdev)
+{
+	struct pci_bus_entry *bus_entry, *t;
+
+	dev_dbg(&pdev->xdev->dev, "cleaning up root buses\n");
+
+	list_for_each_entry_safe(bus_entry, t, &pdev->root_buses, list) {
+		list_del(&bus_entry->list);
+
+		free_root_bus_devs(bus_entry->bus);
+
+		kfree(bus_entry->bus->sysdata);
+
+		device_unregister(bus_entry->bus->bridge);
+		pci_remove_bus(bus_entry->bus);
+
+		kfree(bus_entry);
+	}
+}
+
+static pci_ers_result_t pcifront_common_process( int cmd, struct pcifront_device *pdev,
+	pci_channel_state_t state)
+{
+	pci_ers_result_t result;
+	struct pci_driver *pdrv;
+	int bus = pdev->sh_info->aer_op.bus;
+	int devfn = pdev->sh_info->aer_op.devfn;
+	struct pci_dev *pcidev;
+	int flag = 0;
+
+	dev_dbg(&pdev->xdev->dev, 
+		"pcifront AER process: cmd %x (bus:%x, devfn%x)",
+		cmd, bus, devfn);
+	result = PCI_ERS_RESULT_NONE;
+
+	pcidev = pci_get_bus_and_slot(bus, devfn);
+	if (!pcidev || !pcidev->driver){
+		dev_err(&pcidev->dev, 
+			"device or driver is NULL\n");
+		return result;
+	}
+	pdrv = pcidev->driver;
+
+	if (get_driver(&pdrv->driver)) {
+		if (pdrv->err_handler && pdrv->err_handler->error_detected) {
+			dev_dbg(&pcidev->dev,
+				"trying to call AER service\n");
+			if (pcidev) {
+				flag = 1;
+				switch(cmd) {
+				case XEN_PCI_OP_aer_detected:
+					result = pdrv->err_handler->error_detected(pcidev, state);
+					break;
+				case XEN_PCI_OP_aer_mmio:
+					result = pdrv->err_handler->mmio_enabled(pcidev);
+					break;
+				case XEN_PCI_OP_aer_slotreset:
+					result = pdrv->err_handler->slot_reset(pcidev);
+					break;
+				case XEN_PCI_OP_aer_resume:
+					pdrv->err_handler->resume(pcidev);
+					break;
+				default:
+					dev_err(&pdev->xdev->dev,
+						"bad request in aer recovery operation!\n");
+
+				}
+			}
+		}
+		put_driver(&pdrv->driver);
+	}
+	if (!flag)
+		result = PCI_ERS_RESULT_NONE;
+
+	return result;
+}
+
+
+void pcifront_do_aer(void *data)
+{
+	struct pcifront_device *pdev = data;
+	int cmd = pdev->sh_info->aer_op.cmd;
+	pci_channel_state_t state = 
+		(pci_channel_state_t)pdev->sh_info->aer_op.err;
+
+	/*If a pci_conf op is in progress, 
+		we have to wait until it is done before service aer op*/
+	dev_dbg(&pdev->xdev->dev, 
+		"pcifront service aer bus %x devfn %x\n", pdev->sh_info->aer_op.bus,
+		pdev->sh_info->aer_op.devfn);
+
+	pdev->sh_info->aer_op.err = pcifront_common_process(cmd, pdev, state);
+
+	wmb();
+	clear_bit(_XEN_PCIB_active, (unsigned long*)&pdev->sh_info->flags);
+	notify_remote_via_evtchn(pdev->evtchn);
+
+	/*in case of we lost an aer request in four lines time_window*/
+	smp_mb__before_clear_bit();
+	clear_bit( _PDEVB_op_active, &pdev->flags);
+	smp_mb__after_clear_bit();
+
+	schedule_pcifront_aer_op(pdev);
+
+}
+
+irqreturn_t pcifront_handler_aer(int irq, void *dev, struct pt_regs *regs)
+{
+	struct pcifront_device *pdev = dev;
+	schedule_pcifront_aer_op(pdev);
+	return IRQ_HANDLED;
+}
diff --git a/drivers/xen/pcifront/pcifront.h b/drivers/xen/pcifront/pcifront.h
new file mode 100644
index 0000000..06cb3e1
--- /dev/null
+++ b/drivers/xen/pcifront/pcifront.h
@@ -0,0 +1,55 @@
+/*
+ * PCI Frontend - Common data structures & function declarations
+ *
+ *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
+ */
+#ifndef __XEN_PCIFRONT_H__
+#define __XEN_PCIFRONT_H__
+
+#include <linux/spinlock.h>
+#include <linux/pci.h>
+#include <xen/xenbus.h>
+#include <xen/interface/io/pciif.h>
+#include <linux/interrupt.h>
+#include <xen/pcifront.h>
+#include <asm/atomic.h>
+#include <linux/workqueue.h>
+
+struct pci_bus_entry {
+	struct list_head list;
+	struct pci_bus *bus;
+};
+
+#define _PDEVB_op_active		(0)
+#define PDEVB_op_active 		(1 << (_PDEVB_op_active))
+
+struct pcifront_device {
+	struct xenbus_device *xdev;
+	struct list_head root_buses;
+	spinlock_t dev_lock;
+
+	int evtchn;
+	int gnt_ref;
+
+	/* Lock this when doing any operations in sh_info */
+	spinlock_t sh_info_lock;
+	struct xen_pci_sharedinfo *sh_info;
+	struct work_struct op_work;
+	unsigned long flags;
+
+};
+
+int pcifront_connect(struct pcifront_device *pdev);
+void pcifront_disconnect(struct pcifront_device *pdev);
+
+int pcifront_scan_root(struct pcifront_device *pdev,
+		       unsigned int domain, unsigned int bus);
+int pcifront_rescan_root(struct pcifront_device *pdev,
+			 unsigned int domain, unsigned int bus);
+void pcifront_free_roots(struct pcifront_device *pdev);
+
+void pcifront_do_aer( void *data);
+
+irqreturn_t pcifront_handler_aer(int irq, void *dev, struct pt_regs *regs);
+
+#endif	/* __XEN_PCIFRONT_H__ */
diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
new file mode 100644
index 0000000..ca40547
--- /dev/null
+++ b/drivers/xen/pcifront/xenbus.c
@@ -0,0 +1,468 @@
+/*
+ * PCI Frontend Xenbus Setup - handles setup with backend (imports page/evtchn)
+ *
+ *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
+ */
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/mm.h>
+#include <xen/xenbus.h>
+#include <xen/evtchn.h>
+#include <xen/gnttab.h>
+#include "pcifront.h"
+
+#ifndef __init_refok
+#define __init_refok
+#endif
+
+#define INVALID_GRANT_REF (0)
+#define INVALID_EVTCHN    (-1)
+
+static struct pcifront_device *alloc_pdev(struct xenbus_device *xdev)
+{
+	struct pcifront_device *pdev;
+
+	pdev = kzalloc(sizeof(struct pcifront_device), GFP_KERNEL);
+	if (pdev == NULL)
+		goto out;
+
+	pdev->sh_info =
+	    (struct xen_pci_sharedinfo *)__get_free_page(GFP_KERNEL);
+	if (pdev->sh_info == NULL) {
+		kfree(pdev);
+		pdev = NULL;
+		goto out;
+	}
+	pdev->sh_info->flags = 0;
+
+	/*Flag for registering PV AER handler*/
+	set_bit(_XEN_PCIB_AERHANDLER, (void*)&pdev->sh_info->flags);
+
+	xdev->dev.driver_data = pdev;
+	pdev->xdev = xdev;
+
+	INIT_LIST_HEAD(&pdev->root_buses);
+
+	spin_lock_init(&pdev->dev_lock);
+	spin_lock_init(&pdev->sh_info_lock);
+
+	pdev->evtchn = INVALID_EVTCHN;
+	pdev->gnt_ref = INVALID_GRANT_REF;
+
+	INIT_WORK(&pdev->op_work, pcifront_do_aer, pdev);
+
+	dev_dbg(&xdev->dev, "Allocated pdev @ 0x%p pdev->sh_info @ 0x%p\n",
+		pdev, pdev->sh_info);
+      out:
+	return pdev;
+}
+
+static void free_pdev(struct pcifront_device *pdev)
+{
+	dev_dbg(&pdev->xdev->dev, "freeing pdev @ 0x%p\n", pdev);
+
+	pcifront_free_roots(pdev);
+
+	/*For PCIE_AER error handling job*/
+	flush_scheduled_work();
+	unbind_from_irqhandler(pdev->evtchn, pdev);
+
+	if (pdev->evtchn != INVALID_EVTCHN)
+		xenbus_free_evtchn(pdev->xdev, pdev->evtchn);
+
+	if (pdev->gnt_ref != INVALID_GRANT_REF)
+		gnttab_end_foreign_access(pdev->gnt_ref,
+					  (unsigned long)pdev->sh_info);
+
+	pdev->xdev->dev.driver_data = NULL;
+
+	kfree(pdev);
+}
+
+static int pcifront_publish_info(struct pcifront_device *pdev)
+{
+	int err = 0;
+	struct xenbus_transaction trans;
+
+	err = xenbus_grant_ring(pdev->xdev, virt_to_mfn(pdev->sh_info));
+	if (err < 0)
+		goto out;
+
+	pdev->gnt_ref = err;
+
+	err = xenbus_alloc_evtchn(pdev->xdev, &pdev->evtchn);
+	if (err)
+		goto out;
+
+	bind_caller_port_to_irqhandler(pdev->evtchn, pcifront_handler_aer, 
+		SA_SAMPLE_RANDOM, "pcifront", pdev); 
+
+      do_publish:
+	err = xenbus_transaction_start(&trans);
+	if (err) {
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error writing configuration for backend "
+				 "(start transaction)");
+		goto out;
+	}
+
+	err = xenbus_printf(trans, pdev->xdev->nodename,
+			    "pci-op-ref", "%u", pdev->gnt_ref);
+	if (!err)
+		err = xenbus_printf(trans, pdev->xdev->nodename,
+				    "event-channel", "%u", pdev->evtchn);
+	if (!err)
+		err = xenbus_printf(trans, pdev->xdev->nodename,
+				    "magic", XEN_PCI_MAGIC);
+
+	if (err) {
+		xenbus_transaction_end(trans, 1);
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error writing configuration for backend");
+		goto out;
+	} else {
+		err = xenbus_transaction_end(trans, 0);
+		if (err == -EAGAIN)
+			goto do_publish;
+		else if (err) {
+			xenbus_dev_fatal(pdev->xdev, err,
+					 "Error completing transaction "
+					 "for backend");
+			goto out;
+		}
+	}
+
+	xenbus_switch_state(pdev->xdev, XenbusStateInitialised);
+
+	dev_dbg(&pdev->xdev->dev, "publishing successful!\n");
+
+      out:
+	return err;
+}
+
+static int __devinit pcifront_try_connect(struct pcifront_device *pdev)
+{
+	int err = -EFAULT;
+	int i, num_roots, len;
+	char str[64];
+	unsigned int domain, bus;
+
+	spin_lock(&pdev->dev_lock);
+
+	/* Only connect once */
+	if (xenbus_read_driver_state(pdev->xdev->nodename) !=
+	    XenbusStateInitialised)
+		goto out;
+
+	err = pcifront_connect(pdev);
+	if (err) {
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error connecting PCI Frontend");
+		goto out;
+	}
+
+	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
+			   "root_num", "%d", &num_roots);
+	if (err == -ENOENT) {
+		xenbus_dev_error(pdev->xdev, err,
+				 "No PCI Roots found, trying 0000:00");
+		err = pcifront_scan_root(pdev, 0, 0);
+		num_roots = 0;
+	} else if (err != 1) {
+		if (err == 0)
+			err = -EINVAL;
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error reading number of PCI roots");
+		goto out;
+	}
+
+	for (i = 0; i < num_roots; i++) {
+		len = snprintf(str, sizeof(str), "root-%d", i);
+		if (unlikely(len >= (sizeof(str) - 1))) {
+			err = -ENOMEM;
+			goto out;
+		}
+
+		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
+				   "%x:%x", &domain, &bus);
+		if (err != 2) {
+			if (err >= 0)
+				err = -EINVAL;
+			xenbus_dev_fatal(pdev->xdev, err,
+					 "Error reading PCI root %d", i);
+			goto out;
+		}
+
+		err = pcifront_scan_root(pdev, domain, bus);
+		if (err) {
+			xenbus_dev_fatal(pdev->xdev, err,
+					 "Error scanning PCI root %04x:%02x",
+					 domain, bus);
+			goto out;
+		}
+	}
+
+	err = xenbus_switch_state(pdev->xdev, XenbusStateConnected);
+	if (err)
+		goto out;
+
+      out:
+	spin_unlock(&pdev->dev_lock);
+	return err;
+}
+
+static int pcifront_try_disconnect(struct pcifront_device *pdev)
+{
+	int err = 0;
+	enum xenbus_state prev_state;
+
+	spin_lock(&pdev->dev_lock);
+
+	prev_state = xenbus_read_driver_state(pdev->xdev->nodename);
+
+	if (prev_state >= XenbusStateClosing)
+		goto out;
+
+	if(prev_state == XenbusStateConnected) {
+		pcifront_free_roots(pdev);
+		pcifront_disconnect(pdev);
+	}
+
+	err = xenbus_switch_state(pdev->xdev, XenbusStateClosed);
+
+      out:
+	spin_unlock(&pdev->dev_lock);
+
+	return err;
+}
+
+static int __devinit pcifront_attach_devices(struct pcifront_device *pdev)
+{
+	int err = -EFAULT;
+	int i, num_roots, len;
+	unsigned int domain, bus;
+	char str[64];
+
+	spin_lock(&pdev->dev_lock);
+
+	if (xenbus_read_driver_state(pdev->xdev->nodename) !=
+	    XenbusStateReconfiguring)
+		goto out;
+
+	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
+			   "root_num", "%d", &num_roots);
+	if (err == -ENOENT) {
+		xenbus_dev_error(pdev->xdev, err,
+				 "No PCI Roots found, trying 0000:00");
+		err = pcifront_rescan_root(pdev, 0, 0);
+		num_roots = 0;
+	} else if (err != 1) {
+		if (err == 0)
+			err = -EINVAL;
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error reading number of PCI roots");
+		goto out;
+	}
+
+	for (i = 0; i < num_roots; i++) {
+		len = snprintf(str, sizeof(str), "root-%d", i);
+		if (unlikely(len >= (sizeof(str) - 1))) {
+			err = -ENOMEM;
+			goto out;
+		}
+
+		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
+				   "%x:%x", &domain, &bus);
+		if (err != 2) {
+			if (err >= 0)
+				err = -EINVAL;
+			xenbus_dev_fatal(pdev->xdev, err,
+					 "Error reading PCI root %d", i);
+			goto out;
+		}
+
+		err = pcifront_rescan_root(pdev, domain, bus);
+		if (err) {
+			xenbus_dev_fatal(pdev->xdev, err,
+					 "Error scanning PCI root %04x:%02x",
+					 domain, bus);
+			goto out;
+		}
+	}
+
+	xenbus_switch_state(pdev->xdev, XenbusStateConnected);
+
+      out:
+	spin_unlock(&pdev->dev_lock);
+	return err;
+}
+
+static int pcifront_detach_devices(struct pcifront_device *pdev)
+{
+	int err = 0;
+	int i, num_devs;
+	unsigned int domain, bus, slot, func;
+	struct pci_bus *pci_bus;
+	struct pci_dev *pci_dev;
+	char str[64];
+
+	spin_lock(&pdev->dev_lock);
+
+	if (xenbus_read_driver_state(pdev->xdev->nodename) !=
+	    XenbusStateConnected)
+		goto out;
+
+	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, "num_devs", "%d",
+			   &num_devs);
+	if (err != 1) {
+		if (err >= 0)
+			err = -EINVAL;
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error reading number of PCI devices");
+		goto out;
+	}
+
+	/* Find devices being detached and remove them. */
+	for (i = 0; i < num_devs; i++) {
+		int l, state;
+		l = snprintf(str, sizeof(str), "state-%d", i);
+		if (unlikely(l >= (sizeof(str) - 1))) {
+			err = -ENOMEM;
+			goto out;
+		}
+		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str, "%d",
+				   &state);
+		if (err != 1)
+			state = XenbusStateUnknown;
+
+		if (state != XenbusStateClosing)
+			continue;
+
+		/* Remove device. */
+		l = snprintf(str, sizeof(str), "vdev-%d", i);
+		if (unlikely(l >= (sizeof(str) - 1))) {
+			err = -ENOMEM;
+			goto out;
+		}
+		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
+			   	   "%x:%x:%x.%x", &domain, &bus, &slot, &func);
+		if (err != 4) {
+			if (err >= 0)
+				err = -EINVAL;
+			xenbus_dev_fatal(pdev->xdev, err,
+				 	 "Error reading PCI device %d", i);
+			goto out;
+		}
+
+		pci_bus = pci_find_bus(domain, bus);
+		if(!pci_bus) {
+			dev_dbg(&pdev->xdev->dev, "Cannot get bus %04x:%02x\n",
+				domain, bus);
+			continue;
+		}
+		pci_dev = pci_get_slot(pci_bus, PCI_DEVFN(slot, func));
+		if(!pci_dev) {
+			dev_dbg(&pdev->xdev->dev,
+				"Cannot get PCI device %04x:%02x:%02x.%02x\n",
+				domain, bus, slot, func);
+			continue;
+		}
+		pci_remove_bus_device(pci_dev);
+		pci_dev_put(pci_dev);
+
+		dev_dbg(&pdev->xdev->dev,
+			"PCI device %04x:%02x:%02x.%02x removed.\n",
+			domain, bus, slot, func);
+	}
+
+	err = xenbus_switch_state(pdev->xdev, XenbusStateReconfiguring);
+
+      out:
+	spin_unlock(&pdev->dev_lock);
+	return err;
+}
+
+static void __init_refok pcifront_backend_changed(struct xenbus_device *xdev,
+						  enum xenbus_state be_state)
+{
+	struct pcifront_device *pdev = xdev->dev.driver_data;
+
+	switch (be_state) {
+	case XenbusStateUnknown:
+	case XenbusStateInitialising:
+	case XenbusStateInitWait:
+	case XenbusStateInitialised:
+	case XenbusStateClosed:
+		break;
+
+	case XenbusStateConnected:
+		pcifront_try_connect(pdev);
+		break;
+
+	case XenbusStateClosing:
+		dev_warn(&xdev->dev, "backend going away!\n");
+		pcifront_try_disconnect(pdev);
+		break;
+
+	case XenbusStateReconfiguring:
+		pcifront_detach_devices(pdev);
+		break;
+
+	case XenbusStateReconfigured:
+		pcifront_attach_devices(pdev);
+		break;
+	}
+}
+
+static int pcifront_xenbus_probe(struct xenbus_device *xdev,
+				 const struct xenbus_device_id *id)
+{
+	int err = 0;
+	struct pcifront_device *pdev = alloc_pdev(xdev);
+
+	if (pdev == NULL) {
+		err = -ENOMEM;
+		xenbus_dev_fatal(xdev, err,
+				 "Error allocating pcifront_device struct");
+		goto out;
+	}
+
+	err = pcifront_publish_info(pdev);
+
+      out:
+	return err;
+}
+
+static int pcifront_xenbus_remove(struct xenbus_device *xdev)
+{
+	if (xdev->dev.driver_data)
+		free_pdev(xdev->dev.driver_data);
+
+	return 0;
+}
+
+static const struct xenbus_device_id xenpci_ids[] = {
+	{"pci"},
+	{{0}},
+};
+MODULE_ALIAS("xen:pci");
+
+static struct xenbus_driver xenbus_pcifront_driver = {
+	.name 			= "pcifront",
+	.owner 			= THIS_MODULE,
+	.ids 			= xenpci_ids,
+	.probe 			= pcifront_xenbus_probe,
+	.remove 		= pcifront_xenbus_remove,
+	.otherend_changed 	= pcifront_backend_changed,
+};
+
+static int __init pcifront_init(void)
+{
+	if (!is_running_on_xen())
+		return -ENODEV;
+
+	return xenbus_register_frontend(&xenbus_pcifront_driver);
+}
+
+/* Initialize after the Xen PCI Frontend Stub is initialized */
+subsys_initcall(pcifront_init);
diff --git a/include/xen/pcifront.h b/include/xen/pcifront.h
new file mode 100644
index 0000000..2557b13
--- /dev/null
+++ b/include/xen/pcifront.h
@@ -0,0 +1,83 @@
+/*
+ * PCI Frontend - arch-dependendent declarations
+ *
+ *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
+ */
+#ifndef __XEN_ASM_PCIFRONT_H__
+#define __XEN_ASM_PCIFRONT_H__
+
+#include <linux/spinlock.h>
+
+#ifdef __KERNEL__
+
+#ifndef __ia64__
+
+struct pcifront_device;
+struct pci_bus;
+
+struct pcifront_sd {
+	int domain;
+	struct pcifront_device *pdev;
+};
+
+static inline struct pcifront_device *
+pcifront_get_pdev(struct pcifront_sd *sd)
+{
+	return sd->pdev;
+}
+
+static inline void pcifront_init_sd(struct pcifront_sd *sd,
+				    unsigned int domain, unsigned int bus,
+				    struct pcifront_device *pdev)
+{
+	sd->domain = domain;
+	sd->pdev = pdev;
+}
+
+#if defined(CONFIG_PCI_DOMAINS)
+static inline int pci_domain_nr(struct pci_bus *bus)
+{
+	struct pcifront_sd *sd = bus->sysdata;
+	return sd->domain;
+}
+static inline int pci_proc_domain(struct pci_bus *bus)
+{
+	return pci_domain_nr(bus);
+}
+#endif /* CONFIG_PCI_DOMAINS */
+
+static inline void pcifront_setup_root_resources(struct pci_bus *bus,
+						 struct pcifront_sd *sd)
+{
+}
+
+#else /* __ia64__ */
+
+#include <linux/acpi.h>
+#include <asm/pci.h>
+#define pcifront_sd pci_controller
+
+extern void xen_add_resource(struct pci_controller *, unsigned int,
+			     unsigned int, struct acpi_resource *);
+extern void xen_pcibios_setup_root_windows(struct pci_bus *,
+					   struct pci_controller *);
+
+static inline struct pcifront_device *
+pcifront_get_pdev(struct pcifront_sd *sd)
+{
+	return (struct pcifront_device *)sd->platform_data;
+}
+
+static inline void pcifront_setup_root_resources(struct pci_bus *bus,
+						 struct pcifront_sd *sd)
+{
+	xen_pcibios_setup_root_windows(bus, sd);
+}
+
+#endif /* __ia64__ */
+
+extern struct rw_semaphore pci_bus_sem;
+
+#endif /* __KERNEL__ */
+
+#endif /* __XEN_ASM_PCIFRONT_H__ */
-- 
1.6.2.5

[PATCH 04/31] Fix include header name change.

[Konrad Rzeszutek Wilk]

evtchn.h is now called events.h

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pci_op.c |    2 +-
 drivers/xen/pcifront/xenbus.c |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/xen/pcifront/pci_op.c b/drivers/xen/pcifront/pci_op.c
index 2034d20..b239cf0 100644
--- a/drivers/xen/pcifront/pci_op.c
+++ b/drivers/xen/pcifront/pci_op.c
@@ -10,7 +10,7 @@
 #include <linux/spinlock.h>
 #include <asm/bitops.h>
 #include <linux/time.h>
-#include <xen/evtchn.h>
+#include <xen/events.h>
 #include "pcifront.h"
 
 static int verbose_request = 0;
diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index ca40547..d5b939c 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -7,7 +7,7 @@
 #include <linux/init.h>
 #include <linux/mm.h>
 #include <xen/xenbus.h>
-#include <xen/evtchn.h>
+#include <xen/events.h>
 #include <xen/gnttab.h>
 #include "pcifront.h"
 
-- 
1.6.2.5

[PATCH 05/31] Fix compile warning: ignoring return value of 'pci_bus_add_device', declared with attribute warn_unused_result

[Konrad Rzeszutek Wilk]

Make sure to check the return value and if it is a failure,
provide a descriptive message and quit.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pci_op.c |    9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)

diff --git a/drivers/xen/pcifront/pci_op.c b/drivers/xen/pcifront/pci_op.c
index b239cf0..03ed1f9 100644
--- a/drivers/xen/pcifront/pci_op.c
+++ b/drivers/xen/pcifront/pci_op.c
@@ -497,6 +497,7 @@ int __devinit pcifront_rescan_root(struct pcifront_device *pdev,
 	struct pci_bus *b;
 	struct pci_dev *d;
 	unsigned int devfn;
+	int err;
 
 #ifndef CONFIG_PCI_DOMAINS
 	if (domain != 0) {
@@ -533,7 +534,13 @@ int __devinit pcifront_rescan_root(struct pcifront_device *pdev,
 			dev_info(&pdev->xdev->dev, "New device on "
 				 "%04x:%02x:%02x.%02x found.\n", domain, bus,
 				 PCI_SLOT(devfn), PCI_FUNC(devfn));
-			pci_bus_add_device(d);
+			err = pci_bus_add_device(d);
+			if (err) {
+				dev_err(&pdev->xdev->dev, "Failed to add "
+				" device to bus.\n");
+				return err;
+			}
+			
 		}
 	}
 
-- 
1.6.2.5

[PATCH 06/31] Fix compile warning: passing argument 2 of 'pci_walk_bus' from incompatible pointer type

[Konrad Rzeszutek Wilk]

Just need to change the return value for the call back function.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pci_op.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/drivers/xen/pcifront/pci_op.c b/drivers/xen/pcifront/pci_op.c
index 03ed1f9..e807cd7 100644
--- a/drivers/xen/pcifront/pci_op.c
+++ b/drivers/xen/pcifront/pci_op.c
@@ -416,7 +416,7 @@ void pci_frontend_disable_msi(struct pci_dev* dev)
 #endif /* CONFIG_PCI_MSI */
 
 /* Claim resources for the PCI frontend as-is, backend won't allow changes */
-static void pcifront_claim_resource(struct pci_dev *dev, void *data)
+static int pcifront_claim_resource(struct pci_dev *dev, void *data)
 {
 	struct pcifront_device *pdev = data;
 	int i;
@@ -431,6 +431,8 @@ static void pcifront_claim_resource(struct pci_dev *dev, void *data)
 			pci_claim_resource(dev, i);
 		}
 	}
+
+	return 0;
 }
 
 int __devinit pcifront_scan_root(struct pcifront_device *pdev,
-- 
1.6.2.5

[PATCH 07/31] Fix compile error. The bind_to_irq_handler has different arguments.

[Konrad Rzeszutek Wilk]

Also fix the function decleration for 'pcifront_handler_aer' as
irqhandler's decleration has changed.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pci_op.c   |    2 +-
 drivers/xen/pcifront/pcifront.h |    2 +-
 drivers/xen/pcifront/xenbus.c   |    4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/xen/pcifront/pci_op.c b/drivers/xen/pcifront/pci_op.c
index e807cd7..550d94d 100644
--- a/drivers/xen/pcifront/pci_op.c
+++ b/drivers/xen/pcifront/pci_op.c
@@ -667,7 +667,7 @@ void pcifront_do_aer(void *data)
 
 }
 
-irqreturn_t pcifront_handler_aer(int irq, void *dev, struct pt_regs *regs)
+irqreturn_t pcifront_handler_aer(int irq, void *dev)
 {
 	struct pcifront_device *pdev = dev;
 	schedule_pcifront_aer_op(pdev);
diff --git a/drivers/xen/pcifront/pcifront.h b/drivers/xen/pcifront/pcifront.h
index 06cb3e1..d67ae95 100644
--- a/drivers/xen/pcifront/pcifront.h
+++ b/drivers/xen/pcifront/pcifront.h
@@ -50,6 +50,6 @@ void pcifront_free_roots(struct pcifront_device *pdev);
 
 void pcifront_do_aer( void *data);
 
-irqreturn_t pcifront_handler_aer(int irq, void *dev, struct pt_regs *regs);
+irqreturn_t pcifront_handler_aer(int irq, void *dev);
 
 #endif	/* __XEN_PCIFRONT_H__ */
diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index d5b939c..3348cff 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -94,8 +94,8 @@ static int pcifront_publish_info(struct pcifront_device *pdev)
 	if (err)
 		goto out;
 
-	bind_caller_port_to_irqhandler(pdev->evtchn, pcifront_handler_aer, 
-		SA_SAMPLE_RANDOM, "pcifront", pdev); 
+	bind_evtchn_to_irqhandler(pdev->evtchn, pcifront_handler_aer, 
+		0, "pcifront", pdev); 
 
       do_publish:
 	err = xenbus_transaction_start(&trans);
-- 
1.6.2.5

[PATCH 08/31] Fix compile error: implicit declaration of function 'virt_to_mfn'

[Konrad Rzeszutek Wilk]

Include the correct header file.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/xenbus.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index 3348cff..34bfe1c 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -9,6 +9,7 @@
 #include <xen/xenbus.h>
 #include <xen/events.h>
 #include <xen/gnttab.h>
+#include <xen/page.h>
 #include "pcifront.h"
 
 #ifndef __init_refok
-- 
1.6.2.5

[PATCH 09/31] Fix compile error: implicit declaration of function 'clear_evtchn'

[Konrad Rzeszutek Wilk]

The function is now called xen_clear_irq_pending.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pci_op.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/xen/pcifront/pci_op.c b/drivers/xen/pcifront/pci_op.c
index 550d94d..725732f 100644
--- a/drivers/xen/pcifront/pci_op.c
+++ b/drivers/xen/pcifront/pci_op.c
@@ -169,6 +169,7 @@ static int do_pci_op(struct pcifront_device *pdev, struct xen_pci_op *op)
 	struct xen_pci_op *active_op = &pdev->sh_info->op;
 	unsigned long irq_flags;
 	evtchn_port_t port = pdev->evtchn;
+	unsigned irq = pdev->irq;
 	s64 ns, ns_timeout;
 	struct timeval tv;
 
@@ -190,13 +191,13 @@ static int do_pci_op(struct pcifront_device *pdev, struct xen_pci_op *op)
 	do_gettimeofday(&tv);
 	ns_timeout = timeval_to_ns(&tv) + 2 * (s64)NSEC_PER_SEC;
 
-	clear_evtchn(port);
+	xen_clear_irq_pending(irq);
 
 	while (test_bit(_XEN_PCIF_active,
 			(unsigned long *)&pdev->sh_info->flags)) {
 		if (HYPERVISOR_poll(&port, 1, jiffies + 3*HZ))
 			BUG();
-		clear_evtchn(port);
+		xen_clear_irq_pending(irq);
 		do_gettimeofday(&tv);
 		ns = timeval_to_ns(&tv);
 		if (ns > ns_timeout) {
-- 
1.6.2.5

[PATCH 10/31] Fix compile error: implicit declaration of function 'gnttab_end_foreign_access'

[Konrad Rzeszutek Wilk]

The function in question now resides in grant-table.h instead of gnttab.h.
Fixed the include file decleration.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/xenbus.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index 34bfe1c..4741b5c 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -8,7 +8,7 @@
 #include <linux/mm.h>
 #include <xen/xenbus.h>
 #include <xen/events.h>
-#include <xen/gnttab.h>
+#include <xen/grant_table.h>
 #include <xen/page.h>
 #include "pcifront.h"
 
-- 
1.6.2.5

[PATCH 11/31] Fix compile error: too few arguments to function 'gnttab_end_foreign_access'

[Konrad Rzeszutek Wilk]

The 'gnttab_end..' function has now a readonly parameter.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/xenbus.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index 4741b5c..5e54696 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -72,7 +72,7 @@ static void free_pdev(struct pcifront_device *pdev)
 		xenbus_free_evtchn(pdev->xdev, pdev->evtchn);
 
 	if (pdev->gnt_ref != INVALID_GRANT_REF)
-		gnttab_end_foreign_access(pdev->gnt_ref,
+		gnttab_end_foreign_access(pdev->gnt_ref, 0 /* r/w page */,
 					  (unsigned long)pdev->sh_info);
 
 	pdev->xdev->dev.driver_data = NULL;
-- 
1.6.2.5

[PATCH 12/31] Remove function declerations (CONFIG_PCI_DOMAIN) that exist in recent kernels.

[Konrad Rzeszutek Wilk]

The 'pci_domain_nr' and  'pci_proc_domain' are defined properly
in the modern kernels. No need for this back-port.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 include/xen/pcifront.h |   11 -----------
 1 files changed, 0 insertions(+), 11 deletions(-)

diff --git a/include/xen/pcifront.h b/include/xen/pcifront.h
index 2557b13..bda9bd6 100644
--- a/include/xen/pcifront.h
+++ b/include/xen/pcifront.h
@@ -34,17 +34,6 @@ static inline void pcifront_init_sd(struct pcifront_sd *sd,
 	sd->pdev = pdev;
 }
 
-#if defined(CONFIG_PCI_DOMAINS)
-static inline int pci_domain_nr(struct pci_bus *bus)
-{
-	struct pcifront_sd *sd = bus->sysdata;
-	return sd->domain;
-}
-static inline int pci_proc_domain(struct pci_bus *bus)
-{
-	return pci_domain_nr(bus);
-}
-#endif /* CONFIG_PCI_DOMAINS */
 
 static inline void pcifront_setup_root_resources(struct pci_bus *bus,
 						 struct pcifront_sd *sd)
-- 
1.6.2.5

[PATCH 13/31] Fix uage of INIT_WORK.

[Konrad Rzeszutek Wilk]

Modern kernels only require two arguments. Also update
the workqueue function with the right prototype.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pci_op.c   |    4 ++--
 drivers/xen/pcifront/pcifront.h |    2 +-
 drivers/xen/pcifront/xenbus.c   |    2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/xen/pcifront/pci_op.c b/drivers/xen/pcifront/pci_op.c
index 725732f..fd47951 100644
--- a/drivers/xen/pcifront/pci_op.c
+++ b/drivers/xen/pcifront/pci_op.c
@@ -640,9 +640,9 @@ static pci_ers_result_t pcifront_common_process( int cmd, struct pcifront_device
 }
 
 
-void pcifront_do_aer(void *data)
+void pcifront_do_aer(struct work_struct *data)
 {
-	struct pcifront_device *pdev = data;
+	struct pcifront_device *pdev = container_of(data, struct pcifront_device, op_work);
 	int cmd = pdev->sh_info->aer_op.cmd;
 	pci_channel_state_t state = 
 		(pci_channel_state_t)pdev->sh_info->aer_op.err;
diff --git a/drivers/xen/pcifront/pcifront.h b/drivers/xen/pcifront/pcifront.h
index d67ae95..82364c4 100644
--- a/drivers/xen/pcifront/pcifront.h
+++ b/drivers/xen/pcifront/pcifront.h
@@ -48,7 +48,7 @@ int pcifront_rescan_root(struct pcifront_device *pdev,
 			 unsigned int domain, unsigned int bus);
 void pcifront_free_roots(struct pcifront_device *pdev);
 
-void pcifront_do_aer( void *data);
+void pcifront_do_aer(struct work_struct *data);
 
 irqreturn_t pcifront_handler_aer(int irq, void *dev);
 
diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index 5e54696..4f34f87 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -50,7 +50,7 @@ static struct pcifront_device *alloc_pdev(struct xenbus_device *xdev)
 	pdev->evtchn = INVALID_EVTCHN;
 	pdev->gnt_ref = INVALID_GRANT_REF;
 
-	INIT_WORK(&pdev->op_work, pcifront_do_aer, pdev);
+	INIT_WORK(&pdev->op_work, pcifront_do_aer);
 
 	dev_dbg(&xdev->dev, "Allocated pdev @ 0x%p pdev->sh_info @ 0x%p\n",
 		pdev, pdev->sh_info);
-- 
1.6.2.5

[PATCH 14/31] Add proper check to see if running under Xen.

[Konrad Rzeszutek Wilk]

The test for this is called differently on modern kernels.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/xenbus.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index 4f34f87..c2ef1de 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -459,7 +459,7 @@ static struct xenbus_driver xenbus_pcifront_driver = {
 
 static int __init pcifront_init(void)
 {
-	if (!is_running_on_xen())
+	if (!xen_domain())
 		return -ENODEV;
 
 	return xenbus_register_frontend(&xenbus_pcifront_driver);
-- 
1.6.2.5

[PATCH 15/31] Improper assumption that event channel == IRQ number.

[Konrad Rzeszutek Wilk]

We save now the IRQ number returned from bind_evtchn_to_irqhandler.
That value is now used by 'unbind_from_irqhandler' instead of
improper event channel number.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pcifront.h |    2 ++
 drivers/xen/pcifront/xenbus.c   |   12 ++++++++++--
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/xen/pcifront/pcifront.h b/drivers/xen/pcifront/pcifront.h
index 82364c4..909726b 100644
--- a/drivers/xen/pcifront/pcifront.h
+++ b/drivers/xen/pcifront/pcifront.h
@@ -31,6 +31,8 @@ struct pcifront_device {
 	int evtchn;
 	int gnt_ref;
 
+	int irq;
+
 	/* Lock this when doing any operations in sh_info */
 	spinlock_t sh_info_lock;
 	struct xen_pci_sharedinfo *sh_info;
diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index c2ef1de..8d8c2ca 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -49,6 +49,7 @@ static struct pcifront_device *alloc_pdev(struct xenbus_device *xdev)
 
 	pdev->evtchn = INVALID_EVTCHN;
 	pdev->gnt_ref = INVALID_GRANT_REF;
+	pdev->irq = -1;
 
 	INIT_WORK(&pdev->op_work, pcifront_do_aer);
 
@@ -66,7 +67,7 @@ static void free_pdev(struct pcifront_device *pdev)
 
 	/*For PCIE_AER error handling job*/
 	flush_scheduled_work();
-	unbind_from_irqhandler(pdev->evtchn, pdev);
+	unbind_from_irqhandler(pdev->irq, pdev);
 
 	if (pdev->evtchn != INVALID_EVTCHN)
 		xenbus_free_evtchn(pdev->xdev, pdev->evtchn);
@@ -95,8 +96,15 @@ static int pcifront_publish_info(struct pcifront_device *pdev)
 	if (err)
 		goto out;
 
-	bind_evtchn_to_irqhandler(pdev->evtchn, pcifront_handler_aer, 
+	err = bind_evtchn_to_irqhandler(pdev->evtchn, pcifront_handler_aer, 
 		0, "pcifront", pdev); 
+	if (err < 0) {
+		xenbus_free_evtchn(pdev->xdev, pdev->evtchn);
+		xenbus_dev_fatal(pdev->xdev, err, "Failed to bind evtchn to "
+				 "irqhandler.\n");
+		return err;
+	}
+	pdev->irq = err;
 
       do_publish:
 	err = xenbus_transaction_start(&trans);
-- 
1.6.2.5

[PATCH 16/31] Replace HYPERVISOR_poll with 'xen_poll_irq_timout' function.

[Konrad Rzeszutek Wilk]

We need a mechanism to poll the IRQ. In previous kernels the
HYPERVISOR_poll was used, but now we can use xen_poll_irq_timeout wrapper.

We still retain the old logic where we wait up to 3 seconds.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pci_op.c |    3 +--
 1 files changed, 1 insertions(+), 2 deletions(-)

diff --git a/drivers/xen/pcifront/pci_op.c b/drivers/xen/pcifront/pci_op.c
index fd47951..e270798 100644
--- a/drivers/xen/pcifront/pci_op.c
+++ b/drivers/xen/pcifront/pci_op.c
@@ -195,8 +195,7 @@ static int do_pci_op(struct pcifront_device *pdev, struct xen_pci_op *op)
 
 	while (test_bit(_XEN_PCIF_active,
 			(unsigned long *)&pdev->sh_info->flags)) {
-		if (HYPERVISOR_poll(&port, 1, jiffies + 3*HZ))
-			BUG();
+		xen_poll_irq_timeout(irq, jiffies + 3*HZ);
 		xen_clear_irq_pending(irq);
 		do_gettimeofday(&tv);
 		ns = timeval_to_ns(&tv);
-- 
1.6.2.5

[PATCH 17/31] Coalesce pci.c functions in xenbus.c.

[Konrad Rzeszutek Wilk]

This is the first commit in the process to relocate the driver to
drivers/pci/xen-pcifront.c.

Moving pcifront_disconnect and pcifront_connect in xenbus.c.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/Makefile   |    2 +-
 drivers/xen/pcifront/pci.c      |   46 ---------------------------------------
 drivers/xen/pcifront/pcifront.h |    2 -
 drivers/xen/pcifront/xenbus.c   |   36 ++++++++++++++++++++++++++++++
 4 files changed, 37 insertions(+), 49 deletions(-)
 delete mode 100644 drivers/xen/pcifront/pci.c

diff --git a/drivers/xen/pcifront/Makefile b/drivers/xen/pcifront/Makefile
index 621e988..1f8fa9d 100644
--- a/drivers/xen/pcifront/Makefile
+++ b/drivers/xen/pcifront/Makefile
@@ -1,6 +1,6 @@
 obj-y += pcifront.o
 
-pcifront-y := pci_op.o xenbus.o pci.o
+pcifront-y := pci_op.o xenbus.o
 
 ifeq ($(CONFIG_XEN_PCIDEV_FE_DEBUG),y)
 EXTRA_CFLAGS += -DDEBUG
diff --git a/drivers/xen/pcifront/pci.c b/drivers/xen/pcifront/pci.c
deleted file mode 100644
index 4239f00..0000000
--- a/drivers/xen/pcifront/pci.c
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * PCI Frontend Operations - ensure only one PCI frontend runs at a time
- *
- *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
- */
-#include <linux/module.h>
-#include <linux/init.h>
-#include <linux/pci.h>
-#include <linux/spinlock.h>
-#include "pcifront.h"
-
-DEFINE_SPINLOCK(pcifront_dev_lock);
-static struct pcifront_device *pcifront_dev = NULL;
-
-int pcifront_connect(struct pcifront_device *pdev)
-{
-	int err = 0;
-
-	spin_lock(&pcifront_dev_lock);
-
-	if (!pcifront_dev) {
-		dev_info(&pdev->xdev->dev, "Installing PCI frontend\n");
-		pcifront_dev = pdev;
-	}
-	else {
-		dev_err(&pdev->xdev->dev, "PCI frontend already installed!\n");
-		err = -EEXIST;
-	}
-
-	spin_unlock(&pcifront_dev_lock);
-
-	return err;
-}
-
-void pcifront_disconnect(struct pcifront_device *pdev)
-{
-	spin_lock(&pcifront_dev_lock);
-
-	if (pdev == pcifront_dev) {
-		dev_info(&pdev->xdev->dev,
-			 "Disconnecting PCI Frontend Buses\n");
-		pcifront_dev = NULL;
-	}
-
-	spin_unlock(&pcifront_dev_lock);
-}
diff --git a/drivers/xen/pcifront/pcifront.h b/drivers/xen/pcifront/pcifront.h
index 909726b..8e24fbe 100644
--- a/drivers/xen/pcifront/pcifront.h
+++ b/drivers/xen/pcifront/pcifront.h
@@ -41,8 +41,6 @@ struct pcifront_device {
 
 };
 
-int pcifront_connect(struct pcifront_device *pdev);
-void pcifront_disconnect(struct pcifront_device *pdev);
 
 int pcifront_scan_root(struct pcifront_device *pdev,
 		       unsigned int domain, unsigned int bus);
diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index 8d8c2ca..23d9f22 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -19,6 +19,42 @@
 #define INVALID_GRANT_REF (0)
 #define INVALID_EVTCHN    (-1)
 
+
+DEFINE_SPINLOCK(pcifront_dev_lock);
+static struct pcifront_device *pcifront_dev = NULL;
+
+int pcifront_connect(struct pcifront_device *pdev)
+{
+	int err = 0;
+
+	spin_lock(&pcifront_dev_lock);
+
+	if (!pcifront_dev) {
+		dev_info(&pdev->xdev->dev, "Installing PCI frontend\n");
+		pcifront_dev = pdev;
+	}
+	else {
+		dev_err(&pdev->xdev->dev, "PCI frontend already installed!\n");
+		err = -EEXIST;
+	}
+
+	spin_unlock(&pcifront_dev_lock);
+
+	return err;
+}
+
+void pcifront_disconnect(struct pcifront_device *pdev)
+{
+	spin_lock(&pcifront_dev_lock);
+
+	if (pdev == pcifront_dev) {
+		dev_info(&pdev->xdev->dev,
+			 "Disconnecting PCI Frontend Buses\n");
+		pcifront_dev = NULL;
+	}
+
+	spin_unlock(&pcifront_dev_lock);
+}
 static struct pcifront_device *alloc_pdev(struct xenbus_device *xdev)
 {
 	struct pcifront_device *pdev;
-- 
1.6.2.5

[PATCH 18/31] Coalesce xen/pcifront.h in drivers/xen/pcifront/pcifront.h

[Konrad Rzeszutek Wilk]

Moved all of the function declerations in the driver's pcifront.h.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pcifront.h |   54 ++++++++++++++++++++++++++++-
 include/xen/pcifront.h          |   72 ---------------------------------------
 2 files changed, 53 insertions(+), 73 deletions(-)
 delete mode 100644 include/xen/pcifront.h

diff --git a/drivers/xen/pcifront/pcifront.h b/drivers/xen/pcifront/pcifront.h
index 8e24fbe..3c6a799 100644
--- a/drivers/xen/pcifront/pcifront.h
+++ b/drivers/xen/pcifront/pcifront.h
@@ -11,7 +11,6 @@
 #include <xen/xenbus.h>
 #include <xen/interface/io/pciif.h>
 #include <linux/interrupt.h>
-#include <xen/pcifront.h>
 #include <asm/atomic.h>
 #include <linux/workqueue.h>
 
@@ -41,6 +40,59 @@ struct pcifront_device {
 
 };
 
+#ifndef __ia64__
+
+struct pcifront_sd {
+	int domain;
+	struct pcifront_device *pdev;
+};
+
+static inline struct pcifront_device *
+pcifront_get_pdev(struct pcifront_sd *sd)
+{
+	return sd->pdev;
+}
+
+static inline void pcifront_init_sd(struct pcifront_sd *sd,
+				    unsigned int domain, unsigned int bus,
+				    struct pcifront_device *pdev)
+{
+	sd->domain = domain;
+	sd->pdev = pdev;
+}
+
+
+static inline void pcifront_setup_root_resources(struct pci_bus *bus,
+						 struct pcifront_sd *sd)
+{
+}
+
+#else /* __ia64__ */
+
+#include <linux/acpi.h>
+#include <asm/pci.h>
+#define pcifront_sd pci_controller
+
+extern void xen_add_resource(struct pci_controller *, unsigned int,
+			     unsigned int, struct acpi_resource *);
+extern void xen_pcibios_setup_root_windows(struct pci_bus *,
+					   struct pci_controller *);
+
+static inline struct pcifront_device *
+pcifront_get_pdev(struct pcifront_sd *sd)
+{
+	return (struct pcifront_device *)sd->platform_data;
+}
+
+static inline void pcifront_setup_root_resources(struct pci_bus *bus,
+						 struct pcifront_sd *sd)
+{
+	xen_pcibios_setup_root_windows(bus, sd);
+}
+
+#endif /* __ia64__ */
+
+extern struct rw_semaphore pci_bus_sem;
 
 int pcifront_scan_root(struct pcifront_device *pdev,
 		       unsigned int domain, unsigned int bus);
diff --git a/include/xen/pcifront.h b/include/xen/pcifront.h
deleted file mode 100644
index bda9bd6..0000000
--- a/include/xen/pcifront.h
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * PCI Frontend - arch-dependendent declarations
- *
- *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
- */
-#ifndef __XEN_ASM_PCIFRONT_H__
-#define __XEN_ASM_PCIFRONT_H__
-
-#include <linux/spinlock.h>
-
-#ifdef __KERNEL__
-
-#ifndef __ia64__
-
-struct pcifront_device;
-struct pci_bus;
-
-struct pcifront_sd {
-	int domain;
-	struct pcifront_device *pdev;
-};
-
-static inline struct pcifront_device *
-pcifront_get_pdev(struct pcifront_sd *sd)
-{
-	return sd->pdev;
-}
-
-static inline void pcifront_init_sd(struct pcifront_sd *sd,
-				    unsigned int domain, unsigned int bus,
-				    struct pcifront_device *pdev)
-{
-	sd->domain = domain;
-	sd->pdev = pdev;
-}
-
-
-static inline void pcifront_setup_root_resources(struct pci_bus *bus,
-						 struct pcifront_sd *sd)
-{
-}
-
-#else /* __ia64__ */
-
-#include <linux/acpi.h>
-#include <asm/pci.h>
-#define pcifront_sd pci_controller
-
-extern void xen_add_resource(struct pci_controller *, unsigned int,
-			     unsigned int, struct acpi_resource *);
-extern void xen_pcibios_setup_root_windows(struct pci_bus *,
-					   struct pci_controller *);
-
-static inline struct pcifront_device *
-pcifront_get_pdev(struct pcifront_sd *sd)
-{
-	return (struct pcifront_device *)sd->platform_data;
-}
-
-static inline void pcifront_setup_root_resources(struct pci_bus *bus,
-						 struct pcifront_sd *sd)
-{
-	xen_pcibios_setup_root_windows(bus, sd);
-}
-
-#endif /* __ia64__ */
-
-extern struct rw_semaphore pci_bus_sem;
-
-#endif /* __KERNEL__ */
-
-#endif /* __XEN_ASM_PCIFRONT_H__ */
-- 
1.6.2.5

[PATCH 19/31] Remove ia64 from pcifront.c support.

[Konrad Rzeszutek Wilk]

The ia64 support of PCI frontend requires changes in the arch/ia64/pci/pci.c
which are currently not present. Will revisit once those are present.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pci_op.c   |  115 ---------------------------------------
 drivers/xen/pcifront/pcifront.h |   29 ----------
 2 files changed, 0 insertions(+), 144 deletions(-)

diff --git a/drivers/xen/pcifront/pci_op.c b/drivers/xen/pcifront/pci_op.c
index e270798..e2bea86 100644
--- a/drivers/xen/pcifront/pci_op.c
+++ b/drivers/xen/pcifront/pci_op.c
@@ -16,121 +16,6 @@
 static int verbose_request = 0;
 module_param(verbose_request, int, 0644);
 
-#ifdef __ia64__
-static void pcifront_init_sd(struct pcifront_sd *sd,
-			     unsigned int domain, unsigned int bus,
-			     struct pcifront_device *pdev)
-{
-	int err, i, j, k, len, root_num, res_count;
-	struct acpi_resource res;
-	unsigned int d, b, byte;
-	unsigned long magic;
-	char str[64], tmp[3];
-	unsigned char *buf, *bufp;
-	u8 *ptr;
-
-	memset(sd, 0, sizeof(*sd));
-
-	sd->segment = domain;
-	sd->node = -1;	/* Revisit for NUMA */
-	sd->platform_data = pdev;
-
-	/* Look for resources for this controller in xenbus. */
-	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, "root_num",
-			   "%d", &root_num);
-	if (err != 1)
-		return;
-
-	for (i = 0; i < root_num; i++) {
-		len = snprintf(str, sizeof(str), "root-%d", i);
-		if (unlikely(len >= (sizeof(str) - 1)))
-			return;
-
-		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
-				   str, "%x:%x", &d, &b);
-		if (err != 2)
-			return;
-
-		if (d == domain && b == bus)
-			break;
-	}
-
-	if (i == root_num)
-		return;
-
-	len = snprintf(str, sizeof(str), "root-resource-magic");
-
-	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
-			   str, "%lx", &magic);
-
-	if (err != 1)
-		return; /* No resources, nothing to do */
-
-	if (magic != (sizeof(res) * 2) + 1) {
-		printk(KERN_WARNING "pcifront: resource magic mismatch\n");
-		return;
-	}
-
-	len = snprintf(str, sizeof(str), "root-%d-resources", i);
-	if (unlikely(len >= (sizeof(str) - 1)))
-		return;
-
-	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
-			   str, "%d", &res_count);
-
-	if (err != 1)
-		return; /* No resources, nothing to do */
-
-	sd->window = kzalloc(sizeof(*sd->window) * res_count, GFP_KERNEL);
-	if (!sd->window)
-		return;
-
-	/* magic is also the size of the byte stream in xenbus */
-	buf = kmalloc(magic, GFP_KERNEL);
-	if (!buf) {
-		kfree(sd->window);
-		sd->window = NULL;
-		return;
-	}
-
-	/* Read the resources out of xenbus */
-	for (j = 0; j < res_count; j++) {
-		memset(&res, 0, sizeof(res));
-		memset(buf, 0, magic);
-
-		len = snprintf(str, sizeof(str), "root-%d-resource-%d", i, j);
-		if (unlikely(len >= (sizeof(str) - 1)))
-			return;
-
-		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
-				   "%s", buf);
-		if (err != 1) {
-			printk(KERN_WARNING "pcifront: error reading "
-			       "resource %d on bus %04x:%02x\n",
-			       j, domain, bus);
-			continue;
-		}
-
-		bufp = buf;
-		ptr = (u8 *)&res;
-		memset(tmp, 0, sizeof(tmp));
-
-		/* Copy ASCII byte stream into structure */
-		for (k = 0; k < magic - 1; k += 2) {
-			memcpy(tmp, bufp, 2);
-			bufp += 2;
-
-			sscanf(tmp, "%02x", &byte);
-			*ptr = byte;
-			ptr++;
-		}
-
-		xen_add_resource(sd, domain, bus, &res);
-		sd->windows++;
-	}
-	kfree(buf);
-}
-#endif
 
 static int errno_to_pcibios_err(int errno)
 {
diff --git a/drivers/xen/pcifront/pcifront.h b/drivers/xen/pcifront/pcifront.h
index 3c6a799..4b34619 100644
--- a/drivers/xen/pcifront/pcifront.h
+++ b/drivers/xen/pcifront/pcifront.h
@@ -40,8 +40,6 @@ struct pcifront_device {
 
 };
 
-#ifndef __ia64__
-
 struct pcifront_sd {
 	int domain;
 	struct pcifront_device *pdev;
@@ -61,37 +59,10 @@ static inline void pcifront_init_sd(struct pcifront_sd *sd,
 	sd->pdev = pdev;
 }
 
-
 static inline void pcifront_setup_root_resources(struct pci_bus *bus,
 						 struct pcifront_sd *sd)
 {
 }
-
-#else /* __ia64__ */
-
-#include <linux/acpi.h>
-#include <asm/pci.h>
-#define pcifront_sd pci_controller
-
-extern void xen_add_resource(struct pci_controller *, unsigned int,
-			     unsigned int, struct acpi_resource *);
-extern void xen_pcibios_setup_root_windows(struct pci_bus *,
-					   struct pci_controller *);
-
-static inline struct pcifront_device *
-pcifront_get_pdev(struct pcifront_sd *sd)
-{
-	return (struct pcifront_device *)sd->platform_data;
-}
-
-static inline void pcifront_setup_root_resources(struct pci_bus *bus,
-						 struct pcifront_sd *sd)
-{
-	xen_pcibios_setup_root_windows(bus, sd);
-}
-
-#endif /* __ia64__ */
-
 extern struct rw_semaphore pci_bus_sem;
 
 int pcifront_scan_root(struct pcifront_device *pdev,
-- 
1.6.2.5

[PATCH 20/31] Remove unused pci_bus_sem extern, as we don't use it.

[Konrad Rzeszutek Wilk]

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pcifront.h |    1 -
 1 files changed, 0 insertions(+), 1 deletions(-)

diff --git a/drivers/xen/pcifront/pcifront.h b/drivers/xen/pcifront/pcifront.h
index 4b34619..0bb4dbe 100644
--- a/drivers/xen/pcifront/pcifront.h
+++ b/drivers/xen/pcifront/pcifront.h
@@ -63,7 +63,6 @@ static inline void pcifront_setup_root_resources(struct pci_bus *bus,
 						 struct pcifront_sd *sd)
 {
 }
-extern struct rw_semaphore pci_bus_sem;
 
 int pcifront_scan_root(struct pcifront_device *pdev,
 		       unsigned int domain, unsigned int bus);
-- 
1.6.2.5

[PATCH 21/31] Coalesce pcifront.h in xenbus.c.

[Konrad Rzeszutek Wilk]

. and the next commit should be no surprise.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/pcifront.h |   77 ---------------------------------------
 drivers/xen/pcifront/xenbus.c   |   69 ++++++++++++++++++++++++++++++++++-
 2 files changed, 68 insertions(+), 78 deletions(-)
 delete mode 100644 drivers/xen/pcifront/pcifront.h

diff --git a/drivers/xen/pcifront/pcifront.h b/drivers/xen/pcifront/pcifront.h
deleted file mode 100644
index 0bb4dbe..0000000
--- a/drivers/xen/pcifront/pcifront.h
+++ /dev/null
@@ -1,77 +0,0 @@
-/*
- * PCI Frontend - Common data structures & function declarations
- *
- *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
- */
-#ifndef __XEN_PCIFRONT_H__
-#define __XEN_PCIFRONT_H__
-
-#include <linux/spinlock.h>
-#include <linux/pci.h>
-#include <xen/xenbus.h>
-#include <xen/interface/io/pciif.h>
-#include <linux/interrupt.h>
-#include <asm/atomic.h>
-#include <linux/workqueue.h>
-
-struct pci_bus_entry {
-	struct list_head list;
-	struct pci_bus *bus;
-};
-
-#define _PDEVB_op_active		(0)
-#define PDEVB_op_active 		(1 << (_PDEVB_op_active))
-
-struct pcifront_device {
-	struct xenbus_device *xdev;
-	struct list_head root_buses;
-	spinlock_t dev_lock;
-
-	int evtchn;
-	int gnt_ref;
-
-	int irq;
-
-	/* Lock this when doing any operations in sh_info */
-	spinlock_t sh_info_lock;
-	struct xen_pci_sharedinfo *sh_info;
-	struct work_struct op_work;
-	unsigned long flags;
-
-};
-
-struct pcifront_sd {
-	int domain;
-	struct pcifront_device *pdev;
-};
-
-static inline struct pcifront_device *
-pcifront_get_pdev(struct pcifront_sd *sd)
-{
-	return sd->pdev;
-}
-
-static inline void pcifront_init_sd(struct pcifront_sd *sd,
-				    unsigned int domain, unsigned int bus,
-				    struct pcifront_device *pdev)
-{
-	sd->domain = domain;
-	sd->pdev = pdev;
-}
-
-static inline void pcifront_setup_root_resources(struct pci_bus *bus,
-						 struct pcifront_sd *sd)
-{
-}
-
-int pcifront_scan_root(struct pcifront_device *pdev,
-		       unsigned int domain, unsigned int bus);
-int pcifront_rescan_root(struct pcifront_device *pdev,
-			 unsigned int domain, unsigned int bus);
-void pcifront_free_roots(struct pcifront_device *pdev);
-
-void pcifront_do_aer(struct work_struct *data);
-
-irqreturn_t pcifront_handler_aer(int irq, void *dev);
-
-#endif	/* __XEN_PCIFRONT_H__ */
diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index 23d9f22..56ad256 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -10,7 +10,14 @@
 #include <xen/events.h>
 #include <xen/grant_table.h>
 #include <xen/page.h>
-#include "pcifront.h"
+#include <linux/spinlock.h>
+#include <linux/pci.h>
+#include <xen/xenbus.h>
+#include <xen/interface/io/pciif.h>
+#include <linux/interrupt.h>
+#include <asm/atomic.h>
+#include <linux/workqueue.h>
+
 
 #ifndef __init_refok
 #define __init_refok
@@ -20,6 +27,66 @@
 #define INVALID_EVTCHN    (-1)
 
 
+struct pci_bus_entry {
+	struct list_head list;
+	struct pci_bus *bus;
+};
+
+#define _PDEVB_op_active		(0)
+#define PDEVB_op_active 		(1 << (_PDEVB_op_active))
+
+struct pcifront_device {
+	struct xenbus_device *xdev;
+	struct list_head root_buses;
+	spinlock_t dev_lock;
+
+	int evtchn;
+	int gnt_ref;
+
+	int irq;
+
+	/* Lock this when doing any operations in sh_info */
+	spinlock_t sh_info_lock;
+	struct xen_pci_sharedinfo *sh_info;
+	struct work_struct op_work;
+	unsigned long flags;
+
+};
+
+struct pcifront_sd {
+	int domain;
+	struct pcifront_device *pdev;
+};
+
+static inline struct pcifront_device *
+pcifront_get_pdev(struct pcifront_sd *sd)
+{
+	return sd->pdev;
+}
+
+static inline void pcifront_init_sd(struct pcifront_sd *sd,
+				    unsigned int domain, unsigned int bus,
+				    struct pcifront_device *pdev)
+{
+	sd->domain = domain;
+	sd->pdev = pdev;
+}
+
+static inline void pcifront_setup_root_resources(struct pci_bus *bus,
+						 struct pcifront_sd *sd)
+{
+}
+
+int pcifront_scan_root(struct pcifront_device *pdev,
+		       unsigned int domain, unsigned int bus);
+int pcifront_rescan_root(struct pcifront_device *pdev,
+			 unsigned int domain, unsigned int bus);
+void pcifront_free_roots(struct pcifront_device *pdev);
+
+void pcifront_do_aer(struct work_struct *data);
+
+irqreturn_t pcifront_handler_aer(int irq, void *dev);
+
 DEFINE_SPINLOCK(pcifront_dev_lock);
 static struct pcifront_device *pcifront_dev = NULL;
 
-- 
1.6.2.5

[PATCH 22/31] Coalesce pci_op.c in xenbus.c.

[Konrad Rzeszutek Wilk]

. and now onto removing obsolete function declerations.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/Makefile |    2 +-
 drivers/xen/pcifront/pci_op.c |  560 -----------------------------------------
 drivers/xen/pcifront/xenbus.c |  548 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 549 insertions(+), 561 deletions(-)
 delete mode 100644 drivers/xen/pcifront/pci_op.c

diff --git a/drivers/xen/pcifront/Makefile b/drivers/xen/pcifront/Makefile
index 1f8fa9d..d49136e 100644
--- a/drivers/xen/pcifront/Makefile
+++ b/drivers/xen/pcifront/Makefile
@@ -1,6 +1,6 @@
 obj-y += pcifront.o
 
-pcifront-y := pci_op.o xenbus.o
+pcifront-y := xenbus.o
 
 ifeq ($(CONFIG_XEN_PCIDEV_FE_DEBUG),y)
 EXTRA_CFLAGS += -DDEBUG
diff --git a/drivers/xen/pcifront/pci_op.c b/drivers/xen/pcifront/pci_op.c
deleted file mode 100644
index e2bea86..0000000
--- a/drivers/xen/pcifront/pci_op.c
+++ /dev/null
@@ -1,560 +0,0 @@
-/*
- * PCI Frontend Operations - Communicates with frontend
- *
- *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
- */
-#include <linux/module.h>
-#include <linux/version.h>
-#include <linux/init.h>
-#include <linux/pci.h>
-#include <linux/spinlock.h>
-#include <asm/bitops.h>
-#include <linux/time.h>
-#include <xen/events.h>
-#include "pcifront.h"
-
-static int verbose_request = 0;
-module_param(verbose_request, int, 0644);
-
-
-static int errno_to_pcibios_err(int errno)
-{
-	switch (errno) {
-	case XEN_PCI_ERR_success:
-		return PCIBIOS_SUCCESSFUL;
-
-	case XEN_PCI_ERR_dev_not_found:
-		return PCIBIOS_DEVICE_NOT_FOUND;
-
-	case XEN_PCI_ERR_invalid_offset:
-	case XEN_PCI_ERR_op_failed:
-		return PCIBIOS_BAD_REGISTER_NUMBER;
-
-	case XEN_PCI_ERR_not_implemented:
-		return PCIBIOS_FUNC_NOT_SUPPORTED;
-
-	case XEN_PCI_ERR_access_denied:
-		return PCIBIOS_SET_FAILED;
-	}
-	return errno;
-}
-
-static inline void schedule_pcifront_aer_op(struct pcifront_device *pdev)
-{
-	if (test_bit(_XEN_PCIB_active, (unsigned long *)&pdev->sh_info->flags)
-		&& !test_and_set_bit(_PDEVB_op_active, &pdev->flags)) {
-		dev_dbg(&pdev->xdev->dev, "schedule aer frontend job\n");
-		schedule_work(&pdev->op_work);
-	}
-}
-
-static int do_pci_op(struct pcifront_device *pdev, struct xen_pci_op *op)
-{
-	int err = 0;
-	struct xen_pci_op *active_op = &pdev->sh_info->op;
-	unsigned long irq_flags;
-	evtchn_port_t port = pdev->evtchn;
-	unsigned irq = pdev->irq;
-	s64 ns, ns_timeout;
-	struct timeval tv;
-
-	spin_lock_irqsave(&pdev->sh_info_lock, irq_flags);
-
-	memcpy(active_op, op, sizeof(struct xen_pci_op));
-
-	/* Go */
-	wmb();
-	set_bit(_XEN_PCIF_active, (unsigned long *)&pdev->sh_info->flags);
-	notify_remote_via_evtchn(port);
-
-	/*
-	 * We set a poll timeout of 3 seconds but give up on return after
-	 * 2 seconds. It is better to time out too late rather than too early
-	 * (in the latter case we end up continually re-executing poll() with a
-	 * timeout in the past). 1s difference gives plenty of slack for error.
-	 */
-	do_gettimeofday(&tv);
-	ns_timeout = timeval_to_ns(&tv) + 2 * (s64)NSEC_PER_SEC;
-
-	xen_clear_irq_pending(irq);
-
-	while (test_bit(_XEN_PCIF_active,
-			(unsigned long *)&pdev->sh_info->flags)) {
-		xen_poll_irq_timeout(irq, jiffies + 3*HZ);
-		xen_clear_irq_pending(irq);
-		do_gettimeofday(&tv);
-		ns = timeval_to_ns(&tv);
-		if (ns > ns_timeout) {
-			dev_err(&pdev->xdev->dev,
-				"pciback not responding!!!\n");
-			clear_bit(_XEN_PCIF_active,
-				  (unsigned long *)&pdev->sh_info->flags);
-			err = XEN_PCI_ERR_dev_not_found;
-			goto out;
-		}
-	}
-
-	/*
-	* We might lose backend service request since we 
-	* reuse same evtchn with pci_conf backend response. So re-schedule
-	* aer pcifront service.
-	*/
-	if (test_bit(_XEN_PCIB_active, 
-			(unsigned long*)&pdev->sh_info->flags)) {
-		dev_err(&pdev->xdev->dev, 
-			"schedule aer pcifront service\n");
-		schedule_pcifront_aer_op(pdev);
-	}
-
-	memcpy(op, active_op, sizeof(struct xen_pci_op));
-
-	err = op->err;
-      out:
-	spin_unlock_irqrestore(&pdev->sh_info_lock, irq_flags);
-	return err;
-}
-
-/* Access to this function is spinlocked in drivers/pci/access.c */
-static int pcifront_bus_read(struct pci_bus *bus, unsigned int devfn,
-			     int where, int size, u32 * val)
-{
-	int err = 0;
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_conf_read,
-		.domain = pci_domain_nr(bus),
-		.bus    = bus->number,
-		.devfn  = devfn,
-		.offset = where,
-		.size   = size,
-	};
-	struct pcifront_sd *sd = bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	if (verbose_request)
-		dev_info(&pdev->xdev->dev,
-			 "read dev=%04x:%02x:%02x.%01x - offset %x size %d\n",
-			 pci_domain_nr(bus), bus->number, PCI_SLOT(devfn),
-			 PCI_FUNC(devfn), where, size);
-
-	err = do_pci_op(pdev, &op);
-
-	if (likely(!err)) {
-		if (verbose_request)
-			dev_info(&pdev->xdev->dev, "read got back value %x\n",
-				 op.value);
-
-		*val = op.value;
-	} else if (err == -ENODEV) {
-		/* No device here, pretend that it just returned 0 */
-		err = 0;
-		*val = 0;
-	}
-
-	return errno_to_pcibios_err(err);
-}
-
-/* Access to this function is spinlocked in drivers/pci/access.c */
-static int pcifront_bus_write(struct pci_bus *bus, unsigned int devfn,
-			      int where, int size, u32 val)
-{
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_conf_write,
-		.domain = pci_domain_nr(bus),
-		.bus    = bus->number,
-		.devfn  = devfn,
-		.offset = where,
-		.size   = size,
-		.value  = val,
-	};
-	struct pcifront_sd *sd = bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	if (verbose_request)
-		dev_info(&pdev->xdev->dev,
-			 "write dev=%04x:%02x:%02x.%01x - "
-			 "offset %x size %d val %x\n",
-			 pci_domain_nr(bus), bus->number,
-			 PCI_SLOT(devfn), PCI_FUNC(devfn), where, size, val);
-
-	return errno_to_pcibios_err(do_pci_op(pdev, &op));
-}
-
-struct pci_ops pcifront_bus_ops = {
-	.read = pcifront_bus_read,
-	.write = pcifront_bus_write,
-};
-
-#ifdef CONFIG_PCI_MSI
-int pci_frontend_enable_msix(struct pci_dev *dev,
-		struct msix_entry *entries,
-		int nvec)
-{
-	int err;
-	int i;
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_enable_msix,
-		.domain = pci_domain_nr(dev->bus),
-		.bus = dev->bus->number,
-		.devfn = dev->devfn,
-		.value = nvec,
-	};
-	struct pcifront_sd *sd = dev->bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	if (nvec > SH_INFO_MAX_VEC) {
-		printk("too much vector for pci frontend%x\n", nvec);
-		return -EINVAL;
-	}
-
-	for (i = 0; i < nvec; i++) {
-		op.msix_entries[i].entry = entries[i].entry;
-		op.msix_entries[i].vector = entries[i].vector;
-	}
-
-	err = do_pci_op(pdev, &op);
-
-	if (!err) {
-		if (!op.value) {
-			/* we get the result */
-			for ( i = 0; i < nvec; i++)
-				entries[i].vector = op.msix_entries[i].vector;
-			return 0;
-		}
-		else {
-            printk("enable msix get value %x\n", op.value);
-			return op.value;
-		}
-	}
-	else {
-        printk("enable msix get err %x\n", err);
-		return err;
-	}
-}
-
-void pci_frontend_disable_msix(struct pci_dev* dev)
-{
-	int err;
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_disable_msix,
-		.domain = pci_domain_nr(dev->bus),
-		.bus = dev->bus->number,
-		.devfn = dev->devfn,
-	};
-	struct pcifront_sd *sd = dev->bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	err = do_pci_op(pdev, &op);
-
-	/* What should do for error ? */
-	if (err)
-		printk("pci_disable_msix get err %x\n", err);
-}
-
-int pci_frontend_enable_msi(struct pci_dev *dev)
-{
-	int err;
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_enable_msi,
-		.domain = pci_domain_nr(dev->bus),
-		.bus = dev->bus->number,
-		.devfn = dev->devfn,
-	};
-	struct pcifront_sd *sd = dev->bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	err = do_pci_op(pdev, &op);
-	if (likely(!err)) {
-		dev->irq = op.value;
-	}
-	else {
-		printk("pci frontend enable msi failed for dev %x:%x \n",
-				op.bus, op.devfn);
-		err = -EINVAL;
-	}
-	return err;
-}
-
-void pci_frontend_disable_msi(struct pci_dev* dev)
-{
-	int err;
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_disable_msi,
-		.domain = pci_domain_nr(dev->bus),
-		.bus = dev->bus->number,
-		.devfn = dev->devfn,
-	};
-	struct pcifront_sd *sd = dev->bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	err = do_pci_op(pdev, &op);
-	if (err == XEN_PCI_ERR_dev_not_found) {
-		/* XXX No response from backend, what shall we do? */
-		printk("get no response from backend for disable MSI\n");
-		return;
-	}
-	if (likely(!err))
-		dev->irq = op.value;
-	else
-		/* how can pciback notify us fail? */
-		printk("get fake response frombackend \n");
-}
-#endif /* CONFIG_PCI_MSI */
-
-/* Claim resources for the PCI frontend as-is, backend won't allow changes */
-static int pcifront_claim_resource(struct pci_dev *dev, void *data)
-{
-	struct pcifront_device *pdev = data;
-	int i;
-	struct resource *r;
-
-	for (i = 0; i < PCI_NUM_RESOURCES; i++) {
-		r = &dev->resource[i];
-
-		if (!r->parent && r->start && r->flags) {
-			dev_dbg(&pdev->xdev->dev, "claiming resource %s/%d\n",
-				pci_name(dev), i);
-			pci_claim_resource(dev, i);
-		}
-	}
-
-	return 0;
-}
-
-int __devinit pcifront_scan_root(struct pcifront_device *pdev,
-				 unsigned int domain, unsigned int bus)
-{
-	struct pci_bus *b;
-	struct pcifront_sd *sd = NULL;
-	struct pci_bus_entry *bus_entry = NULL;
-	int err = 0;
-
-#ifndef CONFIG_PCI_DOMAINS
-	if (domain != 0) {
-		dev_err(&pdev->xdev->dev,
-			"PCI Root in non-zero PCI Domain! domain=%d\n", domain);
-		dev_err(&pdev->xdev->dev,
-			"Please compile with CONFIG_PCI_DOMAINS\n");
-		err = -EINVAL;
-		goto err_out;
-	}
-#endif
-
-	dev_info(&pdev->xdev->dev, "Creating PCI Frontend Bus %04x:%02x\n",
-		 domain, bus);
-
-	bus_entry = kmalloc(sizeof(*bus_entry), GFP_KERNEL);
-	sd = kmalloc(sizeof(*sd), GFP_KERNEL);
-	if (!bus_entry || !sd) {
-		err = -ENOMEM;
-		goto err_out;
-	}
-	pcifront_init_sd(sd, domain, bus, pdev);
-
-	b = pci_scan_bus_parented(&pdev->xdev->dev, bus,
-				  &pcifront_bus_ops, sd);
-	if (!b) {
-		dev_err(&pdev->xdev->dev,
-			"Error creating PCI Frontend Bus!\n");
-		err = -ENOMEM;
-		goto err_out;
-	}
-
-	pcifront_setup_root_resources(b, sd);
-	bus_entry->bus = b;
-
-	list_add(&bus_entry->list, &pdev->root_buses);
-
-	/* Claim resources before going "live" with our devices */
-	pci_walk_bus(b, pcifront_claim_resource, pdev);
-
-	pci_bus_add_devices(b);
-
-	return 0;
-
-      err_out:
-	kfree(bus_entry);
-	kfree(sd);
-
-	return err;
-}
-
-int __devinit pcifront_rescan_root(struct pcifront_device *pdev,
-				   unsigned int domain, unsigned int bus)
-{
-	struct pci_bus *b;
-	struct pci_dev *d;
-	unsigned int devfn;
-	int err;
-
-#ifndef CONFIG_PCI_DOMAINS
-	if (domain != 0) {
-		dev_err(&pdev->xdev->dev,
-			"PCI Root in non-zero PCI Domain! domain=%d\n", domain);
-		dev_err(&pdev->xdev->dev,
-			"Please compile with CONFIG_PCI_DOMAINS\n");
-		return -EINVAL;
-	}
-#endif
-
-	dev_info(&pdev->xdev->dev, "Rescanning PCI Frontend Bus %04x:%02x\n",
-		 domain, bus);
-
-	b = pci_find_bus(domain, bus);
-	if(!b)
-		/* If the bus is unknown, create it. */
-		return pcifront_scan_root(pdev, domain, bus);
-
-	/* Rescan the bus for newly attached functions and add.
-	 * We omit handling of PCI bridge attachment because pciback prevents
-	 * bridges from being exported.
-	 */ 
-	for (devfn = 0; devfn < 0x100; devfn++) {
-		d = pci_get_slot(b, devfn);
-		if(d) {
-			/* Device is already known. */
-			pci_dev_put(d);
-			continue;
-		}
-
-		d = pci_scan_single_device(b, devfn);
-		if (d) {
-			dev_info(&pdev->xdev->dev, "New device on "
-				 "%04x:%02x:%02x.%02x found.\n", domain, bus,
-				 PCI_SLOT(devfn), PCI_FUNC(devfn));
-			err = pci_bus_add_device(d);
-			if (err) {
-				dev_err(&pdev->xdev->dev, "Failed to add "
-				" device to bus.\n");
-				return err;
-			}
-			
-		}
-	}
-
-	return 0;
-}
-
-static void free_root_bus_devs(struct pci_bus *bus)
-{
-	struct pci_dev *dev;
-
-	while (!list_empty(&bus->devices)) {
-		dev = container_of(bus->devices.next, struct pci_dev,
-				   bus_list);
-		dev_dbg(&dev->dev, "removing device\n");
-		pci_remove_bus_device(dev);
-	}
-}
-
-void pcifront_free_roots(struct pcifront_device *pdev)
-{
-	struct pci_bus_entry *bus_entry, *t;
-
-	dev_dbg(&pdev->xdev->dev, "cleaning up root buses\n");
-
-	list_for_each_entry_safe(bus_entry, t, &pdev->root_buses, list) {
-		list_del(&bus_entry->list);
-
-		free_root_bus_devs(bus_entry->bus);
-
-		kfree(bus_entry->bus->sysdata);
-
-		device_unregister(bus_entry->bus->bridge);
-		pci_remove_bus(bus_entry->bus);
-
-		kfree(bus_entry);
-	}
-}
-
-static pci_ers_result_t pcifront_common_process( int cmd, struct pcifront_device *pdev,
-	pci_channel_state_t state)
-{
-	pci_ers_result_t result;
-	struct pci_driver *pdrv;
-	int bus = pdev->sh_info->aer_op.bus;
-	int devfn = pdev->sh_info->aer_op.devfn;
-	struct pci_dev *pcidev;
-	int flag = 0;
-
-	dev_dbg(&pdev->xdev->dev, 
-		"pcifront AER process: cmd %x (bus:%x, devfn%x)",
-		cmd, bus, devfn);
-	result = PCI_ERS_RESULT_NONE;
-
-	pcidev = pci_get_bus_and_slot(bus, devfn);
-	if (!pcidev || !pcidev->driver){
-		dev_err(&pcidev->dev, 
-			"device or driver is NULL\n");
-		return result;
-	}
-	pdrv = pcidev->driver;
-
-	if (get_driver(&pdrv->driver)) {
-		if (pdrv->err_handler && pdrv->err_handler->error_detected) {
-			dev_dbg(&pcidev->dev,
-				"trying to call AER service\n");
-			if (pcidev) {
-				flag = 1;
-				switch(cmd) {
-				case XEN_PCI_OP_aer_detected:
-					result = pdrv->err_handler->error_detected(pcidev, state);
-					break;
-				case XEN_PCI_OP_aer_mmio:
-					result = pdrv->err_handler->mmio_enabled(pcidev);
-					break;
-				case XEN_PCI_OP_aer_slotreset:
-					result = pdrv->err_handler->slot_reset(pcidev);
-					break;
-				case XEN_PCI_OP_aer_resume:
-					pdrv->err_handler->resume(pcidev);
-					break;
-				default:
-					dev_err(&pdev->xdev->dev,
-						"bad request in aer recovery operation!\n");
-
-				}
-			}
-		}
-		put_driver(&pdrv->driver);
-	}
-	if (!flag)
-		result = PCI_ERS_RESULT_NONE;
-
-	return result;
-}
-
-
-void pcifront_do_aer(struct work_struct *data)
-{
-	struct pcifront_device *pdev = container_of(data, struct pcifront_device, op_work);
-	int cmd = pdev->sh_info->aer_op.cmd;
-	pci_channel_state_t state = 
-		(pci_channel_state_t)pdev->sh_info->aer_op.err;
-
-	/*If a pci_conf op is in progress, 
-		we have to wait until it is done before service aer op*/
-	dev_dbg(&pdev->xdev->dev, 
-		"pcifront service aer bus %x devfn %x\n", pdev->sh_info->aer_op.bus,
-		pdev->sh_info->aer_op.devfn);
-
-	pdev->sh_info->aer_op.err = pcifront_common_process(cmd, pdev, state);
-
-	wmb();
-	clear_bit(_XEN_PCIB_active, (unsigned long*)&pdev->sh_info->flags);
-	notify_remote_via_evtchn(pdev->evtchn);
-
-	/*in case of we lost an aer request in four lines time_window*/
-	smp_mb__before_clear_bit();
-	clear_bit( _PDEVB_op_active, &pdev->flags);
-	smp_mb__after_clear_bit();
-
-	schedule_pcifront_aer_op(pdev);
-
-}
-
-irqreturn_t pcifront_handler_aer(int irq, void *dev)
-{
-	struct pcifront_device *pdev = dev;
-	schedule_pcifront_aer_op(pdev);
-	return IRQ_HANDLED;
-}
diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index 56ad256..62e0008 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -17,6 +17,9 @@
 #include <linux/interrupt.h>
 #include <asm/atomic.h>
 #include <linux/workqueue.h>
+#include <asm/bitops.h>
+#include <linux/time.h>
+
 
 
 #ifndef __init_refok
@@ -90,6 +93,551 @@ irqreturn_t pcifront_handler_aer(int irq, void *dev);
 DEFINE_SPINLOCK(pcifront_dev_lock);
 static struct pcifront_device *pcifront_dev = NULL;
 
+static int verbose_request = 0;
+module_param(verbose_request, int, 0644);
+
+
+static int errno_to_pcibios_err(int errno)
+{
+	switch (errno) {
+	case XEN_PCI_ERR_success:
+		return PCIBIOS_SUCCESSFUL;
+
+	case XEN_PCI_ERR_dev_not_found:
+		return PCIBIOS_DEVICE_NOT_FOUND;
+
+	case XEN_PCI_ERR_invalid_offset:
+	case XEN_PCI_ERR_op_failed:
+		return PCIBIOS_BAD_REGISTER_NUMBER;
+
+	case XEN_PCI_ERR_not_implemented:
+		return PCIBIOS_FUNC_NOT_SUPPORTED;
+
+	case XEN_PCI_ERR_access_denied:
+		return PCIBIOS_SET_FAILED;
+	}
+	return errno;
+}
+
+static inline void schedule_pcifront_aer_op(struct pcifront_device *pdev)
+{
+	if (test_bit(_XEN_PCIB_active, (unsigned long *)&pdev->sh_info->flags)
+		&& !test_and_set_bit(_PDEVB_op_active, &pdev->flags)) {
+		dev_dbg(&pdev->xdev->dev, "schedule aer frontend job\n");
+		schedule_work(&pdev->op_work);
+	}
+}
+
+static int do_pci_op(struct pcifront_device *pdev, struct xen_pci_op *op)
+{
+	int err = 0;
+	struct xen_pci_op *active_op = &pdev->sh_info->op;
+	unsigned long irq_flags;
+	evtchn_port_t port = pdev->evtchn;
+	unsigned irq = pdev->irq;
+	s64 ns, ns_timeout;
+	struct timeval tv;
+
+	spin_lock_irqsave(&pdev->sh_info_lock, irq_flags);
+
+	memcpy(active_op, op, sizeof(struct xen_pci_op));
+
+	/* Go */
+	wmb();
+	set_bit(_XEN_PCIF_active, (unsigned long *)&pdev->sh_info->flags);
+	notify_remote_via_evtchn(port);
+
+	/*
+	 * We set a poll timeout of 3 seconds but give up on return after
+	 * 2 seconds. It is better to time out too late rather than too early
+	 * (in the latter case we end up continually re-executing poll() with a
+	 * timeout in the past). 1s difference gives plenty of slack for error.
+	 */
+	do_gettimeofday(&tv);
+	ns_timeout = timeval_to_ns(&tv) + 2 * (s64)NSEC_PER_SEC;
+
+	xen_clear_irq_pending(irq);
+
+	while (test_bit(_XEN_PCIF_active,
+			(unsigned long *)&pdev->sh_info->flags)) {
+		xen_poll_irq_timeout(irq, jiffies + 3*HZ);
+		xen_clear_irq_pending(irq);
+		do_gettimeofday(&tv);
+		ns = timeval_to_ns(&tv);
+		if (ns > ns_timeout) {
+			dev_err(&pdev->xdev->dev,
+				"pciback not responding!!!\n");
+			clear_bit(_XEN_PCIF_active,
+				  (unsigned long *)&pdev->sh_info->flags);
+			err = XEN_PCI_ERR_dev_not_found;
+			goto out;
+		}
+	}
+
+	/*
+	* We might lose backend service request since we 
+	* reuse same evtchn with pci_conf backend response. So re-schedule
+	* aer pcifront service.
+	*/
+	if (test_bit(_XEN_PCIB_active, 
+			(unsigned long*)&pdev->sh_info->flags)) {
+		dev_err(&pdev->xdev->dev, 
+			"schedule aer pcifront service\n");
+		schedule_pcifront_aer_op(pdev);
+	}
+
+	memcpy(op, active_op, sizeof(struct xen_pci_op));
+
+	err = op->err;
+      out:
+	spin_unlock_irqrestore(&pdev->sh_info_lock, irq_flags);
+	return err;
+}
+
+/* Access to this function is spinlocked in drivers/pci/access.c */
+static int pcifront_bus_read(struct pci_bus *bus, unsigned int devfn,
+			     int where, int size, u32 * val)
+{
+	int err = 0;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_conf_read,
+		.domain = pci_domain_nr(bus),
+		.bus    = bus->number,
+		.devfn  = devfn,
+		.offset = where,
+		.size   = size,
+	};
+	struct pcifront_sd *sd = bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	if (verbose_request)
+		dev_info(&pdev->xdev->dev,
+			 "read dev=%04x:%02x:%02x.%01x - offset %x size %d\n",
+			 pci_domain_nr(bus), bus->number, PCI_SLOT(devfn),
+			 PCI_FUNC(devfn), where, size);
+
+	err = do_pci_op(pdev, &op);
+
+	if (likely(!err)) {
+		if (verbose_request)
+			dev_info(&pdev->xdev->dev, "read got back value %x\n",
+				 op.value);
+
+		*val = op.value;
+	} else if (err == -ENODEV) {
+		/* No device here, pretend that it just returned 0 */
+		err = 0;
+		*val = 0;
+	}
+
+	return errno_to_pcibios_err(err);
+}
+
+/* Access to this function is spinlocked in drivers/pci/access.c */
+static int pcifront_bus_write(struct pci_bus *bus, unsigned int devfn,
+			      int where, int size, u32 val)
+{
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_conf_write,
+		.domain = pci_domain_nr(bus),
+		.bus    = bus->number,
+		.devfn  = devfn,
+		.offset = where,
+		.size   = size,
+		.value  = val,
+	};
+	struct pcifront_sd *sd = bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	if (verbose_request)
+		dev_info(&pdev->xdev->dev,
+			 "write dev=%04x:%02x:%02x.%01x - "
+			 "offset %x size %d val %x\n",
+			 pci_domain_nr(bus), bus->number,
+			 PCI_SLOT(devfn), PCI_FUNC(devfn), where, size, val);
+
+	return errno_to_pcibios_err(do_pci_op(pdev, &op));
+}
+
+struct pci_ops pcifront_bus_ops = {
+	.read = pcifront_bus_read,
+	.write = pcifront_bus_write,
+};
+
+#ifdef CONFIG_PCI_MSI
+int pci_frontend_enable_msix(struct pci_dev *dev,
+		struct msix_entry *entries,
+		int nvec)
+{
+	int err;
+	int i;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_enable_msix,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+		.value = nvec,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	if (nvec > SH_INFO_MAX_VEC) {
+		printk("too much vector for pci frontend%x\n", nvec);
+		return -EINVAL;
+	}
+
+	for (i = 0; i < nvec; i++) {
+		op.msix_entries[i].entry = entries[i].entry;
+		op.msix_entries[i].vector = entries[i].vector;
+	}
+
+	err = do_pci_op(pdev, &op);
+
+	if (!err) {
+		if (!op.value) {
+			/* we get the result */
+			for ( i = 0; i < nvec; i++)
+				entries[i].vector = op.msix_entries[i].vector;
+			return 0;
+		}
+		else {
+            printk("enable msix get value %x\n", op.value);
+			return op.value;
+		}
+	}
+	else {
+        printk("enable msix get err %x\n", err);
+		return err;
+	}
+}
+
+void pci_frontend_disable_msix(struct pci_dev* dev)
+{
+	int err;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_disable_msix,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	err = do_pci_op(pdev, &op);
+
+	/* What should do for error ? */
+	if (err)
+		printk("pci_disable_msix get err %x\n", err);
+}
+
+int pci_frontend_enable_msi(struct pci_dev *dev)
+{
+	int err;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_enable_msi,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	err = do_pci_op(pdev, &op);
+	if (likely(!err)) {
+		dev->irq = op.value;
+	}
+	else {
+		printk("pci frontend enable msi failed for dev %x:%x \n",
+				op.bus, op.devfn);
+		err = -EINVAL;
+	}
+	return err;
+}
+
+void pci_frontend_disable_msi(struct pci_dev* dev)
+{
+	int err;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_disable_msi,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	err = do_pci_op(pdev, &op);
+	if (err == XEN_PCI_ERR_dev_not_found) {
+		/* XXX No response from backend, what shall we do? */
+		printk("get no response from backend for disable MSI\n");
+		return;
+	}
+	if (likely(!err))
+		dev->irq = op.value;
+	else
+		/* how can pciback notify us fail? */
+		printk("get fake response frombackend \n");
+}
+#endif /* CONFIG_PCI_MSI */
+
+/* Claim resources for the PCI frontend as-is, backend won't allow changes */
+static int pcifront_claim_resource(struct pci_dev *dev, void *data)
+{
+	struct pcifront_device *pdev = data;
+	int i;
+	struct resource *r;
+
+	for (i = 0; i < PCI_NUM_RESOURCES; i++) {
+		r = &dev->resource[i];
+
+		if (!r->parent && r->start && r->flags) {
+			dev_dbg(&pdev->xdev->dev, "claiming resource %s/%d\n",
+				pci_name(dev), i);
+			pci_claim_resource(dev, i);
+		}
+	}
+
+	return 0;
+}
+
+int __devinit pcifront_scan_root(struct pcifront_device *pdev,
+				 unsigned int domain, unsigned int bus)
+{
+	struct pci_bus *b;
+	struct pcifront_sd *sd = NULL;
+	struct pci_bus_entry *bus_entry = NULL;
+	int err = 0;
+
+#ifndef CONFIG_PCI_DOMAINS
+	if (domain != 0) {
+		dev_err(&pdev->xdev->dev,
+			"PCI Root in non-zero PCI Domain! domain=%d\n", domain);
+		dev_err(&pdev->xdev->dev,
+			"Please compile with CONFIG_PCI_DOMAINS\n");
+		err = -EINVAL;
+		goto err_out;
+	}
+#endif
+
+	dev_info(&pdev->xdev->dev, "Creating PCI Frontend Bus %04x:%02x\n",
+		 domain, bus);
+
+	bus_entry = kmalloc(sizeof(*bus_entry), GFP_KERNEL);
+	sd = kmalloc(sizeof(*sd), GFP_KERNEL);
+	if (!bus_entry || !sd) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	pcifront_init_sd(sd, domain, bus, pdev);
+
+	b = pci_scan_bus_parented(&pdev->xdev->dev, bus,
+				  &pcifront_bus_ops, sd);
+	if (!b) {
+		dev_err(&pdev->xdev->dev,
+			"Error creating PCI Frontend Bus!\n");
+		err = -ENOMEM;
+		goto err_out;
+	}
+
+	pcifront_setup_root_resources(b, sd);
+	bus_entry->bus = b;
+
+	list_add(&bus_entry->list, &pdev->root_buses);
+
+	/* Claim resources before going "live" with our devices */
+	pci_walk_bus(b, pcifront_claim_resource, pdev);
+
+	pci_bus_add_devices(b);
+
+	return 0;
+
+      err_out:
+	kfree(bus_entry);
+	kfree(sd);
+
+	return err;
+}
+
+int __devinit pcifront_rescan_root(struct pcifront_device *pdev,
+				   unsigned int domain, unsigned int bus)
+{
+	struct pci_bus *b;
+	struct pci_dev *d;
+	unsigned int devfn;
+	int err;
+
+#ifndef CONFIG_PCI_DOMAINS
+	if (domain != 0) {
+		dev_err(&pdev->xdev->dev,
+			"PCI Root in non-zero PCI Domain! domain=%d\n", domain);
+		dev_err(&pdev->xdev->dev,
+			"Please compile with CONFIG_PCI_DOMAINS\n");
+		return -EINVAL;
+	}
+#endif
+
+	dev_info(&pdev->xdev->dev, "Rescanning PCI Frontend Bus %04x:%02x\n",
+		 domain, bus);
+
+	b = pci_find_bus(domain, bus);
+	if(!b)
+		/* If the bus is unknown, create it. */
+		return pcifront_scan_root(pdev, domain, bus);
+
+	/* Rescan the bus for newly attached functions and add.
+	 * We omit handling of PCI bridge attachment because pciback prevents
+	 * bridges from being exported.
+	 */ 
+	for (devfn = 0; devfn < 0x100; devfn++) {
+		d = pci_get_slot(b, devfn);
+		if(d) {
+			/* Device is already known. */
+			pci_dev_put(d);
+			continue;
+		}
+
+		d = pci_scan_single_device(b, devfn);
+		if (d) {
+			dev_info(&pdev->xdev->dev, "New device on "
+				 "%04x:%02x:%02x.%02x found.\n", domain, bus,
+				 PCI_SLOT(devfn), PCI_FUNC(devfn));
+			err = pci_bus_add_device(d);
+			if (err) {
+				dev_err(&pdev->xdev->dev, "Failed to add "
+				" device to bus.\n");
+				return err;
+			}
+			
+		}
+	}
+
+	return 0;
+}
+
+static void free_root_bus_devs(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+
+	while (!list_empty(&bus->devices)) {
+		dev = container_of(bus->devices.next, struct pci_dev,
+				   bus_list);
+		dev_dbg(&dev->dev, "removing device\n");
+		pci_remove_bus_device(dev);
+	}
+}
+
+void pcifront_free_roots(struct pcifront_device *pdev)
+{
+	struct pci_bus_entry *bus_entry, *t;
+
+	dev_dbg(&pdev->xdev->dev, "cleaning up root buses\n");
+
+	list_for_each_entry_safe(bus_entry, t, &pdev->root_buses, list) {
+		list_del(&bus_entry->list);
+
+		free_root_bus_devs(bus_entry->bus);
+
+		kfree(bus_entry->bus->sysdata);
+
+		device_unregister(bus_entry->bus->bridge);
+		pci_remove_bus(bus_entry->bus);
+
+		kfree(bus_entry);
+	}
+}
+
+static pci_ers_result_t pcifront_common_process( int cmd, struct pcifront_device *pdev,
+	pci_channel_state_t state)
+{
+	pci_ers_result_t result;
+	struct pci_driver *pdrv;
+	int bus = pdev->sh_info->aer_op.bus;
+	int devfn = pdev->sh_info->aer_op.devfn;
+	struct pci_dev *pcidev;
+	int flag = 0;
+
+	dev_dbg(&pdev->xdev->dev, 
+		"pcifront AER process: cmd %x (bus:%x, devfn%x)",
+		cmd, bus, devfn);
+	result = PCI_ERS_RESULT_NONE;
+
+	pcidev = pci_get_bus_and_slot(bus, devfn);
+	if (!pcidev || !pcidev->driver){
+		dev_err(&pcidev->dev, 
+			"device or driver is NULL\n");
+		return result;
+	}
+	pdrv = pcidev->driver;
+
+	if (get_driver(&pdrv->driver)) {
+		if (pdrv->err_handler && pdrv->err_handler->error_detected) {
+			dev_dbg(&pcidev->dev,
+				"trying to call AER service\n");
+			if (pcidev) {
+				flag = 1;
+				switch(cmd) {
+				case XEN_PCI_OP_aer_detected:
+					result = pdrv->err_handler->error_detected(pcidev, state);
+					break;
+				case XEN_PCI_OP_aer_mmio:
+					result = pdrv->err_handler->mmio_enabled(pcidev);
+					break;
+				case XEN_PCI_OP_aer_slotreset:
+					result = pdrv->err_handler->slot_reset(pcidev);
+					break;
+				case XEN_PCI_OP_aer_resume:
+					pdrv->err_handler->resume(pcidev);
+					break;
+				default:
+					dev_err(&pdev->xdev->dev,
+						"bad request in aer recovery operation!\n");
+
+				}
+			}
+		}
+		put_driver(&pdrv->driver);
+	}
+	if (!flag)
+		result = PCI_ERS_RESULT_NONE;
+
+	return result;
+}
+
+
+void pcifront_do_aer(struct work_struct *data)
+{
+	struct pcifront_device *pdev = container_of(data, struct pcifront_device, op_work);
+	int cmd = pdev->sh_info->aer_op.cmd;
+	pci_channel_state_t state = 
+		(pci_channel_state_t)pdev->sh_info->aer_op.err;
+
+	/*If a pci_conf op is in progress, 
+		we have to wait until it is done before service aer op*/
+	dev_dbg(&pdev->xdev->dev, 
+		"pcifront service aer bus %x devfn %x\n", pdev->sh_info->aer_op.bus,
+		pdev->sh_info->aer_op.devfn);
+
+	pdev->sh_info->aer_op.err = pcifront_common_process(cmd, pdev, state);
+
+	wmb();
+	clear_bit(_XEN_PCIB_active, (unsigned long*)&pdev->sh_info->flags);
+	notify_remote_via_evtchn(pdev->evtchn);
+
+	/*in case of we lost an aer request in four lines time_window*/
+	smp_mb__before_clear_bit();
+	clear_bit( _PDEVB_op_active, &pdev->flags);
+	smp_mb__after_clear_bit();
+
+	schedule_pcifront_aer_op(pdev);
+
+}
+
+irqreturn_t pcifront_handler_aer(int irq, void *dev)
+{
+	struct pcifront_device *pdev = dev;
+	schedule_pcifront_aer_op(pdev);
+	return IRQ_HANDLED;
+}
 int pcifront_connect(struct pcifront_device *pdev)
 {
 	int err = 0;
-- 
1.6.2.5

[PATCH 23/31] Remove unnecessary function declerations.

[Konrad Rzeszutek Wilk]

As a result of the merge of the files, we do not
need these function declerations anymore.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/pcifront/xenbus.c |   11 -----------
 1 files changed, 0 insertions(+), 11 deletions(-)

diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
index 62e0008..854850d 100644
--- a/drivers/xen/pcifront/xenbus.c
+++ b/drivers/xen/pcifront/xenbus.c
@@ -21,7 +21,6 @@
 #include <linux/time.h>
 
 
-
 #ifndef __init_refok
 #define __init_refok
 #endif
@@ -80,15 +79,6 @@ static inline void pcifront_setup_root_resources(struct pci_bus *bus,
 {
 }
 
-int pcifront_scan_root(struct pcifront_device *pdev,
-		       unsigned int domain, unsigned int bus);
-int pcifront_rescan_root(struct pcifront_device *pdev,
-			 unsigned int domain, unsigned int bus);
-void pcifront_free_roots(struct pcifront_device *pdev);
-
-void pcifront_do_aer(struct work_struct *data);
-
-irqreturn_t pcifront_handler_aer(int irq, void *dev);
 
 DEFINE_SPINLOCK(pcifront_dev_lock);
 static struct pcifront_device *pcifront_dev = NULL;
@@ -96,7 +86,6 @@ static struct pcifront_device *pcifront_dev = NULL;
 static int verbose_request = 0;
 module_param(verbose_request, int, 0644);
 
-
 static int errno_to_pcibios_err(int errno)
 {
 	switch (errno) {
-- 
1.6.2.5

[PATCH 24/31] Rename the drivers/xen/pcifront/* driver to drivers/pci/xen-pcifront.c.

[Konrad Rzeszutek Wilk]

Also the Kconfig and Makefile are appropiately updated.

This is the last step in the rename process.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/pci/Kconfig           |    9 +
 drivers/pci/Makefile          |    2 +
 drivers/pci/xen-pcifront.c    | 1117 +++++++++++++++++++++++++++++++++++++++++
 drivers/xen/Kconfig           |   17 -
 drivers/xen/Makefile          |    1 -
 drivers/xen/pcifront/Makefile |    7 -
 drivers/xen/pcifront/xenbus.c | 1117 -----------------------------------------
 7 files changed, 1128 insertions(+), 1142 deletions(-)
 create mode 100644 drivers/pci/xen-pcifront.c
 delete mode 100644 drivers/xen/pcifront/Makefile
 delete mode 100644 drivers/xen/pcifront/xenbus.c

diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index fdc864f..97a59d5 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -51,6 +51,15 @@ config PCI_STUB
 
 	  When in doubt, say N.
 
+config XEN_PCIDEV_FRONTEND
+        bool "Xen PCI Frontend"
+        depends on PCI && X86_64
+        select HOTPLUG
+        default y
+        help
+          The PCI device frontend driver allows the kernel to import arbitrary
+          PCI devices from a PCI backend to support PCI driver domains.
+
 config HT_IRQ
 	bool "Interrupts on hypertransport devices"
 	default y
diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
index 3590082..f44cc7c 100644
--- a/drivers/pci/Makefile
+++ b/drivers/pci/Makefile
@@ -59,6 +59,8 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o
 
 obj-$(CONFIG_PCI_STUB) += pci-stub.o
 
+obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o
+
 ifeq ($(CONFIG_PCI_DEBUG),y)
 EXTRA_CFLAGS += -DDEBUG
 endif
diff --git a/drivers/pci/xen-pcifront.c b/drivers/pci/xen-pcifront.c
new file mode 100644
index 0000000..854850d
--- /dev/null
+++ b/drivers/pci/xen-pcifront.c
@@ -0,0 +1,1117 @@
+/*
+ * PCI Frontend Xenbus Setup - handles setup with backend (imports page/evtchn)
+ *
+ *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
+ */
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/mm.h>
+#include <xen/xenbus.h>
+#include <xen/events.h>
+#include <xen/grant_table.h>
+#include <xen/page.h>
+#include <linux/spinlock.h>
+#include <linux/pci.h>
+#include <xen/xenbus.h>
+#include <xen/interface/io/pciif.h>
+#include <linux/interrupt.h>
+#include <asm/atomic.h>
+#include <linux/workqueue.h>
+#include <asm/bitops.h>
+#include <linux/time.h>
+
+
+#ifndef __init_refok
+#define __init_refok
+#endif
+
+#define INVALID_GRANT_REF (0)
+#define INVALID_EVTCHN    (-1)
+
+
+struct pci_bus_entry {
+	struct list_head list;
+	struct pci_bus *bus;
+};
+
+#define _PDEVB_op_active		(0)
+#define PDEVB_op_active 		(1 << (_PDEVB_op_active))
+
+struct pcifront_device {
+	struct xenbus_device *xdev;
+	struct list_head root_buses;
+	spinlock_t dev_lock;
+
+	int evtchn;
+	int gnt_ref;
+
+	int irq;
+
+	/* Lock this when doing any operations in sh_info */
+	spinlock_t sh_info_lock;
+	struct xen_pci_sharedinfo *sh_info;
+	struct work_struct op_work;
+	unsigned long flags;
+
+};
+
+struct pcifront_sd {
+	int domain;
+	struct pcifront_device *pdev;
+};
+
+static inline struct pcifront_device *
+pcifront_get_pdev(struct pcifront_sd *sd)
+{
+	return sd->pdev;
+}
+
+static inline void pcifront_init_sd(struct pcifront_sd *sd,
+				    unsigned int domain, unsigned int bus,
+				    struct pcifront_device *pdev)
+{
+	sd->domain = domain;
+	sd->pdev = pdev;
+}
+
+static inline void pcifront_setup_root_resources(struct pci_bus *bus,
+						 struct pcifront_sd *sd)
+{
+}
+
+
+DEFINE_SPINLOCK(pcifront_dev_lock);
+static struct pcifront_device *pcifront_dev = NULL;
+
+static int verbose_request = 0;
+module_param(verbose_request, int, 0644);
+
+static int errno_to_pcibios_err(int errno)
+{
+	switch (errno) {
+	case XEN_PCI_ERR_success:
+		return PCIBIOS_SUCCESSFUL;
+
+	case XEN_PCI_ERR_dev_not_found:
+		return PCIBIOS_DEVICE_NOT_FOUND;
+
+	case XEN_PCI_ERR_invalid_offset:
+	case XEN_PCI_ERR_op_failed:
+		return PCIBIOS_BAD_REGISTER_NUMBER;
+
+	case XEN_PCI_ERR_not_implemented:
+		return PCIBIOS_FUNC_NOT_SUPPORTED;
+
+	case XEN_PCI_ERR_access_denied:
+		return PCIBIOS_SET_FAILED;
+	}
+	return errno;
+}
+
+static inline void schedule_pcifront_aer_op(struct pcifront_device *pdev)
+{
+	if (test_bit(_XEN_PCIB_active, (unsigned long *)&pdev->sh_info->flags)
+		&& !test_and_set_bit(_PDEVB_op_active, &pdev->flags)) {
+		dev_dbg(&pdev->xdev->dev, "schedule aer frontend job\n");
+		schedule_work(&pdev->op_work);
+	}
+}
+
+static int do_pci_op(struct pcifront_device *pdev, struct xen_pci_op *op)
+{
+	int err = 0;
+	struct xen_pci_op *active_op = &pdev->sh_info->op;
+	unsigned long irq_flags;
+	evtchn_port_t port = pdev->evtchn;
+	unsigned irq = pdev->irq;
+	s64 ns, ns_timeout;
+	struct timeval tv;
+
+	spin_lock_irqsave(&pdev->sh_info_lock, irq_flags);
+
+	memcpy(active_op, op, sizeof(struct xen_pci_op));
+
+	/* Go */
+	wmb();
+	set_bit(_XEN_PCIF_active, (unsigned long *)&pdev->sh_info->flags);
+	notify_remote_via_evtchn(port);
+
+	/*
+	 * We set a poll timeout of 3 seconds but give up on return after
+	 * 2 seconds. It is better to time out too late rather than too early
+	 * (in the latter case we end up continually re-executing poll() with a
+	 * timeout in the past). 1s difference gives plenty of slack for error.
+	 */
+	do_gettimeofday(&tv);
+	ns_timeout = timeval_to_ns(&tv) + 2 * (s64)NSEC_PER_SEC;
+
+	xen_clear_irq_pending(irq);
+
+	while (test_bit(_XEN_PCIF_active,
+			(unsigned long *)&pdev->sh_info->flags)) {
+		xen_poll_irq_timeout(irq, jiffies + 3*HZ);
+		xen_clear_irq_pending(irq);
+		do_gettimeofday(&tv);
+		ns = timeval_to_ns(&tv);
+		if (ns > ns_timeout) {
+			dev_err(&pdev->xdev->dev,
+				"pciback not responding!!!\n");
+			clear_bit(_XEN_PCIF_active,
+				  (unsigned long *)&pdev->sh_info->flags);
+			err = XEN_PCI_ERR_dev_not_found;
+			goto out;
+		}
+	}
+
+	/*
+	* We might lose backend service request since we 
+	* reuse same evtchn with pci_conf backend response. So re-schedule
+	* aer pcifront service.
+	*/
+	if (test_bit(_XEN_PCIB_active, 
+			(unsigned long*)&pdev->sh_info->flags)) {
+		dev_err(&pdev->xdev->dev, 
+			"schedule aer pcifront service\n");
+		schedule_pcifront_aer_op(pdev);
+	}
+
+	memcpy(op, active_op, sizeof(struct xen_pci_op));
+
+	err = op->err;
+      out:
+	spin_unlock_irqrestore(&pdev->sh_info_lock, irq_flags);
+	return err;
+}
+
+/* Access to this function is spinlocked in drivers/pci/access.c */
+static int pcifront_bus_read(struct pci_bus *bus, unsigned int devfn,
+			     int where, int size, u32 * val)
+{
+	int err = 0;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_conf_read,
+		.domain = pci_domain_nr(bus),
+		.bus    = bus->number,
+		.devfn  = devfn,
+		.offset = where,
+		.size   = size,
+	};
+	struct pcifront_sd *sd = bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	if (verbose_request)
+		dev_info(&pdev->xdev->dev,
+			 "read dev=%04x:%02x:%02x.%01x - offset %x size %d\n",
+			 pci_domain_nr(bus), bus->number, PCI_SLOT(devfn),
+			 PCI_FUNC(devfn), where, size);
+
+	err = do_pci_op(pdev, &op);
+
+	if (likely(!err)) {
+		if (verbose_request)
+			dev_info(&pdev->xdev->dev, "read got back value %x\n",
+				 op.value);
+
+		*val = op.value;
+	} else if (err == -ENODEV) {
+		/* No device here, pretend that it just returned 0 */
+		err = 0;
+		*val = 0;
+	}
+
+	return errno_to_pcibios_err(err);
+}
+
+/* Access to this function is spinlocked in drivers/pci/access.c */
+static int pcifront_bus_write(struct pci_bus *bus, unsigned int devfn,
+			      int where, int size, u32 val)
+{
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_conf_write,
+		.domain = pci_domain_nr(bus),
+		.bus    = bus->number,
+		.devfn  = devfn,
+		.offset = where,
+		.size   = size,
+		.value  = val,
+	};
+	struct pcifront_sd *sd = bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	if (verbose_request)
+		dev_info(&pdev->xdev->dev,
+			 "write dev=%04x:%02x:%02x.%01x - "
+			 "offset %x size %d val %x\n",
+			 pci_domain_nr(bus), bus->number,
+			 PCI_SLOT(devfn), PCI_FUNC(devfn), where, size, val);
+
+	return errno_to_pcibios_err(do_pci_op(pdev, &op));
+}
+
+struct pci_ops pcifront_bus_ops = {
+	.read = pcifront_bus_read,
+	.write = pcifront_bus_write,
+};
+
+#ifdef CONFIG_PCI_MSI
+int pci_frontend_enable_msix(struct pci_dev *dev,
+		struct msix_entry *entries,
+		int nvec)
+{
+	int err;
+	int i;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_enable_msix,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+		.value = nvec,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	if (nvec > SH_INFO_MAX_VEC) {
+		printk("too much vector for pci frontend%x\n", nvec);
+		return -EINVAL;
+	}
+
+	for (i = 0; i < nvec; i++) {
+		op.msix_entries[i].entry = entries[i].entry;
+		op.msix_entries[i].vector = entries[i].vector;
+	}
+
+	err = do_pci_op(pdev, &op);
+
+	if (!err) {
+		if (!op.value) {
+			/* we get the result */
+			for ( i = 0; i < nvec; i++)
+				entries[i].vector = op.msix_entries[i].vector;
+			return 0;
+		}
+		else {
+            printk("enable msix get value %x\n", op.value);
+			return op.value;
+		}
+	}
+	else {
+        printk("enable msix get err %x\n", err);
+		return err;
+	}
+}
+
+void pci_frontend_disable_msix(struct pci_dev* dev)
+{
+	int err;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_disable_msix,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	err = do_pci_op(pdev, &op);
+
+	/* What should do for error ? */
+	if (err)
+		printk("pci_disable_msix get err %x\n", err);
+}
+
+int pci_frontend_enable_msi(struct pci_dev *dev)
+{
+	int err;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_enable_msi,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	err = do_pci_op(pdev, &op);
+	if (likely(!err)) {
+		dev->irq = op.value;
+	}
+	else {
+		printk("pci frontend enable msi failed for dev %x:%x \n",
+				op.bus, op.devfn);
+		err = -EINVAL;
+	}
+	return err;
+}
+
+void pci_frontend_disable_msi(struct pci_dev* dev)
+{
+	int err;
+	struct xen_pci_op op = {
+		.cmd    = XEN_PCI_OP_disable_msi,
+		.domain = pci_domain_nr(dev->bus),
+		.bus = dev->bus->number,
+		.devfn = dev->devfn,
+	};
+	struct pcifront_sd *sd = dev->bus->sysdata;
+	struct pcifront_device *pdev = pcifront_get_pdev(sd);
+
+	err = do_pci_op(pdev, &op);
+	if (err == XEN_PCI_ERR_dev_not_found) {
+		/* XXX No response from backend, what shall we do? */
+		printk("get no response from backend for disable MSI\n");
+		return;
+	}
+	if (likely(!err))
+		dev->irq = op.value;
+	else
+		/* how can pciback notify us fail? */
+		printk("get fake response frombackend \n");
+}
+#endif /* CONFIG_PCI_MSI */
+
+/* Claim resources for the PCI frontend as-is, backend won't allow changes */
+static int pcifront_claim_resource(struct pci_dev *dev, void *data)
+{
+	struct pcifront_device *pdev = data;
+	int i;
+	struct resource *r;
+
+	for (i = 0; i < PCI_NUM_RESOURCES; i++) {
+		r = &dev->resource[i];
+
+		if (!r->parent && r->start && r->flags) {
+			dev_dbg(&pdev->xdev->dev, "claiming resource %s/%d\n",
+				pci_name(dev), i);
+			pci_claim_resource(dev, i);
+		}
+	}
+
+	return 0;
+}
+
+int __devinit pcifront_scan_root(struct pcifront_device *pdev,
+				 unsigned int domain, unsigned int bus)
+{
+	struct pci_bus *b;
+	struct pcifront_sd *sd = NULL;
+	struct pci_bus_entry *bus_entry = NULL;
+	int err = 0;
+
+#ifndef CONFIG_PCI_DOMAINS
+	if (domain != 0) {
+		dev_err(&pdev->xdev->dev,
+			"PCI Root in non-zero PCI Domain! domain=%d\n", domain);
+		dev_err(&pdev->xdev->dev,
+			"Please compile with CONFIG_PCI_DOMAINS\n");
+		err = -EINVAL;
+		goto err_out;
+	}
+#endif
+
+	dev_info(&pdev->xdev->dev, "Creating PCI Frontend Bus %04x:%02x\n",
+		 domain, bus);
+
+	bus_entry = kmalloc(sizeof(*bus_entry), GFP_KERNEL);
+	sd = kmalloc(sizeof(*sd), GFP_KERNEL);
+	if (!bus_entry || !sd) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	pcifront_init_sd(sd, domain, bus, pdev);
+
+	b = pci_scan_bus_parented(&pdev->xdev->dev, bus,
+				  &pcifront_bus_ops, sd);
+	if (!b) {
+		dev_err(&pdev->xdev->dev,
+			"Error creating PCI Frontend Bus!\n");
+		err = -ENOMEM;
+		goto err_out;
+	}
+
+	pcifront_setup_root_resources(b, sd);
+	bus_entry->bus = b;
+
+	list_add(&bus_entry->list, &pdev->root_buses);
+
+	/* Claim resources before going "live" with our devices */
+	pci_walk_bus(b, pcifront_claim_resource, pdev);
+
+	pci_bus_add_devices(b);
+
+	return 0;
+
+      err_out:
+	kfree(bus_entry);
+	kfree(sd);
+
+	return err;
+}
+
+int __devinit pcifront_rescan_root(struct pcifront_device *pdev,
+				   unsigned int domain, unsigned int bus)
+{
+	struct pci_bus *b;
+	struct pci_dev *d;
+	unsigned int devfn;
+	int err;
+
+#ifndef CONFIG_PCI_DOMAINS
+	if (domain != 0) {
+		dev_err(&pdev->xdev->dev,
+			"PCI Root in non-zero PCI Domain! domain=%d\n", domain);
+		dev_err(&pdev->xdev->dev,
+			"Please compile with CONFIG_PCI_DOMAINS\n");
+		return -EINVAL;
+	}
+#endif
+
+	dev_info(&pdev->xdev->dev, "Rescanning PCI Frontend Bus %04x:%02x\n",
+		 domain, bus);
+
+	b = pci_find_bus(domain, bus);
+	if(!b)
+		/* If the bus is unknown, create it. */
+		return pcifront_scan_root(pdev, domain, bus);
+
+	/* Rescan the bus for newly attached functions and add.
+	 * We omit handling of PCI bridge attachment because pciback prevents
+	 * bridges from being exported.
+	 */ 
+	for (devfn = 0; devfn < 0x100; devfn++) {
+		d = pci_get_slot(b, devfn);
+		if(d) {
+			/* Device is already known. */
+			pci_dev_put(d);
+			continue;
+		}
+
+		d = pci_scan_single_device(b, devfn);
+		if (d) {
+			dev_info(&pdev->xdev->dev, "New device on "
+				 "%04x:%02x:%02x.%02x found.\n", domain, bus,
+				 PCI_SLOT(devfn), PCI_FUNC(devfn));
+			err = pci_bus_add_device(d);
+			if (err) {
+				dev_err(&pdev->xdev->dev, "Failed to add "
+				" device to bus.\n");
+				return err;
+			}
+			
+		}
+	}
+
+	return 0;
+}
+
+static void free_root_bus_devs(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+
+	while (!list_empty(&bus->devices)) {
+		dev = container_of(bus->devices.next, struct pci_dev,
+				   bus_list);
+		dev_dbg(&dev->dev, "removing device\n");
+		pci_remove_bus_device(dev);
+	}
+}
+
+void pcifront_free_roots(struct pcifront_device *pdev)
+{
+	struct pci_bus_entry *bus_entry, *t;
+
+	dev_dbg(&pdev->xdev->dev, "cleaning up root buses\n");
+
+	list_for_each_entry_safe(bus_entry, t, &pdev->root_buses, list) {
+		list_del(&bus_entry->list);
+
+		free_root_bus_devs(bus_entry->bus);
+
+		kfree(bus_entry->bus->sysdata);
+
+		device_unregister(bus_entry->bus->bridge);
+		pci_remove_bus(bus_entry->bus);
+
+		kfree(bus_entry);
+	}
+}
+
+static pci_ers_result_t pcifront_common_process( int cmd, struct pcifront_device *pdev,
+	pci_channel_state_t state)
+{
+	pci_ers_result_t result;
+	struct pci_driver *pdrv;
+	int bus = pdev->sh_info->aer_op.bus;
+	int devfn = pdev->sh_info->aer_op.devfn;
+	struct pci_dev *pcidev;
+	int flag = 0;
+
+	dev_dbg(&pdev->xdev->dev, 
+		"pcifront AER process: cmd %x (bus:%x, devfn%x)",
+		cmd, bus, devfn);
+	result = PCI_ERS_RESULT_NONE;
+
+	pcidev = pci_get_bus_and_slot(bus, devfn);
+	if (!pcidev || !pcidev->driver){
+		dev_err(&pcidev->dev, 
+			"device or driver is NULL\n");
+		return result;
+	}
+	pdrv = pcidev->driver;
+
+	if (get_driver(&pdrv->driver)) {
+		if (pdrv->err_handler && pdrv->err_handler->error_detected) {
+			dev_dbg(&pcidev->dev,
+				"trying to call AER service\n");
+			if (pcidev) {
+				flag = 1;
+				switch(cmd) {
+				case XEN_PCI_OP_aer_detected:
+					result = pdrv->err_handler->error_detected(pcidev, state);
+					break;
+				case XEN_PCI_OP_aer_mmio:
+					result = pdrv->err_handler->mmio_enabled(pcidev);
+					break;
+				case XEN_PCI_OP_aer_slotreset:
+					result = pdrv->err_handler->slot_reset(pcidev);
+					break;
+				case XEN_PCI_OP_aer_resume:
+					pdrv->err_handler->resume(pcidev);
+					break;
+				default:
+					dev_err(&pdev->xdev->dev,
+						"bad request in aer recovery operation!\n");
+
+				}
+			}
+		}
+		put_driver(&pdrv->driver);
+	}
+	if (!flag)
+		result = PCI_ERS_RESULT_NONE;
+
+	return result;
+}
+
+
+void pcifront_do_aer(struct work_struct *data)
+{
+	struct pcifront_device *pdev = container_of(data, struct pcifront_device, op_work);
+	int cmd = pdev->sh_info->aer_op.cmd;
+	pci_channel_state_t state = 
+		(pci_channel_state_t)pdev->sh_info->aer_op.err;
+
+	/*If a pci_conf op is in progress, 
+		we have to wait until it is done before service aer op*/
+	dev_dbg(&pdev->xdev->dev, 
+		"pcifront service aer bus %x devfn %x\n", pdev->sh_info->aer_op.bus,
+		pdev->sh_info->aer_op.devfn);
+
+	pdev->sh_info->aer_op.err = pcifront_common_process(cmd, pdev, state);
+
+	wmb();
+	clear_bit(_XEN_PCIB_active, (unsigned long*)&pdev->sh_info->flags);
+	notify_remote_via_evtchn(pdev->evtchn);
+
+	/*in case of we lost an aer request in four lines time_window*/
+	smp_mb__before_clear_bit();
+	clear_bit( _PDEVB_op_active, &pdev->flags);
+	smp_mb__after_clear_bit();
+
+	schedule_pcifront_aer_op(pdev);
+
+}
+
+irqreturn_t pcifront_handler_aer(int irq, void *dev)
+{
+	struct pcifront_device *pdev = dev;
+	schedule_pcifront_aer_op(pdev);
+	return IRQ_HANDLED;
+}
+int pcifront_connect(struct pcifront_device *pdev)
+{
+	int err = 0;
+
+	spin_lock(&pcifront_dev_lock);
+
+	if (!pcifront_dev) {
+		dev_info(&pdev->xdev->dev, "Installing PCI frontend\n");
+		pcifront_dev = pdev;
+	}
+	else {
+		dev_err(&pdev->xdev->dev, "PCI frontend already installed!\n");
+		err = -EEXIST;
+	}
+
+	spin_unlock(&pcifront_dev_lock);
+
+	return err;
+}
+
+void pcifront_disconnect(struct pcifront_device *pdev)
+{
+	spin_lock(&pcifront_dev_lock);
+
+	if (pdev == pcifront_dev) {
+		dev_info(&pdev->xdev->dev,
+			 "Disconnecting PCI Frontend Buses\n");
+		pcifront_dev = NULL;
+	}
+
+	spin_unlock(&pcifront_dev_lock);
+}
+static struct pcifront_device *alloc_pdev(struct xenbus_device *xdev)
+{
+	struct pcifront_device *pdev;
+
+	pdev = kzalloc(sizeof(struct pcifront_device), GFP_KERNEL);
+	if (pdev == NULL)
+		goto out;
+
+	pdev->sh_info =
+	    (struct xen_pci_sharedinfo *)__get_free_page(GFP_KERNEL);
+	if (pdev->sh_info == NULL) {
+		kfree(pdev);
+		pdev = NULL;
+		goto out;
+	}
+	pdev->sh_info->flags = 0;
+
+	/*Flag for registering PV AER handler*/
+	set_bit(_XEN_PCIB_AERHANDLER, (void*)&pdev->sh_info->flags);
+
+	xdev->dev.driver_data = pdev;
+	pdev->xdev = xdev;
+
+	INIT_LIST_HEAD(&pdev->root_buses);
+
+	spin_lock_init(&pdev->dev_lock);
+	spin_lock_init(&pdev->sh_info_lock);
+
+	pdev->evtchn = INVALID_EVTCHN;
+	pdev->gnt_ref = INVALID_GRANT_REF;
+	pdev->irq = -1;
+
+	INIT_WORK(&pdev->op_work, pcifront_do_aer);
+
+	dev_dbg(&xdev->dev, "Allocated pdev @ 0x%p pdev->sh_info @ 0x%p\n",
+		pdev, pdev->sh_info);
+      out:
+	return pdev;
+}
+
+static void free_pdev(struct pcifront_device *pdev)
+{
+	dev_dbg(&pdev->xdev->dev, "freeing pdev @ 0x%p\n", pdev);
+
+	pcifront_free_roots(pdev);
+
+	/*For PCIE_AER error handling job*/
+	flush_scheduled_work();
+	unbind_from_irqhandler(pdev->irq, pdev);
+
+	if (pdev->evtchn != INVALID_EVTCHN)
+		xenbus_free_evtchn(pdev->xdev, pdev->evtchn);
+
+	if (pdev->gnt_ref != INVALID_GRANT_REF)
+		gnttab_end_foreign_access(pdev->gnt_ref, 0 /* r/w page */,
+					  (unsigned long)pdev->sh_info);
+
+	pdev->xdev->dev.driver_data = NULL;
+
+	kfree(pdev);
+}
+
+static int pcifront_publish_info(struct pcifront_device *pdev)
+{
+	int err = 0;
+	struct xenbus_transaction trans;
+
+	err = xenbus_grant_ring(pdev->xdev, virt_to_mfn(pdev->sh_info));
+	if (err < 0)
+		goto out;
+
+	pdev->gnt_ref = err;
+
+	err = xenbus_alloc_evtchn(pdev->xdev, &pdev->evtchn);
+	if (err)
+		goto out;
+
+	err = bind_evtchn_to_irqhandler(pdev->evtchn, pcifront_handler_aer, 
+		0, "pcifront", pdev); 
+	if (err < 0) {
+		xenbus_free_evtchn(pdev->xdev, pdev->evtchn);
+		xenbus_dev_fatal(pdev->xdev, err, "Failed to bind evtchn to "
+				 "irqhandler.\n");
+		return err;
+	}
+	pdev->irq = err;
+
+      do_publish:
+	err = xenbus_transaction_start(&trans);
+	if (err) {
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error writing configuration for backend "
+				 "(start transaction)");
+		goto out;
+	}
+
+	err = xenbus_printf(trans, pdev->xdev->nodename,
+			    "pci-op-ref", "%u", pdev->gnt_ref);
+	if (!err)
+		err = xenbus_printf(trans, pdev->xdev->nodename,
+				    "event-channel", "%u", pdev->evtchn);
+	if (!err)
+		err = xenbus_printf(trans, pdev->xdev->nodename,
+				    "magic", XEN_PCI_MAGIC);
+
+	if (err) {
+		xenbus_transaction_end(trans, 1);
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error writing configuration for backend");
+		goto out;
+	} else {
+		err = xenbus_transaction_end(trans, 0);
+		if (err == -EAGAIN)
+			goto do_publish;
+		else if (err) {
+			xenbus_dev_fatal(pdev->xdev, err,
+					 "Error completing transaction "
+					 "for backend");
+			goto out;
+		}
+	}
+
+	xenbus_switch_state(pdev->xdev, XenbusStateInitialised);
+
+	dev_dbg(&pdev->xdev->dev, "publishing successful!\n");
+
+      out:
+	return err;
+}
+
+static int __devinit pcifront_try_connect(struct pcifront_device *pdev)
+{
+	int err = -EFAULT;
+	int i, num_roots, len;
+	char str[64];
+	unsigned int domain, bus;
+
+	spin_lock(&pdev->dev_lock);
+
+	/* Only connect once */
+	if (xenbus_read_driver_state(pdev->xdev->nodename) !=
+	    XenbusStateInitialised)
+		goto out;
+
+	err = pcifront_connect(pdev);
+	if (err) {
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error connecting PCI Frontend");
+		goto out;
+	}
+
+	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
+			   "root_num", "%d", &num_roots);
+	if (err == -ENOENT) {
+		xenbus_dev_error(pdev->xdev, err,
+				 "No PCI Roots found, trying 0000:00");
+		err = pcifront_scan_root(pdev, 0, 0);
+		num_roots = 0;
+	} else if (err != 1) {
+		if (err == 0)
+			err = -EINVAL;
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error reading number of PCI roots");
+		goto out;
+	}
+
+	for (i = 0; i < num_roots; i++) {
+		len = snprintf(str, sizeof(str), "root-%d", i);
+		if (unlikely(len >= (sizeof(str) - 1))) {
+			err = -ENOMEM;
+			goto out;
+		}
+
+		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
+				   "%x:%x", &domain, &bus);
+		if (err != 2) {
+			if (err >= 0)
+				err = -EINVAL;
+			xenbus_dev_fatal(pdev->xdev, err,
+					 "Error reading PCI root %d", i);
+			goto out;
+		}
+
+		err = pcifront_scan_root(pdev, domain, bus);
+		if (err) {
+			xenbus_dev_fatal(pdev->xdev, err,
+					 "Error scanning PCI root %04x:%02x",
+					 domain, bus);
+			goto out;
+		}
+	}
+
+	err = xenbus_switch_state(pdev->xdev, XenbusStateConnected);
+	if (err)
+		goto out;
+
+      out:
+	spin_unlock(&pdev->dev_lock);
+	return err;
+}
+
+static int pcifront_try_disconnect(struct pcifront_device *pdev)
+{
+	int err = 0;
+	enum xenbus_state prev_state;
+
+	spin_lock(&pdev->dev_lock);
+
+	prev_state = xenbus_read_driver_state(pdev->xdev->nodename);
+
+	if (prev_state >= XenbusStateClosing)
+		goto out;
+
+	if(prev_state == XenbusStateConnected) {
+		pcifront_free_roots(pdev);
+		pcifront_disconnect(pdev);
+	}
+
+	err = xenbus_switch_state(pdev->xdev, XenbusStateClosed);
+
+      out:
+	spin_unlock(&pdev->dev_lock);
+
+	return err;
+}
+
+static int __devinit pcifront_attach_devices(struct pcifront_device *pdev)
+{
+	int err = -EFAULT;
+	int i, num_roots, len;
+	unsigned int domain, bus;
+	char str[64];
+
+	spin_lock(&pdev->dev_lock);
+
+	if (xenbus_read_driver_state(pdev->xdev->nodename) !=
+	    XenbusStateReconfiguring)
+		goto out;
+
+	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
+			   "root_num", "%d", &num_roots);
+	if (err == -ENOENT) {
+		xenbus_dev_error(pdev->xdev, err,
+				 "No PCI Roots found, trying 0000:00");
+		err = pcifront_rescan_root(pdev, 0, 0);
+		num_roots = 0;
+	} else if (err != 1) {
+		if (err == 0)
+			err = -EINVAL;
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error reading number of PCI roots");
+		goto out;
+	}
+
+	for (i = 0; i < num_roots; i++) {
+		len = snprintf(str, sizeof(str), "root-%d", i);
+		if (unlikely(len >= (sizeof(str) - 1))) {
+			err = -ENOMEM;
+			goto out;
+		}
+
+		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
+				   "%x:%x", &domain, &bus);
+		if (err != 2) {
+			if (err >= 0)
+				err = -EINVAL;
+			xenbus_dev_fatal(pdev->xdev, err,
+					 "Error reading PCI root %d", i);
+			goto out;
+		}
+
+		err = pcifront_rescan_root(pdev, domain, bus);
+		if (err) {
+			xenbus_dev_fatal(pdev->xdev, err,
+					 "Error scanning PCI root %04x:%02x",
+					 domain, bus);
+			goto out;
+		}
+	}
+
+	xenbus_switch_state(pdev->xdev, XenbusStateConnected);
+
+      out:
+	spin_unlock(&pdev->dev_lock);
+	return err;
+}
+
+static int pcifront_detach_devices(struct pcifront_device *pdev)
+{
+	int err = 0;
+	int i, num_devs;
+	unsigned int domain, bus, slot, func;
+	struct pci_bus *pci_bus;
+	struct pci_dev *pci_dev;
+	char str[64];
+
+	spin_lock(&pdev->dev_lock);
+
+	if (xenbus_read_driver_state(pdev->xdev->nodename) !=
+	    XenbusStateConnected)
+		goto out;
+
+	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, "num_devs", "%d",
+			   &num_devs);
+	if (err != 1) {
+		if (err >= 0)
+			err = -EINVAL;
+		xenbus_dev_fatal(pdev->xdev, err,
+				 "Error reading number of PCI devices");
+		goto out;
+	}
+
+	/* Find devices being detached and remove them. */
+	for (i = 0; i < num_devs; i++) {
+		int l, state;
+		l = snprintf(str, sizeof(str), "state-%d", i);
+		if (unlikely(l >= (sizeof(str) - 1))) {
+			err = -ENOMEM;
+			goto out;
+		}
+		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str, "%d",
+				   &state);
+		if (err != 1)
+			state = XenbusStateUnknown;
+
+		if (state != XenbusStateClosing)
+			continue;
+
+		/* Remove device. */
+		l = snprintf(str, sizeof(str), "vdev-%d", i);
+		if (unlikely(l >= (sizeof(str) - 1))) {
+			err = -ENOMEM;
+			goto out;
+		}
+		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
+			   	   "%x:%x:%x.%x", &domain, &bus, &slot, &func);
+		if (err != 4) {
+			if (err >= 0)
+				err = -EINVAL;
+			xenbus_dev_fatal(pdev->xdev, err,
+				 	 "Error reading PCI device %d", i);
+			goto out;
+		}
+
+		pci_bus = pci_find_bus(domain, bus);
+		if(!pci_bus) {
+			dev_dbg(&pdev->xdev->dev, "Cannot get bus %04x:%02x\n",
+				domain, bus);
+			continue;
+		}
+		pci_dev = pci_get_slot(pci_bus, PCI_DEVFN(slot, func));
+		if(!pci_dev) {
+			dev_dbg(&pdev->xdev->dev,
+				"Cannot get PCI device %04x:%02x:%02x.%02x\n",
+				domain, bus, slot, func);
+			continue;
+		}
+		pci_remove_bus_device(pci_dev);
+		pci_dev_put(pci_dev);
+
+		dev_dbg(&pdev->xdev->dev,
+			"PCI device %04x:%02x:%02x.%02x removed.\n",
+			domain, bus, slot, func);
+	}
+
+	err = xenbus_switch_state(pdev->xdev, XenbusStateReconfiguring);
+
+      out:
+	spin_unlock(&pdev->dev_lock);
+	return err;
+}
+
+static void __init_refok pcifront_backend_changed(struct xenbus_device *xdev,
+						  enum xenbus_state be_state)
+{
+	struct pcifront_device *pdev = xdev->dev.driver_data;
+
+	switch (be_state) {
+	case XenbusStateUnknown:
+	case XenbusStateInitialising:
+	case XenbusStateInitWait:
+	case XenbusStateInitialised:
+	case XenbusStateClosed:
+		break;
+
+	case XenbusStateConnected:
+		pcifront_try_connect(pdev);
+		break;
+
+	case XenbusStateClosing:
+		dev_warn(&xdev->dev, "backend going away!\n");
+		pcifront_try_disconnect(pdev);
+		break;
+
+	case XenbusStateReconfiguring:
+		pcifront_detach_devices(pdev);
+		break;
+
+	case XenbusStateReconfigured:
+		pcifront_attach_devices(pdev);
+		break;
+	}
+}
+
+static int pcifront_xenbus_probe(struct xenbus_device *xdev,
+				 const struct xenbus_device_id *id)
+{
+	int err = 0;
+	struct pcifront_device *pdev = alloc_pdev(xdev);
+
+	if (pdev == NULL) {
+		err = -ENOMEM;
+		xenbus_dev_fatal(xdev, err,
+				 "Error allocating pcifront_device struct");
+		goto out;
+	}
+
+	err = pcifront_publish_info(pdev);
+
+      out:
+	return err;
+}
+
+static int pcifront_xenbus_remove(struct xenbus_device *xdev)
+{
+	if (xdev->dev.driver_data)
+		free_pdev(xdev->dev.driver_data);
+
+	return 0;
+}
+
+static const struct xenbus_device_id xenpci_ids[] = {
+	{"pci"},
+	{{0}},
+};
+MODULE_ALIAS("xen:pci");
+
+static struct xenbus_driver xenbus_pcifront_driver = {
+	.name 			= "pcifront",
+	.owner 			= THIS_MODULE,
+	.ids 			= xenpci_ids,
+	.probe 			= pcifront_xenbus_probe,
+	.remove 		= pcifront_xenbus_remove,
+	.otherend_changed 	= pcifront_backend_changed,
+};
+
+static int __init pcifront_init(void)
+{
+	if (!xen_domain())
+		return -ENODEV;
+
+	return xenbus_register_frontend(&xenbus_pcifront_driver);
+}
+
+/* Initialize after the Xen PCI Frontend Stub is initialized */
+subsys_initcall(pcifront_init);
diff --git a/drivers/xen/Kconfig b/drivers/xen/Kconfig
index d4a33b6..88eb94c 100644
--- a/drivers/xen/Kconfig
+++ b/drivers/xen/Kconfig
@@ -174,20 +174,3 @@ config ACPI_PROCESSOR_XEN
 	depends on XEN_DOM0 && ACPI_PROCESSOR && CPU_FREQ
 	default y
 
-config XEN_PCIDEV_FRONTEND
-        bool "Xen PCI Frontend"
-        depends on PCI && X86_64
-        select HOTPLUG
-        default y
-        help
-          The PCI device frontend driver allows the kernel to import arbitrary
-          PCI devices from a PCI backend to support PCI driver domains.
-
-config XEN_PCIDEV_FE_DEBUG
-        bool "Xen PCI Frontend Debugging"
-        depends on XEN_PCIDEV_FRONTEND
-        default n
-        help
-          Enables some debug statements within the PCI Frontend.
-
-
diff --git a/drivers/xen/Makefile b/drivers/xen/Makefile
index 0bba5f2..cddfffb 100644
--- a/drivers/xen/Makefile
+++ b/drivers/xen/Makefile
@@ -13,7 +13,6 @@ obj-$(CONFIG_XEN_GNTDEV)		+= gntdev.o
 obj-$(CONFIG_XEN_BLKDEV_BACKEND)	+= blkback/
 obj-$(CONFIG_XEN_NETDEV_BACKEND)	+= netback/
 obj-$(CONFIG_XEN_PCIDEV_BACKEND)	+= pciback/
-obj-$(CONFIG_XEN_PCIDEV_FRONTEND)       += pcifront/
 obj-$(CONFIG_XENFS)			+= xenfs/
 obj-$(CONFIG_XEN_SYS_HYPERVISOR)	+= sys-hypervisor.o
 obj-$(CONFIG_XEN_S3)			+= acpi.o
diff --git a/drivers/xen/pcifront/Makefile b/drivers/xen/pcifront/Makefile
deleted file mode 100644
index d49136e..0000000
--- a/drivers/xen/pcifront/Makefile
+++ /dev/null
@@ -1,7 +0,0 @@
-obj-y += pcifront.o
-
-pcifront-y := xenbus.o
-
-ifeq ($(CONFIG_XEN_PCIDEV_FE_DEBUG),y)
-EXTRA_CFLAGS += -DDEBUG
-endif
diff --git a/drivers/xen/pcifront/xenbus.c b/drivers/xen/pcifront/xenbus.c
deleted file mode 100644
index 854850d..0000000
--- a/drivers/xen/pcifront/xenbus.c
+++ /dev/null
@@ -1,1117 +0,0 @@
-/*
- * PCI Frontend Xenbus Setup - handles setup with backend (imports page/evtchn)
- *
- *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
- */
-#include <linux/module.h>
-#include <linux/init.h>
-#include <linux/mm.h>
-#include <xen/xenbus.h>
-#include <xen/events.h>
-#include <xen/grant_table.h>
-#include <xen/page.h>
-#include <linux/spinlock.h>
-#include <linux/pci.h>
-#include <xen/xenbus.h>
-#include <xen/interface/io/pciif.h>
-#include <linux/interrupt.h>
-#include <asm/atomic.h>
-#include <linux/workqueue.h>
-#include <asm/bitops.h>
-#include <linux/time.h>
-
-
-#ifndef __init_refok
-#define __init_refok
-#endif
-
-#define INVALID_GRANT_REF (0)
-#define INVALID_EVTCHN    (-1)
-
-
-struct pci_bus_entry {
-	struct list_head list;
-	struct pci_bus *bus;
-};
-
-#define _PDEVB_op_active		(0)
-#define PDEVB_op_active 		(1 << (_PDEVB_op_active))
-
-struct pcifront_device {
-	struct xenbus_device *xdev;
-	struct list_head root_buses;
-	spinlock_t dev_lock;
-
-	int evtchn;
-	int gnt_ref;
-
-	int irq;
-
-	/* Lock this when doing any operations in sh_info */
-	spinlock_t sh_info_lock;
-	struct xen_pci_sharedinfo *sh_info;
-	struct work_struct op_work;
-	unsigned long flags;
-
-};
-
-struct pcifront_sd {
-	int domain;
-	struct pcifront_device *pdev;
-};
-
-static inline struct pcifront_device *
-pcifront_get_pdev(struct pcifront_sd *sd)
-{
-	return sd->pdev;
-}
-
-static inline void pcifront_init_sd(struct pcifront_sd *sd,
-				    unsigned int domain, unsigned int bus,
-				    struct pcifront_device *pdev)
-{
-	sd->domain = domain;
-	sd->pdev = pdev;
-}
-
-static inline void pcifront_setup_root_resources(struct pci_bus *bus,
-						 struct pcifront_sd *sd)
-{
-}
-
-
-DEFINE_SPINLOCK(pcifront_dev_lock);
-static struct pcifront_device *pcifront_dev = NULL;
-
-static int verbose_request = 0;
-module_param(verbose_request, int, 0644);
-
-static int errno_to_pcibios_err(int errno)
-{
-	switch (errno) {
-	case XEN_PCI_ERR_success:
-		return PCIBIOS_SUCCESSFUL;
-
-	case XEN_PCI_ERR_dev_not_found:
-		return PCIBIOS_DEVICE_NOT_FOUND;
-
-	case XEN_PCI_ERR_invalid_offset:
-	case XEN_PCI_ERR_op_failed:
-		return PCIBIOS_BAD_REGISTER_NUMBER;
-
-	case XEN_PCI_ERR_not_implemented:
-		return PCIBIOS_FUNC_NOT_SUPPORTED;
-
-	case XEN_PCI_ERR_access_denied:
-		return PCIBIOS_SET_FAILED;
-	}
-	return errno;
-}
-
-static inline void schedule_pcifront_aer_op(struct pcifront_device *pdev)
-{
-	if (test_bit(_XEN_PCIB_active, (unsigned long *)&pdev->sh_info->flags)
-		&& !test_and_set_bit(_PDEVB_op_active, &pdev->flags)) {
-		dev_dbg(&pdev->xdev->dev, "schedule aer frontend job\n");
-		schedule_work(&pdev->op_work);
-	}
-}
-
-static int do_pci_op(struct pcifront_device *pdev, struct xen_pci_op *op)
-{
-	int err = 0;
-	struct xen_pci_op *active_op = &pdev->sh_info->op;
-	unsigned long irq_flags;
-	evtchn_port_t port = pdev->evtchn;
-	unsigned irq = pdev->irq;
-	s64 ns, ns_timeout;
-	struct timeval tv;
-
-	spin_lock_irqsave(&pdev->sh_info_lock, irq_flags);
-
-	memcpy(active_op, op, sizeof(struct xen_pci_op));
-
-	/* Go */
-	wmb();
-	set_bit(_XEN_PCIF_active, (unsigned long *)&pdev->sh_info->flags);
-	notify_remote_via_evtchn(port);
-
-	/*
-	 * We set a poll timeout of 3 seconds but give up on return after
-	 * 2 seconds. It is better to time out too late rather than too early
-	 * (in the latter case we end up continually re-executing poll() with a
-	 * timeout in the past). 1s difference gives plenty of slack for error.
-	 */
-	do_gettimeofday(&tv);
-	ns_timeout = timeval_to_ns(&tv) + 2 * (s64)NSEC_PER_SEC;
-
-	xen_clear_irq_pending(irq);
-
-	while (test_bit(_XEN_PCIF_active,
-			(unsigned long *)&pdev->sh_info->flags)) {
-		xen_poll_irq_timeout(irq, jiffies + 3*HZ);
-		xen_clear_irq_pending(irq);
-		do_gettimeofday(&tv);
-		ns = timeval_to_ns(&tv);
-		if (ns > ns_timeout) {
-			dev_err(&pdev->xdev->dev,
-				"pciback not responding!!!\n");
-			clear_bit(_XEN_PCIF_active,
-				  (unsigned long *)&pdev->sh_info->flags);
-			err = XEN_PCI_ERR_dev_not_found;
-			goto out;
-		}
-	}
-
-	/*
-	* We might lose backend service request since we 
-	* reuse same evtchn with pci_conf backend response. So re-schedule
-	* aer pcifront service.
-	*/
-	if (test_bit(_XEN_PCIB_active, 
-			(unsigned long*)&pdev->sh_info->flags)) {
-		dev_err(&pdev->xdev->dev, 
-			"schedule aer pcifront service\n");
-		schedule_pcifront_aer_op(pdev);
-	}
-
-	memcpy(op, active_op, sizeof(struct xen_pci_op));
-
-	err = op->err;
-      out:
-	spin_unlock_irqrestore(&pdev->sh_info_lock, irq_flags);
-	return err;
-}
-
-/* Access to this function is spinlocked in drivers/pci/access.c */
-static int pcifront_bus_read(struct pci_bus *bus, unsigned int devfn,
-			     int where, int size, u32 * val)
-{
-	int err = 0;
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_conf_read,
-		.domain = pci_domain_nr(bus),
-		.bus    = bus->number,
-		.devfn  = devfn,
-		.offset = where,
-		.size   = size,
-	};
-	struct pcifront_sd *sd = bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	if (verbose_request)
-		dev_info(&pdev->xdev->dev,
-			 "read dev=%04x:%02x:%02x.%01x - offset %x size %d\n",
-			 pci_domain_nr(bus), bus->number, PCI_SLOT(devfn),
-			 PCI_FUNC(devfn), where, size);
-
-	err = do_pci_op(pdev, &op);
-
-	if (likely(!err)) {
-		if (verbose_request)
-			dev_info(&pdev->xdev->dev, "read got back value %x\n",
-				 op.value);
-
-		*val = op.value;
-	} else if (err == -ENODEV) {
-		/* No device here, pretend that it just returned 0 */
-		err = 0;
-		*val = 0;
-	}
-
-	return errno_to_pcibios_err(err);
-}
-
-/* Access to this function is spinlocked in drivers/pci/access.c */
-static int pcifront_bus_write(struct pci_bus *bus, unsigned int devfn,
-			      int where, int size, u32 val)
-{
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_conf_write,
-		.domain = pci_domain_nr(bus),
-		.bus    = bus->number,
-		.devfn  = devfn,
-		.offset = where,
-		.size   = size,
-		.value  = val,
-	};
-	struct pcifront_sd *sd = bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	if (verbose_request)
-		dev_info(&pdev->xdev->dev,
-			 "write dev=%04x:%02x:%02x.%01x - "
-			 "offset %x size %d val %x\n",
-			 pci_domain_nr(bus), bus->number,
-			 PCI_SLOT(devfn), PCI_FUNC(devfn), where, size, val);
-
-	return errno_to_pcibios_err(do_pci_op(pdev, &op));
-}
-
-struct pci_ops pcifront_bus_ops = {
-	.read = pcifront_bus_read,
-	.write = pcifront_bus_write,
-};
-
-#ifdef CONFIG_PCI_MSI
-int pci_frontend_enable_msix(struct pci_dev *dev,
-		struct msix_entry *entries,
-		int nvec)
-{
-	int err;
-	int i;
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_enable_msix,
-		.domain = pci_domain_nr(dev->bus),
-		.bus = dev->bus->number,
-		.devfn = dev->devfn,
-		.value = nvec,
-	};
-	struct pcifront_sd *sd = dev->bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	if (nvec > SH_INFO_MAX_VEC) {
-		printk("too much vector for pci frontend%x\n", nvec);
-		return -EINVAL;
-	}
-
-	for (i = 0; i < nvec; i++) {
-		op.msix_entries[i].entry = entries[i].entry;
-		op.msix_entries[i].vector = entries[i].vector;
-	}
-
-	err = do_pci_op(pdev, &op);
-
-	if (!err) {
-		if (!op.value) {
-			/* we get the result */
-			for ( i = 0; i < nvec; i++)
-				entries[i].vector = op.msix_entries[i].vector;
-			return 0;
-		}
-		else {
-            printk("enable msix get value %x\n", op.value);
-			return op.value;
-		}
-	}
-	else {
-        printk("enable msix get err %x\n", err);
-		return err;
-	}
-}
-
-void pci_frontend_disable_msix(struct pci_dev* dev)
-{
-	int err;
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_disable_msix,
-		.domain = pci_domain_nr(dev->bus),
-		.bus = dev->bus->number,
-		.devfn = dev->devfn,
-	};
-	struct pcifront_sd *sd = dev->bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	err = do_pci_op(pdev, &op);
-
-	/* What should do for error ? */
-	if (err)
-		printk("pci_disable_msix get err %x\n", err);
-}
-
-int pci_frontend_enable_msi(struct pci_dev *dev)
-{
-	int err;
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_enable_msi,
-		.domain = pci_domain_nr(dev->bus),
-		.bus = dev->bus->number,
-		.devfn = dev->devfn,
-	};
-	struct pcifront_sd *sd = dev->bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	err = do_pci_op(pdev, &op);
-	if (likely(!err)) {
-		dev->irq = op.value;
-	}
-	else {
-		printk("pci frontend enable msi failed for dev %x:%x \n",
-				op.bus, op.devfn);
-		err = -EINVAL;
-	}
-	return err;
-}
-
-void pci_frontend_disable_msi(struct pci_dev* dev)
-{
-	int err;
-	struct xen_pci_op op = {
-		.cmd    = XEN_PCI_OP_disable_msi,
-		.domain = pci_domain_nr(dev->bus),
-		.bus = dev->bus->number,
-		.devfn = dev->devfn,
-	};
-	struct pcifront_sd *sd = dev->bus->sysdata;
-	struct pcifront_device *pdev = pcifront_get_pdev(sd);
-
-	err = do_pci_op(pdev, &op);
-	if (err == XEN_PCI_ERR_dev_not_found) {
-		/* XXX No response from backend, what shall we do? */
-		printk("get no response from backend for disable MSI\n");
-		return;
-	}
-	if (likely(!err))
-		dev->irq = op.value;
-	else
-		/* how can pciback notify us fail? */
-		printk("get fake response frombackend \n");
-}
-#endif /* CONFIG_PCI_MSI */
-
-/* Claim resources for the PCI frontend as-is, backend won't allow changes */
-static int pcifront_claim_resource(struct pci_dev *dev, void *data)
-{
-	struct pcifront_device *pdev = data;
-	int i;
-	struct resource *r;
-
-	for (i = 0; i < PCI_NUM_RESOURCES; i++) {
-		r = &dev->resource[i];
-
-		if (!r->parent && r->start && r->flags) {
-			dev_dbg(&pdev->xdev->dev, "claiming resource %s/%d\n",
-				pci_name(dev), i);
-			pci_claim_resource(dev, i);
-		}
-	}
-
-	return 0;
-}
-
-int __devinit pcifront_scan_root(struct pcifront_device *pdev,
-				 unsigned int domain, unsigned int bus)
-{
-	struct pci_bus *b;
-	struct pcifront_sd *sd = NULL;
-	struct pci_bus_entry *bus_entry = NULL;
-	int err = 0;
-
-#ifndef CONFIG_PCI_DOMAINS
-	if (domain != 0) {
-		dev_err(&pdev->xdev->dev,
-			"PCI Root in non-zero PCI Domain! domain=%d\n", domain);
-		dev_err(&pdev->xdev->dev,
-			"Please compile with CONFIG_PCI_DOMAINS\n");
-		err = -EINVAL;
-		goto err_out;
-	}
-#endif
-
-	dev_info(&pdev->xdev->dev, "Creating PCI Frontend Bus %04x:%02x\n",
-		 domain, bus);
-
-	bus_entry = kmalloc(sizeof(*bus_entry), GFP_KERNEL);
-	sd = kmalloc(sizeof(*sd), GFP_KERNEL);
-	if (!bus_entry || !sd) {
-		err = -ENOMEM;
-		goto err_out;
-	}
-	pcifront_init_sd(sd, domain, bus, pdev);
-
-	b = pci_scan_bus_parented(&pdev->xdev->dev, bus,
-				  &pcifront_bus_ops, sd);
-	if (!b) {
-		dev_err(&pdev->xdev->dev,
-			"Error creating PCI Frontend Bus!\n");
-		err = -ENOMEM;
-		goto err_out;
-	}
-
-	pcifront_setup_root_resources(b, sd);
-	bus_entry->bus = b;
-
-	list_add(&bus_entry->list, &pdev->root_buses);
-
-	/* Claim resources before going "live" with our devices */
-	pci_walk_bus(b, pcifront_claim_resource, pdev);
-
-	pci_bus_add_devices(b);
-
-	return 0;
-
-      err_out:
-	kfree(bus_entry);
-	kfree(sd);
-
-	return err;
-}
-
-int __devinit pcifront_rescan_root(struct pcifront_device *pdev,
-				   unsigned int domain, unsigned int bus)
-{
-	struct pci_bus *b;
-	struct pci_dev *d;
-	unsigned int devfn;
-	int err;
-
-#ifndef CONFIG_PCI_DOMAINS
-	if (domain != 0) {
-		dev_err(&pdev->xdev->dev,
-			"PCI Root in non-zero PCI Domain! domain=%d\n", domain);
-		dev_err(&pdev->xdev->dev,
-			"Please compile with CONFIG_PCI_DOMAINS\n");
-		return -EINVAL;
-	}
-#endif
-
-	dev_info(&pdev->xdev->dev, "Rescanning PCI Frontend Bus %04x:%02x\n",
-		 domain, bus);
-
-	b = pci_find_bus(domain, bus);
-	if(!b)
-		/* If the bus is unknown, create it. */
-		return pcifront_scan_root(pdev, domain, bus);
-
-	/* Rescan the bus for newly attached functions and add.
-	 * We omit handling of PCI bridge attachment because pciback prevents
-	 * bridges from being exported.
-	 */ 
-	for (devfn = 0; devfn < 0x100; devfn++) {
-		d = pci_get_slot(b, devfn);
-		if(d) {
-			/* Device is already known. */
-			pci_dev_put(d);
-			continue;
-		}
-
-		d = pci_scan_single_device(b, devfn);
-		if (d) {
-			dev_info(&pdev->xdev->dev, "New device on "
-				 "%04x:%02x:%02x.%02x found.\n", domain, bus,
-				 PCI_SLOT(devfn), PCI_FUNC(devfn));
-			err = pci_bus_add_device(d);
-			if (err) {
-				dev_err(&pdev->xdev->dev, "Failed to add "
-				" device to bus.\n");
-				return err;
-			}
-			
-		}
-	}
-
-	return 0;
-}
-
-static void free_root_bus_devs(struct pci_bus *bus)
-{
-	struct pci_dev *dev;
-
-	while (!list_empty(&bus->devices)) {
-		dev = container_of(bus->devices.next, struct pci_dev,
-				   bus_list);
-		dev_dbg(&dev->dev, "removing device\n");
-		pci_remove_bus_device(dev);
-	}
-}
-
-void pcifront_free_roots(struct pcifront_device *pdev)
-{
-	struct pci_bus_entry *bus_entry, *t;
-
-	dev_dbg(&pdev->xdev->dev, "cleaning up root buses\n");
-
-	list_for_each_entry_safe(bus_entry, t, &pdev->root_buses, list) {
-		list_del(&bus_entry->list);
-
-		free_root_bus_devs(bus_entry->bus);
-
-		kfree(bus_entry->bus->sysdata);
-
-		device_unregister(bus_entry->bus->bridge);
-		pci_remove_bus(bus_entry->bus);
-
-		kfree(bus_entry);
-	}
-}
-
-static pci_ers_result_t pcifront_common_process( int cmd, struct pcifront_device *pdev,
-	pci_channel_state_t state)
-{
-	pci_ers_result_t result;
-	struct pci_driver *pdrv;
-	int bus = pdev->sh_info->aer_op.bus;
-	int devfn = pdev->sh_info->aer_op.devfn;
-	struct pci_dev *pcidev;
-	int flag = 0;
-
-	dev_dbg(&pdev->xdev->dev, 
-		"pcifront AER process: cmd %x (bus:%x, devfn%x)",
-		cmd, bus, devfn);
-	result = PCI_ERS_RESULT_NONE;
-
-	pcidev = pci_get_bus_and_slot(bus, devfn);
-	if (!pcidev || !pcidev->driver){
-		dev_err(&pcidev->dev, 
-			"device or driver is NULL\n");
-		return result;
-	}
-	pdrv = pcidev->driver;
-
-	if (get_driver(&pdrv->driver)) {
-		if (pdrv->err_handler && pdrv->err_handler->error_detected) {
-			dev_dbg(&pcidev->dev,
-				"trying to call AER service\n");
-			if (pcidev) {
-				flag = 1;
-				switch(cmd) {
-				case XEN_PCI_OP_aer_detected:
-					result = pdrv->err_handler->error_detected(pcidev, state);
-					break;
-				case XEN_PCI_OP_aer_mmio:
-					result = pdrv->err_handler->mmio_enabled(pcidev);
-					break;
-				case XEN_PCI_OP_aer_slotreset:
-					result = pdrv->err_handler->slot_reset(pcidev);
-					break;
-				case XEN_PCI_OP_aer_resume:
-					pdrv->err_handler->resume(pcidev);
-					break;
-				default:
-					dev_err(&pdev->xdev->dev,
-						"bad request in aer recovery operation!\n");
-
-				}
-			}
-		}
-		put_driver(&pdrv->driver);
-	}
-	if (!flag)
-		result = PCI_ERS_RESULT_NONE;
-
-	return result;
-}
-
-
-void pcifront_do_aer(struct work_struct *data)
-{
-	struct pcifront_device *pdev = container_of(data, struct pcifront_device, op_work);
-	int cmd = pdev->sh_info->aer_op.cmd;
-	pci_channel_state_t state = 
-		(pci_channel_state_t)pdev->sh_info->aer_op.err;
-
-	/*If a pci_conf op is in progress, 
-		we have to wait until it is done before service aer op*/
-	dev_dbg(&pdev->xdev->dev, 
-		"pcifront service aer bus %x devfn %x\n", pdev->sh_info->aer_op.bus,
-		pdev->sh_info->aer_op.devfn);
-
-	pdev->sh_info->aer_op.err = pcifront_common_process(cmd, pdev, state);
-
-	wmb();
-	clear_bit(_XEN_PCIB_active, (unsigned long*)&pdev->sh_info->flags);
-	notify_remote_via_evtchn(pdev->evtchn);
-
-	/*in case of we lost an aer request in four lines time_window*/
-	smp_mb__before_clear_bit();
-	clear_bit( _PDEVB_op_active, &pdev->flags);
-	smp_mb__after_clear_bit();
-
-	schedule_pcifront_aer_op(pdev);
-
-}
-
-irqreturn_t pcifront_handler_aer(int irq, void *dev)
-{
-	struct pcifront_device *pdev = dev;
-	schedule_pcifront_aer_op(pdev);
-	return IRQ_HANDLED;
-}
-int pcifront_connect(struct pcifront_device *pdev)
-{
-	int err = 0;
-
-	spin_lock(&pcifront_dev_lock);
-
-	if (!pcifront_dev) {
-		dev_info(&pdev->xdev->dev, "Installing PCI frontend\n");
-		pcifront_dev = pdev;
-	}
-	else {
-		dev_err(&pdev->xdev->dev, "PCI frontend already installed!\n");
-		err = -EEXIST;
-	}
-
-	spin_unlock(&pcifront_dev_lock);
-
-	return err;
-}
-
-void pcifront_disconnect(struct pcifront_device *pdev)
-{
-	spin_lock(&pcifront_dev_lock);
-
-	if (pdev == pcifront_dev) {
-		dev_info(&pdev->xdev->dev,
-			 "Disconnecting PCI Frontend Buses\n");
-		pcifront_dev = NULL;
-	}
-
-	spin_unlock(&pcifront_dev_lock);
-}
-static struct pcifront_device *alloc_pdev(struct xenbus_device *xdev)
-{
-	struct pcifront_device *pdev;
-
-	pdev = kzalloc(sizeof(struct pcifront_device), GFP_KERNEL);
-	if (pdev == NULL)
-		goto out;
-
-	pdev->sh_info =
-	    (struct xen_pci_sharedinfo *)__get_free_page(GFP_KERNEL);
-	if (pdev->sh_info == NULL) {
-		kfree(pdev);
-		pdev = NULL;
-		goto out;
-	}
-	pdev->sh_info->flags = 0;
-
-	/*Flag for registering PV AER handler*/
-	set_bit(_XEN_PCIB_AERHANDLER, (void*)&pdev->sh_info->flags);
-
-	xdev->dev.driver_data = pdev;
-	pdev->xdev = xdev;
-
-	INIT_LIST_HEAD(&pdev->root_buses);
-
-	spin_lock_init(&pdev->dev_lock);
-	spin_lock_init(&pdev->sh_info_lock);
-
-	pdev->evtchn = INVALID_EVTCHN;
-	pdev->gnt_ref = INVALID_GRANT_REF;
-	pdev->irq = -1;
-
-	INIT_WORK(&pdev->op_work, pcifront_do_aer);
-
-	dev_dbg(&xdev->dev, "Allocated pdev @ 0x%p pdev->sh_info @ 0x%p\n",
-		pdev, pdev->sh_info);
-      out:
-	return pdev;
-}
-
-static void free_pdev(struct pcifront_device *pdev)
-{
-	dev_dbg(&pdev->xdev->dev, "freeing pdev @ 0x%p\n", pdev);
-
-	pcifront_free_roots(pdev);
-
-	/*For PCIE_AER error handling job*/
-	flush_scheduled_work();
-	unbind_from_irqhandler(pdev->irq, pdev);
-
-	if (pdev->evtchn != INVALID_EVTCHN)
-		xenbus_free_evtchn(pdev->xdev, pdev->evtchn);
-
-	if (pdev->gnt_ref != INVALID_GRANT_REF)
-		gnttab_end_foreign_access(pdev->gnt_ref, 0 /* r/w page */,
-					  (unsigned long)pdev->sh_info);
-
-	pdev->xdev->dev.driver_data = NULL;
-
-	kfree(pdev);
-}
-
-static int pcifront_publish_info(struct pcifront_device *pdev)
-{
-	int err = 0;
-	struct xenbus_transaction trans;
-
-	err = xenbus_grant_ring(pdev->xdev, virt_to_mfn(pdev->sh_info));
-	if (err < 0)
-		goto out;
-
-	pdev->gnt_ref = err;
-
-	err = xenbus_alloc_evtchn(pdev->xdev, &pdev->evtchn);
-	if (err)
-		goto out;
-
-	err = bind_evtchn_to_irqhandler(pdev->evtchn, pcifront_handler_aer, 
-		0, "pcifront", pdev); 
-	if (err < 0) {
-		xenbus_free_evtchn(pdev->xdev, pdev->evtchn);
-		xenbus_dev_fatal(pdev->xdev, err, "Failed to bind evtchn to "
-				 "irqhandler.\n");
-		return err;
-	}
-	pdev->irq = err;
-
-      do_publish:
-	err = xenbus_transaction_start(&trans);
-	if (err) {
-		xenbus_dev_fatal(pdev->xdev, err,
-				 "Error writing configuration for backend "
-				 "(start transaction)");
-		goto out;
-	}
-
-	err = xenbus_printf(trans, pdev->xdev->nodename,
-			    "pci-op-ref", "%u", pdev->gnt_ref);
-	if (!err)
-		err = xenbus_printf(trans, pdev->xdev->nodename,
-				    "event-channel", "%u", pdev->evtchn);
-	if (!err)
-		err = xenbus_printf(trans, pdev->xdev->nodename,
-				    "magic", XEN_PCI_MAGIC);
-
-	if (err) {
-		xenbus_transaction_end(trans, 1);
-		xenbus_dev_fatal(pdev->xdev, err,
-				 "Error writing configuration for backend");
-		goto out;
-	} else {
-		err = xenbus_transaction_end(trans, 0);
-		if (err == -EAGAIN)
-			goto do_publish;
-		else if (err) {
-			xenbus_dev_fatal(pdev->xdev, err,
-					 "Error completing transaction "
-					 "for backend");
-			goto out;
-		}
-	}
-
-	xenbus_switch_state(pdev->xdev, XenbusStateInitialised);
-
-	dev_dbg(&pdev->xdev->dev, "publishing successful!\n");
-
-      out:
-	return err;
-}
-
-static int __devinit pcifront_try_connect(struct pcifront_device *pdev)
-{
-	int err = -EFAULT;
-	int i, num_roots, len;
-	char str[64];
-	unsigned int domain, bus;
-
-	spin_lock(&pdev->dev_lock);
-
-	/* Only connect once */
-	if (xenbus_read_driver_state(pdev->xdev->nodename) !=
-	    XenbusStateInitialised)
-		goto out;
-
-	err = pcifront_connect(pdev);
-	if (err) {
-		xenbus_dev_fatal(pdev->xdev, err,
-				 "Error connecting PCI Frontend");
-		goto out;
-	}
-
-	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
-			   "root_num", "%d", &num_roots);
-	if (err == -ENOENT) {
-		xenbus_dev_error(pdev->xdev, err,
-				 "No PCI Roots found, trying 0000:00");
-		err = pcifront_scan_root(pdev, 0, 0);
-		num_roots = 0;
-	} else if (err != 1) {
-		if (err == 0)
-			err = -EINVAL;
-		xenbus_dev_fatal(pdev->xdev, err,
-				 "Error reading number of PCI roots");
-		goto out;
-	}
-
-	for (i = 0; i < num_roots; i++) {
-		len = snprintf(str, sizeof(str), "root-%d", i);
-		if (unlikely(len >= (sizeof(str) - 1))) {
-			err = -ENOMEM;
-			goto out;
-		}
-
-		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
-				   "%x:%x", &domain, &bus);
-		if (err != 2) {
-			if (err >= 0)
-				err = -EINVAL;
-			xenbus_dev_fatal(pdev->xdev, err,
-					 "Error reading PCI root %d", i);
-			goto out;
-		}
-
-		err = pcifront_scan_root(pdev, domain, bus);
-		if (err) {
-			xenbus_dev_fatal(pdev->xdev, err,
-					 "Error scanning PCI root %04x:%02x",
-					 domain, bus);
-			goto out;
-		}
-	}
-
-	err = xenbus_switch_state(pdev->xdev, XenbusStateConnected);
-	if (err)
-		goto out;
-
-      out:
-	spin_unlock(&pdev->dev_lock);
-	return err;
-}
-
-static int pcifront_try_disconnect(struct pcifront_device *pdev)
-{
-	int err = 0;
-	enum xenbus_state prev_state;
-
-	spin_lock(&pdev->dev_lock);
-
-	prev_state = xenbus_read_driver_state(pdev->xdev->nodename);
-
-	if (prev_state >= XenbusStateClosing)
-		goto out;
-
-	if(prev_state == XenbusStateConnected) {
-		pcifront_free_roots(pdev);
-		pcifront_disconnect(pdev);
-	}
-
-	err = xenbus_switch_state(pdev->xdev, XenbusStateClosed);
-
-      out:
-	spin_unlock(&pdev->dev_lock);
-
-	return err;
-}
-
-static int __devinit pcifront_attach_devices(struct pcifront_device *pdev)
-{
-	int err = -EFAULT;
-	int i, num_roots, len;
-	unsigned int domain, bus;
-	char str[64];
-
-	spin_lock(&pdev->dev_lock);
-
-	if (xenbus_read_driver_state(pdev->xdev->nodename) !=
-	    XenbusStateReconfiguring)
-		goto out;
-
-	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend,
-			   "root_num", "%d", &num_roots);
-	if (err == -ENOENT) {
-		xenbus_dev_error(pdev->xdev, err,
-				 "No PCI Roots found, trying 0000:00");
-		err = pcifront_rescan_root(pdev, 0, 0);
-		num_roots = 0;
-	} else if (err != 1) {
-		if (err == 0)
-			err = -EINVAL;
-		xenbus_dev_fatal(pdev->xdev, err,
-				 "Error reading number of PCI roots");
-		goto out;
-	}
-
-	for (i = 0; i < num_roots; i++) {
-		len = snprintf(str, sizeof(str), "root-%d", i);
-		if (unlikely(len >= (sizeof(str) - 1))) {
-			err = -ENOMEM;
-			goto out;
-		}
-
-		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
-				   "%x:%x", &domain, &bus);
-		if (err != 2) {
-			if (err >= 0)
-				err = -EINVAL;
-			xenbus_dev_fatal(pdev->xdev, err,
-					 "Error reading PCI root %d", i);
-			goto out;
-		}
-
-		err = pcifront_rescan_root(pdev, domain, bus);
-		if (err) {
-			xenbus_dev_fatal(pdev->xdev, err,
-					 "Error scanning PCI root %04x:%02x",
-					 domain, bus);
-			goto out;
-		}
-	}
-
-	xenbus_switch_state(pdev->xdev, XenbusStateConnected);
-
-      out:
-	spin_unlock(&pdev->dev_lock);
-	return err;
-}
-
-static int pcifront_detach_devices(struct pcifront_device *pdev)
-{
-	int err = 0;
-	int i, num_devs;
-	unsigned int domain, bus, slot, func;
-	struct pci_bus *pci_bus;
-	struct pci_dev *pci_dev;
-	char str[64];
-
-	spin_lock(&pdev->dev_lock);
-
-	if (xenbus_read_driver_state(pdev->xdev->nodename) !=
-	    XenbusStateConnected)
-		goto out;
-
-	err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, "num_devs", "%d",
-			   &num_devs);
-	if (err != 1) {
-		if (err >= 0)
-			err = -EINVAL;
-		xenbus_dev_fatal(pdev->xdev, err,
-				 "Error reading number of PCI devices");
-		goto out;
-	}
-
-	/* Find devices being detached and remove them. */
-	for (i = 0; i < num_devs; i++) {
-		int l, state;
-		l = snprintf(str, sizeof(str), "state-%d", i);
-		if (unlikely(l >= (sizeof(str) - 1))) {
-			err = -ENOMEM;
-			goto out;
-		}
-		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str, "%d",
-				   &state);
-		if (err != 1)
-			state = XenbusStateUnknown;
-
-		if (state != XenbusStateClosing)
-			continue;
-
-		/* Remove device. */
-		l = snprintf(str, sizeof(str), "vdev-%d", i);
-		if (unlikely(l >= (sizeof(str) - 1))) {
-			err = -ENOMEM;
-			goto out;
-		}
-		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
-			   	   "%x:%x:%x.%x", &domain, &bus, &slot, &func);
-		if (err != 4) {
-			if (err >= 0)
-				err = -EINVAL;
-			xenbus_dev_fatal(pdev->xdev, err,
-				 	 "Error reading PCI device %d", i);
-			goto out;
-		}
-
-		pci_bus = pci_find_bus(domain, bus);
-		if(!pci_bus) {
-			dev_dbg(&pdev->xdev->dev, "Cannot get bus %04x:%02x\n",
-				domain, bus);
-			continue;
-		}
-		pci_dev = pci_get_slot(pci_bus, PCI_DEVFN(slot, func));
-		if(!pci_dev) {
-			dev_dbg(&pdev->xdev->dev,
-				"Cannot get PCI device %04x:%02x:%02x.%02x\n",
-				domain, bus, slot, func);
-			continue;
-		}
-		pci_remove_bus_device(pci_dev);
-		pci_dev_put(pci_dev);
-
-		dev_dbg(&pdev->xdev->dev,
-			"PCI device %04x:%02x:%02x.%02x removed.\n",
-			domain, bus, slot, func);
-	}
-
-	err = xenbus_switch_state(pdev->xdev, XenbusStateReconfiguring);
-
-      out:
-	spin_unlock(&pdev->dev_lock);
-	return err;
-}
-
-static void __init_refok pcifront_backend_changed(struct xenbus_device *xdev,
-						  enum xenbus_state be_state)
-{
-	struct pcifront_device *pdev = xdev->dev.driver_data;
-
-	switch (be_state) {
-	case XenbusStateUnknown:
-	case XenbusStateInitialising:
-	case XenbusStateInitWait:
-	case XenbusStateInitialised:
-	case XenbusStateClosed:
-		break;
-
-	case XenbusStateConnected:
-		pcifront_try_connect(pdev);
-		break;
-
-	case XenbusStateClosing:
-		dev_warn(&xdev->dev, "backend going away!\n");
-		pcifront_try_disconnect(pdev);
-		break;
-
-	case XenbusStateReconfiguring:
-		pcifront_detach_devices(pdev);
-		break;
-
-	case XenbusStateReconfigured:
-		pcifront_attach_devices(pdev);
-		break;
-	}
-}
-
-static int pcifront_xenbus_probe(struct xenbus_device *xdev,
-				 const struct xenbus_device_id *id)
-{
-	int err = 0;
-	struct pcifront_device *pdev = alloc_pdev(xdev);
-
-	if (pdev == NULL) {
-		err = -ENOMEM;
-		xenbus_dev_fatal(xdev, err,
-				 "Error allocating pcifront_device struct");
-		goto out;
-	}
-
-	err = pcifront_publish_info(pdev);
-
-      out:
-	return err;
-}
-
-static int pcifront_xenbus_remove(struct xenbus_device *xdev)
-{
-	if (xdev->dev.driver_data)
-		free_pdev(xdev->dev.driver_data);
-
-	return 0;
-}
-
-static const struct xenbus_device_id xenpci_ids[] = {
-	{"pci"},
-	{{0}},
-};
-MODULE_ALIAS("xen:pci");
-
-static struct xenbus_driver xenbus_pcifront_driver = {
-	.name 			= "pcifront",
-	.owner 			= THIS_MODULE,
-	.ids 			= xenpci_ids,
-	.probe 			= pcifront_xenbus_probe,
-	.remove 		= pcifront_xenbus_remove,
-	.otherend_changed 	= pcifront_backend_changed,
-};
-
-static int __init pcifront_init(void)
-{
-	if (!xen_domain())
-		return -ENODEV;
-
-	return xenbus_register_frontend(&xenbus_pcifront_driver);
-}
-
-/* Initialize after the Xen PCI Frontend Stub is initialized */
-subsys_initcall(pcifront_init);
-- 
1.6.2.5

[PATCH 25/31] Change the boot-order of initialising the PCI frontend.

[Konrad Rzeszutek Wilk]

In the past PCI front-end would be booted as part of the subsystem.
That was OK as the XenBus was started at that point as well.
But with the recent kernels XenBus is initialized much later and the
the registration of the PCI bus would miserably fail (BUG). This
fixes that issue.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/pci/Kconfig        |    7 ++++---
 drivers/pci/xen-pcifront.c |   13 ++++++++++---
 2 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index 97a59d5..3a858a8 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -52,10 +52,11 @@ config PCI_STUB
 	  When in doubt, say N.
 
 config XEN_PCIDEV_FRONTEND
-        bool "Xen PCI Frontend"
-        depends on PCI && X86_64
+        tristate "Xen PCI Frontend"
+        depends on XEN && PCI && X86_64
         select HOTPLUG
-        default y
+	select XEN_XENBUS_FRONTEND
+	default y
         help
           The PCI device frontend driver allows the kernel to import arbitrary
           PCI devices from a PCI backend to support PCI driver domains.
diff --git a/drivers/pci/xen-pcifront.c b/drivers/pci/xen-pcifront.c
index 854850d..8ec3aca 100644
--- a/drivers/pci/xen-pcifront.c
+++ b/drivers/pci/xen-pcifront.c
@@ -1094,7 +1094,6 @@ static const struct xenbus_device_id xenpci_ids[] = {
 	{"pci"},
 	{{0}},
 };
-MODULE_ALIAS("xen:pci");
 
 static struct xenbus_driver xenbus_pcifront_driver = {
 	.name 			= "pcifront",
@@ -1113,5 +1112,13 @@ static int __init pcifront_init(void)
 	return xenbus_register_frontend(&xenbus_pcifront_driver);
 }
 
-/* Initialize after the Xen PCI Frontend Stub is initialized */
-subsys_initcall(pcifront_init);
+static void __exit pcifront_cleanup(void)
+{
+	xenbus_unregister_driver(&xenbus_pcifront_driver);
+}
+module_init(pcifront_init);
+module_exit(pcifront_cleanup);
+
+MODULE_DESCRIPTION("Xen PCI passthrough frontend.");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("xen:pci");
-- 
1.6.2.5

[PATCH 26/31] Fix warnings/errors reported by checkpatch.pl on xen-pcifront.c

[Konrad Rzeszutek Wilk]

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/pci/xen-pcifront.c |  136 ++++++++++++++++++++++----------------------
 1 files changed, 69 insertions(+), 67 deletions(-)

diff --git a/drivers/pci/xen-pcifront.c b/drivers/pci/xen-pcifront.c
index 8ec3aca..768df28 100644
--- a/drivers/pci/xen-pcifront.c
+++ b/drivers/pci/xen-pcifront.c
@@ -17,7 +17,7 @@
 #include <linux/interrupt.h>
 #include <asm/atomic.h>
 #include <linux/workqueue.h>
-#include <asm/bitops.h>
+#include <linux/bitops.h>
 #include <linux/time.h>
 
 
@@ -81,9 +81,9 @@ static inline void pcifront_setup_root_resources(struct pci_bus *bus,
 
 
 DEFINE_SPINLOCK(pcifront_dev_lock);
-static struct pcifront_device *pcifront_dev = NULL;
+static struct pcifront_device *pcifront_dev;
 
-static int verbose_request = 0;
+static int verbose_request;
 module_param(verbose_request, int, 0644);
 
 static int errno_to_pcibios_err(int errno)
@@ -164,13 +164,13 @@ static int do_pci_op(struct pcifront_device *pdev, struct xen_pci_op *op)
 	}
 
 	/*
-	* We might lose backend service request since we 
+	* We might lose backend service request since we
 	* reuse same evtchn with pci_conf backend response. So re-schedule
 	* aer pcifront service.
 	*/
-	if (test_bit(_XEN_PCIB_active, 
-			(unsigned long*)&pdev->sh_info->flags)) {
-		dev_err(&pdev->xdev->dev, 
+	if (test_bit(_XEN_PCIB_active,
+			(unsigned long *)&pdev->sh_info->flags)) {
+		dev_err(&pdev->xdev->dev,
 			"schedule aer pcifront service\n");
 		schedule_pcifront_aer_op(pdev);
 	}
@@ -178,14 +178,14 @@ static int do_pci_op(struct pcifront_device *pdev, struct xen_pci_op *op)
 	memcpy(op, active_op, sizeof(struct xen_pci_op));
 
 	err = op->err;
-      out:
+out:
 	spin_unlock_irqrestore(&pdev->sh_info_lock, irq_flags);
 	return err;
 }
 
 /* Access to this function is spinlocked in drivers/pci/access.c */
 static int pcifront_bus_read(struct pci_bus *bus, unsigned int devfn,
-			     int where, int size, u32 * val)
+			     int where, int size, u32 *val)
 {
 	int err = 0;
 	struct xen_pci_op op = {
@@ -271,7 +271,7 @@ int pci_frontend_enable_msix(struct pci_dev *dev,
 	struct pcifront_device *pdev = pcifront_get_pdev(sd);
 
 	if (nvec > SH_INFO_MAX_VEC) {
-		printk("too much vector for pci frontend%x\n", nvec);
+		printk(KERN_ERR "too much vector for pci frontend%x\n", nvec);
 		return -EINVAL;
 	}
 
@@ -285,22 +285,21 @@ int pci_frontend_enable_msix(struct pci_dev *dev,
 	if (!err) {
 		if (!op.value) {
 			/* we get the result */
-			for ( i = 0; i < nvec; i++)
+			for (i = 0; i < nvec; i++)
 				entries[i].vector = op.msix_entries[i].vector;
 			return 0;
-		}
-		else {
-            printk("enable msix get value %x\n", op.value);
+		} else {
+			printk(KERN_DEBUG "enable msix get value %x\n",
+				op.value);
 			return op.value;
 		}
-	}
-	else {
-        printk("enable msix get err %x\n", err);
+	} else {
+		printk(KERN_ERR "enable msix get err %x\n", err);
 		return err;
 	}
 }
 
-void pci_frontend_disable_msix(struct pci_dev* dev)
+void pci_frontend_disable_msix(struct pci_dev *dev)
 {
 	int err;
 	struct xen_pci_op op = {
@@ -316,7 +315,7 @@ void pci_frontend_disable_msix(struct pci_dev* dev)
 
 	/* What should do for error ? */
 	if (err)
-		printk("pci_disable_msix get err %x\n", err);
+		printk(KERN_ERR "pci_disable_msix get err %x\n", err);
 }
 
 int pci_frontend_enable_msi(struct pci_dev *dev)
@@ -334,16 +333,15 @@ int pci_frontend_enable_msi(struct pci_dev *dev)
 	err = do_pci_op(pdev, &op);
 	if (likely(!err)) {
 		dev->irq = op.value;
-	}
-	else {
-		printk("pci frontend enable msi failed for dev %x:%x \n",
+	} else {
+		printk(KERN_ERR "pci frontend enable msi failed for dev %x:%x \n",
 				op.bus, op.devfn);
 		err = -EINVAL;
 	}
 	return err;
 }
 
-void pci_frontend_disable_msi(struct pci_dev* dev)
+void pci_frontend_disable_msi(struct pci_dev *dev)
 {
 	int err;
 	struct xen_pci_op op = {
@@ -358,14 +356,14 @@ void pci_frontend_disable_msi(struct pci_dev* dev)
 	err = do_pci_op(pdev, &op);
 	if (err == XEN_PCI_ERR_dev_not_found) {
 		/* XXX No response from backend, what shall we do? */
-		printk("get no response from backend for disable MSI\n");
+		printk(KERN_DEBUG "get no response from backend for disable MSI\n");
 		return;
 	}
 	if (likely(!err))
 		dev->irq = op.value;
 	else
 		/* how can pciback notify us fail? */
-		printk("get fake response frombackend \n");
+		printk(KERN_DEBUG "get fake response frombackend \n");
 }
 #endif /* CONFIG_PCI_MSI */
 
@@ -440,7 +438,7 @@ int __devinit pcifront_scan_root(struct pcifront_device *pdev,
 
 	return 0;
 
-      err_out:
+err_out:
 	kfree(bus_entry);
 	kfree(sd);
 
@@ -469,17 +467,17 @@ int __devinit pcifront_rescan_root(struct pcifront_device *pdev,
 		 domain, bus);
 
 	b = pci_find_bus(domain, bus);
-	if(!b)
+	if (!b)
 		/* If the bus is unknown, create it. */
 		return pcifront_scan_root(pdev, domain, bus);
 
 	/* Rescan the bus for newly attached functions and add.
 	 * We omit handling of PCI bridge attachment because pciback prevents
 	 * bridges from being exported.
-	 */ 
+	 */
 	for (devfn = 0; devfn < 0x100; devfn++) {
 		d = pci_get_slot(b, devfn);
-		if(d) {
+		if (d) {
 			/* Device is already known. */
 			pci_dev_put(d);
 			continue;
@@ -496,7 +494,6 @@ int __devinit pcifront_rescan_root(struct pcifront_device *pdev,
 				" device to bus.\n");
 				return err;
 			}
-			
 		}
 	}
 
@@ -535,8 +532,9 @@ void pcifront_free_roots(struct pcifront_device *pdev)
 	}
 }
 
-static pci_ers_result_t pcifront_common_process( int cmd, struct pcifront_device *pdev,
-	pci_channel_state_t state)
+static pci_ers_result_t pcifront_common_process(int cmd,
+						struct pcifront_device *pdev,
+						pci_channel_state_t state)
 {
 	pci_ers_result_t result;
 	struct pci_driver *pdrv;
@@ -545,14 +543,14 @@ static pci_ers_result_t pcifront_common_process( int cmd, struct pcifront_device
 	struct pci_dev *pcidev;
 	int flag = 0;
 
-	dev_dbg(&pdev->xdev->dev, 
+	dev_dbg(&pdev->xdev->dev,
 		"pcifront AER process: cmd %x (bus:%x, devfn%x)",
 		cmd, bus, devfn);
 	result = PCI_ERS_RESULT_NONE;
 
 	pcidev = pci_get_bus_and_slot(bus, devfn);
-	if (!pcidev || !pcidev->driver){
-		dev_err(&pcidev->dev, 
+	if (!pcidev || !pcidev->driver) {
+		dev_err(&pcidev->dev,
 			"device or driver is NULL\n");
 		return result;
 	}
@@ -564,22 +562,26 @@ static pci_ers_result_t pcifront_common_process( int cmd, struct pcifront_device
 				"trying to call AER service\n");
 			if (pcidev) {
 				flag = 1;
-				switch(cmd) {
+				switch (cmd) {
 				case XEN_PCI_OP_aer_detected:
-					result = pdrv->err_handler->error_detected(pcidev, state);
+					result = pdrv->err_handler->
+						 error_detected(pcidev, state);
 					break;
 				case XEN_PCI_OP_aer_mmio:
-					result = pdrv->err_handler->mmio_enabled(pcidev);
+					result = pdrv->err_handler->
+						 mmio_enabled(pcidev);
 					break;
 				case XEN_PCI_OP_aer_slotreset:
-					result = pdrv->err_handler->slot_reset(pcidev);
+					result = pdrv->err_handler->
+						 slot_reset(pcidev);
 					break;
 				case XEN_PCI_OP_aer_resume:
 					pdrv->err_handler->resume(pcidev);
 					break;
 				default:
 					dev_err(&pdev->xdev->dev,
-						"bad request in aer recovery operation!\n");
+						"bad request in aer recovery "
+						"operation!\n");
 
 				}
 			}
@@ -595,26 +597,27 @@ static pci_ers_result_t pcifront_common_process( int cmd, struct pcifront_device
 
 void pcifront_do_aer(struct work_struct *data)
 {
-	struct pcifront_device *pdev = container_of(data, struct pcifront_device, op_work);
+	struct pcifront_device *pdev =
+		container_of(data, struct pcifront_device, op_work);
 	int cmd = pdev->sh_info->aer_op.cmd;
-	pci_channel_state_t state = 
+	pci_channel_state_t state =
 		(pci_channel_state_t)pdev->sh_info->aer_op.err;
 
-	/*If a pci_conf op is in progress, 
+	/*If a pci_conf op is in progress,
 		we have to wait until it is done before service aer op*/
-	dev_dbg(&pdev->xdev->dev, 
-		"pcifront service aer bus %x devfn %x\n", pdev->sh_info->aer_op.bus,
-		pdev->sh_info->aer_op.devfn);
+	dev_dbg(&pdev->xdev->dev,
+		"pcifront service aer bus %x devfn %x\n",
+		pdev->sh_info->aer_op.bus, pdev->sh_info->aer_op.devfn);
 
 	pdev->sh_info->aer_op.err = pcifront_common_process(cmd, pdev, state);
 
 	wmb();
-	clear_bit(_XEN_PCIB_active, (unsigned long*)&pdev->sh_info->flags);
+	clear_bit(_XEN_PCIB_active, (unsigned long *)&pdev->sh_info->flags);
 	notify_remote_via_evtchn(pdev->evtchn);
 
 	/*in case of we lost an aer request in four lines time_window*/
 	smp_mb__before_clear_bit();
-	clear_bit( _PDEVB_op_active, &pdev->flags);
+	clear_bit(_PDEVB_op_active, &pdev->flags);
 	smp_mb__after_clear_bit();
 
 	schedule_pcifront_aer_op(pdev);
@@ -636,8 +639,7 @@ int pcifront_connect(struct pcifront_device *pdev)
 	if (!pcifront_dev) {
 		dev_info(&pdev->xdev->dev, "Installing PCI frontend\n");
 		pcifront_dev = pdev;
-	}
-	else {
+	} else {
 		dev_err(&pdev->xdev->dev, "PCI frontend already installed!\n");
 		err = -EEXIST;
 	}
@@ -677,7 +679,7 @@ static struct pcifront_device *alloc_pdev(struct xenbus_device *xdev)
 	pdev->sh_info->flags = 0;
 
 	/*Flag for registering PV AER handler*/
-	set_bit(_XEN_PCIB_AERHANDLER, (void*)&pdev->sh_info->flags);
+	set_bit(_XEN_PCIB_AERHANDLER, (void *)&pdev->sh_info->flags);
 
 	xdev->dev.driver_data = pdev;
 	pdev->xdev = xdev;
@@ -695,7 +697,7 @@ static struct pcifront_device *alloc_pdev(struct xenbus_device *xdev)
 
 	dev_dbg(&xdev->dev, "Allocated pdev @ 0x%p pdev->sh_info @ 0x%p\n",
 		pdev, pdev->sh_info);
-      out:
+out:
 	return pdev;
 }
 
@@ -736,8 +738,8 @@ static int pcifront_publish_info(struct pcifront_device *pdev)
 	if (err)
 		goto out;
 
-	err = bind_evtchn_to_irqhandler(pdev->evtchn, pcifront_handler_aer, 
-		0, "pcifront", pdev); 
+	err = bind_evtchn_to_irqhandler(pdev->evtchn, pcifront_handler_aer,
+		0, "pcifront", pdev);
 	if (err < 0) {
 		xenbus_free_evtchn(pdev->xdev, pdev->evtchn);
 		xenbus_dev_fatal(pdev->xdev, err, "Failed to bind evtchn to "
@@ -746,7 +748,7 @@ static int pcifront_publish_info(struct pcifront_device *pdev)
 	}
 	pdev->irq = err;
 
-      do_publish:
+do_publish:
 	err = xenbus_transaction_start(&trans);
 	if (err) {
 		xenbus_dev_fatal(pdev->xdev, err,
@@ -785,7 +787,7 @@ static int pcifront_publish_info(struct pcifront_device *pdev)
 
 	dev_dbg(&pdev->xdev->dev, "publishing successful!\n");
 
-      out:
+out:
 	return err;
 }
 
@@ -855,7 +857,7 @@ static int __devinit pcifront_try_connect(struct pcifront_device *pdev)
 	if (err)
 		goto out;
 
-      out:
+out:
 	spin_unlock(&pdev->dev_lock);
 	return err;
 }
@@ -872,14 +874,14 @@ static int pcifront_try_disconnect(struct pcifront_device *pdev)
 	if (prev_state >= XenbusStateClosing)
 		goto out;
 
-	if(prev_state == XenbusStateConnected) {
+	if (prev_state == XenbusStateConnected) {
 		pcifront_free_roots(pdev);
 		pcifront_disconnect(pdev);
 	}
 
 	err = xenbus_switch_state(pdev->xdev, XenbusStateClosed);
 
-      out:
+out:
 	spin_unlock(&pdev->dev_lock);
 
 	return err;
@@ -941,7 +943,7 @@ static int __devinit pcifront_attach_devices(struct pcifront_device *pdev)
 
 	xenbus_switch_state(pdev->xdev, XenbusStateConnected);
 
-      out:
+out:
 	spin_unlock(&pdev->dev_lock);
 	return err;
 }
@@ -994,23 +996,23 @@ static int pcifront_detach_devices(struct pcifront_device *pdev)
 			goto out;
 		}
 		err = xenbus_scanf(XBT_NIL, pdev->xdev->otherend, str,
-			   	   "%x:%x:%x.%x", &domain, &bus, &slot, &func);
+				   "%x:%x:%x.%x", &domain, &bus, &slot, &func);
 		if (err != 4) {
 			if (err >= 0)
 				err = -EINVAL;
 			xenbus_dev_fatal(pdev->xdev, err,
-				 	 "Error reading PCI device %d", i);
+					 "Error reading PCI device %d", i);
 			goto out;
 		}
 
 		pci_bus = pci_find_bus(domain, bus);
-		if(!pci_bus) {
+		if (!pci_bus) {
 			dev_dbg(&pdev->xdev->dev, "Cannot get bus %04x:%02x\n",
 				domain, bus);
 			continue;
 		}
 		pci_dev = pci_get_slot(pci_bus, PCI_DEVFN(slot, func));
-		if(!pci_dev) {
+		if (!pci_dev) {
 			dev_dbg(&pdev->xdev->dev,
 				"Cannot get PCI device %04x:%02x:%02x.%02x\n",
 				domain, bus, slot, func);
@@ -1026,7 +1028,7 @@ static int pcifront_detach_devices(struct pcifront_device *pdev)
 
 	err = xenbus_switch_state(pdev->xdev, XenbusStateReconfiguring);
 
-      out:
+out:
 	spin_unlock(&pdev->dev_lock);
 	return err;
 }
@@ -1078,7 +1080,7 @@ static int pcifront_xenbus_probe(struct xenbus_device *xdev,
 
 	err = pcifront_publish_info(pdev);
 
-      out:
+out:
 	return err;
 }
 
@@ -1092,7 +1094,7 @@ static int pcifront_xenbus_remove(struct xenbus_device *xdev)
 
 static const struct xenbus_device_id xenpci_ids[] = {
 	{"pci"},
-	{{0}},
+	{""},
 };
 
 static struct xenbus_driver xenbus_pcifront_driver = {
-- 
1.6.2.5

[PATCH 27/31] Find an unbound irq number in reverse order (high to low).

[Konrad Rzeszutek Wilk]

From: root <root@localhost.localdomain>

In earlier Xen Linux kernels, the IRQ mapping was a straight 1:1 and the
find_unbound_irq started looking around 256 for open IRQs and up. IRQs
from 0 to 255 were reserved for PCI devices.  Previous to this patch,
the 'find_unbound_irq'  started looking at get_nr_hw_irqs() number.
For privileged  domain where the ACPI information is available that
returns the upper-bound of what the GSIs. For non-privileged PV domains,
where ACPI is no-existent the get_nr_hw_irqs() reports the IRQ_LEGACY (16).
With PCI passthrough enabled, and with PCI cards that have IRQs pinned
to a higher number than 16 we collide with previously allocated IRQs.
Specifically the PCI IRQs collide with the IPI's for Xen functions
(as they are allocated earlier).
For example:

00:00.11 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller (prog-if 10 [OHCI])
	...
	Interrupt: pin A routed to IRQ 18

[root@localhost ~]# cat /proc/interrupts | head
           CPU0       CPU1       CPU2
 16:      38186          0          0   xen-dyn-virq      timer0
 17:        149          0          0   xen-dyn-ipi       spinlock0
 18:        962          0          0   xen-dyn-ipi       resched0

and when the USB controller is loaded, the kernel reports:
IRQ handler type mismatch for IRQ 18
current handler: resched0

One way to fix this is to reverse the logic when looking for un-used
IRQ numbers and start with the highest available number. With that,
we would get:

           CPU0       CPU1       CPU2
... snip ..
292:         35          0          0   xen-dyn-ipi       callfunc0
293:       3992          0          0   xen-dyn-ipi       resched0
294:        224          0          0   xen-dyn-ipi       spinlock0
295:      57183          0          0   xen-dyn-virq      timer0
NMI:          0          0          0   Non-maskable interrupts
.. snip ..

And interrupts for PCI cards are now accessible.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/events.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index e9f7551..b2c1b09 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -371,11 +371,12 @@ static int find_unbound_irq(void)
 	struct irq_desc *desc;
 	int start = get_nr_hw_irqs();
 
-	for (irq = start; irq < nr_irqs; irq++)
+	/* nr_irqs is a magic value. Must not use it.*/
+	for (irq = nr_irqs-1; irq > start; irq--)
 		if (irq_info[irq].type == IRQT_UNBOUND)
 			break;
 
-	if (irq == nr_irqs)
+	if (irq == start || irq == nr_irqs)
 		panic("No available IRQ to bind to: increase nr_irqs!\n");
 
 	desc = irq_to_desc_alloc_node(irq, 0);
-- 
1.6.2.5

[PATCH 28/31] For non-privileged domains, implement a pcibios_enable_irq (xen_pcifront_enable_irq) function.

[Konrad Rzeszutek Wilk]

This function calls the 'xen_allocate_pirq' which does the 
majority of work: allocate a free IRQ descriptor, setup an IRQ chip, etc.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/pci/xen.c   |   15 +++++++++++++++
 drivers/xen/events.c |   12 ++++++++++--
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c
index 1b922aa..c4580b8 100644
--- a/arch/x86/pci/xen.c
+++ b/arch/x86/pci/xen.c
@@ -14,8 +14,23 @@
 
 #include <asm/xen/hypervisor.h>
 
+#include <xen/events.h>
+
 static int xen_pcifront_enable_irq(struct pci_dev *dev)
 {
+	int rc;
+
+	dev_info(&dev->dev, "Xen PCI enabling IRQ: %d\n", dev->irq);
+
+	if (dev->irq < 0)
+		return -EINVAL;
+
+	rc = xen_allocate_pirq(dev->irq, "pcifront");
+	if (rc < 0) {
+		dev_warn(&dev->dev, "Xen PCI IRQ: %d, failed to register:%d\n",
+			 dev->irq, rc);
+		return rc;
+	}
 	return 0;
 }
 
diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index b2c1b09..40bcbde 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -561,7 +561,9 @@ int xen_allocate_pirq(unsigned gsi, char *name)
 		goto out;	/* XXX need refcount? */
 	}
 
-	if (identity_mapped_irq(gsi)) {
+	/* If we are a PV guest, we don't have GSIs (no ACPI passed). Therefore
+	 * we are using the !xen_initial_domain() to drop in the function.*/
+	if (identity_mapped_irq(gsi) || !xen_initial_domain()) {
 		irq = gsi;
 		irq_to_desc_alloc_node(irq, 0);
 		dynamic_irq_init(irq);
@@ -572,7 +574,13 @@ int xen_allocate_pirq(unsigned gsi, char *name)
 				      handle_level_irq, name);
 
 	irq_op.irq = gsi;
-	if (HYPERVISOR_physdev_op(PHYSDEVOP_alloc_irq_vector, &irq_op)) {
+	irq_op.vector = 0;
+
+	/* Only the privileged domain can do this. For non-priv, the pcifront
+	 * driver provides a PCI bus that does the call to do exactly
+	 * this in the priv domain. */
+	if (xen_initial_domain() &&
+	    HYPERVISOR_physdev_op(PHYSDEVOP_alloc_irq_vector, &irq_op)) {
 		dynamic_irq_cleanup(irq);
 		irq = -ENOSPC;
  		goto out;
-- 
1.6.2.5

[PATCH 29/31] xen_destroy_irq + xen_allocate_pirq in PV non-priv mode should not make certain Xen-HYPERCALLs.

[Konrad Rzeszutek Wilk]

The privileged domain should make these calls for the guest.
The non-privileged domain should make these calls via the pcifront
interface (patches that augment these two functions to do so
will follow).

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/events.c |   30 ++++++++++++++++--------------
 1 files changed, 16 insertions(+), 14 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 40bcbde..fd55b5b 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -606,14 +606,15 @@ int xen_destroy_irq(int irq)
 	if (!desc)
 		goto out;
 
-	unmap_irq.pirq = info->u.pirq.nr;
-	unmap_irq.domid = info->u.pirq.domid;
-	rc = HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq);
-	if (rc) {
-		printk(KERN_WARNING "unmap irq failed %d\n", rc);
-		goto out;
+	if (xen_initial_domain()) {
+		unmap_irq.pirq = info->u.pirq.nr;
+		unmap_irq.domid = info->u.pirq.domid;
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq);
+		if (rc) {
+			printk(KERN_WARNING "unmap irq failed %d\n", rc);
+			goto out;
+		}
 	}
-
 	irq_info[irq] = mk_unbound_info();
 
 	dynamic_irq_cleanup(irq);
@@ -701,17 +702,18 @@ int xen_create_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc, int type)
 	if (irq == -1)
 		goto out;
 
-	rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);
-	if (rc) {
+	if (xen_initial_domain()) {
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);
+		if (rc) {
 
-		printk(KERN_WARNING "xen map irq failed %d\n", rc);
+			printk(KERN_WARNING "xen map irq failed %d\n", rc);
 
-		dynamic_irq_cleanup(irq);
+			dynamic_irq_cleanup(irq);
 
-		irq = -1;
-		goto out;
+			irq = -1;
+			goto out;
+		}
 	}
-
 	irq_info[irq] = mk_pirq_info(0, map_irq.pirq, map_irq.index);
 	if (domid)
 		irq_info[irq].u.pirq.domid = domid;
-- 
1.6.2.5

[PATCH 30/31] Add pci_frontend_[enable|disable]_[msi|msix] function decleration and EXPORT_SYMBOL_GPL.

[Konrad Rzeszutek Wilk]

Non-privileged domains can enable/disable MSI devices via the
pci_frontend_* functions. Those functions in turn make calls to
the pciback (on the privileged domain) which in turn calls the
appropiate pci_[enable|disable]_[msi|misx] function.

Actual usage to follow.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/include/asm/xen/pci.h |   18 +++++++++++++++++-
 drivers/pci/xen-pcifront.c     |    9 +++++++--
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/xen/pci.h b/arch/x86/include/asm/xen/pci.h
index cb84abe..cb37499 100644
--- a/arch/x86/include/asm/xen/pci.h
+++ b/arch/x86/include/asm/xen/pci.h
@@ -33,5 +33,21 @@ static inline int xen_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 	return -1;
 }
 #endif
-
+#if defined(CONFIG_PCI_MSI) && defined(CONFIG_XEN_PCIDEV_FRONTEND)
+/* Defined in drivers/pci/xen-pcifront.c */
+int pci_frontend_enable_msi(struct pci_dev *dev, int *pirq);
+void pci_frontend_disable_msi(struct pci_dev *dev);
+int pci_frontend_enable_msix(struct pci_dev *dev,
+			     struct msix_entry *entries, int nvec);
+void pci_frontend_disable_msix(struct pci_dev *dev);
+#else
+static inline int pci_frontend_enable_msi(struct pci_dev *dev) { return -1; }
+static void pci_frontend_disable_msi(struct pci_dev *dev) { }
+static int pci_frontend_enable_msix(struct pci_dev *dev,
+			     struct msix_entry *entries, int nvec)
+{
+	return -1;
+}
+static void pci_frontend_disable_msix(struct pci_dev *dev) { }
+#endif
 #endif	/* _ASM_X86_XEN_PCI_H */
diff --git a/drivers/pci/xen-pcifront.c b/drivers/pci/xen-pcifront.c
index 768df28..adef0c3 100644
--- a/drivers/pci/xen-pcifront.c
+++ b/drivers/pci/xen-pcifront.c
@@ -298,6 +298,7 @@ int pci_frontend_enable_msix(struct pci_dev *dev,
 		return err;
 	}
 }
+EXPORT_SYMBOL_GPL(pci_frontend_enable_msix);
 
 void pci_frontend_disable_msix(struct pci_dev *dev)
 {
@@ -317,8 +318,9 @@ void pci_frontend_disable_msix(struct pci_dev *dev)
 	if (err)
 		printk(KERN_ERR "pci_disable_msix get err %x\n", err);
 }
+EXPORT_SYMBOL_GPL(pci_frontend_disable_msix);
 
-int pci_frontend_enable_msi(struct pci_dev *dev)
+int pci_frontend_enable_msi(struct pci_dev *dev, int *pirq)
 {
 	int err;
 	struct xen_pci_op op = {
@@ -330,9 +332,10 @@ int pci_frontend_enable_msi(struct pci_dev *dev)
 	struct pcifront_sd *sd = dev->bus->sysdata;
 	struct pcifront_device *pdev = pcifront_get_pdev(sd);
 
+	*pirq = -1;
 	err = do_pci_op(pdev, &op);
 	if (likely(!err)) {
-		dev->irq = op.value;
+		*pirq = op.value;
 	} else {
 		printk(KERN_ERR "pci frontend enable msi failed for dev %x:%x \n",
 				op.bus, op.devfn);
@@ -340,6 +343,7 @@ int pci_frontend_enable_msi(struct pci_dev *dev)
 	}
 	return err;
 }
+EXPORT_SYMBOL(pci_frontend_enable_msi);
 
 void pci_frontend_disable_msi(struct pci_dev *dev)
 {
@@ -365,6 +369,7 @@ void pci_frontend_disable_msi(struct pci_dev *dev)
 		/* how can pciback notify us fail? */
 		printk(KERN_DEBUG "get fake response frombackend \n");
 }
+EXPORT_SYMBOL_GPL(pci_frontend_disable_msi);
 #endif /* CONFIG_PCI_MSI */
 
 /* Claim resources for the PCI frontend as-is, backend won't allow changes */
-- 
1.6.2.5

[PATCH 31/31] To enable MSI devices in a non-privileged PV domain use pci_frontend_enable_msi.

[Konrad Rzeszutek Wilk]

The return argument of pci_frontend_enable_msi provides the PIRQ
number allocated by Xen Hypervisor. We will use that to bind
the local IRQ number to it.

Tested succesfully with MSI devices.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/events.c |   19 +++++++++++++++++++
 1 files changed, 19 insertions(+), 0 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index fd55b5b..f432c13 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -39,6 +39,7 @@
 #include <asm/sync_bitops.h>
 #include <asm/xen/hypercall.h>
 #include <asm/xen/hypervisor.h>
+#include <asm/xen/pci.h>
 
 #include <xen/xen-ops.h>
 #include <xen/events.h>
@@ -713,6 +714,24 @@ int xen_create_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc, int type)
 			irq = -1;
 			goto out;
 		}
+	} else {
+		int pirq = 0;
+		/* TODO: MSI-X not yet supported. */
+		if (type == PCI_CAP_ID_MSIX) {
+			dev_warn(&dev->dev, "pcifront: MSI-X not supported.\n");
+			rc = -ENODEV;
+		} else
+			rc = pci_frontend_enable_msi(dev, &pirq);
+		if (rc) {
+			printk(KERN_WARNING "xen map irq failed %d\n", rc);
+			dynamic_irq_cleanup(irq);
+			irq = -1;
+			goto out;
+		}
+		map_irq.pirq = pirq;
+		map_irq.index = 0;
+		dev_info(&dev->dev, "Allocated IRQ %d (vector: %d)\n",
+				     irq, map_irq.pirq);
 	}
 	irq_info[irq] = mk_pirq_info(0, map_irq.pirq, map_irq.index);
 	if (domid)
-- 
1.6.2.5

Re: [PATCH 29/31] xen_destroy_irq + xen_allocate_pirq in PV non-priv mode should not make certain Xen-HYPERCALLs.

[Jeremy Fitzhardinge]

On 11/05/09 13:33, Konrad Rzeszutek Wilk wrote:
> The privileged domain should make these calls for the guest.
> The non-privileged domain should make these calls via the pcifront
> interface (patches that augment these two functions to do so
> will follow).
>
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  drivers/xen/events.c |   30 ++++++++++++++++--------------
>  1 files changed, 16 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/xen/events.c b/drivers/xen/events.c
> index 40bcbde..fd55b5b 100644
> --- a/drivers/xen/events.c
> +++ b/drivers/xen/events.c
> @@ -606,14 +606,15 @@ int xen_destroy_irq(int irq)
>  	if (!desc)
>  		goto out;
>  
> -	unmap_irq.pirq = info->u.pirq.nr;
> -	unmap_irq.domid = info->u.pirq.domid;
>   

This doesn't apply to my tree because these lines read:
    unmap_irq.pirq = info->u.pirq.nr;
    unmap_irq.domid = DOMID_SELF;

I'm not sure where the creep happened.  Oh, perhaps because I applied
the pcifront and back series in different branches?  Ah, yes, that's
it.  I'll see if I can sort it out.

    J

Re: [PATCH PV_OPS PCIFRONT]

[Konrad Rzeszutek Wilk]

> That is it for right now. The driver works with INTx and MSI cards. I've tested
> with USB and network (Broadcom) succesfully. There is still some more work to do:
>  - MSI disable is not yet in,
>  - no MSI-X enable/disable functionality.
and:
 - If 4GB or more are allocated to the domain, you get this:

PCI: Warning: Cannot find a gap in the 32bit address range
PCI: Unassigned devices with 32bit resource registers may break!

   and the device shows up as disabled and is not usuable.

 - No support yet for 'reassign_device' which page aligns the BARs
   on PCI devices.

 - Haven't tested FLR or PCI AER.
 - Saw this once:

WARNING: at drivers/pci/msi.c:602 pci_enable_msi_block+0xcd/0x339()
.. snip ..
Call Trace:
 [<ffffffff8107ed59>] warn_slowpath_common+0xc9/0x10c
 [<ffffffff8107edcc>] warn_slowpath_null+0x30/0x4d
 [<ffffffff81362cec>] pci_enable_msi_block+0xcd/0x339
 [<ffffffff814006ec>] ? pciback_do_op+0x0/0x1b4
 [<ffffffff8140469e>] pciback_enable_msi+0x3e/0xb0
 [<ffffffff814007b9>] pciback_do_op+0xcd/0x1b4
..snip..

But besides that it works :-)

Re: [PATCH PV_OPS PCIFRONT]

[Jeremy Fitzhardinge]

On 11/05/09 14:04, Konrad Rzeszutek Wilk wrote:
>> That is it for right now. The driver works with INTx and MSI cards. I've tested
>> with USB and network (Broadcom) succesfully. There is still some more work to do:
>>  - MSI disable is not yet in,
>>  - no MSI-X enable/disable functionality.
>>     
> and:
>  - If 4GB or more are allocated to the domain, you get this:
>
> PCI: Warning: Cannot find a gap in the 32bit address range
> PCI: Unassigned devices with 32bit resource registers may break!
>
>    and the device shows up as disabled and is not usuable.
>   

Presumably less than 4G can trigger this.  My rough thought about this
was to always reserve a chunk of memory under 4G to make space for this
kind of thing, and push the displaced memory higher.

>  - No support yet for 'reassign_device' which page aligns the BARs
>    on PCI devices.
>
>  - Haven't tested FLR or PCI AER.
>  - Saw this once:
>
> WARNING: at drivers/pci/msi.c:602 pci_enable_msi_block+0xcd/0x339()
> .. snip ..
> Call Trace:
>  [<ffffffff8107ed59>] warn_slowpath_common+0xc9/0x10c
>  [<ffffffff8107edcc>] warn_slowpath_null+0x30/0x4d
>  [<ffffffff81362cec>] pci_enable_msi_block+0xcd/0x339
>  [<ffffffff814006ec>] ? pciback_do_op+0x0/0x1b4
>  [<ffffffff8140469e>] pciback_enable_msi+0x3e/0xb0
>  [<ffffffff814007b9>] pciback_do_op+0xcd/0x1b4
> ..snip..
>   

What does that warning mean?

> But besides that it works :-)
>   

Good!  I'll throw it into the mix and see how it flies.

    J

Re: [PATCH PV_OPS PCIFRONT]

[Konrad Rzeszutek Wilk]

On Fri, Nov 06, 2009 at 01:40:14PM -0800, Jeremy Fitzhardinge wrote:
> On 11/05/09 14:04, Konrad Rzeszutek Wilk wrote:
> >> That is it for right now. The driver works with INTx and MSI cards. I've tested
> >> with USB and network (Broadcom) succesfully. There is still some more work to do:
> >>  - MSI disable is not yet in,
> >>  - no MSI-X enable/disable functionality.
> >>     
> > and:
> >  - If 4GB or more are allocated to the domain, you get this:
> >
> > PCI: Warning: Cannot find a gap in the 32bit address range
> > PCI: Unassigned devices with 32bit resource registers may break!
> >
> >    and the device shows up as disabled and is not usuable.
> >   
> 
> Presumably less than 4G can trigger this.  My rough thought about this
> was to always reserve a chunk of memory under 4G to make space for this
> kind of thing, and push the displaced memory higher.

<nods> This would be done in the xc_build_domain_linux_something ?

> >
> > WARNING: at drivers/pci/msi.c:602 pci_enable_msi_block+0xcd/0x339()
> > .. snip ..
> > Call Trace:
> >  [<ffffffff8107ed59>] warn_slowpath_common+0xc9/0x10c
> >  [<ffffffff8107edcc>] warn_slowpath_null+0x30/0x4d
> >  [<ffffffff81362cec>] pci_enable_msi_block+0xcd/0x339
> >  [<ffffffff814006ec>] ? pciback_do_op+0x0/0x1b4
> >  [<ffffffff8140469e>] pciback_enable_msi+0x3e/0xb0
> >  [<ffffffff814007b9>] pciback_do_op+0xcd/0x1b4
> > ..snip..
> >   
> 
> What does that warning mean?

Something about slowpath. Didn't dive any deeper in this. I think it complains
about the workqueue taking too long to do its job.
> 
> > But besides that it works :-)
> >   
> 
> Good!  I'll throw it into the mix and see how it flies.

<crosses his fingers>

Re: [PATCH PV_OPS PCIFRONT]

[Konrad Rzeszutek Wilk]

> > What does that warning mean?
> 
> Something about slowpath. Didn't dive any deeper in this. I think it complains
> about the workqueue taking too long to do its job.

Here is another one I get (it is unrelated to Xen - a general USB floppy issue):

scsi 9:0:0:0: Direct-Access     TEAC     FD-05PUB         3200 PQ: 0 ANSI: 0 CCS
sd 9:0:0:0: Attached scsi generic sg4 type 0
sd 8:0:0:0: [sdc] Assuming drive cache: write through
 sdc: sdc1 sdc2 sdc3
------------[ cut here ]------------
WARNING: at /mnt/tmp/fc11/xen/lib/dma-debug.c:860 check_for_stack+0xfa/0x15e()
Hardware name: To Be Filled By O.E.M.
ohci_hcd 0000:00:12.1: DMA-API: device driver maps memory fromstack [addr=ffff8801d4129de8]
Modules linked in: dvb_core tuner_simple tuner_types tda9887 tda8290 snd_hda_codec_atihdmi snd_hda_codec_realtek radeon tuner snd_hda_intel snd_usb_audio snd_hda_codec cx8800 snd_usb_lib cx8802 cx88xx snd_rawmidi snd_pcm snd_seq_device ttm snd_hwdep pwc snd_timer ir_common snd drm v4l2_common videodev ata_generic tveeprom firewire_ohci pl2303 btcx_risc r8169 usb_storage firewire_core i2c_piix4 soundcore pata_acpi usbserial v4l1_compat videobuf_dma_sg tg3 v4l2_compat_ioctl32 floppy videobuf_core i2c_algo_bit mii pata_atiixp snd_page_alloc i2c_core crc_itu_t shpchp serio_raw wmi pcspkr [last unloaded: scsi_wait_scan]
Pid: 999, comm: usb-storage Tainted: G        W  2.6.31.4 #1
Call Trace:
 [<ffffffff8107ed59>] warn_slowpath_common+0xc9/0x10c
 [<ffffffff8107ee49>] warn_slowpath_fmt+0x60/0x84
 [<ffffffff81010dff>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8133b473>] check_for_stack+0xfa/0x15e
 [<ffffffff8133c7a6>] debug_dma_map_page+0x111/0x163
 [<ffffffff814ecc21>] dma_map_single_attrs.clone.2+0xfa/0x128
 [<ffffffff814ece0e>] usb_hcd_submit_urb+0x1bf/0xd84
 [<ffffffff81012463>] ? __xen_spin_lock+0x11f/0x150
 [<ffffffff8101200c>] ? xen_spin_unlock+0x27/0x7d
 [<ffffffff814ee3c1>] usb_submit_urb+0x463/0x487
 [<ffffffffa00d8407>] usb_stor_msg_common+0x105/0x1e1 [usb_storage]
 [<ffffffffa00d94d3>] usb_stor_ctrl_transfer+0xb5/0x100 [usb_storage]
 [<ffffffffa00d97f8>] usb_stor_CB_transport+0x59/0x2cd [usb_storage]
 [<ffffffffa00d8946>] usb_stor_invoke_transport+0x1fe/0x494 [usb_storage]
 [<ffffffffa00d804e>] usb_stor_ufi_command+0x8f/0xac [usb_storage]
 [<ffffffffa00dafbc>] usb_stor_control_thread+0x1e9/0x31f [usb_storage]
 [<ffffffff81010dff>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffffa00dadd3>] ? usb_stor_control_thread+0x0/0x31f [usb_storage]
 [<ffffffff810a4731>] kthread+0xbe/0xcd
 [<ffffffff81018aaa>] child_rip+0xa/0x20
 [<ffffffff81017c67>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101841d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81018aa0>] ? child_rip+0x0/0x20
---[ end trace a7919e7f17c0a727 ]---
sd 8:0:0:0: [sdc] Assuming drive cache: write through
sd 8:0:0:0: [sdc] Attached SCSI removable disk
usb 1-5:1.0: uevent
usb 1-5: uevent
sd 9:0:0:0: [sdd] 2880 512-byte logical blocks: (1.47 MB/1.40 MiB)
ohci_hcd 0000:00:12.1: urb ffff8801d89f0e40 path 3 ep1in 93120000 cc 9 --> status -121
sd 9:0:0:0: [sdd] Write Protect is off
sd 9:0:0:0: [sdd] Mode Sense: 00 46 94 00
sd 9:0:0:0: [sdd] Assuming drive cache: write through
ohci_hcd 0000:00:12.1: urb ffff8801d2dd6a80 path 3 ep1in 92120000 cc 9 --> status -121

Re: [PATCH PV_OPS PCIFRONT]

[Jeremy Fitzhardinge]

On 11/06/09 13:50, Konrad Rzeszutek Wilk wrote:
> On Fri, Nov 06, 2009 at 01:40:14PM -0800, Jeremy Fitzhardinge wrote:
>   
>> On 11/05/09 14:04, Konrad Rzeszutek Wilk wrote:
>>     
>>>> That is it for right now. The driver works with INTx and MSI cards. I've tested
>>>> with USB and network (Broadcom) succesfully. There is still some more work to do:
>>>>  - MSI disable is not yet in,
>>>>  - no MSI-X enable/disable functionality.
>>>>     
>>>>         
>>> and:
>>>  - If 4GB or more are allocated to the domain, you get this:
>>>
>>> PCI: Warning: Cannot find a gap in the 32bit address range
>>> PCI: Unassigned devices with 32bit resource registers may break!
>>>
>>>    and the device shows up as disabled and is not usuable.
>>>   
>>>       
>> Presumably less than 4G can trigger this.  My rough thought about this
>> was to always reserve a chunk of memory under 4G to make space for this
>> kind of thing, and push the displaced memory higher.
>>     
> <nods> This would be done in the xc_build_domain_linux_something ?
>   

I was thinking of doing it within the domain itself.  Fairly early, it
would clear out the range and modify the E820 table before handing it to
the rest of the kernel.  The code's actually already there, but it
hasn't really been exercised yet.  And it just releases the memory, but
doesn't relocate it.  You'd need to balloon it in again later.

>>> WARNING: at drivers/pci/msi.c:602 pci_enable_msi_block+0xcd/0x339()
>>> .. snip ..
>>> Call Trace:
>>>  [<ffffffff8107ed59>] warn_slowpath_common+0xc9/0x10c
>>>  [<ffffffff8107edcc>] warn_slowpath_null+0x30/0x4d
>>>  [<ffffffff81362cec>] pci_enable_msi_block+0xcd/0x339
>>>  [<ffffffff814006ec>] ? pciback_do_op+0x0/0x1b4
>>>  [<ffffffff8140469e>] pciback_enable_msi+0x3e/0xb0
>>>  [<ffffffff814007b9>] pciback_do_op+0xcd/0x1b4
>>> ..snip..
>>>   
>>>       
>> What does that warning mean?
>>     
> Something about slowpath. Didn't dive any deeper in this. I think it complains
> about the workqueue taking too long to do its job.
>   

"slowpath" here just means the slow path of the warning code; ie, a
warning happened.  The warning itself appears to be:
    WARN_ON(!!dev->msi_enabled);
in pci_enable_msi_block().

    J