[PATCH v7] Xen PCI + Xen PCI frontend driver.

[PATCH v7] Xen PCI + Xen PCI frontend driver.

[Konrad Rzeszutek Wilk]

This patch set contains the supporting patches and the driver itself for
Xen Paravirtualized (PV) domains to use PCI pass-through devices (the git tree
is git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/xen-pcifront-0.7).
This patch-set is also utilized in Stefano's PV on HVM MSI/MSI-X patchset [1].

The Xen PCI frontend driver can be used by PV guests on IOMMU hardware
(or IOMMU-less). Without the hardware IOMMU you have a potential security
hole wherein a guest domain can use the hardware to map pages outside its
memory range and slurp pages up. As such, this is more restricted to a
Privileged PV domain, aka - device driver domain (similar to Qubes but a
poor-man mechanism [2]).

The first set of patches are specific to the Xen subsystem, where
we introduce an IRQ chip for Physical IRQs, along with the supporting
harness code:
 xen: Don't disable the I/O space
 xen: define BIOVEC_PHYS_MERGEABLE()
 xen: implement pirq type event channels
 xen: identity map gsi->irqs
 xen: dynamically allocate irq & event structures
 xen: set pirq name to something useful.
 xen: statically initialize cpu_evtchn_mask_p
 xen: Find an unbound irq number in reverse order (high to low).
 xen: Provide a variant of xen_poll_irq with timeout.
 xen: fix shared irq device passthrough

The next set of patches expose functionality for module drivers to be able to
enumerate and iomap (using the _PAGE_IOMAP flag) PCI devices.

 x86/io_apic: add get_nr_irqs_gsi()
 x86/PCI: Clean up pci_cache_line_size
 x86/PCI: make sure _PAGE_IOMAP it set on pci mappings
 x86/PCI: Export pci_walk_bus function.

The next two patches abstract the MSI/MSI-X architecture calls so that the
default native one (used on bare-metal) can be overwritten when running
in virtualized mode (right now on Xen). The implementation is a simple
function pointer structure.

 x86: Copy-n-paste arch_teardown_msi_irqs from msi.c to io_apic.c.
 x86: Introduce x86_msi_ops

Next, the Xen PCI stub driver. I've put it in the same location
where other sub-platform PCI drivers are. It hooks up to the
PCI legacy IRQ setup ('pcibios_enable_irq'), and MSI/MSI-X
allocation/de-allocation (via the x86_msi_ops introduced in earlier patches).

 xen/x86/PCI: Add support for the Xen PCI subsystem

Lastly, the Xen PCI front-end driver which is responsible for hooking up
to the PCI configuration read/write methods via the 'pci_scan_bus_parented' call.
In essence all pci_conf_read/write in the guest will be tunneled via
pcifront_bus_[read|write] methods. The MSI/MSI-X calls will be handled
by the Xen-PCI front-end driver as well.

 xen-pcifront: Xen PCI frontend driver.

The next two are a bug fix and updating the MAINTAINER file:

 MAINTAINERS: Add myself for Xen PCI and Xen SWIOTLB maintainer.
 xen/pci: Request ACS when Xen-SWIOTLB is activated.


The shortlog and the diffstat:

Alex Nixon (3):
      xen: Don't disable the I/O space
      x86/PCI: Clean up pci_cache_line_size
      xen/x86/PCI: Add support for the Xen PCI subsystem

Gerd Hoffmann (1):
      xen: set pirq name to something useful.

Jeremy Fitzhardinge (5):
      xen: define BIOVEC_PHYS_MERGEABLE()
      xen: implement pirq type event channels
      x86/io_apic: add get_nr_irqs_gsi()
      xen: statically initialize cpu_evtchn_mask_p
      x86/PCI: make sure _PAGE_IOMAP it set on pci mappings

Konrad Rzeszutek Wilk (8):
      xen: identity map gsi->irqs
      xen: dynamically allocate irq & event structures
      xen: Find an unbound irq number in reverse order (high to low).
      xen: Provide a variant of xen_poll_irq with timeout.
      x86/PCI: Export pci_walk_bus function.
      x86: Copy-n-paste arch_teardown_msi_irqs from msi.c to io_apic.c.
      xen/pci: Request ACS when Xen-SWIOTLB is activated.
      MAINTAINERS: Add myself for Xen PCI and Xen SWIOTLB maintainer.

Noboru Iwamatsu (1):
      xenbus: prevent warnings on unhandled enumeration values

Ryan Wilson (1):
      xen-pcifront: Xen PCI frontend driver.

Stefano Stabellini (1):
      x86: Introduce x86_msi_ops

Weidong Han (1):
      xen: fix shared irq device passthrough

Yosuke Iwamatsu (1):
      xenbus: Xen paravirtualised PCI hotplug support.


 MAINTAINERS                        |   14 +
 arch/x86/Kconfig                   |    5 +
 arch/x86/include/asm/io.h          |   13 +
 arch/x86/include/asm/io_apic.h     |    1 +
 arch/x86/include/asm/pci.h         |   25 +-
 arch/x86/include/asm/pci_x86.h     |    1 +
 arch/x86/include/asm/x86_init.h    |    9 +
 arch/x86/include/asm/xen/pci.h     |   53 ++
 arch/x86/kernel/apic/io_apic.c     |   23 +-
 arch/x86/kernel/x86_init.c         |    6 +
 arch/x86/pci/Makefile              |    1 +
 arch/x86/pci/common.c              |   17 +-
 arch/x86/pci/i386.c                |    2 +
 arch/x86/pci/xen.c                 |  147 +++++
 arch/x86/xen/enlighten.c           |    3 +
 arch/x86/xen/pci-swiotlb-xen.c     |    4 +
 arch/x86/xen/setup.c               |    2 -
 drivers/block/xen-blkfront.c       |    2 +
 drivers/input/xen-kbdfront.c       |    2 +
 drivers/net/xen-netfront.c         |    2 +
 drivers/pci/Kconfig                |   16 +
 drivers/pci/Makefile               |    2 +
 drivers/pci/bus.c                  |    1 +
 drivers/pci/xen-pcifront.c         | 1157 ++++++++++++++++++++++++++++++++++++
 drivers/video/xen-fbfront.c        |    2 +
 drivers/xen/Makefile               |    2 +-
 drivers/xen/biomerge.c             |   13 +
 drivers/xen/events.c               |  338 ++++++++++-
 drivers/xen/xenbus/xenbus_client.c |    2 +
 include/xen/events.h               |   18 +
 include/xen/interface/io/pciif.h   |  122 ++++
 include/xen/interface/io/xenbus.h  |    8 +-
 32 files changed, 1982 insertions(+), 31 deletions(-)

P.S.
[1]. git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git 2.6.36-rc1-pvhvm-pirq-v3

[2]: http://qubes-os.org/ which utilizes hardware IOMMU to run seperate domains wherein
each has specific access to hardware.

[3] Some of the authors of the patches have moved on, so their e-mails
are bouncing. Any thoughts of what to do about that? Just leave it as is?  

[PATCH 01/22] xen: Don't disable the I/O space

[Konrad Rzeszutek Wilk]

From: Alex Nixon <alex.nixon@citrix.com>

If a guest domain wants to access PCI devices through the frontend
driver (coming later in the patch series), it will need access to the
I/O space.

[ Impact: Allow for domU IO access, preparing for pci passthrough ]

Signed-off-by: Alex Nixon <alex.nixon@citrix.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/xen/setup.c |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)

diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index 328b003..c413132 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -260,7 +260,5 @@ void __init xen_arch_setup(void)
 
 	pm_idle = xen_idle;
 
-	paravirt_disable_iospace();
-
 	fiddle_vdso();
 }
-- 
1.7.0.4

[PATCH 02/22] xen: define BIOVEC_PHYS_MERGEABLE()

[Konrad Rzeszutek Wilk]

From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

Impact: allow Xen control of bio merging

When running in Xen domain with device access, we need to make sure
the block subsystem doesn't merge requests across pages which aren't
machine physically contiguous.  To do this, we define our own
BIOVEC_PHYS_MERGEABLE.  When CONFIG_XEN isn't enabled, or we're not
running in a Xen domain, this has identical behaviour to the normal
implementation.  When running under Xen, we also make sure the
underlying machine pages are the same or adjacent.

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/include/asm/io.h |   13 +++++++++++++
 drivers/xen/Makefile      |    2 +-
 drivers/xen/biomerge.c    |   13 +++++++++++++
 3 files changed, 27 insertions(+), 1 deletions(-)
 create mode 100644 drivers/xen/biomerge.c

diff --git a/arch/x86/include/asm/io.h b/arch/x86/include/asm/io.h
index 30a3e97..0ad29d4 100644
--- a/arch/x86/include/asm/io.h
+++ b/arch/x86/include/asm/io.h
@@ -41,6 +41,8 @@
 #include <asm-generic/int-ll64.h>
 #include <asm/page.h>
 
+#include <xen/xen.h>
+
 #define build_mmio_read(name, size, type, reg, barrier) \
 static inline type name(const volatile void __iomem *addr) \
 { type ret; asm volatile("mov" size " %1,%0":reg (ret) \
@@ -349,6 +351,17 @@ extern void __iomem *early_memremap(resource_size_t phys_addr,
 extern void early_iounmap(void __iomem *addr, unsigned long size);
 extern void fixup_early_ioremap(void);
 
+#ifdef CONFIG_XEN
+struct bio_vec;
+
+extern bool xen_biovec_phys_mergeable(const struct bio_vec *vec1,
+				      const struct bio_vec *vec2);
+
+#define BIOVEC_PHYS_MERGEABLE(vec1, vec2)				\
+	(__BIOVEC_PHYS_MERGEABLE(vec1, vec2) &&				\
+	 (!xen_domain() || xen_biovec_phys_mergeable(vec1, vec2)))
+#endif	/* CONFIG_XEN */
+
 #define IO_SPACE_LIMIT 0xffff
 
 #endif /* _ASM_X86_IO_H */
diff --git a/drivers/xen/Makefile b/drivers/xen/Makefile
index fcaf838..b47f5da 100644
--- a/drivers/xen/Makefile
+++ b/drivers/xen/Makefile
@@ -1,4 +1,4 @@
-obj-y	+= grant-table.o features.o events.o manage.o
+obj-y	+= grant-table.o features.o events.o manage.o biomerge.o
 obj-y	+= xenbus/
 
 nostackp := $(call cc-option, -fno-stack-protector)
diff --git a/drivers/xen/biomerge.c b/drivers/xen/biomerge.c
new file mode 100644
index 0000000..ba6eda4
--- /dev/null
+++ b/drivers/xen/biomerge.c
@@ -0,0 +1,13 @@
+#include <linux/bio.h>
+#include <linux/io.h>
+#include <xen/page.h>
+
+bool xen_biovec_phys_mergeable(const struct bio_vec *vec1,
+			       const struct bio_vec *vec2)
+{
+	unsigned long mfn1 = pfn_to_mfn(page_to_pfn(vec1->bv_page));
+	unsigned long mfn2 = pfn_to_mfn(page_to_pfn(vec2->bv_page));
+
+	return __BIOVEC_PHYS_MERGEABLE(vec1, vec2) &&
+		((mfn1 == mfn2) || ((mfn1+1) == mfn2));
+}
-- 
1.7.0.4

[PATCH 03/22] xen: implement pirq type event channels

[Konrad Rzeszutek Wilk]

From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

A privileged PV Xen domain can get direct access to hardware.  In
order for this to be useful, it must be able to get hardware
interrupts.

Being a PV Xen domain, all interrupts are delivered as event channels.
PIRQ event channels are bound to a pirq number and an interrupt
vector.  When a IO APIC raises a hardware interrupt on that vector, it
is delivered as an event channel, which we can deliver to the
appropriate device driver(s).

This patch simply implements the infrastructure for dealing with pirq
event channels.

[ Impact: integrate hardware interrupts into Xen's event scheme ]

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/events.c |  243 +++++++++++++++++++++++++++++++++++++++++++++++++-
 include/xen/events.h |   11 +++
 2 files changed, 252 insertions(+), 2 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 13365ba..b8f030a 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -16,7 +16,7 @@
  *    (typically dom0).
  * 2. VIRQs, typically used for timers.  These are per-cpu events.
  * 3. IPIs.
- * 4. Hardware interrupts. Not supported at present.
+ * 4. PIRQs - Hardware interrupts.
  *
  * Jeremy Fitzhardinge <jeremy@xensource.com>, XenSource Inc, 2007
  */
@@ -46,6 +46,9 @@
 #include <xen/interface/hvm/hvm_op.h>
 #include <xen/interface/hvm/params.h>
 
+/* Leave low irqs free for identity mapping */
+#define LEGACY_IRQS	16
+
 /*
  * This lock protects updates to the following mapping and reference-count
  * arrays. The lock does not need to be acquired to read the mapping tables.
@@ -89,10 +92,12 @@ struct irq_info
 		enum ipi_vector ipi;
 		struct {
 			unsigned short gsi;
-			unsigned short vector;
+			unsigned char vector;
+			unsigned char flags;
 		} pirq;
 	} u;
 };
+#define PIRQ_NEEDS_EOI	(1 << 0)
 
 static struct irq_info irq_info[NR_IRQS];
 
@@ -113,6 +118,7 @@ static inline unsigned long *cpu_evtchn_mask(int cpu)
 
 static struct irq_chip xen_dynamic_chip;
 static struct irq_chip xen_percpu_chip;
+static struct irq_chip xen_pirq_chip;
 
 /* Constructor for packed IRQ information. */
 static struct irq_info mk_unbound_info(void)
@@ -225,6 +231,15 @@ static unsigned int cpu_from_evtchn(unsigned int evtchn)
 	return ret;
 }
 
+static bool pirq_needs_eoi(unsigned irq)
+{
+	struct irq_info *info = info_for_irq(irq);
+
+	BUG_ON(info->type != IRQT_PIRQ);
+
+	return info->u.pirq.flags & PIRQ_NEEDS_EOI;
+}
+
 static inline unsigned long active_evtchns(unsigned int cpu,
 					   struct shared_info *sh,
 					   unsigned int idx)
@@ -366,6 +381,210 @@ static int find_unbound_irq(void)
 	return irq;
 }
 
+static bool identity_mapped_irq(unsigned irq)
+{
+	/* only identity map legacy irqs */
+	return irq < LEGACY_IRQS;
+}
+
+static void pirq_unmask_notify(int irq)
+{
+	struct physdev_eoi eoi = { .irq = irq };
+
+	if (unlikely(pirq_needs_eoi(irq))) {
+		int rc = HYPERVISOR_physdev_op(PHYSDEVOP_eoi, &eoi);
+		WARN_ON(rc);
+	}
+}
+
+static void pirq_query_unmask(int irq)
+{
+	struct physdev_irq_status_query irq_status;
+	struct irq_info *info = info_for_irq(irq);
+
+	BUG_ON(info->type != IRQT_PIRQ);
+
+	irq_status.irq = irq;
+	if (HYPERVISOR_physdev_op(PHYSDEVOP_irq_status_query, &irq_status))
+		irq_status.flags = 0;
+
+	info->u.pirq.flags &= ~PIRQ_NEEDS_EOI;
+	if (irq_status.flags & XENIRQSTAT_needs_eoi)
+		info->u.pirq.flags |= PIRQ_NEEDS_EOI;
+}
+
+static bool probing_irq(int irq)
+{
+	struct irq_desc *desc = irq_to_desc(irq);
+
+	return desc && desc->action == NULL;
+}
+
+static unsigned int startup_pirq(unsigned int irq)
+{
+	struct evtchn_bind_pirq bind_pirq;
+	struct irq_info *info = info_for_irq(irq);
+	int evtchn = evtchn_from_irq(irq);
+
+	BUG_ON(info->type != IRQT_PIRQ);
+
+	if (VALID_EVTCHN(evtchn))
+		goto out;
+
+	bind_pirq.pirq = irq;
+	/* NB. We are happy to share unless we are probing. */
+	bind_pirq.flags = probing_irq(irq) ? 0 : BIND_PIRQ__WILL_SHARE;
+	if (HYPERVISOR_event_channel_op(EVTCHNOP_bind_pirq, &bind_pirq) != 0) {
+		if (!probing_irq(irq))
+			printk(KERN_INFO "Failed to obtain physical IRQ %d\n",
+			       irq);
+		return 0;
+	}
+	evtchn = bind_pirq.port;
+
+	pirq_query_unmask(irq);
+
+	evtchn_to_irq[evtchn] = irq;
+	bind_evtchn_to_cpu(evtchn, 0);
+	info->evtchn = evtchn;
+
+out:
+	unmask_evtchn(evtchn);
+	pirq_unmask_notify(irq);
+
+	return 0;
+}
+
+static void shutdown_pirq(unsigned int irq)
+{
+	struct evtchn_close close;
+	struct irq_info *info = info_for_irq(irq);
+	int evtchn = evtchn_from_irq(irq);
+
+	BUG_ON(info->type != IRQT_PIRQ);
+
+	if (!VALID_EVTCHN(evtchn))
+		return;
+
+	mask_evtchn(evtchn);
+
+	close.port = evtchn;
+	if (HYPERVISOR_event_channel_op(EVTCHNOP_close, &close) != 0)
+		BUG();
+
+	bind_evtchn_to_cpu(evtchn, 0);
+	evtchn_to_irq[evtchn] = -1;
+	info->evtchn = 0;
+}
+
+static void enable_pirq(unsigned int irq)
+{
+	startup_pirq(irq);
+}
+
+static void disable_pirq(unsigned int irq)
+{
+}
+
+static void ack_pirq(unsigned int irq)
+{
+	int evtchn = evtchn_from_irq(irq);
+
+	move_native_irq(irq);
+
+	if (VALID_EVTCHN(evtchn)) {
+		mask_evtchn(evtchn);
+		clear_evtchn(evtchn);
+	}
+}
+
+static void end_pirq(unsigned int irq)
+{
+	int evtchn = evtchn_from_irq(irq);
+	struct irq_desc *desc = irq_to_desc(irq);
+
+	if (WARN_ON(!desc))
+		return;
+
+	if ((desc->status & (IRQ_DISABLED|IRQ_PENDING)) ==
+	    (IRQ_DISABLED|IRQ_PENDING)) {
+		shutdown_pirq(irq);
+	} else if (VALID_EVTCHN(evtchn)) {
+		unmask_evtchn(evtchn);
+		pirq_unmask_notify(irq);
+	}
+}
+
+static int find_irq_by_gsi(unsigned gsi)
+{
+	int irq;
+
+	for (irq = 0; irq < NR_IRQS; irq++) {
+		struct irq_info *info = info_for_irq(irq);
+
+		if (info == NULL || info->type != IRQT_PIRQ)
+			continue;
+
+		if (gsi_from_irq(irq) == gsi)
+			return irq;
+	}
+
+	return -1;
+}
+
+/*
+ * Allocate a physical irq, along with a vector.  We don't assign an
+ * event channel until the irq actually started up.  Return an
+ * existing irq if we've already got one for the gsi.
+ */
+int xen_allocate_pirq(unsigned gsi)
+{
+	int irq;
+	struct physdev_irq irq_op;
+
+	spin_lock(&irq_mapping_update_lock);
+
+	irq = find_irq_by_gsi(gsi);
+	if (irq != -1) {
+		printk(KERN_INFO "xen_allocate_pirq: returning irq %d for gsi %u\n",
+		       irq, gsi);
+		goto out;	/* XXX need refcount? */
+	}
+
+	if (identity_mapped_irq(gsi)) {
+		irq = gsi;
+		dynamic_irq_init(irq);
+	} else
+		irq = find_unbound_irq();
+
+	set_irq_chip_and_handler_name(irq, &xen_pirq_chip,
+				      handle_level_irq, "pirq");
+
+	irq_op.irq = irq;
+	if (HYPERVISOR_physdev_op(PHYSDEVOP_alloc_irq_vector, &irq_op)) {
+		dynamic_irq_cleanup(irq);
+		irq = -ENOSPC;
+		goto out;
+	}
+
+	irq_info[irq] = mk_pirq_info(0, gsi, irq_op.vector);
+
+out:
+	spin_unlock(&irq_mapping_update_lock);
+
+	return irq;
+}
+
+int xen_vector_from_irq(unsigned irq)
+{
+	return vector_from_irq(irq);
+}
+
+int xen_gsi_from_irq(unsigned irq)
+{
+	return gsi_from_irq(irq);
+}
+
 int bind_evtchn_to_irq(unsigned int evtchn)
 {
 	int irq;
@@ -965,6 +1184,26 @@ static struct irq_chip xen_dynamic_chip __read_mostly = {
 	.retrigger	= retrigger_dynirq,
 };
 
+static struct irq_chip xen_pirq_chip __read_mostly = {
+	.name		= "xen-pirq",
+
+	.startup	= startup_pirq,
+	.shutdown	= shutdown_pirq,
+
+	.enable		= enable_pirq,
+	.unmask		= enable_pirq,
+
+	.disable	= disable_pirq,
+	.mask		= disable_pirq,
+
+	.ack		= ack_pirq,
+	.end		= end_pirq,
+
+	.set_affinity	= set_affinity_irq,
+
+	.retrigger	= retrigger_dynirq,
+};
+
 static struct irq_chip xen_percpu_chip __read_mostly = {
 	.name		= "xen-percpu",
 
diff --git a/include/xen/events.h b/include/xen/events.h
index a15d932..8f62320 100644
--- a/include/xen/events.h
+++ b/include/xen/events.h
@@ -63,4 +63,15 @@ int xen_set_callback_via(uint64_t via);
 void xen_evtchn_do_upcall(struct pt_regs *regs);
 void xen_hvm_evtchn_do_upcall(void);
 
+/* Allocate an irq for a physical interrupt, given a gsi.  "Legacy"
+ * GSIs are identity mapped; others are dynamically allocated as
+ * usual. */
+int xen_allocate_pirq(unsigned gsi);
+
+/* Return vector allocated to pirq */
+int xen_vector_from_irq(unsigned pirq);
+
+/* Return gsi allocated to pirq */
+int xen_gsi_from_irq(unsigned pirq);
+
 #endif	/* _XEN_EVENTS_H */
-- 
1.7.0.4

[PATCH 04/22] x86/io_apic: add get_nr_irqs_gsi()

[Konrad Rzeszutek Wilk]

From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

Impact: new interface to get max GSI

Add get_nr_irqs_gsi() to return nr_irqs_gsi.  Xen will use this to
determine how many irqs it needs to reserve for hardware irqs.

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: "H. Peter Anvin" <hpa@zytor.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: x86@kernel.org
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
---
 arch/x86/include/asm/io_apic.h |    1 +
 arch/x86/kernel/apic/io_apic.c |    5 +++++
 2 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
index 9cb2edb..f27c681 100644
--- a/arch/x86/include/asm/io_apic.h
+++ b/arch/x86/include/asm/io_apic.h
@@ -169,6 +169,7 @@ extern void mask_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries);
 extern int restore_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries);
 
 extern void probe_nr_irqs_gsi(void);
+extern int get_nr_irqs_gsi(void);
 
 extern int setup_ioapic_entry(int apic, int irq,
 			      struct IO_APIC_route_entry *entry,
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index f1efeba..4c9b2b9 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -3862,6 +3862,11 @@ void __init probe_nr_irqs_gsi(void)
 	printk(KERN_DEBUG "nr_irqs_gsi: %d\n", nr_irqs_gsi);
 }
 
+int get_nr_irqs_gsi(void)
+{
+	return nr_irqs_gsi;
+}
+
 #ifdef CONFIG_SPARSE_IRQ
 int __init arch_probe_nr_irqs(void)
 {
-- 
1.7.0.4

Re: [PATCH 04/22] x86/io_apic: add get_nr_irqs_gsi()

[Thomas Gleixner]

On Mon, 4 Oct 2010, Konrad Rzeszutek Wilk wrote:

> From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> 
> Impact: new interface to get max GSI
> 
> Add get_nr_irqs_gsi() to return nr_irqs_gsi.  Xen will use this to
> determine how many irqs it needs to reserve for hardware irqs.

Acked.by: Thomas Gleixner <tglx@linutronix.de>

> Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Reviewed-by: "H. Peter Anvin" <hpa@zytor.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: x86@kernel.org
> Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
> ---
>  arch/x86/include/asm/io_apic.h |    1 +
>  arch/x86/kernel/apic/io_apic.c |    5 +++++
>  2 files changed, 6 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
> index 9cb2edb..f27c681 100644
> --- a/arch/x86/include/asm/io_apic.h
> +++ b/arch/x86/include/asm/io_apic.h
> @@ -169,6 +169,7 @@ extern void mask_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries);
>  extern int restore_IO_APIC_setup(struct IO_APIC_route_entry **ioapic_entries);
>  
>  extern void probe_nr_irqs_gsi(void);
> +extern int get_nr_irqs_gsi(void);
>  
>  extern int setup_ioapic_entry(int apic, int irq,
>  			      struct IO_APIC_route_entry *entry,
> diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
> index f1efeba..4c9b2b9 100644
> --- a/arch/x86/kernel/apic/io_apic.c
> +++ b/arch/x86/kernel/apic/io_apic.c
> @@ -3862,6 +3862,11 @@ void __init probe_nr_irqs_gsi(void)
>  	printk(KERN_DEBUG "nr_irqs_gsi: %d\n", nr_irqs_gsi);
>  }
>  
> +int get_nr_irqs_gsi(void)
> +{
> +	return nr_irqs_gsi;
> +}
> +
>  #ifdef CONFIG_SPARSE_IRQ
>  int __init arch_probe_nr_irqs(void)
>  {
> -- 
> 1.7.0.4
> 

[PATCH 05/22] xen: identity map gsi->irqs

[Konrad Rzeszutek Wilk]

Impact: preserve compat with native

Reserve the lower irq range for use for hardware interrupts so we
can identity-map them.

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/events.c |   23 +++++++++++++++++------
 1 files changed, 17 insertions(+), 6 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index b8f030a..8eeb808 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -33,6 +33,7 @@
 #include <asm/ptrace.h>
 #include <asm/irq.h>
 #include <asm/idle.h>
+#include <asm/io_apic.h>
 #include <asm/sync_bitops.h>
 #include <asm/xen/hypercall.h>
 #include <asm/xen/hypervisor.h>
@@ -46,9 +47,6 @@
 #include <xen/interface/hvm/hvm_op.h>
 #include <xen/interface/hvm/params.h>
 
-/* Leave low irqs free for identity mapping */
-#define LEGACY_IRQS	16
-
 /*
  * This lock protects updates to the following mapping and reference-count
  * arrays. The lock does not need to be acquired to read the mapping tables.
@@ -351,12 +349,24 @@ static void unmask_evtchn(int port)
 	put_cpu();
 }
 
+static int get_nr_hw_irqs(void)
+{
+	int ret = 1;
+
+#ifdef CONFIG_X86_IO_APIC
+	ret = get_nr_irqs_gsi();
+#endif
+
+	return ret;
+}
+
 static int find_unbound_irq(void)
 {
 	int irq;
 	struct irq_desc *desc;
+	int start = get_nr_hw_irqs();
 
-	for (irq = 0; irq < nr_irqs; irq++) {
+	for (irq = start; irq < nr_irqs; irq++) {
 		desc = irq_to_desc(irq);
 		/* only 0->15 have init'd desc; handle irq > 16 */
 		if (desc == NULL)
@@ -383,8 +393,8 @@ static int find_unbound_irq(void)
 
 static bool identity_mapped_irq(unsigned irq)
 {
-	/* only identity map legacy irqs */
-	return irq < LEGACY_IRQS;
+	/* identity map all the hardware irqs */
+	return irq < get_nr_hw_irqs();
 }
 
 static void pirq_unmask_notify(int irq)
@@ -553,6 +563,7 @@ int xen_allocate_pirq(unsigned gsi)
 
 	if (identity_mapped_irq(gsi)) {
 		irq = gsi;
+		irq_to_desc_alloc_node(irq, 0);
 		dynamic_irq_init(irq);
 	} else
 		irq = find_unbound_irq();
-- 
1.7.0.4

[PATCH 07/22] xen: set pirq name to something useful.

[Konrad Rzeszutek Wilk]

Impact: cleanup

Make pirq show useful information in /proc/interrupts

[v2: Removed the parts for arch/x86/xen/pci.c ]

Signed-off-by: Gerd Hoffmann <kraxel@xeni.home.kraxel.org>
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/events.c |    4 ++--
 include/xen/events.h |    2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 25412ef..a98e720 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -546,7 +546,7 @@ static int find_irq_by_gsi(unsigned gsi)
  * event channel until the irq actually started up.  Return an
  * existing irq if we've already got one for the gsi.
  */
-int xen_allocate_pirq(unsigned gsi)
+int xen_allocate_pirq(unsigned gsi, char *name)
 {
 	int irq;
 	struct physdev_irq irq_op;
@@ -568,7 +568,7 @@ int xen_allocate_pirq(unsigned gsi)
 		irq = find_unbound_irq();
 
 	set_irq_chip_and_handler_name(irq, &xen_pirq_chip,
-				      handle_level_irq, "pirq");
+				      handle_level_irq, name);
 
 	irq_op.irq = irq;
 	if (HYPERVISOR_physdev_op(PHYSDEVOP_alloc_irq_vector, &irq_op)) {
diff --git a/include/xen/events.h b/include/xen/events.h
index 8f62320..8227da8 100644
--- a/include/xen/events.h
+++ b/include/xen/events.h
@@ -66,7 +66,7 @@ void xen_hvm_evtchn_do_upcall(void);
 /* Allocate an irq for a physical interrupt, given a gsi.  "Legacy"
  * GSIs are identity mapped; others are dynamically allocated as
  * usual. */
-int xen_allocate_pirq(unsigned gsi);
+int xen_allocate_pirq(unsigned gsi, char *name);
 
 /* Return vector allocated to pirq */
 int xen_vector_from_irq(unsigned pirq);
-- 
1.7.0.4

[PATCH 08/22] xen: statically initialize cpu_evtchn_mask_p

[Konrad Rzeszutek Wilk]

From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

Sometimes cpu_evtchn_mask_p can get used early, before it has been
allocated.  Statically initialize it with an initdata version to catch
any early references.

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 drivers/xen/events.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index a98e720..e1bd0be 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -104,7 +104,12 @@ static int *evtchn_to_irq;
 struct cpu_evtchn_s {
 	unsigned long bits[NR_EVENT_CHANNELS/BITS_PER_LONG];
 };
-static struct cpu_evtchn_s *cpu_evtchn_mask_p;
+
+static __initdata struct cpu_evtchn_s init_evtchn_mask = {
+	.bits[0 ... (NR_EVENT_CHANNELS/BITS_PER_LONG)-1] = ~0ul,
+};
+static struct cpu_evtchn_s *cpu_evtchn_mask_p = &init_evtchn_mask;
+
 static inline unsigned long *cpu_evtchn_mask(int cpu)
 {
 	return cpu_evtchn_mask_p[cpu].bits;
-- 
1.7.0.4

[PATCH 09/22] xen: Find an unbound irq number in reverse order (high to low).

[Konrad Rzeszutek Wilk]

In earlier Xen Linux kernels, the IRQ mapping was a straight 1:1 and the
find_unbound_irq started looking around 256 for open IRQs and up. IRQs
from 0 to 255 were reserved for PCI devices.  Previous to this patch,
the 'find_unbound_irq'  started looking at get_nr_hw_irqs() number.
For privileged  domain where the ACPI information is available that
returns the upper-bound of what the GSIs. For non-privileged PV domains,
where ACPI is no-existent the get_nr_hw_irqs() reports the IRQ_LEGACY (16).
With PCI passthrough enabled, and with PCI cards that have IRQs pinned
to a higher number than 16 we collide with previously allocated IRQs.
Specifically the PCI IRQs collide with the IPI's for Xen functions
(as they are allocated earlier).
For example:

00:00.11 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller (prog-if 10 [OHCI])
	...
	Interrupt: pin A routed to IRQ 18

[root@localhost ~]# cat /proc/interrupts | head
           CPU0       CPU1       CPU2
 16:      38186          0          0   xen-dyn-virq      timer0
 17:        149          0          0   xen-dyn-ipi       spinlock0
 18:        962          0          0   xen-dyn-ipi       resched0

and when the USB controller is loaded, the kernel reports:
IRQ handler type mismatch for IRQ 18
current handler: resched0

One way to fix this is to reverse the logic when looking for un-used
IRQ numbers and start with the highest available number. With that,
we would get:

           CPU0       CPU1       CPU2
... snip ..
292:         35          0          0   xen-dyn-ipi       callfunc0
293:       3992          0          0   xen-dyn-ipi       resched0
294:        224          0          0   xen-dyn-ipi       spinlock0
295:      57183          0          0   xen-dyn-virq      timer0
NMI:          0          0          0   Non-maskable interrupts
.. snip ..

And interrupts for PCI cards are now accessible.

This patch also includes the fix, found by Ian Campbell, titled
"xen: fix off-by-one error in find_unbound_irq."

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
---
 drivers/xen/events.c |   13 ++++++++++---
 1 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index e1bd0be..385734f 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -370,7 +370,11 @@ static int find_unbound_irq(void)
 	struct irq_desc *desc;
 	int start = get_nr_hw_irqs();
 
-	for (irq = start; irq < nr_irqs; irq++) {
+	if (start == nr_irqs)
+		goto no_irqs;
+
+	/* nr_irqs is a magic value. Must not use it.*/
+	for (irq = nr_irqs-1; irq > start; irq--) {
 		desc = irq_to_desc(irq);
 		/* only 0->15 have init'd desc; handle irq > 16 */
 		if (desc == NULL)
@@ -383,8 +387,8 @@ static int find_unbound_irq(void)
 			break;
 	}
 
-	if (irq == nr_irqs)
-		panic("No available IRQ to bind to: increase nr_irqs!\n");
+	if (irq == start)
+		goto no_irqs;
 
 	desc = irq_to_desc_alloc_node(irq, 0);
 	if (WARN_ON(desc == NULL))
@@ -393,6 +397,9 @@ static int find_unbound_irq(void)
 	dynamic_irq_init_keep_chip_data(irq);
 
 	return irq;
+
+no_irqs:
+	panic("No available IRQ to bind to: increase nr_irqs!\n");
 }
 
 static bool identity_mapped_irq(unsigned irq)
-- 
1.7.0.4

Re: [PATCH 09/22] xen: Find an unbound irq number in reverse order (high to low).

[Stefano Stabellini]

On Mon, 4 Oct 2010, Konrad Rzeszutek Wilk wrote:
> In earlier Xen Linux kernels, the IRQ mapping was a straight 1:1 and the
> find_unbound_irq started looking around 256 for open IRQs and up. IRQs
> from 0 to 255 were reserved for PCI devices.  Previous to this patch,
> the 'find_unbound_irq'  started looking at get_nr_hw_irqs() number.
> For privileged  domain where the ACPI information is available that
> returns the upper-bound of what the GSIs. For non-privileged PV domains,
> where ACPI is no-existent the get_nr_hw_irqs() reports the IRQ_LEGACY (16).
> With PCI passthrough enabled, and with PCI cards that have IRQs pinned
> to a higher number than 16 we collide with previously allocated IRQs.
> Specifically the PCI IRQs collide with the IPI's for Xen functions
> (as they are allocated earlier).
> For example:
> 
> 00:00.11 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller (prog-if 10 [OHCI])
> 	...
> 	Interrupt: pin A routed to IRQ 18
> 
> [root@localhost ~]# cat /proc/interrupts | head
>            CPU0       CPU1       CPU2
>  16:      38186          0          0   xen-dyn-virq      timer0
>  17:        149          0          0   xen-dyn-ipi       spinlock0
>  18:        962          0          0   xen-dyn-ipi       resched0
> 
> and when the USB controller is loaded, the kernel reports:
> IRQ handler type mismatch for IRQ 18
> current handler: resched0
> 
> One way to fix this is to reverse the logic when looking for un-used
> IRQ numbers and start with the highest available number. With that,
> we would get:
> 
>            CPU0       CPU1       CPU2
> ... snip ..
> 292:         35          0          0   xen-dyn-ipi       callfunc0
> 293:       3992          0          0   xen-dyn-ipi       resched0
> 294:        224          0          0   xen-dyn-ipi       spinlock0
> 295:      57183          0          0   xen-dyn-virq      timer0
> NMI:          0          0          0   Non-maskable interrupts
> .. snip ..
> 
> And interrupts for PCI cards are now accessible.
> 

Unfortunately this is the wrong way to fix the issue: Xen has a range of
allowed pirq for each domain and we don't know exactly what is the
maximum pirq (see my patch "xen: get the maximum number of pirqs from
xen" [1]).
Considering that we might use the irq number returned by
find_unbound_irq through xen_allocate_pirq as pirq number in some cases,
starting from the highest value could be unsafe.
In practice it should be impossible to see this issue because it can
only happen if the irq returned by xen_allocate_pirq is higher than the
max pirq in xen. However AFAIK when we call xen_allocate_pirq with the
intention of using the return value as pirq we always fall in the if
(identity_mapped_irq(gsi) || !xen_initial_domain()) that avoid calling
find_unbound_irq.
In any case I still think that it is a good idea to try to avoid the problem.

In order to solve this issue you can:

- add a static offset to get_nr_hw_irqs in case of pv domains;

- use the patch "xen: get the maximum number of pirqs from
xen" [1] to get the highest possible pirq number from xen so that we can be
sure we are using an allowed value;

- if the xen interface allows it, disjoin the concept of pirq from the
concept of linux irq; then you can let xen choose any pirq number it
wants (see my patch "xen: support pirq != irq" [2] and how you can exploit
it in the patch "xen: implement xen_hvm_register_pirq" [3]).
Of course this doesn't help if it must be linux the one that chooses the
pirq number.

Feel free to pick up any of the patches I sent as part of the "PV on
HVM: receive interrupts as xen events" series.


References:
[1] http://lkml.org/lkml/2010/8/30/177
[2] http://lkml.org/lkml/2010/8/30/171
[3] http://lkml.org/lkml/2010/8/30/172

Re: [PATCH 09/22] xen: Find an unbound irq number in reverse order (high to low).

[Konrad Rzeszutek Wilk]

> 
> Unfortunately this is the wrong way to fix the issue: Xen has a range of
> allowed pirq for each domain and we don't know exactly what is the
> maximum pirq (see my patch "xen: get the maximum number of pirqs from
> xen" [1]).

> Considering that we might use the irq number returned by
> find_unbound_irq through xen_allocate_pirq as pirq number in some cases,

Ah, but we wouldn't! We would end up only using the 'find_unbound_irq' for
event channels. For IRQs that are for physical devices (either being
real devices passed in or QEMU PCI devices) we end up requesting an IRQ that
matches whatever the device has defined in dev->irq (or whatever the
vectors values for MSI/MSI-X devices that is provided) via the Xen PCI frontend
driver (in case of QEMU whatever its emulation provides).

> starting from the highest value could be unsafe.
> In practice it should be impossible to see this issue because it can
> only happen if the irq returned by xen_allocate_pirq is higher than the
> max pirq in xen. However AFAIK when we call xen_allocate_pirq with the
> intention of using the return value as pirq we always fall in the if
> (identity_mapped_irq(gsi) || !xen_initial_domain()) that avoid calling
> find_unbound_irq.

Right, and we end up using an the pirq/gsi number at that point. This
patch would not touch that logic.

> In any case I still think that it is a good idea to try to avoid the problem.
> 
> In order to solve this issue you can:
> 
> - add a static offset to get_nr_hw_irqs in case of pv domains;
> 
> - use the patch "xen: get the maximum number of pirqs from
> xen" [1] to get the highest possible pirq number from xen so that we can be
> sure we are using an allowed value;
> 
> - if the xen interface allows it, disjoin the concept of pirq from the
> concept of linux irq; then you can let xen choose any pirq number it
> wants (see my patch "xen: support pirq != irq" [2] and how you can exploit
> it in the patch "xen: implement xen_hvm_register_pirq" [3]).
> Of course this doesn't help if it must be linux the one that chooses the
> pirq number.

All of those deal with PCI devices. This patch is intended for the bug 
where we want to deal with event channels and we don't want them to
usurp the irq_desc that are above NR_IRQs and where we get a PCI device for
which its IRQ falls within what info_for_irq(irq) would return: IRQT_VIRQ or
 IRQT_IPI or IRQT_EVTCHN.

In other words, avoid colliding with event channels b/c those are allocated
earlier than the PCI devices.

FYI, the reason this is not triggered under HVM is that it has an IOAPIC.
But under PV no such beast. Here is an output from /proc/interrupts:

A) HVM guest (2.6.36-rc6 w/ PCI passthrough):
            CPU0       CPU1       

   0:        191          0   IO-APIC-edge      timer

   1:          4          3   IO-APIC-edge      i8042

   4:         46         25   IO-APIC-edge      serial

   8:          0          1   IO-APIC-edge      rtc0

   9:          0          0   IO-APIC-fasteoi   acpi

  12:         54         45   IO-APIC-edge      i8042

  14:          0          0   IO-APIC-edge      ata_piix

  15:         20         17   IO-APIC-edge      ata_piix

  16:      10670          0  xen-percpu-virq      timer0

  17:          0       9252  xen-percpu-virq      timer1

  18:        138          0   xen-dyn-event     xenbus

  19:          0          0   IO-APIC-fasteoi   uhci_hcd:usb4

  21:          0          0   IO-APIC-fasteoi   ehci_hcd:usb1

  40:          0          0   IO-APIC-fasteoi   uhci_hcd:usb2

  45:          0          0   IO-APIC-fasteoi   uhci_hcd:usb3

  64:         30         31   PCI-MSI-edge      eth1

..snip..

B) PV guest (2.6.36-rc6 w/ PCI passthrough)
           CPU0       CPU1       CPU2       CPU3       

  0:       1760          0          0          0  xen-percpu-virq      timer0

  1:          2          0          0          0  xen-percpu-ipi       spinlock0

  2:       2481          0          0          0  xen-percpu-ipi       resched0

  3:         23          0          0          0  xen-percpu-ipi       callfunc0

  4:          0          0          0          0  xen-percpu-virq      debug0

  5:         94          0          0          0  xen-percpu-ipi       callfuncsingle0

  6:          0       3474          0          0  xen-percpu-virq      timer1

..snip..

C) PV guest (2.6.36-rc6 devel/xen-pcifront-0.8 with PCI devices.):

 18:          0          0          0          0  xen-pirq-pcifront  uhci_hcd:usb4

 19:          0          0          0          0  xen-pirq-pcifront  uhci_hcd:usb3

 23:          0          0          0          0  xen-pirq-pcifront  ehci_hcd:usb1, uhci_hcd:usb2

103:         38          0          0          0  xen-pirq-pcifront-msi  eth1

276:         24          0          0          0   xen-dyn-event     eth0

277:        188          0          0          0   xen-dyn-event     hvc_console

278:         69          0          0          0   xen-dyn-event     pcifront

279:        259          0          0          0   xen-dyn-event     xenbus

280:          0          0          0        101  xen-percpu-ipi       callfuncsingle3

... snip..

D) HVM 2.6.36-rc6 (devel/xen-pcifront-0.8)
   0:        223          0   IO-APIC-edge      timer
   1:          3          4   IO-APIC-edge      i8042
   4:         44         53   IO-APIC-edge      serial
   8:          0          1   IO-APIC-edge      rtc0
   9:          0          0   IO-APIC-fasteoi   acpi
  12:         47         50   IO-APIC-edge      i8042
  14:          0          0   IO-APIC-edge      ata_piix
  15:         17         20   IO-APIC-edge      ata_piix
  19:          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
  21:          0          0   IO-APIC-fasteoi   ehci_hcd:usb1
  40:          0          0   IO-APIC-fasteoi   uhci_hcd:usb2
  45:          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
  64:         30         30   PCI-MSI-edge      eth1
1149:        137          0   xen-dyn-event     xenbus
1150:          0       8920  xen-percpu-virq      timer1
1151:      11156          0  xen-percpu-virq      timer0

 ..snip..

As you can see from B), the IRQs would be allocated from the start and inhibit
PCI devices from using them.

A) and D) show just the different off what this patch does.
> 
> Feel free to pick up any of the patches I sent as part of the "PV on
> HVM: receive interrupts as xen events" series.
> 
> 
> References:
> [1] http://lkml.org/lkml/2010/8/30/177
> [2] http://lkml.org/lkml/2010/8/30/171
> [3] http://lkml.org/lkml/2010/8/30/172

Re: [PATCH 09/22] xen: Find an unbound irq number in reverse order (high to low).

[Stefano Stabellini]

On Wed, 6 Oct 2010, Konrad Rzeszutek Wilk wrote:
> > 
> > Unfortunately this is the wrong way to fix the issue: Xen has a range of
> > allowed pirq for each domain and we don't know exactly what is the
> > maximum pirq (see my patch "xen: get the maximum number of pirqs from
> > xen" [1]).
> 
> > Considering that we might use the irq number returned by
> > find_unbound_irq through xen_allocate_pirq as pirq number in some cases,
> 
> Ah, but we wouldn't! We would end up only using the 'find_unbound_irq' for
> event channels. For IRQs that are for physical devices (either being
> real devices passed in or QEMU PCI devices) we end up requesting an IRQ that
> matches whatever the device has defined in dev->irq (or whatever the
> vectors values for MSI/MSI-X devices that is provided) via the Xen PCI frontend
> driver (in case of QEMU whatever its emulation provides).
> 
> > starting from the highest value could be unsafe.
> > In practice it should be impossible to see this issue because it can
> > only happen if the irq returned by xen_allocate_pirq is higher than the
> > max pirq in xen. However AFAIK when we call xen_allocate_pirq with the
> > intention of using the return value as pirq we always fall in the if
> > (identity_mapped_irq(gsi) || !xen_initial_domain()) that avoid calling
> > find_unbound_irq.
> 
> Right, and we end up using an the pirq/gsi number at that point. This
> patch would not touch that logic.

What about adding a comment on top of xen_allocate_pirq like the
following:

/* xen_allocate_irq might allocate irqs from the top down, as a
 * consequence don't assume that the irq number returned has a low value
 * or can be used as a pirq number unless you know otherwise.
 *
 * One notable exception is when xen_allocate_irq is called passing an
 * hardware gsi as argument, in that case the irq number returned
 * matches the gsi number passed as first argument.
 */

Re: [Xen-devel] Re: [PATCH 09/22] xen: Find an unbound irq number in reverse order (high to low).

[Konrad Rzeszutek Wilk]

On Thu, Oct 07, 2010 at 11:38:49AM +0100, Stefano Stabellini wrote:
> On Wed, 6 Oct 2010, Konrad Rzeszutek Wilk wrote:
> > > 
> > > Unfortunately this is the wrong way to fix the issue: Xen has a range of
> > > allowed pirq for each domain and we don't know exactly what is the
> > > maximum pirq (see my patch "xen: get the maximum number of pirqs from
> > > xen" [1]).
> > 
> > > Considering that we might use the irq number returned by
> > > find_unbound_irq through xen_allocate_pirq as pirq number in some cases,
> > 
> > Ah, but we wouldn't! We would end up only using the 'find_unbound_irq' for
> > event channels. For IRQs that are for physical devices (either being
> > real devices passed in or QEMU PCI devices) we end up requesting an IRQ that
> > matches whatever the device has defined in dev->irq (or whatever the
> > vectors values for MSI/MSI-X devices that is provided) via the Xen PCI frontend
> > driver (in case of QEMU whatever its emulation provides).
> > 
> > > starting from the highest value could be unsafe.
> > > In practice it should be impossible to see this issue because it can
> > > only happen if the irq returned by xen_allocate_pirq is higher than the
> > > max pirq in xen. However AFAIK when we call xen_allocate_pirq with the
> > > intention of using the return value as pirq we always fall in the if
> > > (identity_mapped_irq(gsi) || !xen_initial_domain()) that avoid calling
> > > find_unbound_irq.
> > 
> > Right, and we end up using an the pirq/gsi number at that point. This
> > patch would not touch that logic.
> 
> What about adding a comment on top of xen_allocate_pirq like the
> following:
> 
> /* xen_allocate_irq might allocate irqs from the top down, as a
>  * consequence don't assume that the irq number returned has a low value
>  * or can be used as a pirq number unless you know otherwise.
>  *
>  * One notable exception is when xen_allocate_irq is called passing an
>  * hardware gsi as argument, in that case the irq number returned
>  * matches the gsi number passed as first argument.
>  */

Done!

[PATCH 10/22] xen: Provide a variant of xen_poll_irq with timeout.

[Konrad Rzeszutek Wilk]

The 'xen_poll_irq_timeout' provides a method to pass in
the poll timeout for IRQs if requested. We also export
those two poll functions as Xen PCI fronted uses them.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
---
 drivers/xen/events.c |   17 ++++++++++++-----
 include/xen/events.h |    4 ++++
 2 files changed, 16 insertions(+), 5 deletions(-)

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 385734f..791932d 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -1133,7 +1133,7 @@ void xen_clear_irq_pending(int irq)
 	if (VALID_EVTCHN(evtchn))
 		clear_evtchn(evtchn);
 }
-
+EXPORT_SYMBOL(xen_clear_irq_pending);
 void xen_set_irq_pending(int irq)
 {
 	int evtchn = evtchn_from_irq(irq);
@@ -1153,9 +1153,9 @@ bool xen_test_irq_pending(int irq)
 	return ret;
 }
 
-/* Poll waiting for an irq to become pending.  In the usual case, the
-   irq will be disabled so it won't deliver an interrupt. */
-void xen_poll_irq(int irq)
+/* Poll waiting for an irq to become pending with timeout.  In the usual case,
+ * the irq will be disabled so it won't deliver an interrupt. */
+void xen_poll_irq_timeout(int irq, u64 timeout)
 {
 	evtchn_port_t evtchn = evtchn_from_irq(irq);
 
@@ -1163,13 +1163,20 @@ void xen_poll_irq(int irq)
 		struct sched_poll poll;
 
 		poll.nr_ports = 1;
-		poll.timeout = 0;
+		poll.timeout = timeout;
 		set_xen_guest_handle(poll.ports, &evtchn);
 
 		if (HYPERVISOR_sched_op(SCHEDOP_poll, &poll) != 0)
 			BUG();
 	}
 }
+EXPORT_SYMBOL(xen_poll_irq_timeout);
+/* Poll waiting for an irq to become pending.  In the usual case, the
+ * irq will be disabled so it won't deliver an interrupt. */
+void xen_poll_irq(int irq)
+{
+	xen_poll_irq_timeout(irq, 0 /* no timeout */);
+}
 
 void xen_irq_resume(void)
 {
diff --git a/include/xen/events.h b/include/xen/events.h
index 8227da8..2532f8b 100644
--- a/include/xen/events.h
+++ b/include/xen/events.h
@@ -53,6 +53,10 @@ bool xen_test_irq_pending(int irq);
    irq will be disabled so it won't deliver an interrupt. */
 void xen_poll_irq(int irq);
 
+/* Poll waiting for an irq to become pending with a timeout.  In the usual case,
+ * the irq will be disabled so it won't deliver an interrupt. */
+void xen_poll_irq_timeout(int irq, u64 timeout);
+
 /* Determine the IRQ which is bound to an event channel */
 unsigned irq_from_evtchn(unsigned int evtchn);
 
-- 
1.7.0.4

[PATCH 12/22] x86/PCI: Clean up pci_cache_line_size

[Konrad Rzeszutek Wilk]

From: Alex Nixon <alex.nixon@citrix.com>

Separate out x86 cache_line_size initialisation code into its own
function (so it can be shared by Xen later in this patch series)

[ Impact: cleanup ]

Signed-off-by: Alex Nixon <alex.nixon@citrix.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: "H. Peter Anvin" <hpa@zytor.com>
Reviewed-by: Matthew Wilcox <willy@linux.intel.com>
Reviewed-by: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: x86@kernel.org
---
 arch/x86/include/asm/pci_x86.h |    1 +
 arch/x86/pci/common.c          |   17 ++++++++++-------
 2 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/pci_x86.h b/arch/x86/include/asm/pci_x86.h
index 49c7219..7045267 100644
--- a/arch/x86/include/asm/pci_x86.h
+++ b/arch/x86/include/asm/pci_x86.h
@@ -47,6 +47,7 @@ enum pci_bf_sort_state {
 extern unsigned int pcibios_max_latency;
 
 void pcibios_resource_survey(void);
+void pcibios_set_cache_line_size(void);
 
 /* pci-pc.c */
 
diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
index a0772af..f7c8a39 100644
--- a/arch/x86/pci/common.c
+++ b/arch/x86/pci/common.c
@@ -421,16 +421,10 @@ struct pci_bus * __devinit pcibios_scan_root(int busnum)
 
 	return bus;
 }
-
-int __init pcibios_init(void)
+void __init pcibios_set_cache_line_size(void)
 {
 	struct cpuinfo_x86 *c = &boot_cpu_data;
 
-	if (!raw_pci_ops) {
-		printk(KERN_WARNING "PCI: System does not support PCI\n");
-		return 0;
-	}
-
 	/*
 	 * Set PCI cacheline size to that of the CPU if the CPU has reported it.
 	 * (For older CPUs that don't support cpuid, we se it to 32 bytes
@@ -445,7 +439,16 @@ int __init pcibios_init(void)
  		pci_dfl_cache_line_size = 32 >> 2;
 		printk(KERN_DEBUG "PCI: Unknown cacheline size. Setting to 32 bytes\n");
 	}
+}
+
+int __init pcibios_init(void)
+{
+	if (!raw_pci_ops) {
+		printk(KERN_WARNING "PCI: System does not support PCI\n");
+		return 0;
+	}
 
+	pcibios_set_cache_line_size();
 	pcibios_resource_survey();
 
 	if (pci_bf_sort >= pci_force_bf)
-- 
1.7.0.4

[PATCH 13/22] x86/PCI: make sure _PAGE_IOMAP it set on pci mappings

[Konrad Rzeszutek Wilk]

From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

When mapping pci space via /sys or /proc, make sure we're really
doing a hardware mapping by setting _PAGE_IOMAP.

[ Impact: bugfix; make PCI mappings map the right pages ]

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Reviewed-by: "H. Peter Anvin" <hpa@zytor.com>
Reviewed-by: Matthew Wilcox <willy@linux.intel.com>
Cc: x86@kernel.org
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
---
 arch/x86/pci/i386.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/arch/x86/pci/i386.c b/arch/x86/pci/i386.c
index 5525309..8379c2c 100644
--- a/arch/x86/pci/i386.c
+++ b/arch/x86/pci/i386.c
@@ -311,6 +311,8 @@ int pci_mmap_page_range(struct pci_dev *dev, struct vm_area_struct *vma,
 		 */
 		prot |= _PAGE_CACHE_UC_MINUS;
 
+	prot |= _PAGE_IOMAP;	/* creating a mapping for IO */
+
 	vma->vm_page_prot = __pgprot(prot);
 
 	if (io_remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
-- 
1.7.0.4

[PATCH 14/22] x86/PCI: Export pci_walk_bus function.

[Konrad Rzeszutek Wilk]

In preperation of modularizing Xen-pcifront the pci_walk_bus
needs to be exported so that the xen-pcifront module can walk
call the pci subsystem to walk the PCI devices and claim them.

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Jesse Barnes <jbarnes@virtuousgeek.org> [http://marc.info/?l=linux-pci&m=126149958010298&w=2]
---
 drivers/pci/bus.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 7f0af0e..69546e9 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -299,6 +299,7 @@ void pci_walk_bus(struct pci_bus *top, int (*cb)(struct pci_dev *, void *),
 	}
 	up_read(&pci_bus_sem);
 }
+EXPORT_SYMBOL_GPL(pci_walk_bus);
 
 EXPORT_SYMBOL(pci_bus_alloc_resource);
 EXPORT_SYMBOL_GPL(pci_bus_add_device);
-- 
1.7.0.4

[PATCH 15/22] x86: Copy-n-paste arch_teardown_msi_irqs from msi.c to io_apic.c.

[Konrad Rzeszutek Wilk]

In preparation for non-privileged domains to disable PCI devices'
MSI/MSIx, we need to augment arch_teardown_msi_irqs to make
a call to the privileged domain (patch to follow).

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
---
 arch/x86/include/asm/pci.h     |    1 +
 arch/x86/kernel/apic/io_apic.c |   14 ++++++++++++++
 2 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/pci.h b/arch/x86/include/asm/pci.h
index d395540..79d6ada 100644
--- a/arch/x86/include/asm/pci.h
+++ b/arch/x86/include/asm/pci.h
@@ -96,6 +96,7 @@ extern void pci_iommu_alloc(void);
 
 /* MSI arch hook */
 #define arch_setup_msi_irqs arch_setup_msi_irqs
+#define arch_teardown_msi_irqs arch_teardown_msi_irqs
 
 #define PCI_DMA_BUS_IS_PHYS (dma_ops->is_phys)
 
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 4c9b2b9..4132eea 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -3599,6 +3599,20 @@ void arch_teardown_msi_irq(unsigned int irq)
 	destroy_irq(irq);
 }
 
+void arch_teardown_msi_irqs(struct pci_dev *dev)
+{
+	struct msi_desc *entry;
+
+	list_for_each_entry(entry, &dev->msi_list, list) {
+		int i, nvec;
+		if (entry->irq == 0)
+			continue;
+		nvec = 1 << entry->msi_attrib.multiple;
+		for (i = 0; i < nvec; i++)
+			arch_teardown_msi_irq(entry->irq + i);
+	}
+}
+
 #if defined (CONFIG_DMAR) || defined (CONFIG_INTR_REMAP)
 #ifdef CONFIG_SMP
 static int dmar_msi_set_affinity(unsigned int irq, const struct cpumask *mask)
-- 
1.7.0.4

Re: [PATCH 15/22] x86: Copy-n-paste arch_teardown_msi_irqs from msi.c to io_apic.c.

[Thomas Gleixner]

On Mon, 4 Oct 2010, Konrad Rzeszutek Wilk wrote:

> In preparation for non-privileged domains to disable PCI devices'
> MSI/MSIx, we need to augment arch_teardown_msi_irqs to make
> a call to the privileged domain (patch to follow).

Can't we find a more clever solution than just copying stuff around ?

Something like this:

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 69b7be3..36d607c 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -60,7 +60,12 @@ int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 #endif
 
 #ifndef arch_teardown_msi_irqs
-void arch_teardown_msi_irqs(struct pci_dev *dev)
+# define arch_teardown_msi_irqs default_teardown_msi_irqs
+# define HAVE_DEFAULT_MSI_TEARDOWN_IRQS
+#endif
+
+#ifdef HAVE_DEFAULT_MSI_TEARDOWN_IRQS
+void default_teardown_msi_irqs(struct pci_dev *dev)
 {
 	struct msi_desc *entry;
 
That way you can override arch_teardown_msi_irqs with x86_msi_ops, define
HAVE_DEFAULT_MSI_TEARDOWN_IRQS in asm/msi.h and set the default
pointer to default_teardown_msi_irqs().

Thanks,

	tglx

Re: [PATCH 15/22] x86: Copy-n-paste arch_teardown_msi_irqs from msi.c to io_apic.c.

[Konrad Rzeszutek Wilk]

[-- Attachment #1: Type: text/plain, Size: 6842 bytes --]

On Tue, Oct 05, 2010 at 11:07:36AM +0200, Thomas Gleixner wrote:
> On Mon, 4 Oct 2010, Konrad Rzeszutek Wilk wrote:
> 
> > In preparation for non-privileged domains to disable PCI devices'
> > MSI/MSIx, we need to augment arch_teardown_msi_irqs to make
> > a call to the privileged domain (patch to follow).
> 
> Can't we find a more clever solution than just copying stuff around ?

Thanks for suggestion. Based on your idea, this is the patch I've rigged up.
I've also included a different re-implementation of the next patch
that would utilize this (x86-Introduce-x86_msi_ops). Tested successfully on x86.
Cross-compiling on PPC, PPC64, and IA64 shows no failure.

Is it OK if I put you as the Author of this?

>From a9d8656100c0c389fe59dcd5e98380f6ef7bcd93 Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Wed, 6 Oct 2010 16:05:35 -0400
Subject: [PATCH 1/2] msi: Introduce default_[teardown|setup]_msi_irqs with fallback.

Introduce an override for the arch_[teardown|setup]_msi_irqs
that can be utilized to fallback to the default arch_* code.

If a platform wants to utilize the code paths defined
in driver/pci/msi.c it has to define HAVE_DEFAULT_MSI_TEARDOWN_IRQS
or HAVE_DEFAULT_MSI_SETUP_IRQS. Otherwise the old mechanism
of over-ridding the arch_* works fine.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: Thomas Gleixner <tglx@linutronix.de>
---
 drivers/pci/msi.c |   14 ++++++++++++--
 1 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 69b7be3..e57bfa9 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -35,7 +35,12 @@ int arch_msi_check_device(struct pci_dev *dev, int nvec, int type)
 #endif
 
 #ifndef arch_setup_msi_irqs
-int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+# define arch_setup_msi_irqs default_setup_msi_irqs
+# define HAVE_DEFAULT_MSI_SETUP_IRQS
+#endif
+
+#ifdef HAVE_DEFAULT_MSI_SETUP_IRQS
+int default_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 {
 	struct msi_desc *entry;
 	int ret;
@@ -60,7 +65,12 @@ int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 #endif
 
 #ifndef arch_teardown_msi_irqs
-void arch_teardown_msi_irqs(struct pci_dev *dev)
+# define arch_teardown_msi_irqs default_teardown_msi_irqs
+# define HAVE_DEFAULT_MSI_TEARDOWN_IRQS
+#endif
+
+#ifdef HAVE_DEFAULT_MSI_TEARDOWN_IRQS
+void default_teardown_msi_irqs(struct pci_dev *dev)
 {
 	struct msi_desc *entry;
 
-- 
1.7.0.4



and:


>From f180620f523b0c2b377fa6600b5444abc714903f Mon Sep 17 00:00:00 2001
From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Date: Wed, 6 Oct 2010 16:12:28 -0400
Subject: [PATCH 2/2] x86: Introduce x86_msi_ops

Introduce an x86 specific indirect mechanism to setup MSIs.
The MSI setup functions become function pointers in an x86_msi_ops
struct, that defaults to the implementation in io_apic.c and msi.c.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
---
 arch/x86/include/asm/pci.h      |   27 +++++++++++++++++++++++++--
 arch/x86/include/asm/x86_init.h |    9 +++++++++
 arch/x86/kernel/apic/io_apic.c  |    4 ++--
 arch/x86/kernel/x86_init.c      |    6 ++++++
 4 files changed, 42 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/pci.h b/arch/x86/include/asm/pci.h
index d395540..5a93f36 100644
--- a/arch/x86/include/asm/pci.h
+++ b/arch/x86/include/asm/pci.h
@@ -7,6 +7,7 @@
 #include <linux/string.h>
 #include <asm/scatterlist.h>
 #include <asm/io.h>
+#include <asm/x86_init.h>
 
 #ifdef __KERNEL__
 
@@ -94,8 +95,30 @@ static inline void early_quirks(void) { }
 
 extern void pci_iommu_alloc(void);
 
-/* MSI arch hook */
-#define arch_setup_msi_irqs arch_setup_msi_irqs
+/* MSI arch specific hooks */
+static inline int x86_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+{
+	return x86_msi.setup_msi_irqs(dev, nvec, type);
+}
+
+static inline void x86_teardown_msi_irqs(struct pci_dev *dev)
+{
+	x86_msi.teardown_msi_irqs(dev);
+}
+
+static inline void x86_teardown_msi_irq(unsigned int irq)
+{
+	x86_msi.teardown_msi_irq(irq);
+}
+#define arch_setup_msi_irqs x86_setup_msi_irqs
+#define arch_teardown_msi_irqs x86_teardown_msi_irqs
+#define arch_teardown_msi_irq x86_teardown_msi_irq
+/* implemented in arch/x86/kernel/apic/io_apic. */
+int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);
+void native_teardown_msi_irq(unsigned int irq);
+/* default to the implementation in drivers/lib/msi.c */
+#define HAVE_DEFAULT_MSI_TEARDOWN_IRQS
+void default_teardown_msi_irqs(struct pci_dev *dev);
 
 #define PCI_DMA_BUS_IS_PHYS (dma_ops->is_phys)
 
diff --git a/arch/x86/include/asm/x86_init.h b/arch/x86/include/asm/x86_init.h
index baa579c..64642ad 100644
--- a/arch/x86/include/asm/x86_init.h
+++ b/arch/x86/include/asm/x86_init.h
@@ -154,9 +154,18 @@ struct x86_platform_ops {
 	int (*i8042_detect)(void);
 };
 
+struct pci_dev;
+
+struct x86_msi_ops {
+	int (*setup_msi_irqs)(struct pci_dev *dev, int nvec, int type);
+	void (*teardown_msi_irq)(unsigned int irq);
+	void (*teardown_msi_irqs)(struct pci_dev *dev);
+};
+
 extern struct x86_init_ops x86_init;
 extern struct x86_cpuinit_ops x86_cpuinit;
 extern struct x86_platform_ops x86_platform;
+extern struct x86_msi_ops x86_msi;
 
 extern void x86_init_noop(void);
 extern void x86_init_uint_noop(unsigned int unused);
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 4c9b2b9..1bfc6d1 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -3533,7 +3533,7 @@ static int setup_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc, int irq)
 	return 0;
 }
 
-int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 {
 	unsigned int irq;
 	int ret, sub_handle;
@@ -3594,7 +3594,7 @@ error:
 	return ret;
 }
 
-void arch_teardown_msi_irq(unsigned int irq)
+void native_teardown_msi_irq(unsigned int irq)
 {
 	destroy_irq(irq);
 }
diff --git a/arch/x86/kernel/x86_init.c b/arch/x86/kernel/x86_init.c
index cd6da6b..2398392 100644
--- a/arch/x86/kernel/x86_init.c
+++ b/arch/x86/kernel/x86_init.c
@@ -6,6 +6,7 @@
 #include <linux/init.h>
 #include <linux/ioport.h>
 #include <linux/module.h>
+#include <linux/pci.h>
 
 #include <asm/bios_ebda.h>
 #include <asm/paravirt.h>
@@ -99,3 +100,8 @@ struct x86_platform_ops x86_platform = {
 };
 
 EXPORT_SYMBOL_GPL(x86_platform);
+struct x86_msi_ops x86_msi = {
+	.setup_msi_irqs = native_setup_msi_irqs,
+	.teardown_msi_irq = native_teardown_msi_irq,
+	.teardown_msi_irqs = default_teardown_msi_irqs,
+};
-- 
1.7.0.4


[-- Attachment #2: 0001-msi-Introduce-default_-teardown-setup-_msi_irqs-with.patch --]
[-- Type: text/x-diff, Size: 1781 bytes --]

>From a9d8656100c0c389fe59dcd5e98380f6ef7bcd93 Mon Sep 17 00:00:00 2001
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date: Wed, 6 Oct 2010 16:05:35 -0400
Subject: [PATCH 1/2] msi: Introduce default_[teardown|setup]_msi_irqs with fallback.

Introduce an override for the arch_[teardown|setup]_msi_irqs
that can be utilized to fallback to the default arch_* code.

If a platform wants to utilize the code paths defined
in driver/pci/msi.c it has to define HAVE_DEFAULT_MSI_TEARDOWN_IRQS
or HAVE_DEFAULT_MSI_SETUP_IRQS. Otherwise the old mechanism
of over-ridding the arch_* works fine.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: Thomas Gleixner <tglx@linutronix.de>
---
 drivers/pci/msi.c |   14 ++++++++++++--
 1 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 69b7be3..e57bfa9 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -35,7 +35,12 @@ int arch_msi_check_device(struct pci_dev *dev, int nvec, int type)
 #endif
 
 #ifndef arch_setup_msi_irqs
-int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+# define arch_setup_msi_irqs default_setup_msi_irqs
+# define HAVE_DEFAULT_MSI_SETUP_IRQS
+#endif
+
+#ifdef HAVE_DEFAULT_MSI_SETUP_IRQS
+int default_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 {
 	struct msi_desc *entry;
 	int ret;
@@ -60,7 +65,12 @@ int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 #endif
 
 #ifndef arch_teardown_msi_irqs
-void arch_teardown_msi_irqs(struct pci_dev *dev)
+# define arch_teardown_msi_irqs default_teardown_msi_irqs
+# define HAVE_DEFAULT_MSI_TEARDOWN_IRQS
+#endif
+
+#ifdef HAVE_DEFAULT_MSI_TEARDOWN_IRQS
+void default_teardown_msi_irqs(struct pci_dev *dev)
 {
 	struct msi_desc *entry;
 
-- 
1.7.0.4


[-- Attachment #3: 0002-x86-Introduce-x86_msi_ops.patch --]
[-- Type: text/x-diff, Size: 4338 bytes --]

>From f180620f523b0c2b377fa6600b5444abc714903f Mon Sep 17 00:00:00 2001
From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Date: Wed, 6 Oct 2010 16:12:28 -0400
Subject: [PATCH 2/2] x86: Introduce x86_msi_ops

Introduce an x86 specific indirect mechanism to setup MSIs.
The MSI setup functions become function pointers in an x86_msi_ops
struct, that defaults to the implementation in io_apic.c and msi.c.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
---
 arch/x86/include/asm/pci.h      |   27 +++++++++++++++++++++++++--
 arch/x86/include/asm/x86_init.h |    9 +++++++++
 arch/x86/kernel/apic/io_apic.c  |    4 ++--
 arch/x86/kernel/x86_init.c      |    6 ++++++
 4 files changed, 42 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/pci.h b/arch/x86/include/asm/pci.h
index d395540..5a93f36 100644
--- a/arch/x86/include/asm/pci.h
+++ b/arch/x86/include/asm/pci.h
@@ -7,6 +7,7 @@
 #include <linux/string.h>
 #include <asm/scatterlist.h>
 #include <asm/io.h>
+#include <asm/x86_init.h>
 
 #ifdef __KERNEL__
 
@@ -94,8 +95,30 @@ static inline void early_quirks(void) { }
 
 extern void pci_iommu_alloc(void);
 
-/* MSI arch hook */
-#define arch_setup_msi_irqs arch_setup_msi_irqs
+/* MSI arch specific hooks */
+static inline int x86_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+{
+	return x86_msi.setup_msi_irqs(dev, nvec, type);
+}
+
+static inline void x86_teardown_msi_irqs(struct pci_dev *dev)
+{
+	x86_msi.teardown_msi_irqs(dev);
+}
+
+static inline void x86_teardown_msi_irq(unsigned int irq)
+{
+	x86_msi.teardown_msi_irq(irq);
+}
+#define arch_setup_msi_irqs x86_setup_msi_irqs
+#define arch_teardown_msi_irqs x86_teardown_msi_irqs
+#define arch_teardown_msi_irq x86_teardown_msi_irq
+/* implemented in arch/x86/kernel/apic/io_apic. */
+int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);
+void native_teardown_msi_irq(unsigned int irq);
+/* default to the implementation in drivers/lib/msi.c */
+#define HAVE_DEFAULT_MSI_TEARDOWN_IRQS
+void default_teardown_msi_irqs(struct pci_dev *dev);
 
 #define PCI_DMA_BUS_IS_PHYS (dma_ops->is_phys)
 
diff --git a/arch/x86/include/asm/x86_init.h b/arch/x86/include/asm/x86_init.h
index baa579c..64642ad 100644
--- a/arch/x86/include/asm/x86_init.h
+++ b/arch/x86/include/asm/x86_init.h
@@ -154,9 +154,18 @@ struct x86_platform_ops {
 	int (*i8042_detect)(void);
 };
 
+struct pci_dev;
+
+struct x86_msi_ops {
+	int (*setup_msi_irqs)(struct pci_dev *dev, int nvec, int type);
+	void (*teardown_msi_irq)(unsigned int irq);
+	void (*teardown_msi_irqs)(struct pci_dev *dev);
+};
+
 extern struct x86_init_ops x86_init;
 extern struct x86_cpuinit_ops x86_cpuinit;
 extern struct x86_platform_ops x86_platform;
+extern struct x86_msi_ops x86_msi;
 
 extern void x86_init_noop(void);
 extern void x86_init_uint_noop(unsigned int unused);
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 4c9b2b9..1bfc6d1 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -3533,7 +3533,7 @@ static int setup_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc, int irq)
 	return 0;
 }
 
-int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 {
 	unsigned int irq;
 	int ret, sub_handle;
@@ -3594,7 +3594,7 @@ error:
 	return ret;
 }
 
-void arch_teardown_msi_irq(unsigned int irq)
+void native_teardown_msi_irq(unsigned int irq)
 {
 	destroy_irq(irq);
 }
diff --git a/arch/x86/kernel/x86_init.c b/arch/x86/kernel/x86_init.c
index cd6da6b..2398392 100644
--- a/arch/x86/kernel/x86_init.c
+++ b/arch/x86/kernel/x86_init.c
@@ -6,6 +6,7 @@
 #include <linux/init.h>
 #include <linux/ioport.h>
 #include <linux/module.h>
+#include <linux/pci.h>
 
 #include <asm/bios_ebda.h>
 #include <asm/paravirt.h>
@@ -99,3 +100,8 @@ struct x86_platform_ops x86_platform = {
 };
 
 EXPORT_SYMBOL_GPL(x86_platform);
+struct x86_msi_ops x86_msi = {
+	.setup_msi_irqs = native_setup_msi_irqs,
+	.teardown_msi_irq = native_teardown_msi_irq,
+	.teardown_msi_irqs = default_teardown_msi_irqs,
+};
-- 
1.7.0.4

Re: [PATCH 15/22] x86: Copy-n-paste arch_teardown_msi_irqs from msi.c to io_apic.c.

[Konrad Rzeszutek Wilk]

[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]

On Wed, Oct 06, 2010 at 09:09:13PM -0400, Konrad Rzeszutek Wilk wrote:
> On Tue, Oct 05, 2010 at 11:07:36AM +0200, Thomas Gleixner wrote:
> > On Mon, 4 Oct 2010, Konrad Rzeszutek Wilk wrote:
> > 
> > > In preparation for non-privileged domains to disable PCI devices'
> > > MSI/MSIx, we need to augment arch_teardown_msi_irqs to make
> > > a call to the privileged domain (patch to follow).
> > 
> > Can't we find a more clever solution than just copying stuff around ?
> 
> Thanks for suggestion. Based on your idea, this is the patch I've rigged up.
> I've also included a different re-implementation of the next patch
> that would utilize this (x86-Introduce-x86_msi_ops). Tested successfully on x86.
> Cross-compiling on PPC, PPC64, and IA64 shows no failure.

But I did find that playing with 'make randconfig' where I got:
# CONFIG_PCI not set

produces interesting errors. So attached is the second patch, but
hardned against the case of building an X86 kernel without PCI support.

The patches are also available at:
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/xen-pcifront-0.8



[-- Attachment #2: 0001-x86-Introduce-x86_msi_ops.patch --]
[-- Type: text/x-diff, Size: 4648 bytes --]

>From bb31ebf713b6651769876d49ea33fa08807c4be6 Mon Sep 17 00:00:00 2001
From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Date: Wed, 6 Oct 2010 16:12:28 -0400
Subject: [PATCH] x86: Introduce x86_msi_ops

Introduce an x86 specific indirect mechanism to setup MSIs.
The MSI setup functions become function pointers in an x86_msi_ops
struct, that defaults to the implementation in io_apic.c and msi.c.

[v2: Use HAVE_DEFAULT_* knobs]
Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
---
 arch/x86/include/asm/pci.h      |   33 +++++++++++++++++++++++++++++++--
 arch/x86/include/asm/x86_init.h |    9 +++++++++
 arch/x86/kernel/apic/io_apic.c  |    4 ++--
 arch/x86/kernel/x86_init.c      |    7 +++++++
 4 files changed, 49 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/pci.h b/arch/x86/include/asm/pci.h
index d395540..ca0437c 100644
--- a/arch/x86/include/asm/pci.h
+++ b/arch/x86/include/asm/pci.h
@@ -7,6 +7,7 @@
 #include <linux/string.h>
 #include <asm/scatterlist.h>
 #include <asm/io.h>
+#include <asm/x86_init.h>
 
 #ifdef __KERNEL__
 
@@ -94,8 +95,36 @@ static inline void early_quirks(void) { }
 
 extern void pci_iommu_alloc(void);
 
-/* MSI arch hook */
-#define arch_setup_msi_irqs arch_setup_msi_irqs
+#ifdef CONFIG_PCI_MSI
+/* MSI arch specific hooks */
+static inline int x86_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+{
+	return x86_msi.setup_msi_irqs(dev, nvec, type);
+}
+
+static inline void x86_teardown_msi_irqs(struct pci_dev *dev)
+{
+	x86_msi.teardown_msi_irqs(dev);
+}
+
+static inline void x86_teardown_msi_irq(unsigned int irq)
+{
+	x86_msi.teardown_msi_irq(irq);
+}
+#define arch_setup_msi_irqs x86_setup_msi_irqs
+#define arch_teardown_msi_irqs x86_teardown_msi_irqs
+#define arch_teardown_msi_irq x86_teardown_msi_irq
+/* implemented in arch/x86/kernel/apic/io_apic. */
+int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);
+void native_teardown_msi_irq(unsigned int irq);
+/* default to the implementation in drivers/lib/msi.c */
+#define HAVE_DEFAULT_MSI_TEARDOWN_IRQS
+void default_teardown_msi_irqs(struct pci_dev *dev);
+#else
+#define native_setup_msi_irqs		NULL
+#define native_teardown_msi_irq		NULL
+#define default_teardown_msi_irqs	NULL
+#endif
 
 #define PCI_DMA_BUS_IS_PHYS (dma_ops->is_phys)
 
diff --git a/arch/x86/include/asm/x86_init.h b/arch/x86/include/asm/x86_init.h
index baa579c..64642ad 100644
--- a/arch/x86/include/asm/x86_init.h
+++ b/arch/x86/include/asm/x86_init.h
@@ -154,9 +154,18 @@ struct x86_platform_ops {
 	int (*i8042_detect)(void);
 };
 
+struct pci_dev;
+
+struct x86_msi_ops {
+	int (*setup_msi_irqs)(struct pci_dev *dev, int nvec, int type);
+	void (*teardown_msi_irq)(unsigned int irq);
+	void (*teardown_msi_irqs)(struct pci_dev *dev);
+};
+
 extern struct x86_init_ops x86_init;
 extern struct x86_cpuinit_ops x86_cpuinit;
 extern struct x86_platform_ops x86_platform;
+extern struct x86_msi_ops x86_msi;
 
 extern void x86_init_noop(void);
 extern void x86_init_uint_noop(unsigned int unused);
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 4c9b2b9..1bfc6d1 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -3533,7 +3533,7 @@ static int setup_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc, int irq)
 	return 0;
 }
 
-int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 {
 	unsigned int irq;
 	int ret, sub_handle;
@@ -3594,7 +3594,7 @@ error:
 	return ret;
 }
 
-void arch_teardown_msi_irq(unsigned int irq)
+void native_teardown_msi_irq(unsigned int irq)
 {
 	destroy_irq(irq);
 }
diff --git a/arch/x86/kernel/x86_init.c b/arch/x86/kernel/x86_init.c
index cd6da6b..ceb2911 100644
--- a/arch/x86/kernel/x86_init.c
+++ b/arch/x86/kernel/x86_init.c
@@ -6,10 +6,12 @@
 #include <linux/init.h>
 #include <linux/ioport.h>
 #include <linux/module.h>
+#include <linux/pci.h>
 
 #include <asm/bios_ebda.h>
 #include <asm/paravirt.h>
 #include <asm/pci_x86.h>
+#include <asm/pci.h>
 #include <asm/mpspec.h>
 #include <asm/setup.h>
 #include <asm/apic.h>
@@ -99,3 +101,8 @@ struct x86_platform_ops x86_platform = {
 };
 
 EXPORT_SYMBOL_GPL(x86_platform);
+struct x86_msi_ops x86_msi = {
+	.setup_msi_irqs = native_setup_msi_irqs,
+	.teardown_msi_irq = native_teardown_msi_irq,
+	.teardown_msi_irqs = default_teardown_msi_irqs,
+};
-- 
1.7.0.4

[PATCH 16/22] x86: Introduce x86_msi_ops

[Konrad Rzeszutek Wilk]

From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

Introduce an x86 specific indirect mechanism to setup MSIs.
The MSI setup functions become function pointers in an x86_msi_ops
struct, that defaults to the implementation in io_apic.c

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: Jesse Barnes <jbarnes@virtuousgeek.org>
---
 arch/x86/include/asm/pci.h      |   26 +++++++++++++++++++++++---
 arch/x86/include/asm/x86_init.h |    9 +++++++++
 arch/x86/kernel/apic/io_apic.c  |    8 ++++----
 arch/x86/kernel/x86_init.c      |    6 ++++++
 4 files changed, 42 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/pci.h b/arch/x86/include/asm/pci.h
index 79d6ada..5b0f853 100644
--- a/arch/x86/include/asm/pci.h
+++ b/arch/x86/include/asm/pci.h
@@ -7,6 +7,7 @@
 #include <linux/string.h>
 #include <asm/scatterlist.h>
 #include <asm/io.h>
+#include <asm/x86_init.h>
 
 #ifdef __KERNEL__
 
@@ -94,9 +95,28 @@ static inline void early_quirks(void) { }
 
 extern void pci_iommu_alloc(void);
 
-/* MSI arch hook */
-#define arch_setup_msi_irqs arch_setup_msi_irqs
-#define arch_teardown_msi_irqs arch_teardown_msi_irqs
+/* MSI arch specific hooks */
+static inline int x86_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+{
+	return x86_msi.setup_msi_irqs(dev, nvec, type);
+}
+
+static inline void x86_teardown_msi_irqs(struct pci_dev *dev)
+{
+	x86_msi.teardown_msi_irqs(dev);
+}
+
+static inline void x86_teardown_msi_irq(unsigned int irq)
+{
+	x86_msi.teardown_msi_irq(irq);
+}
+
+#define arch_setup_msi_irqs x86_setup_msi_irqs
+#define arch_teardown_msi_irqs x86_teardown_msi_irqs
+#define arch_teardown_msi_irq x86_teardown_msi_irq
+int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);
+void native_teardown_msi_irq(unsigned int irq);
+void native_teardown_msi_irqs(struct pci_dev *dev);
 
 #define PCI_DMA_BUS_IS_PHYS (dma_ops->is_phys)
 
diff --git a/arch/x86/include/asm/x86_init.h b/arch/x86/include/asm/x86_init.h
index baa579c..64642ad 100644
--- a/arch/x86/include/asm/x86_init.h
+++ b/arch/x86/include/asm/x86_init.h
@@ -154,9 +154,18 @@ struct x86_platform_ops {
 	int (*i8042_detect)(void);
 };
 
+struct pci_dev;
+
+struct x86_msi_ops {
+	int (*setup_msi_irqs)(struct pci_dev *dev, int nvec, int type);
+	void (*teardown_msi_irq)(unsigned int irq);
+	void (*teardown_msi_irqs)(struct pci_dev *dev);
+};
+
 extern struct x86_init_ops x86_init;
 extern struct x86_cpuinit_ops x86_cpuinit;
 extern struct x86_platform_ops x86_platform;
+extern struct x86_msi_ops x86_msi;
 
 extern void x86_init_noop(void);
 extern void x86_init_uint_noop(unsigned int unused);
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 4132eea..af80132 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -3533,7 +3533,7 @@ static int setup_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc, int irq)
 	return 0;
 }
 
-int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 {
 	unsigned int irq;
 	int ret, sub_handle;
@@ -3594,12 +3594,12 @@ error:
 	return ret;
 }
 
-void arch_teardown_msi_irq(unsigned int irq)
+void native_teardown_msi_irq(unsigned int irq)
 {
 	destroy_irq(irq);
 }
 
-void arch_teardown_msi_irqs(struct pci_dev *dev)
+void native_teardown_msi_irqs(struct pci_dev *dev)
 {
 	struct msi_desc *entry;
 
@@ -3609,7 +3609,7 @@ void arch_teardown_msi_irqs(struct pci_dev *dev)
 			continue;
 		nvec = 1 << entry->msi_attrib.multiple;
 		for (i = 0; i < nvec; i++)
-			arch_teardown_msi_irq(entry->irq + i);
+			x86_msi.teardown_msi_irq(entry->irq + i);
 	}
 }
 
diff --git a/arch/x86/kernel/x86_init.c b/arch/x86/kernel/x86_init.c
index cd6da6b..91b5209 100644
--- a/arch/x86/kernel/x86_init.c
+++ b/arch/x86/kernel/x86_init.c
@@ -6,6 +6,7 @@
 #include <linux/init.h>
 #include <linux/ioport.h>
 #include <linux/module.h>
+#include <linux/pci.h>
 
 #include <asm/bios_ebda.h>
 #include <asm/paravirt.h>
@@ -99,3 +100,8 @@ struct x86_platform_ops x86_platform = {
 };
 
 EXPORT_SYMBOL_GPL(x86_platform);
+struct x86_msi_ops x86_msi = {
+	.setup_msi_irqs = native_setup_msi_irqs,
+	.teardown_msi_irq = native_teardown_msi_irq,
+	.teardown_msi_irqs = native_teardown_msi_irqs,
+};
-- 
1.7.0.4

[PATCH 17/22] xen/x86/PCI: Add support for the Xen PCI subsystem

[Konrad Rzeszutek Wilk]

From: Alex Nixon <alex.nixon@citrix.com>

The frontend stub lives in arch/x86/pci/xen.c, alongside other
sub-arch PCI init code (e.g. olpc.c).

It provides a mechanism for Xen PCI frontend to setup/destroy
legacy interrupts, MSI/MSI-X, and PCI configuration operations.

[ Impact: add core of Xen PCI support ]
[ v2: Removed the IOMMU code and only focusing on PCI.]
[ v3: removed usage of pci_scan_all_fns as that does not exist]
[ v4: introduced pci_xen value to fix compile warnings]
[ v5: squished fixes+features in one patch, changed Reviewed-by to Ccs]
[ v7: added Acked-by]
Signed-off-by: Alex Nixon <alex.nixon@citrix.com>
Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Jesse Barnes <jbarnes@virtuousgeek.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Matthew Wilcox <willy@linux.intel.com>
Cc: Qing He <qing.he@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: x86@kernel.org
---
 arch/x86/Kconfig               |    5 ++
 arch/x86/include/asm/xen/pci.h |   53 ++++++++++++++
 arch/x86/pci/Makefile          |    1 +
 arch/x86/pci/xen.c             |  147 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/xen/enlighten.c       |    3 +
 drivers/xen/events.c           |   32 ++++++++-
 include/xen/events.h           |    3 +
 7 files changed, 242 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/include/asm/xen/pci.h
 create mode 100644 arch/x86/pci/xen.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index cea0cd9..11cfde8 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1925,6 +1925,11 @@ config PCI_OLPC
 	def_bool y
 	depends on PCI && OLPC && (PCI_GOOLPC || PCI_GOANY)
 
+config PCI_XEN
+	def_bool n
+	depends on XEN
+	select SWIOTLB_XEN
+
 config PCI_DOMAINS
 	def_bool y
 	depends on PCI
diff --git a/arch/x86/include/asm/xen/pci.h b/arch/x86/include/asm/xen/pci.h
new file mode 100644
index 0000000..449c82f
--- /dev/null
+++ b/arch/x86/include/asm/xen/pci.h
@@ -0,0 +1,53 @@
+#ifndef _ASM_X86_XEN_PCI_H
+#define _ASM_X86_XEN_PCI_H
+
+#if defined(CONFIG_PCI_XEN)
+extern int __init pci_xen_init(void);
+#define pci_xen 1
+#else
+#define pci_xen 0
+#define pci_xen_init (0)
+#endif
+
+#if defined(CONFIG_PCI_MSI)
+#if defined(CONFIG_PCI_XEN)
+/* The drivers/pci/xen-pcifront.c sets this structure to
+ * its own functions.
+ */
+struct xen_pci_frontend_ops {
+	int (*enable_msi)(struct pci_dev *dev, int **vectors);
+	void (*disable_msi)(struct pci_dev *dev);
+	int (*enable_msix)(struct pci_dev *dev, int **vectors, int nvec);
+	void (*disable_msix)(struct pci_dev *dev);
+};
+
+extern struct xen_pci_frontend_ops *xen_pci_frontend;
+
+static inline int xen_pci_frontend_enable_msi(struct pci_dev *dev,
+					      int **vectors)
+{
+	if (xen_pci_frontend && xen_pci_frontend->enable_msi)
+		return xen_pci_frontend->enable_msi(dev, vectors);
+	return -ENODEV;
+}
+static inline void xen_pci_frontend_disable_msi(struct pci_dev *dev)
+{
+	if (xen_pci_frontend && xen_pci_frontend->disable_msi)
+			xen_pci_frontend->disable_msi(dev);
+}
+static inline int xen_pci_frontend_enable_msix(struct pci_dev *dev,
+					       int **vectors, int nvec)
+{
+	if (xen_pci_frontend && xen_pci_frontend->enable_msix)
+		return xen_pci_frontend->enable_msix(dev, vectors, nvec);
+	return -ENODEV;
+}
+static inline void xen_pci_frontend_disable_msix(struct pci_dev *dev)
+{
+	if (xen_pci_frontend && xen_pci_frontend->disable_msix)
+			xen_pci_frontend->disable_msix(dev);
+}
+#endif /* CONFIG_PCI_XEN */
+#endif /* CONFIG_PCI_MSI */
+
+#endif	/* _ASM_X86_XEN_PCI_H */
diff --git a/arch/x86/pci/Makefile b/arch/x86/pci/Makefile
index a0207a7..effd96e 100644
--- a/arch/x86/pci/Makefile
+++ b/arch/x86/pci/Makefile
@@ -4,6 +4,7 @@ obj-$(CONFIG_PCI_BIOS)		+= pcbios.o
 obj-$(CONFIG_PCI_MMCONFIG)	+= mmconfig_$(BITS).o direct.o mmconfig-shared.o
 obj-$(CONFIG_PCI_DIRECT)	+= direct.o
 obj-$(CONFIG_PCI_OLPC)		+= olpc.o
+obj-$(CONFIG_PCI_XEN)		+= xen.o
 
 obj-y				+= fixup.o
 obj-$(CONFIG_ACPI)		+= acpi.o
diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c
new file mode 100644
index 0000000..b19c873
--- /dev/null
+++ b/arch/x86/pci/xen.c
@@ -0,0 +1,147 @@
+/*
+ * Xen PCI Frontend Stub - puts some "dummy" functions in to the Linux
+ *			   x86 PCI core to support the Xen PCI Frontend
+ *
+ *   Author: Ryan Wilson <hap9@epoch.ncsc.mil>
+ */
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/pci.h>
+#include <linux/acpi.h>
+
+#include <linux/io.h>
+#include <asm/pci_x86.h>
+
+#include <asm/xen/hypervisor.h>
+
+#include <xen/events.h>
+#include <asm/xen/pci.h>
+
+#if defined(CONFIG_PCI_MSI)
+#include <linux/msi.h>
+
+struct xen_pci_frontend_ops *xen_pci_frontend;
+EXPORT_SYMBOL_GPL(xen_pci_frontend);
+
+/*
+ * For MSI interrupts we have to use drivers/xen/event.s functions to
+ * allocate an irq_desc and setup the right */
+
+
+static int xen_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+{
+	int irq, ret, i;
+	struct msi_desc *msidesc;
+	int *v;
+
+	v = kzalloc(sizeof(int) * max(1, nvec), GFP_KERNEL);
+	if (!v)
+		return -ENOMEM;
+
+	if (!xen_initial_domain()) {
+		if (type == PCI_CAP_ID_MSIX)
+			ret = xen_pci_frontend_enable_msix(dev, &v, nvec);
+		else
+			ret = xen_pci_frontend_enable_msi(dev, &v);
+		if (ret)
+			goto error;
+	}
+	i = 0;
+	list_for_each_entry(msidesc, &dev->msi_list, list) {
+		irq = xen_allocate_pirq(v[i], 0, /* not sharable */
+			(type == PCI_CAP_ID_MSIX) ?
+			"pcifront-msi-x" : "pcifront-msi");
+		if (irq < 0)
+			return -1;
+
+		ret = set_irq_msi(irq, msidesc);
+		if (ret)
+			goto error_while;
+		i++;
+	}
+	kfree(v);
+	return 0;
+
+error_while:
+	unbind_from_irqhandler(irq, NULL);
+error:
+	if (ret == -ENODEV)
+		dev_err(&dev->dev, "Xen PCI frontend has not registered" \
+			" MSI/MSI-X support!\n");
+
+	kfree(v);
+	return ret;
+}
+
+static void xen_teardown_msi_irqs(struct pci_dev *dev)
+{
+	/* Only do this when were are in non-privileged mode.*/
+	if (!xen_initial_domain()) {
+		struct msi_desc *msidesc;
+
+		msidesc = list_entry(dev->msi_list.next, struct msi_desc, list);
+		if (msidesc->msi_attrib.is_msix)
+			xen_pci_frontend_disable_msix(dev);
+		else
+			xen_pci_frontend_disable_msi(dev);
+	}
+
+}
+
+static void xen_teardown_msi_irq(unsigned int irq)
+{
+	xen_destroy_irq(irq);
+}
+#endif
+
+static int xen_pcifront_enable_irq(struct pci_dev *dev)
+{
+	int rc;
+	int share = 1;
+
+	dev_info(&dev->dev, "Xen PCI enabling IRQ: %d\n", dev->irq);
+
+	if (dev->irq < 0)
+		return -EINVAL;
+
+	if (dev->irq < NR_IRQS_LEGACY)
+		share = 0;
+
+	rc = xen_allocate_pirq(dev->irq, share, "pcifront");
+	if (rc < 0) {
+		dev_warn(&dev->dev, "Xen PCI IRQ: %d, failed to register:%d\n",
+			 dev->irq, rc);
+		return rc;
+	}
+	return 0;
+}
+
+int __init pci_xen_init(void)
+{
+	if (!xen_pv_domain() || xen_initial_domain())
+		return -ENODEV;
+
+	printk(KERN_INFO "PCI: setting up Xen PCI frontend stub\n");
+
+	pcibios_set_cache_line_size();
+
+	pcibios_enable_irq = xen_pcifront_enable_irq;
+	pcibios_disable_irq = NULL;
+
+#ifdef CONFIG_ACPI
+	/* Keep ACPI out of the picture */
+	acpi_noirq = 1;
+#endif
+
+#ifdef CONFIG_ISAPNP
+	/* Stop isapnp from probing */
+	isapnp_disable = 1;
+#endif
+
+#ifdef CONFIG_PCI_MSI
+	x86_msi.setup_msi_irqs = xen_setup_msi_irqs;
+	x86_msi.teardown_msi_irq = xen_teardown_msi_irq;
+	x86_msi.teardown_msi_irqs = xen_teardown_msi_irqs;
+#endif
+	return 0;
+}
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 7d46c84..1ccfa1b 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -45,6 +45,7 @@
 #include <asm/paravirt.h>
 #include <asm/apic.h>
 #include <asm/page.h>
+#include <asm/xen/pci.h>
 #include <asm/xen/hypercall.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/fixmap.h>
@@ -1220,6 +1221,8 @@ asmlinkage void __init xen_start_kernel(void)
 		add_preferred_console("xenboot", 0, NULL);
 		add_preferred_console("tty", 0, NULL);
 		add_preferred_console("hvc", 0, NULL);
+		if (pci_xen)
+			x86_init.pci.arch_init = pci_xen_init;
 	} else {
 		/* Make sure ACS will be enabled */
 		pci_request_acs();
diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index e97e11a..4e8002e 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -576,7 +576,9 @@ int xen_allocate_pirq(unsigned gsi, int shareable, char *name)
 		goto out;	/* XXX need refcount? */
 	}
 
-	if (identity_mapped_irq(gsi)) {
+	/* If we are a PV guest, we don't have GSIs (no ACPI passed). Therefore
+	 * we are using the !xen_initial_domain() to drop in the function.*/
+	if (identity_mapped_irq(gsi) || !xen_initial_domain()) {
 		irq = gsi;
 		irq_to_desc_alloc_node(irq, 0);
 		dynamic_irq_init(irq);
@@ -587,7 +589,13 @@ int xen_allocate_pirq(unsigned gsi, int shareable, char *name)
 				      handle_level_irq, name);
 
 	irq_op.irq = irq;
-	if (HYPERVISOR_physdev_op(PHYSDEVOP_alloc_irq_vector, &irq_op)) {
+	irq_op.vector = 0;
+
+	/* Only the privileged domain can do this. For non-priv, the pcifront
+	 * driver provides a PCI bus that does the call to do exactly
+	 * this in the priv domain. */
+	if (xen_initial_domain() &&
+	    HYPERVISOR_physdev_op(PHYSDEVOP_alloc_irq_vector, &irq_op)) {
 		dynamic_irq_cleanup(irq);
 		irq = -ENOSPC;
 		goto out;
@@ -602,6 +610,26 @@ out:
 	return irq;
 }
 
+int xen_destroy_irq(int irq)
+{
+	struct irq_desc *desc;
+	int rc = -ENOENT;
+
+	spin_lock(&irq_mapping_update_lock);
+
+	desc = irq_to_desc(irq);
+	if (!desc)
+		goto out;
+
+	irq_info[irq] = mk_unbound_info();
+
+	dynamic_irq_cleanup(irq);
+
+out:
+	spin_unlock(&irq_mapping_update_lock);
+	return rc;
+}
+
 int xen_vector_from_irq(unsigned irq)
 {
 	return vector_from_irq(irq);
diff --git a/include/xen/events.h b/include/xen/events.h
index d7a4ca7..c1717ca 100644
--- a/include/xen/events.h
+++ b/include/xen/events.h
@@ -72,6 +72,9 @@ void xen_hvm_evtchn_do_upcall(void);
  * usual. */
 int xen_allocate_pirq(unsigned gsi, int shareable, char *name);
 
+/* De-allocates the above mentioned physical interrupt. */
+int xen_destroy_irq(int irq);
+
 /* Return vector allocated to pirq */
 int xen_vector_from_irq(unsigned pirq);
 
-- 
1.7.0.4

[PATCH 21/22] xen/pci: Request ACS when Xen-SWIOTLB is activated.

[Konrad Rzeszutek Wilk]

It used to done in the Xen startup code but that is not really
appropiate.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/xen/pci-swiotlb-xen.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/arch/x86/xen/pci-swiotlb-xen.c b/arch/x86/xen/pci-swiotlb-xen.c
index a013ec9..be4d80a 100644
--- a/arch/x86/xen/pci-swiotlb-xen.c
+++ b/arch/x86/xen/pci-swiotlb-xen.c
@@ -1,6 +1,7 @@
 /* Glue code to lib/swiotlb-xen.c */
 
 #include <linux/dma-mapping.h>
+#include <linux/pci.h>
 #include <xen/swiotlb-xen.h>
 
 #include <asm/xen/hypervisor.h>
@@ -54,5 +55,8 @@ void __init pci_xen_swiotlb_init(void)
 	if (xen_swiotlb) {
 		xen_swiotlb_init(1);
 		dma_ops = &xen_swiotlb_dma_ops;
+
+		/* Make sure ACS will be enabled */
+		pci_request_acs();
 	}
 }
-- 
1.7.0.4

[PATCH 22/22] MAINTAINERS: Add myself for Xen PCI and Xen SWIOTLB maintainer.

[Konrad Rzeszutek Wilk]

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 MAINTAINERS |   14 ++++++++++++++
 1 files changed, 14 insertions(+), 0 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 668682d..662aa75 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6486,6 +6486,20 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/mjg59/platform-drivers-x86.
 S:	Maintained
 F:	drivers/platform/x86
 
+XEN PCI SUBSYSTEM
+M:	Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
+L:	xen-devel@lists.xensource.com
+S:	Supported
+F:	arch/x86/pci/*xen*
+F:	drivers/pci/*xen*
+
+XEN SWIOTLB SUBSYSTEM
+M:	Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
+L:	xen-devel@lists.xensource.com
+S:	Supported
+F:	arch/x86/xen/*swiotlb*
+F:	drivers/xen/*swiotlb*
+
 XEN HYPERVISOR INTERFACE
 M:	Jeremy Fitzhardinge <jeremy@xensource.com>
 M:	Chris Wright <chrisw@sous-sol.org>
-- 
1.7.0.4

Re: [Xen-devel] [PATCH v7] Xen PCI + Xen PCI frontend driver.

[Jan Beulich]

>>> On 04.10.10 at 20:13, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> This patch set contains the supporting patches and the driver itself for
> Xen Paravirtualized (PV) domains to use PCI pass-through devices (the git 
> tree
> is git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git 
> devel/xen-pcifront-0.7).
> This patch-set is also utilized in Stefano's PV on HVM MSI/MSI-X patchset [1].
> 
> The Xen PCI frontend driver can be used by PV guests on IOMMU hardware
> (or IOMMU-less). Without the hardware IOMMU you have a potential security
> hole wherein a guest domain can use the hardware to map pages outside its
> memory range and slurp pages up. As such, this is more restricted to a
> Privileged PV domain, aka - device driver domain (similar to Qubes but a
> poor-man mechanism [2]).

Somehow this patch series appears to be incomplete: Neither on
xen-devel nor on patchwork I can find patches 6, 11, 18, 19, and 20.

> The first set of patches are specific to the Xen subsystem, where
> we introduce an IRQ chip for Physical IRQs, along with the supporting
> harness code:
>  xen: Don't disable the I/O space
>  xen: define BIOVEC_PHYS_MERGEABLE()
>  xen: implement pirq type event channels
>  xen: identity map gsi->irqs
>  xen: dynamically allocate irq & event structures
>  xen: set pirq name to something useful.
>  xen: statically initialize cpu_evtchn_mask_p
>  xen: Find an unbound irq number in reverse order (high to low).
>  xen: Provide a variant of xen_poll_irq with timeout.
>  xen: fix shared irq device passthrough
> 
> The next set of patches expose functionality for module drivers to be able 
> to
> enumerate and iomap (using the _PAGE_IOMAP flag) PCI devices.
> 
>  x86/io_apic: add get_nr_irqs_gsi()
>  x86/PCI: Clean up pci_cache_line_size
>  x86/PCI: make sure _PAGE_IOMAP it set on pci mappings
>  x86/PCI: Export pci_walk_bus function.
> 
> The next two patches abstract the MSI/MSI-X architecture calls so that the
> default native one (used on bare-metal) can be overwritten when running
> in virtualized mode (right now on Xen). The implementation is a simple
> function pointer structure.
> 
>  x86: Copy-n-paste arch_teardown_msi_irqs from msi.c to io_apic.c.
>  x86: Introduce x86_msi_ops
> 
> Next, the Xen PCI stub driver. I've put it in the same location
> where other sub-platform PCI drivers are. It hooks up to the
> PCI legacy IRQ setup ('pcibios_enable_irq'), and MSI/MSI-X
> allocation/de-allocation (via the x86_msi_ops introduced in earlier patches).
> 
>  xen/x86/PCI: Add support for the Xen PCI subsystem
> 
> Lastly, the Xen PCI front-end driver which is responsible for hooking up
> to the PCI configuration read/write methods via the 'pci_scan_bus_parented' 
> call.
> In essence all pci_conf_read/write in the guest will be tunneled via
> pcifront_bus_[read|write] methods. The MSI/MSI-X calls will be handled
> by the Xen-PCI front-end driver as well.
> 
>  xen-pcifront: Xen PCI frontend driver.
> 
> The next two are a bug fix and updating the MAINTAINER file:
> 
>  MAINTAINERS: Add myself for Xen PCI and Xen SWIOTLB maintainer.
>  xen/pci: Request ACS when Xen-SWIOTLB is activated.

Also, up to here I count only 20 patch descriptions...

> The shortlog and the diffstat:
> 
> Alex Nixon (3):
>       xen: Don't disable the I/O space
>       x86/PCI: Clean up pci_cache_line_size
>       xen/x86/PCI: Add support for the Xen PCI subsystem
> 
> Gerd Hoffmann (1):
>       xen: set pirq name to something useful.
> 
> Jeremy Fitzhardinge (5):
>       xen: define BIOVEC_PHYS_MERGEABLE()
>       xen: implement pirq type event channels
>       x86/io_apic: add get_nr_irqs_gsi()
>       xen: statically initialize cpu_evtchn_mask_p
>       x86/PCI: make sure _PAGE_IOMAP it set on pci mappings
> 
> Konrad Rzeszutek Wilk (8):
>       xen: identity map gsi->irqs
>       xen: dynamically allocate irq & event structures
>       xen: Find an unbound irq number in reverse order (high to low).
>       xen: Provide a variant of xen_poll_irq with timeout.
>       x86/PCI: Export pci_walk_bus function.
>       x86: Copy-n-paste arch_teardown_msi_irqs from msi.c to io_apic.c.
>       xen/pci: Request ACS when Xen-SWIOTLB is activated.
>       MAINTAINERS: Add myself for Xen PCI and Xen SWIOTLB maintainer.
> 
> Noboru Iwamatsu (1):
>       xenbus: prevent warnings on unhandled enumeration values
> 
> Ryan Wilson (1):
>       xen-pcifront: Xen PCI frontend driver.
> 
> Stefano Stabellini (1):
>       x86: Introduce x86_msi_ops
> 
> Weidong Han (1):
>       xen: fix shared irq device passthrough
> 
> Yosuke Iwamatsu (1):
>       xenbus: Xen paravirtualised PCI hotplug support.

... while these sum up to the announced (in the actual series) 22.

Jan

>  MAINTAINERS                        |   14 +
>  arch/x86/Kconfig                   |    5 +
>  arch/x86/include/asm/io.h          |   13 +
>  arch/x86/include/asm/io_apic.h     |    1 +
>  arch/x86/include/asm/pci.h         |   25 +-
>  arch/x86/include/asm/pci_x86.h     |    1 +
>  arch/x86/include/asm/x86_init.h    |    9 +
>  arch/x86/include/asm/xen/pci.h     |   53 ++
>  arch/x86/kernel/apic/io_apic.c     |   23 +-
>  arch/x86/kernel/x86_init.c         |    6 +
>  arch/x86/pci/Makefile              |    1 +
>  arch/x86/pci/common.c              |   17 +-
>  arch/x86/pci/i386.c                |    2 +
>  arch/x86/pci/xen.c                 |  147 +++++
>  arch/x86/xen/enlighten.c           |    3 +
>  arch/x86/xen/pci-swiotlb-xen.c     |    4 +
>  arch/x86/xen/setup.c               |    2 -
>  drivers/block/xen-blkfront.c       |    2 +
>  drivers/input/xen-kbdfront.c       |    2 +
>  drivers/net/xen-netfront.c         |    2 +
>  drivers/pci/Kconfig                |   16 +
>  drivers/pci/Makefile               |    2 +
>  drivers/pci/bus.c                  |    1 +
>  drivers/pci/xen-pcifront.c         | 1157 
> ++++++++++++++++++++++++++++++++++++
>  drivers/video/xen-fbfront.c        |    2 +
>  drivers/xen/Makefile               |    2 +-
>  drivers/xen/biomerge.c             |   13 +
>  drivers/xen/events.c               |  338 ++++++++++-
>  drivers/xen/xenbus/xenbus_client.c |    2 +
>  include/xen/events.h               |   18 +
>  include/xen/interface/io/pciif.h   |  122 ++++
>  include/xen/interface/io/xenbus.h  |    8 +-
>  32 files changed, 1982 insertions(+), 31 deletions(-)
> 
> P.S.
> [1]. git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git 
> 2.6.36-rc1-pvhvm-pirq-v3
> 
> [2]: http://qubes-os.org/ which utilizes hardware IOMMU to run seperate 
> domains wherein
> each has specific access to hardware.
> 
> [3] Some of the authors of the patches have moved on, so their e-mails
> are bouncing. Any thoughts of what to do about that? Just leave it as is?  
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com 
> http://lists.xensource.com/xen-devel 

Re: [Xen-devel] [PATCH v7] Xen PCI + Xen PCI frontend driver.

[Konrad Rzeszutek Wilk]

On Tue, Oct 05, 2010 at 08:47:20AM +0100, Jan Beulich wrote:
> >>> On 04.10.10 at 20:13, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > This patch set contains the supporting patches and the driver itself for
> > Xen Paravirtualized (PV) domains to use PCI pass-through devices (the git 
> > tree
> > is git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git 
> > devel/xen-pcifront-0.7).
.. snip..
> Somehow this patch series appears to be incomplete: Neither on
> xen-devel nor on patchwork I can find patches 6, 11, 18, 19, and 20.

Shucks! I am not sure what happend, the git-send-email looked to have sent them 
all.  Let me redo them and resend them with more attention to this.

Are you able to use the git tree to look at the patches as an internim solution?

> >  MAINTAINERS: Add myself for Xen PCI and Xen SWIOTLB maintainer.
> >  xen/pci: Request ACS when Xen-SWIOTLB is activated.
> 
> Also, up to here I count only 20 patch descriptions...
> 
> > The shortlog and the diffstat:
.. snip ..
> 
> ... while these sum up to the announced (in the actual series) 22.

<scratches his head in confusion>

Re: [PATCH v7] Xen PCI + Xen PCI frontend driver.

[Konrad Rzeszutek Wilk]

On Wed, Oct 06, 2010 at 06:50:43PM -0400, Konrad Rzeszutek Wilk wrote:
> On Tue, Oct 05, 2010 at 08:47:20AM +0100, Jan Beulich wrote:
> > >>> On 04.10.10 at 20:13, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > > This patch set contains the supporting patches and the driver itself for
> > > Xen Paravirtualized (PV) domains to use PCI pass-through devices (the git 
> > > tree
> > > is git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git 
> > > devel/xen-pcifront-0.7).
> .. snip..
> > Somehow this patch series appears to be incomplete: Neither on
> > xen-devel nor on patchwork I can find patches 6, 11, 18, 19, and 20.
> 
> Shucks! I am not sure what happend, the git-send-email looked to have sent them 
> all.  Let me redo them and resend them with more attention to this.

AHA! I think I know the answer. Those are the patches for which the Author:
is bouncing. Any ideas of what to do except next time just manually respond
and paste in the patches?