[PATCH 0/5] Fix Intel IRQ remapping initialization order

[PATCH 0/5] Fix Intel IRQ remapping initialization order

[Joerg Roedel]

Hi,

here is a patch-set against tip/x86/apic to fix an initialization order
problem with the IRQ remapping code.  The problem is in the ordering of
the irq_remapping_prepare and irq_remapping_supported functions.

Currently the call-order is irq_remapping_prepare ->
irq_remapping_supported, so that 'prepare' can succeed but 'supported'
fails, so that interrupt remapping gets initialized but not enabled.
This causes a broken interrupt setup on affected systems (machines with
an Intel IOMMU without, or broken, IRQ remapping support). The result
are lost interrupts and a non-bootable system.

Both functions do checks whether IRQ remapping can be enabled on the
machine.  The reason for this is that some checks rely on
dmar_table_init() and thus have to be done in irq_remapping_prepare().

This patch-set moves all these checks into the irq_remapping_prepare()
path with the right ordering and removes the irq_remapping_supported()
function and its call-backs. This fixes the initializion order problem
and simplifies the exported API from the IOMMU code.

Please review.

Thanks,

	Joerg

Joerg Roedel (5):
  iommu/vt-d: Allocate IRQ remapping data structures only for all IOMMUs
  iommu/vt-d: Don't check for ecap_ir_support in
    intel_irq_remapping_supported
  iommu/vt-d: Move supported-checks to intel_prepare_irq_remapping()
  iommu/amd: Check for irq-remap support amd_iommu_prepare()
  iommu, x86, apic: Remove irq_remapping_supported()

 arch/x86/kernel/apic/apic.c         |  5 ---
 drivers/iommu/amd_iommu.c           |  1 -
 drivers/iommu/amd_iommu_init.c      |  8 ++---
 drivers/iommu/amd_iommu_proto.h     |  1 -
 drivers/iommu/intel_irq_remapping.c | 62 +++++++++++++++++--------------------
 drivers/iommu/irq_remapping.c       | 11 -------
 drivers/iommu/irq_remapping.h       |  3 --
 7 files changed, 31 insertions(+), 60 deletions(-)

-- 
1.9.1

[PATCH 1/5] iommu/vt-d: Allocate IRQ remapping data structures only for all IOMMUs

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

IRQ remapping is only supported when all IOMMUs in the
system support it. So check if all IOMMUs in the system
support IRQ remapping before doing the allocations.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 drivers/iommu/intel_irq_remapping.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/iommu/intel_irq_remapping.c b/drivers/iommu/intel_irq_remapping.c
index ff35b03..3ef63b8 100644
--- a/drivers/iommu/intel_irq_remapping.c
+++ b/drivers/iommu/intel_irq_remapping.c
@@ -568,14 +568,16 @@ static int __init intel_prepare_irq_remapping(void)
 		goto error;
 	}
 
-	for_each_iommu(iommu, drhd) {
+	/* First make sure all IOMMUs support IRQ remapping */
+	for_each_iommu(iommu, drhd)
 		if (!ecap_ir_support(iommu->ecap))
-			continue;
+			goto error;
 
-		/* Do the allocations early */
+	/* Do the allocations early */
+	for_each_iommu(iommu, drhd)
 		if (intel_setup_irq_remapping(iommu))
 			goto error;
-	}
+
 	return 0;
 error:
 	intel_cleanup_irq_remapping();
-- 
1.9.1

[PATCH 2/5] iommu/vt-d: Don't check for ecap_ir_support in intel_irq_remapping_supported

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

This check needs an initialized dmar-table and is already
done in intel_prepare_irq_remapping().

Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 drivers/iommu/intel_irq_remapping.c | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/drivers/iommu/intel_irq_remapping.c b/drivers/iommu/intel_irq_remapping.c
index 3ef63b8..45223ea 100644
--- a/drivers/iommu/intel_irq_remapping.c
+++ b/drivers/iommu/intel_irq_remapping.c
@@ -512,9 +512,6 @@ static int __init dmar_x2apic_optout(void)
 
 static int __init intel_irq_remapping_supported(void)
 {
-	struct dmar_drhd_unit *drhd;
-	struct intel_iommu *iommu;
-
 	if (disable_irq_remap)
 		return 0;
 	if (irq_remap_broken) {
@@ -532,10 +529,6 @@ static int __init intel_irq_remapping_supported(void)
 	if (!dmar_ir_support())
 		return 0;
 
-	for_each_iommu(iommu, drhd)
-		if (!ecap_ir_support(iommu->ecap))
-			return 0;
-
 	return 1;
 }
 
-- 
1.9.1

[PATCH 3/5] iommu/vt-d: Move supported-checks to intel_prepare_irq_remapping()

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

The checks for IRQ remapping support rely on ordering with
dmar_table_init(), which is called in
intel_prepare_irq_remapping(). Move the checks for IRQ
remapping support to this function to ensure the right
ordering.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 drivers/iommu/intel_irq_remapping.c | 39 +++++++++++++++++++++----------------
 1 file changed, 22 insertions(+), 17 deletions(-)

diff --git a/drivers/iommu/intel_irq_remapping.c b/drivers/iommu/intel_irq_remapping.c
index 45223ea..0c9226f 100644
--- a/drivers/iommu/intel_irq_remapping.c
+++ b/drivers/iommu/intel_irq_remapping.c
@@ -512,23 +512,6 @@ static int __init dmar_x2apic_optout(void)
 
 static int __init intel_irq_remapping_supported(void)
 {
-	if (disable_irq_remap)
-		return 0;
-	if (irq_remap_broken) {
-		printk(KERN_WARNING
-			"This system BIOS has enabled interrupt remapping\n"
-			"on a chipset that contains an erratum making that\n"
-			"feature unstable.  To maintain system stability\n"
-			"interrupt remapping is being disabled.  Please\n"
-			"contact your BIOS vendor for an update\n");
-		add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_STILL_OK);
-		disable_irq_remap = 1;
-		return 0;
-	}
-
-	if (!dmar_ir_support())
-		return 0;
-
 	return 1;
 }
 
@@ -553,6 +536,24 @@ static int __init intel_prepare_irq_remapping(void)
 	struct dmar_drhd_unit *drhd;
 	struct intel_iommu *iommu;
 
+	/* First check whether IRQ remapping should be enabled */
+	if (disable_irq_remap)
+		return -1;
+
+	/* Are we running on sane hardware? */
+	if (irq_remap_broken) {
+		printk(KERN_WARNING
+			"This system BIOS has enabled interrupt remapping\n"
+			"on a chipset that contains an erratum making that\n"
+			"feature unstable.  To maintain system stability\n"
+			"interrupt remapping is being disabled.  Please\n"
+			"contact your BIOS vendor for an update\n");
+		add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_STILL_OK);
+		disable_irq_remap = 1;
+		return -1;
+	}
+
+	/* Everthing seems fine so far, parse the DMAR table */
 	if (dmar_table_init() < 0)
 		return -1;
 
@@ -561,6 +562,10 @@ static int __init intel_prepare_irq_remapping(void)
 		goto error;
 	}
 
+	/* Is interrupt remapping supported by hardware? */
+	if (!dmar_ir_support())
+		return 0;
+
 	/* First make sure all IOMMUs support IRQ remapping */
 	for_each_iommu(iommu, drhd)
 		if (!ecap_ir_support(iommu->ecap))
-- 
1.9.1

[PATCH 4/5] iommu/amd: Check for irq-remap support amd_iommu_prepare()

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

This allows to get rid of the irq_remapping_supported()
function and all its call-backs into the Intel and AMD
IOMMU drivers.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 drivers/iommu/amd_iommu_init.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/iommu/amd_iommu_init.c b/drivers/iommu/amd_iommu_init.c
index de3390a..fa63016 100644
--- a/drivers/iommu/amd_iommu_init.c
+++ b/drivers/iommu/amd_iommu_init.c
@@ -2127,6 +2127,9 @@ static int __init iommu_go_to_state(enum iommu_init_state state)
 #ifdef CONFIG_IRQ_REMAP
 int __init amd_iommu_prepare(void)
 {
+	if (!amd_iommu_irq_remap)
+		return -1;
+
 	return iommu_go_to_state(IOMMU_ACPI_FINISHED);
 }
 
-- 
1.9.1

[PATCH 5/5] iommu, x86, apic: Remove irq_remapping_supported()

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

The functionality has been moved to irq_remapping_prepare,
so remove this function and all its call-backs.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 arch/x86/kernel/apic/apic.c         |  5 -----
 drivers/iommu/amd_iommu.c           |  1 -
 drivers/iommu/amd_iommu_init.c      |  5 -----
 drivers/iommu/amd_iommu_proto.h     |  1 -
 drivers/iommu/intel_irq_remapping.c |  6 ------
 drivers/iommu/irq_remapping.c       | 11 -----------
 drivers/iommu/irq_remapping.h       |  3 ---
 7 files changed, 32 deletions(-)

diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 141f103..4484483 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -1575,11 +1575,6 @@ void enable_x2apic(void)
 int __init enable_IR(void)
 {
 #ifdef CONFIG_IRQ_REMAP
-	if (!irq_remapping_supported()) {
-		pr_debug("intr-remapping not supported\n");
-		return -1;
-	}
-
 	if (!x2apic_preenabled && skip_ioapic_setup) {
 		pr_info("Skipped enabling intr-remap because of skipping "
 			"io-apic setup\n");
diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index a5c59b4..05ffb7c 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -4114,7 +4114,6 @@ static struct irq_domain *get_irq_domain(struct irq_alloc_info *info)
 }
 
 struct irq_remap_ops amd_iommu_irq_ops = {
-	.supported		= amd_iommu_supported,
 	.prepare		= amd_iommu_prepare,
 	.enable			= amd_iommu_enable,
 	.disable		= amd_iommu_disable,
diff --git a/drivers/iommu/amd_iommu_init.c b/drivers/iommu/amd_iommu_init.c
index fa63016..b9ce87e 100644
--- a/drivers/iommu/amd_iommu_init.c
+++ b/drivers/iommu/amd_iommu_init.c
@@ -2133,11 +2133,6 @@ int __init amd_iommu_prepare(void)
 	return iommu_go_to_state(IOMMU_ACPI_FINISHED);
 }
 
-int __init amd_iommu_supported(void)
-{
-	return amd_iommu_irq_remap ? 1 : 0;
-}
-
 int __init amd_iommu_enable(void)
 {
 	int ret;
diff --git a/drivers/iommu/amd_iommu_proto.h b/drivers/iommu/amd_iommu_proto.h
index 612a221..fcf61ca 100644
--- a/drivers/iommu/amd_iommu_proto.h
+++ b/drivers/iommu/amd_iommu_proto.h
@@ -33,7 +33,6 @@ extern void amd_iommu_init_notifier(void);
 extern void amd_iommu_init_api(void);
 
 /* Needed for interrupt remapping */
-extern int amd_iommu_supported(void);
 extern int amd_iommu_prepare(void);
 extern int amd_iommu_enable(void);
 extern void amd_iommu_disable(void);
diff --git a/drivers/iommu/intel_irq_remapping.c b/drivers/iommu/intel_irq_remapping.c
index 0c9226f..8918f47 100644
--- a/drivers/iommu/intel_irq_remapping.c
+++ b/drivers/iommu/intel_irq_remapping.c
@@ -510,11 +510,6 @@ static int __init dmar_x2apic_optout(void)
 	return dmar->flags & DMAR_X2APIC_OPT_OUT;
 }
 
-static int __init intel_irq_remapping_supported(void)
-{
-	return 1;
-}
-
 static void __init intel_cleanup_irq_remapping(void)
 {
 	struct dmar_drhd_unit *drhd;
@@ -970,7 +965,6 @@ static struct irq_domain *intel_get_irq_domain(struct irq_alloc_info *info)
 }
 
 struct irq_remap_ops intel_irq_remap_ops = {
-	.supported		= intel_irq_remapping_supported,
 	.prepare		= intel_prepare_irq_remapping,
 	.enable			= intel_enable_irq_remapping,
 	.disable		= disable_irq_remapping,
diff --git a/drivers/iommu/irq_remapping.c b/drivers/iommu/irq_remapping.c
index 66517e7..6e7acd1 100644
--- a/drivers/iommu/irq_remapping.c
+++ b/drivers/iommu/irq_remapping.c
@@ -80,17 +80,6 @@ void set_irq_remapping_broken(void)
 	irq_remap_broken = 1;
 }
 
-int irq_remapping_supported(void)
-{
-	if (disable_irq_remap)
-		return 0;
-
-	if (!remap_ops || !remap_ops->supported)
-		return 0;
-
-	return remap_ops->supported();
-}
-
 int __init irq_remapping_prepare(void)
 {
 	remap_ops = &intel_irq_remap_ops;
diff --git a/drivers/iommu/irq_remapping.h b/drivers/iommu/irq_remapping.h
index 4bd791d..961dc49 100644
--- a/drivers/iommu/irq_remapping.h
+++ b/drivers/iommu/irq_remapping.h
@@ -36,9 +36,6 @@ extern int no_x2apic_optout;
 extern int irq_remapping_enabled;
 
 struct irq_remap_ops {
-	/* Check whether Interrupt Remapping is supported */
-	int (*supported)(void);
-
 	/* Initializes hardware and makes it ready for remapping interrupts */
 	int  (*prepare)(void);
 
-- 
1.9.1

Re: [PATCH 0/5] Fix Intel IRQ remapping initialization order

[Jiang Liu]

On 2014/12/15 23:13, Joerg Roedel wrote:
> Hi,
> 
> here is a patch-set against tip/x86/apic to fix an initialization order
> problem with the IRQ remapping code.  The problem is in the ordering of
> the irq_remapping_prepare and irq_remapping_supported functions.
> 
> Currently the call-order is irq_remapping_prepare ->
> irq_remapping_supported, so that 'prepare' can succeed but 'supported'
> fails, so that interrupt remapping gets initialized but not enabled.
> This causes a broken interrupt setup on affected systems (machines with
> an Intel IOMMU without, or broken, IRQ remapping support). The result
> are lost interrupts and a non-bootable system.
> 
> Both functions do checks whether IRQ remapping can be enabled on the
> machine.  The reason for this is that some checks rely on
> dmar_table_init() and thus have to be done in irq_remapping_prepare().
> 
> This patch-set moves all these checks into the irq_remapping_prepare()
> path with the right ordering and removes the irq_remapping_supported()
> function and its call-backs. This fixes the initializion order problem
> and simplifies the exported API from the IOMMU code.
> 
> Please review.
Hi Joerg,
	I have posted a patch set for the same purpose at:
https://lkml.org/lkml/2014/12/10/20
	Seems we need to combine these two patch sets:)
Regards!
Gerry


> 
> Thanks,
> 
> 	Joerg
> 
> Joerg Roedel (5):
>   iommu/vt-d: Allocate IRQ remapping data structures only for all IOMMUs
>   iommu/vt-d: Don't check for ecap_ir_support in
>     intel_irq_remapping_supported
>   iommu/vt-d: Move supported-checks to intel_prepare_irq_remapping()
>   iommu/amd: Check for irq-remap support amd_iommu_prepare()
>   iommu, x86, apic: Remove irq_remapping_supported()
> 
>  arch/x86/kernel/apic/apic.c         |  5 ---
>  drivers/iommu/amd_iommu.c           |  1 -
>  drivers/iommu/amd_iommu_init.c      |  8 ++---
>  drivers/iommu/amd_iommu_proto.h     |  1 -
>  drivers/iommu/intel_irq_remapping.c | 62 +++++++++++++++++--------------------
>  drivers/iommu/irq_remapping.c       | 11 -------
>  drivers/iommu/irq_remapping.h       |  3 --
>  7 files changed, 31 insertions(+), 60 deletions(-)
> 

Re: [PATCH 0/5] Fix Intel IRQ remapping initialization order

[Thomas Gleixner]

On Mon, 15 Dec 2014, Jiang Liu wrote:
> On 2014/12/15 23:13, Joerg Roedel wrote:
> > Hi,
> > 
> > here is a patch-set against tip/x86/apic to fix an initialization order
> > problem with the IRQ remapping code.  The problem is in the ordering of
> > the irq_remapping_prepare and irq_remapping_supported functions.
> > 
> > Currently the call-order is irq_remapping_prepare ->
> > irq_remapping_supported, so that 'prepare' can succeed but 'supported'
> > fails, so that interrupt remapping gets initialized but not enabled.
> > This causes a broken interrupt setup on affected systems (machines with
> > an Intel IOMMU without, or broken, IRQ remapping support). The result
> > are lost interrupts and a non-bootable system.
> > 
> > Both functions do checks whether IRQ remapping can be enabled on the
> > machine.  The reason for this is that some checks rely on
> > dmar_table_init() and thus have to be done in irq_remapping_prepare().
> > 
> > This patch-set moves all these checks into the irq_remapping_prepare()
> > path with the right ordering and removes the irq_remapping_supported()
> > function and its call-backs. This fixes the initializion order problem
> > and simplifies the exported API from the IOMMU code.
> > 
> > Please review.
> Hi Joerg,
> 	I have posted a patch set for the same purpose at:
> https://lkml.org/lkml/2014/12/10/20
> 	Seems we need to combine these two patch sets:)

Actually you want to combine it also with these patches:

326c2bb2c526: iommu/vt-d: Convert allocations to GFP_KERNEL
e9220e591375: iommu/vt-d: Move iommu preparatory allocations to irq_remap_ops.prepare
e88edbd316ea: iommu, x86: Restructure setup of the irq remapping feature
dd60143c04f2: x86, smpboot: Remove pointless preempt_disable() in native_smp_prepare_cpus()

against 3.19 independent of the irqdomain stuff.

So that would be a clean base to put the rest of the irqdomain and
hotplug stuff on top.

So the ordering of the patches for 3.20 would become:

   iommu cleanup (init and allocations)
   acpi cleanup and parser extensions
   ioapic hotplug
   irqdomain conversion

I will route dd60143c04f2 "x86, smpboot: Remove pointless
preempt_disable() in native_smp_prepare_cpus()" into -rc1.  I'm going
to do so for a few other cherry-picks from x86/apic.

So can you please create a combined series, which just deals with the
init cleanup and the allocation conversion (ATOMIC -> GFP) based on
current Linus tree should be fine.

Thanks,

	tglx

Re: [PATCH 0/5] Fix Intel IRQ remapping initialization order

[Jiang Liu]

On 2014/12/16 3:54, Thomas Gleixner wrote:
> On Mon, 15 Dec 2014, Jiang Liu wrote:
>> On 2014/12/15 23:13, Joerg Roedel wrote:
>>> Hi,
>>>
>>> here is a patch-set against tip/x86/apic to fix an initialization order
>>> problem with the IRQ remapping code.  The problem is in the ordering of
>>> the irq_remapping_prepare and irq_remapping_supported functions.
>>>
>>> Currently the call-order is irq_remapping_prepare ->
>>> irq_remapping_supported, so that 'prepare' can succeed but 'supported'
>>> fails, so that interrupt remapping gets initialized but not enabled.
>>> This causes a broken interrupt setup on affected systems (machines with
>>> an Intel IOMMU without, or broken, IRQ remapping support). The result
>>> are lost interrupts and a non-bootable system.
>>>
>>> Both functions do checks whether IRQ remapping can be enabled on the
>>> machine.  The reason for this is that some checks rely on
>>> dmar_table_init() and thus have to be done in irq_remapping_prepare().
>>>
>>> This patch-set moves all these checks into the irq_remapping_prepare()
>>> path with the right ordering and removes the irq_remapping_supported()
>>> function and its call-backs. This fixes the initializion order problem
>>> and simplifies the exported API from the IOMMU code.
>>>
>>> Please review.
>> Hi Joerg,
>> 	I have posted a patch set for the same purpose at:
>> https://lkml.org/lkml/2014/12/10/20
>> 	Seems we need to combine these two patch sets:)
> 
> Actually you want to combine it also with these patches:
> 
> 326c2bb2c526: iommu/vt-d: Convert allocations to GFP_KERNEL
> e9220e591375: iommu/vt-d: Move iommu preparatory allocations to irq_remap_ops.prepare
> e88edbd316ea: iommu, x86: Restructure setup of the irq remapping feature
> dd60143c04f2: x86, smpboot: Remove pointless preempt_disable() in native_smp_prepare_cpus()
> 
> against 3.19 independent of the irqdomain stuff.
> 
> So that would be a clean base to put the rest of the irqdomain and
> hotplug stuff on top.
> 
> So the ordering of the patches for 3.20 would become:
> 
>    iommu cleanup (init and allocations)
>    acpi cleanup and parser extensions
>    ioapic hotplug
>    irqdomain conversion
> 
> I will route dd60143c04f2 "x86, smpboot: Remove pointless
> preempt_disable() in native_smp_prepare_cpus()" into -rc1.  I'm going
> to do so for a few other cherry-picks from x86/apic.
> 
> So can you please create a combined series, which just deals with the
> init cleanup and the allocation conversion (ATOMIC -> GFP) based on
> current Linus tree should be fine.
Hi Thomas,
	I will work on that, hope we will have something after Christmas Holiday:)
Regards!
Gerry
> 
> Thanks,
> 
> 	tglx
>