Booting dom0, various issues since v3.16 kernels

Booting dom0, various issues since v3.16 kernels

[Stefan Bader]

Hey David,

after just being in that pain, I thought I might as well give a summary to
you/the list. Maybe helpful to not forget which piece should go to which stable...

So:
v3.16...v3.17.8: Somewhen in between those, the acpi irq seems to have broken.
                 I have not yet verified that, but at least three changes in
                 3.19-rc6 seem to look related:

 * "x86/xen: Treat SCI interrupt as normal GSI interrupt",
 * "ACPI: pci: Do not clear pci_dev->irq in acpi_pci_irq_disable()", and
 * "x86/xen: Override ACPI IRQ management callback __acpi_unregister_gsi"

v3.17.8...v3.18.4: Beside the acpi interrupt, no USB devices (beyond the
                   hubs) get initialized. Not sure what fixed it, but it
                   looks ok in v3.19-rc7.
                   Beside that, there also was a regression in swiotlb
                   that I think was passed on to some stable maintainers:

 * "Revert "swiotlb-xen: pass dev_addr to swiotlb_tbl_unmap_single""

v3.18.4..v3.19-rc7: The issues above look to be fixed. Only some Haswell
                    based box now crashed on boot as dom0 while parsing
                    some ACPI tables (will send more detail seperately).
                    This happens only on that host and only when running
                    as dom0. Bare-metal is ok and an Opteron based different
                    host is also fine.

-Stefan

Re: Booting dom0, various issues since v3.16 kernels

[Sander Eikelenboom]

Thursday, February 5, 2015, 3:22:49 PM, you wrote:

> Hey David,

> after just being in that pain, I thought I might as well give a summary to
> you/the list. Maybe helpful to not forget which piece should go to which stable...

> So:
> v3.16...v3.17.8: Somewhen in between those, the acpi irq seems to have broken.
>                  I have not yet verified that, but at least three changes in
>                  3.19-rc6 seem to look related:

>  * "x86/xen: Treat SCI interrupt as normal GSI interrupt",
>  * "ACPI: pci: Do not clear pci_dev->irq in acpi_pci_irq_disable()", and
>  * "x86/xen: Override ACPI IRQ management callback __acpi_unregister_gsi"

Yes Jiang Liu fixed those and said he would backport the required fixes once
they where accepted in mainline, put perhaps a polite ping is necessary there.

> v3.17.8...v3.18.4: Beside the acpi interrupt, no USB devices (beyond the
>                    hubs) get initialized. Not sure what fixed it, but it
>                    looks ok in v3.19-rc7.
>                    Beside that, there also was a regression in swiotlb
>                    that I think was passed on to some stable maintainers:

Probably the same issue as above (for me it fixed a powerbutton issue and some
pci-passthrough problems).

And there seems to be more refactoring coming for 3.20 .. so fingerscrossed.

--
Sander

>  * "Revert "swiotlb-xen: pass dev_addr to swiotlb_tbl_unmap_single""

> v3.18.4..v3.19-rc7: The issues above look to be fixed. Only some Haswell
>                     based box now crashed on boot as dom0 while parsing
>                     some ACPI tables (will send more detail seperately).
>                     This happens only on that host and only when running
>                     as dom0. Bare-metal is ok and an Opteron based different
>                     host is also fine.

> -Stefan

Re: Booting dom0, various issues since v3.16 kernels

[Konrad Rzeszutek Wilk]

On Thu, Feb 05, 2015 at 03:47:11PM +0100, Sander Eikelenboom wrote:
> 
> Thursday, February 5, 2015, 3:22:49 PM, you wrote:
> 
> > Hey David,
> 
> > after just being in that pain, I thought I might as well give a summary to
> > you/the list. Maybe helpful to not forget which piece should go to which stable...
> 
> > So:
> > v3.16...v3.17.8: Somewhen in between those, the acpi irq seems to have broken.
> >                  I have not yet verified that, but at least three changes in
> >                  3.19-rc6 seem to look related:
> 
> >  * "x86/xen: Treat SCI interrupt as normal GSI interrupt",
> >  * "ACPI: pci: Do not clear pci_dev->irq in acpi_pci_irq_disable()", and
> >  * "x86/xen: Override ACPI IRQ management callback __acpi_unregister_gsi"
> 
> Yes Jiang Liu fixed those and said he would backport the required fixes once
> they where accepted in mainline, put perhaps a polite ping is necessary there.

Would you be OK pinging him please?
> 
> > v3.17.8...v3.18.4: Beside the acpi interrupt, no USB devices (beyond the
> >                    hubs) get initialized. Not sure what fixed it, but it
> >                    looks ok in v3.19-rc7.
> >                    Beside that, there also was a regression in swiotlb
> >                    that I think was passed on to some stable maintainers:
> 
> Probably the same issue as above (for me it fixed a powerbutton issue and some
> pci-passthrough problems).
> 
> And there seems to be more refactoring coming for 3.20 .. so fingerscrossed.
> 
> --
> Sander
> 
> >  * "Revert "swiotlb-xen: pass dev_addr to swiotlb_tbl_unmap_single""
> 
> > v3.18.4..v3.19-rc7: The issues above look to be fixed. Only some Haswell
> >                     based box now crashed on boot as dom0 while parsing
> >                     some ACPI tables (will send more detail seperately).
> >                     This happens only on that host and only when running
> >                     as dom0. Bare-metal is ok and an Opteron based different
> >                     host is also fine.
> 
> > -Stefan
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Booting dom0, various issues since v3.16 kernels

[Stefan Bader]

[-- Attachment #1.1.1: Type: text/plain, Size: 2583 bytes --]

On 05.02.2015 15:47, Sander Eikelenboom wrote:
> 
> Thursday, February 5, 2015, 3:22:49 PM, you wrote:
> 
>> Hey David,
> 
>> after just being in that pain, I thought I might as well give a summary to
>> you/the list. Maybe helpful to not forget which piece should go to which stable...
> 
>> So:
>> v3.16...v3.17.8: Somewhen in between those, the acpi irq seems to have broken.
>>                  I have not yet verified that, but at least three changes in
>>                  3.19-rc6 seem to look related:
> 
>>  * "x86/xen: Treat SCI interrupt as normal GSI interrupt",
>>  * "ACPI: pci: Do not clear pci_dev->irq in acpi_pci_irq_disable()", and
>>  * "x86/xen: Override ACPI IRQ management callback __acpi_unregister_gsi"
> 
> Yes Jiang Liu fixed those and said he would backport the required fixes once
> they where accepted in mainline, put perhaps a polite ping is necessary there.
> 
>> v3.17.8...v3.18.4: Beside the acpi interrupt, no USB devices (beyond the
>>                    hubs) get initialized. Not sure what fixed it, but it
>>                    looks ok in v3.19-rc7.
>>                    Beside that, there also was a regression in swiotlb
>>                    that I think was passed on to some stable maintainers:
> 
> Probably the same issue as above (for me it fixed a powerbutton issue and some
> pci-passthrough problems).
> 
> And there seems to be more refactoring coming for 3.20 .. so fingerscrossed.
> 
> --
> Sander

Hi Sander,

sorry, I know you did the ping somewhere in another thread. The one which I
cannot find again right now. :/ Maybe you can forward the following two patches
which would be my backport to 3.18. I hope things are correct. I dropped the
middle patch as it did not seem to be needed and mushed around the other two.
At least things seemed to be fixed up on a quick test-boot.

-Stefan


> 
>>  * "Revert "swiotlb-xen: pass dev_addr to swiotlb_tbl_unmap_single""
> 
>> v3.18.4..v3.19-rc7: The issues above look to be fixed. Only some Haswell
>>                     based box now crashed on boot as dom0 while parsing
>>                     some ACPI tables (will send more detail seperately).
>>                     This happens only on that host and only when running
>>                     as dom0. Bare-metal is ok and an Opteron based different
>>                     host is also fine.
> 
>> -Stefan
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.1.2: 0001-x86-xen-Treat-SCI-interrupt-as-normal-GSI-interrupt.patch --]
[-- Type: text/x-diff; name="0001-x86-xen-Treat-SCI-interrupt-as-normal-GSI-interrupt.patch", Size: 5360 bytes --]

From 8b5b328b62248d95743ca9af7aa71c06dd808dfe Mon Sep 17 00:00:00 2001
From: Jiang Liu <jiang.liu@linux.intel.com>
Date: Tue, 20 Jan 2015 10:21:05 +0800
Subject: [PATCH 1/2] x86/xen: Treat SCI interrupt as normal GSI interrupt

Currently Xen Domain0 has special treatment for ACPI SCI interrupt,
that is initialize irq for ACPI SCI at early stage in a special way as:
xen_init_IRQ()
	->pci_xen_initial_domain()
		->xen_setup_acpi_sci()
			Allocate and initialize irq for ACPI SCI

Function xen_setup_acpi_sci() calls acpi_gsi_to_irq() to get an irq
number for ACPI SCI. But unfortunately acpi_gsi_to_irq() depends on
IOAPIC irqdomains through following path
acpi_gsi_to_irq()
	->mp_map_gsi_to_irq()
		->mp_map_pin_to_irq()
			->check IOAPIC irqdomain

For PV domains, it uses Xen event based interrupt manangement and
doesn't make uses of native IOAPIC, so no irqdomains created for IOAPIC.
This causes Xen domain0 fail to install interrupt handler for ACPI SCI
and all ACPI events will be lost. Please refer to:
https://lkml.org/lkml/2014/12/19/178

So the fix is to get rid of special treatment for ACPI SCI, just treat
ACPI SCI as normal GSI interrupt as:
acpi_gsi_to_irq()
	->acpi_register_gsi()
		->acpi_register_gsi_xen()
			->xen_register_gsi()

With above change, there's no need for xen_setup_acpi_sci() anymore.
The above change also works with bare metal kernel too.

Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
Tested-by: Sander Eikelenboom <linux@eikelenboom.it>
Cc: Tony Luck <tony.luck@intel.com>
Cc: xen-devel@lists.xenproject.org
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Len Brown <len.brown@intel.com>
Cc: Pavel Machek <pavel@ucw.cz>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Link: http://lkml.kernel.org/r/1421720467-7709-2-git-send-email-jiang.liu@linux.intel.com
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Stefan Bader <stefan.bader@canonical.com>
---
 arch/x86/kernel/acpi/boot.c | 23 +++++++++++-----------
 arch/x86/pci/xen.c          | 47 ---------------------------------------------
 2 files changed, 12 insertions(+), 58 deletions(-)

diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
index a142e77..460f498 100644
--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -604,18 +604,19 @@ void __init acpi_pic_sci_set_trigger(unsigned int irq, u16 trigger)
 
 int acpi_gsi_to_irq(u32 gsi, unsigned int *irqp)
 {
-	int irq;
-
-	if (acpi_irq_model == ACPI_IRQ_MODEL_PIC) {
-		*irqp = gsi;
-	} else {
-		irq = mp_map_gsi_to_irq(gsi,
-					IOAPIC_MAP_ALLOC | IOAPIC_MAP_CHECK);
-		if (irq < 0)
-			return -1;
-		*irqp = irq;
+	int rc, irq, trigger, polarity;
+
+	rc = acpi_get_override_irq(gsi, &trigger, &polarity);
+	if (rc == 0) {
+		trigger = trigger ? ACPI_LEVEL_SENSITIVE : ACPI_EDGE_SENSITIVE;
+		polarity = polarity ? ACPI_ACTIVE_LOW : ACPI_ACTIVE_HIGH;
+		irq = acpi_register_gsi(NULL, gsi, trigger, polarity);
+		if (irq >= 0) {
+			*irqp = irq;
+			return 0;
+		}
 	}
-	return 0;
+	return -1;
 }
 EXPORT_SYMBOL_GPL(acpi_gsi_to_irq);
 
diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c
index 093f5f4..6b3cf7c 100644
--- a/arch/x86/pci/xen.c
+++ b/arch/x86/pci/xen.c
@@ -452,52 +452,6 @@ int __init pci_xen_hvm_init(void)
 }
 
 #ifdef CONFIG_XEN_DOM0
-static __init void xen_setup_acpi_sci(void)
-{
-	int rc;
-	int trigger, polarity;
-	int gsi = acpi_sci_override_gsi;
-	int irq = -1;
-	int gsi_override = -1;
-
-	if (!gsi)
-		return;
-
-	rc = acpi_get_override_irq(gsi, &trigger, &polarity);
-	if (rc) {
-		printk(KERN_WARNING "xen: acpi_get_override_irq failed for acpi"
-				" sci, rc=%d\n", rc);
-		return;
-	}
-	trigger = trigger ? ACPI_LEVEL_SENSITIVE : ACPI_EDGE_SENSITIVE;
-	polarity = polarity ? ACPI_ACTIVE_LOW : ACPI_ACTIVE_HIGH;
-
-	printk(KERN_INFO "xen: sci override: global_irq=%d trigger=%d "
-			"polarity=%d\n", gsi, trigger, polarity);
-
-	/* Before we bind the GSI to a Linux IRQ, check whether
-	 * we need to override it with bus_irq (IRQ) value. Usually for
-	 * IRQs below IRQ_LEGACY_IRQ this holds IRQ == GSI, as so:
-	 *  ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
-	 * but there are oddballs where the IRQ != GSI:
-	 *  ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 20 low level)
-	 * which ends up being: gsi_to_irq[9] == 20
-	 * (which is what acpi_gsi_to_irq ends up calling when starting the
-	 * the ACPI interpreter and keels over since IRQ 9 has not been
-	 * setup as we had setup IRQ 20 for it).
-	 */
-	if (acpi_gsi_to_irq(gsi, &irq) == 0) {
-		/* Use the provided value if it's valid. */
-		if (irq >= 0)
-			gsi_override = irq;
-	}
-
-	gsi = xen_register_gsi(gsi, gsi_override, trigger, polarity);
-	printk(KERN_INFO "xen: acpi sci %d\n", gsi);
-
-	return;
-}
-
 int __init pci_xen_initial_domain(void)
 {
 	int irq;
@@ -509,7 +463,6 @@ int __init pci_xen_initial_domain(void)
 	x86_msi.msi_mask_irq = xen_nop_msi_mask_irq;
 	x86_msi.msix_mask_irq = xen_nop_msix_mask_irq;
 #endif
-	xen_setup_acpi_sci();
 	__acpi_register_gsi = acpi_register_gsi_xen;
 	/* Pre-allocate legacy irqs */
 	for (irq = 0; irq < nr_legacy_irqs(); irq++) {
-- 
1.9.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.1.3: 0002-x86-xen-Override-ACPI-IRQ-management-callback-__acpi.patch --]
[-- Type: text/x-diff; name="0002-x86-xen-Override-ACPI-IRQ-management-callback-__acpi.patch", Size: 2397 bytes --]

From 96f1d05b484334ff01ea2ee2b66798e294e34d96 Mon Sep 17 00:00:00 2001
From: Jiang Liu <jiang.liu@linux.intel.com>
Date: Tue, 20 Jan 2015 10:21:07 +0800
Subject: [PATCH 2/2] x86/xen: Override ACPI IRQ management callback
 __acpi_unregister_gsi

Xen overrides __acpi_register_gsi and leaves __acpi_unregister_gsi as is.
That means, an IRQ allocated by acpi_register_gsi_xen_hvm() or
acpi_register_gsi_xen() will be freed by acpi_unregister_gsi_ioapic(),
which may cause undesired effects. So override __acpi_unregister_gsi to
NULL for safety.

Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
Tested-by: Sander Eikelenboom <linux@eikelenboom.it>
Cc: Tony Luck <tony.luck@intel.com>
Cc: xen-devel@lists.xenproject.org
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: David Vrabel <david.vrabel@citrix.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Graeme Gregory <graeme.gregory@linaro.org>
Cc: Lv Zheng <lv.zheng@intel.com>
Link: http://lkml.kernel.org/r/1421720467-7709-4-git-send-email-jiang.liu@linux.intel.com
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Stefan Bader <stefan.bader@canonical.com>
---
 arch/x86/include/asm/acpi.h | 1 +
 arch/x86/pci/xen.c          | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/arch/x86/include/asm/acpi.h b/arch/x86/include/asm/acpi.h
index 0ab4f9f..3a45668 100644
--- a/arch/x86/include/asm/acpi.h
+++ b/arch/x86/include/asm/acpi.h
@@ -50,6 +50,7 @@ void acpi_pic_sci_set_trigger(unsigned int, u16);
 
 extern int (*__acpi_register_gsi)(struct device *dev, u32 gsi,
 				  int trigger, int polarity);
+extern void (*__acpi_unregister_gsi)(u32 gsi);
 
 static inline void disable_acpi(void)
 {
diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c
index 6b3cf7c..3c41716 100644
--- a/arch/x86/pci/xen.c
+++ b/arch/x86/pci/xen.c
@@ -442,6 +442,7 @@ int __init pci_xen_hvm_init(void)
 	 * just how GSIs get registered.
 	 */
 	__acpi_register_gsi = acpi_register_gsi_xen_hvm;
+	__acpi_unregister_gsi = NULL;
 #endif
 
 #ifdef CONFIG_PCI_MSI
@@ -464,6 +465,7 @@ int __init pci_xen_initial_domain(void)
 	x86_msi.msix_mask_irq = xen_nop_msix_mask_irq;
 #endif
 	__acpi_register_gsi = acpi_register_gsi_xen;
+	__acpi_unregister_gsi = NULL;
 	/* Pre-allocate legacy irqs */
 	for (irq = 0; irq < nr_legacy_irqs(); irq++) {
 		int trigger, polarity;
-- 
1.9.1


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Booting dom0, various issues since v3.16 kernels

[Sander Eikelenboom]

Monday, February 9, 2015, 10:31:15 AM, you wrote:

> On 05.02.2015 15:47, Sander Eikelenboom wrote:
>> 
>> Thursday, February 5, 2015, 3:22:49 PM, you wrote:
>> 
>>> Hey David,
>> 
>>> after just being in that pain, I thought I might as well give a summary to
>>> you/the list. Maybe helpful to not forget which piece should go to which stable...
>> 
>>> So:
>>> v3.16...v3.17.8: Somewhen in between those, the acpi irq seems to have broken.
>>>                  I have not yet verified that, but at least three changes in
>>>                  3.19-rc6 seem to look related:
>> 
>>>  * "x86/xen: Treat SCI interrupt as normal GSI interrupt",
>>>  * "ACPI: pci: Do not clear pci_dev->irq in acpi_pci_irq_disable()", and
>>>  * "x86/xen: Override ACPI IRQ management callback __acpi_unregister_gsi"
>> 
>> Yes Jiang Liu fixed those and said he would backport the required fixes once
>> they where accepted in mainline, put perhaps a polite ping is necessary there.
>> 
>>> v3.17.8...v3.18.4: Beside the acpi interrupt, no USB devices (beyond the
>>>                    hubs) get initialized. Not sure what fixed it, but it
>>>                    looks ok in v3.19-rc7.
>>>                    Beside that, there also was a regression in swiotlb
>>>                    that I think was passed on to some stable maintainers:
>> 
>> Probably the same issue as above (for me it fixed a powerbutton issue and some
>> pci-passthrough problems).
>> 
>> And there seems to be more refactoring coming for 3.20 .. so fingerscrossed.
>> 
>> --
>> Sander

> Hi Sander,

> sorry, I know you did the ping somewhere in another thread. The one which I
> cannot find again right now. :/ Maybe you can forward the following two patches
> which would be my backport to 3.18. I hope things are correct. I dropped the
> middle patch as it did not seem to be needed and mushed around the other two.
> At least things seemed to be fixed up on a quick test-boot.

> -Stefan

Hi Stefan,

I will also do a short test to see if it also fixes the problems i was seeing,
and report back to you and/or forward your patches.

Thanks for the effort !

--
Sander



>> 
>>>  * "Revert "swiotlb-xen: pass dev_addr to swiotlb_tbl_unmap_single""
>> 
>>> v3.18.4..v3.19-rc7: The issues above look to be fixed. Only some Haswell
>>>                     based box now crashed on boot as dom0 while parsing
>>>                     some ACPI tables (will send more detail seperately).
>>>                     This happens only on that host and only when running
>>>                     as dom0. Bare-metal is ok and an Opteron based different
>>>                     host is also fine.
>> 
>>> -Stefan
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>>