[PATCH] PCI: vmd: Free up IRQs on suspend path

[PATCH] PCI: vmd: Free up IRQs on suspend path

[Sushma Kalakota]

commit e2b1820bd5d09 upstream

Free up the IRQs we request on the suspend path and reallocate them on the
resume path.

Fixes this error:

  CPU 111 disable failed: CPU has 9 vectors assigned and there are only 0 available.
  Error taking CPU111 down: -34
  Non-boot CPUs are not disabled
  Enabling non-boot CPUs ...

For consistency, this patch also includes the VMD portion of:
3eefa790c9681: PCI: host: Mark PCIe/PCI (MSI) cascade ISR as IRQF_NO_THREAD

CC: Scott Bauer <scott.bauer@intel.com>
CC: Bjorn Helgaas <bhelgaas@google.com>
CC: Keith Busch <keith.busch@intel.com>
Reviewed-by: Jon Derrick <jonathan.derrick@intel.com>
Signed-off-by: Sushma Kalakota <sushmax.kalakota@intel.com>
---
 drivers/pci/host/vmd.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
index 0e7f8f319fe3..5c2258ad3589 100644
--- a/drivers/pci/host/vmd.c
+++ b/drivers/pci/host/vmd.c
@@ -698,7 +698,8 @@ static int vmd_probe(struct pci_dev *dev, const struct pci_device_id *id)
 	for (i = 0; i < vmd->msix_count; i++) {
 		INIT_LIST_HEAD(&vmd->irqs[i].irq_list);
 		err = devm_request_irq(&dev->dev, pci_irq_vector(dev, i),
-				       vmd_irq, 0, "vmd", &vmd->irqs[i]);
+				       vmd_irq, IRQF_NO_THREAD,
+				       "vmd", &vmd->irqs[i]);
 		if (err)
 			return err;
 	}
@@ -731,6 +732,11 @@ static void vmd_remove(struct pci_dev *dev)
 static int vmd_suspend(struct device *dev)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
+	struct vmd_dev *vmd = pci_get_drvdata(pdev);
+	int i;
+
+	for (i = 0; i < vmd->msix_count; i++)
+                devm_free_irq(dev, pci_irq_vector(pdev, i), &vmd->irqs[i]);
 
 	pci_save_state(pdev);
 	return 0;
@@ -739,6 +745,16 @@ static int vmd_suspend(struct device *dev)
 static int vmd_resume(struct device *dev)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
+	struct vmd_dev *vmd = pci_get_drvdata(pdev);
+	int err, i;
+
+	for (i = 0; i < vmd->msix_count; i++) {
+		err = devm_request_irq(dev, pci_irq_vector(pdev, i),
+				       vmd_irq, IRQF_NO_THREAD,
+				       "vmd", &vmd->irqs[i]);
+		if (err)
+			return err;
+	}
 
 	pci_restore_state(pdev);
 	return 0;
-- 
2.17.1

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[Greg KH]

On Wed, Feb 06, 2019 at 02:36:16PM -0700, Sushma Kalakota wrote:
> commit e2b1820bd5d09 upstream
> 
> Free up the IRQs we request on the suspend path and reallocate them on the
> resume path.
> 
> Fixes this error:
> 
>   CPU 111 disable failed: CPU has 9 vectors assigned and there are only 0 available.
>   Error taking CPU111 down: -34
>   Non-boot CPUs are not disabled
>   Enabling non-boot CPUs ...
> 
> For consistency, this patch also includes the VMD portion of:
> 3eefa790c9681: PCI: host: Mark PCIe/PCI (MSI) cascade ISR as IRQF_NO_THREAD
> 
> CC: Scott Bauer <scott.bauer@intel.com>
> CC: Bjorn Helgaas <bhelgaas@google.com>
> CC: Keith Busch <keith.busch@intel.com>
> Reviewed-by: Jon Derrick <jonathan.derrick@intel.com>
> Signed-off-by: Sushma Kalakota <sushmax.kalakota@intel.com>
> ---
>  drivers/pci/host/vmd.c | 18 +++++++++++++++++-
>  1 file changed, 17 insertions(+), 1 deletion(-)
> 

What stable kernel tree(s) do you want this patch applied to?  And why?

thanks,

greg k-h

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[Derrick, Jonathan]

[-- Attachment #1: Type: text/plain, Size: 1463 bytes --]

On Thu, 2019-02-07 at 12:07 +0100, Greg KH wrote:
> On Wed, Feb 06, 2019 at 02:36:16PM -0700, Sushma Kalakota wrote:
> > commit e2b1820bd5d09 upstream
> > 
> > Free up the IRQs we request on the suspend path and reallocate them
> > on the
> > resume path.
> > 
> > Fixes this error:
> > 
> >   CPU 111 disable failed: CPU has 9 vectors assigned and there are
> > only 0 available.
> >   Error taking CPU111 down: -34
> >   Non-boot CPUs are not disabled
> >   Enabling non-boot CPUs ...
> > 
> > For consistency, this patch also includes the VMD portion of:
> > 3eefa790c9681: PCI: host: Mark PCIe/PCI (MSI) cascade ISR as
> > IRQF_NO_THREAD
> > 
> > CC: Scott Bauer <scott.bauer@intel.com>
> > CC: Bjorn Helgaas <bhelgaas@google.com>
> > CC: Keith Busch <keith.busch@intel.com>
> > Reviewed-by: Jon Derrick <jonathan.derrick@intel.com>
> > Signed-off-by: Sushma Kalakota <sushmax.kalakota@intel.com>
> > ---
> >  drivers/pci/host/vmd.c | 18 +++++++++++++++++-
> >  1 file changed, 17 insertions(+), 1 deletion(-)
> > 
> 
> What stable kernel tree(s) do you want this patch applied to?  And
> why?
> 
> thanks,
> 
> greg k-h

Hi Greg,

This is for 4.9.y and intended to increase reliability of
suspend/resume not leading to a failure to suspend or failure to
resume, either of which would be undesireable to users of the feature,
and potentially make the overall feature unusable if suspend/resume
were user requirement.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3278 bytes --]

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[gregkh]

On Thu, Feb 07, 2019 at 03:10:09PM +0000, Derrick, Jonathan wrote:
> On Thu, 2019-02-07 at 12:07 +0100, Greg KH wrote:
> > On Wed, Feb 06, 2019 at 02:36:16PM -0700, Sushma Kalakota wrote:
> > > commit e2b1820bd5d09 upstream
> > > 
> > > Free up the IRQs we request on the suspend path and reallocate them
> > > on the
> > > resume path.
> > > 
> > > Fixes this error:
> > > 
> > >   CPU 111 disable failed: CPU has 9 vectors assigned and there are
> > > only 0 available.
> > >   Error taking CPU111 down: -34
> > >   Non-boot CPUs are not disabled
> > >   Enabling non-boot CPUs ...
> > > 
> > > For consistency, this patch also includes the VMD portion of:
> > > 3eefa790c9681: PCI: host: Mark PCIe/PCI (MSI) cascade ISR as
> > > IRQF_NO_THREAD
> > > 
> > > CC: Scott Bauer <scott.bauer@intel.com>
> > > CC: Bjorn Helgaas <bhelgaas@google.com>
> > > CC: Keith Busch <keith.busch@intel.com>
> > > Reviewed-by: Jon Derrick <jonathan.derrick@intel.com>
> > > Signed-off-by: Sushma Kalakota <sushmax.kalakota@intel.com>
> > > ---
> > >  drivers/pci/host/vmd.c | 18 +++++++++++++++++-
> > >  1 file changed, 17 insertions(+), 1 deletion(-)
> > > 
> > 
> > What stable kernel tree(s) do you want this patch applied to?  And
> > why?
> > 
> > thanks,
> > 
> > greg k-h
> 
> Hi Greg,
> 
> This is for 4.9.y and intended to increase reliability of
> suspend/resume not leading to a failure to suspend or failure to
> resume, either of which would be undesireable to users of the feature,
> and potentially make the overall feature unusable if suspend/resume
> were user requirement.

Ok, but who is still using this old kernel on these types of machines?
Why haven't they moved to 4.14 or newer by now?  The normal systems that
use 4.9 or older should not need this, as they are the horrid SoC trees.

Who has reported this problem in their systems?

thanks,

greg k-h

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[Greg KH]

On Wed, Feb 06, 2019 at 02:36:16PM -0700, Sushma Kalakota wrote:
> commit e2b1820bd5d09 upstream
> 
> Free up the IRQs we request on the suspend path and reallocate them on the
> resume path.
> 
> Fixes this error:
> 
>   CPU 111 disable failed: CPU has 9 vectors assigned and there are only 0 available.
>   Error taking CPU111 down: -34
>   Non-boot CPUs are not disabled
>   Enabling non-boot CPUs ...
> 
> For consistency, this patch also includes the VMD portion of:
> 3eefa790c9681: PCI: host: Mark PCIe/PCI (MSI) cascade ISR as IRQF_NO_THREAD
> 
> CC: Scott Bauer <scott.bauer@intel.com>
> CC: Bjorn Helgaas <bhelgaas@google.com>
> CC: Keith Busch <keith.busch@intel.com>
> Reviewed-by: Jon Derrick <jonathan.derrick@intel.com>
> Signed-off-by: Sushma Kalakota <sushmax.kalakota@intel.com>

Now queued up, thanks.

greg k-h

[PATCH] PCI: vmd: Free up IRQs on suspend path

[Scott Bauer]

This patch frees up the IRQs we request on the suspend path,
and reallocates them on the resume path.

Fixes:
[  559.964386] CPU 111 disable failed: CPU has 9 vectors assigned and there are only 0 available.
[  559.966824] Error taking CPU111 down: -34
[  559.966825] Non-boot CPUs are not disabled
[  559.966826] Enabling non-boot CPUs ...

Signed-off-by: Scott Bauer <scott.bauer@intel.com>
---
 drivers/pci/host/vmd.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
index 7e967a8784b2..4fe1756af010 100644
--- a/drivers/pci/host/vmd.c
+++ b/drivers/pci/host/vmd.c
@@ -763,6 +763,11 @@ static void vmd_remove(struct pci_dev *dev)
 static int vmd_suspend(struct device *dev)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
+	struct vmd_dev *vmd = pci_get_drvdata(pdev);
+	int i;
+
+	for (i = 0; i < vmd->msix_count; i++)
+                devm_free_irq(dev, pci_irq_vector(pdev, i), &vmd->irqs[i]);
 
 	pci_save_state(pdev);
 	return 0;
@@ -771,6 +776,16 @@ static int vmd_suspend(struct device *dev)
 static int vmd_resume(struct device *dev)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
+	struct vmd_dev *vmd = pci_get_drvdata(pdev);
+	int err, i;
+
+	for (i = 0; i < vmd->msix_count; i++) {
+		err = devm_request_irq(dev, pci_irq_vector(pdev, i),
+				       vmd_irq, IRQF_NO_THREAD,
+				       "vmd", &vmd->irqs[i]);
+		if (err)
+			return err;
+	}
 
 	pci_restore_state(pdev);
 	return 0;
-- 
2.11.0

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[Bjorn Helgaas]

On Fri, Aug 11, 2017 at 02:54:32PM -0600, Scott Bauer wrote:
> This patch frees up the IRQs we request on the suspend path,
> and reallocates them on the resume path.
> 
> Fixes:
> [  559.964386] CPU 111 disable failed: CPU has 9 vectors assigned and there are only 0 available.
> [  559.966824] Error taking CPU111 down: -34
> [  559.966825] Non-boot CPUs are not disabled
> [  559.966826] Enabling non-boot CPUs ...
> 
> Signed-off-by: Scott Bauer <scott.bauer@intel.com>

Keith acked the previous version (which contained the enable/disable),
but since this is different, I'll wait for his ack again.

> ---
>  drivers/pci/host/vmd.c | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
> index 7e967a8784b2..4fe1756af010 100644
> --- a/drivers/pci/host/vmd.c
> +++ b/drivers/pci/host/vmd.c
> @@ -763,6 +763,11 @@ static void vmd_remove(struct pci_dev *dev)
>  static int vmd_suspend(struct device *dev)
>  {
>  	struct pci_dev *pdev = to_pci_dev(dev);
> +	struct vmd_dev *vmd = pci_get_drvdata(pdev);
> +	int i;
> +
> +	for (i = 0; i < vmd->msix_count; i++)
> +                devm_free_irq(dev, pci_irq_vector(pdev, i), &vmd->irqs[i]);
>  
>  	pci_save_state(pdev);
>  	return 0;
> @@ -771,6 +776,16 @@ static int vmd_suspend(struct device *dev)
>  static int vmd_resume(struct device *dev)
>  {
>  	struct pci_dev *pdev = to_pci_dev(dev);
> +	struct vmd_dev *vmd = pci_get_drvdata(pdev);
> +	int err, i;
> +
> +	for (i = 0; i < vmd->msix_count; i++) {
> +		err = devm_request_irq(dev, pci_irq_vector(pdev, i),
> +				       vmd_irq, IRQF_NO_THREAD,
> +				       "vmd", &vmd->irqs[i]);
> +		if (err)
> +			return err;
> +	}
>  
>  	pci_restore_state(pdev);
>  	return 0;
> -- 
> 2.11.0
> 

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[Keith Busch]

On Mon, Aug 14, 2017 at 01:06:52PM -0500, Bjorn Helgaas wrote:
> On Fri, Aug 11, 2017 at 02:54:32PM -0600, Scott Bauer wrote:
> > This patch frees up the IRQs we request on the suspend path,
> > and reallocates them on the resume path.
> > 
> > Fixes:
> > [  559.964386] CPU 111 disable failed: CPU has 9 vectors assigned and there are only 0 available.
> > [  559.966824] Error taking CPU111 down: -34
> > [  559.966825] Non-boot CPUs are not disabled
> > [  559.966826] Enabling non-boot CPUs ...
> > 
> > Signed-off-by: Scott Bauer <scott.bauer@intel.com>
> 
> Keith acked the previous version (which contained the enable/disable),
> but since this is different, I'll wait for his ack again.

Thanks for checking. This is actually the patch that was intended for
submission.

Acked-by: Keith Busch <keith.busch@intel.com>
 
> > ---
> >  drivers/pci/host/vmd.c | 15 +++++++++++++++
> >  1 file changed, 15 insertions(+)
> > 
> > diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
> > index 7e967a8784b2..4fe1756af010 100644
> > --- a/drivers/pci/host/vmd.c
> > +++ b/drivers/pci/host/vmd.c
> > @@ -763,6 +763,11 @@ static void vmd_remove(struct pci_dev *dev)
> >  static int vmd_suspend(struct device *dev)
> >  {
> >  	struct pci_dev *pdev = to_pci_dev(dev);
> > +	struct vmd_dev *vmd = pci_get_drvdata(pdev);
> > +	int i;
> > +
> > +	for (i = 0; i < vmd->msix_count; i++)
> > +                devm_free_irq(dev, pci_irq_vector(pdev, i), &vmd->irqs[i]);
> >  
> >  	pci_save_state(pdev);
> >  	return 0;
> > @@ -771,6 +776,16 @@ static int vmd_suspend(struct device *dev)
> >  static int vmd_resume(struct device *dev)
> >  {
> >  	struct pci_dev *pdev = to_pci_dev(dev);
> > +	struct vmd_dev *vmd = pci_get_drvdata(pdev);
> > +	int err, i;
> > +
> > +	for (i = 0; i < vmd->msix_count; i++) {
> > +		err = devm_request_irq(dev, pci_irq_vector(pdev, i),
> > +				       vmd_irq, IRQF_NO_THREAD,
> > +				       "vmd", &vmd->irqs[i]);
> > +		if (err)
> > +			return err;
> > +	}
> >  
> >  	pci_restore_state(pdev);
> >  	return 0;
> > -- 
> > 2.11.0
> > 

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[Bjorn Helgaas]

On Fri, Aug 11, 2017 at 02:54:32PM -0600, Scott Bauer wrote:
> This patch frees up the IRQs we request on the suspend path,
> and reallocates them on the resume path.
> 
> Fixes:
> [  559.964386] CPU 111 disable failed: CPU has 9 vectors assigned and there are only 0 available.
> [  559.966824] Error taking CPU111 down: -34
> [  559.966825] Non-boot CPUs are not disabled
> [  559.966826] Enabling non-boot CPUs ...
> 
> Signed-off-by: Scott Bauer <scott.bauer@intel.com>

Applied with Keith's ack to pci/host-vmd for v4.14, thanks!

> ---
>  drivers/pci/host/vmd.c | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
> index 7e967a8784b2..4fe1756af010 100644
> --- a/drivers/pci/host/vmd.c
> +++ b/drivers/pci/host/vmd.c
> @@ -763,6 +763,11 @@ static void vmd_remove(struct pci_dev *dev)
>  static int vmd_suspend(struct device *dev)
>  {
>  	struct pci_dev *pdev = to_pci_dev(dev);
> +	struct vmd_dev *vmd = pci_get_drvdata(pdev);
> +	int i;
> +
> +	for (i = 0; i < vmd->msix_count; i++)
> +                devm_free_irq(dev, pci_irq_vector(pdev, i), &vmd->irqs[i]);
>  
>  	pci_save_state(pdev);
>  	return 0;
> @@ -771,6 +776,16 @@ static int vmd_suspend(struct device *dev)
>  static int vmd_resume(struct device *dev)
>  {
>  	struct pci_dev *pdev = to_pci_dev(dev);
> +	struct vmd_dev *vmd = pci_get_drvdata(pdev);
> +	int err, i;
> +
> +	for (i = 0; i < vmd->msix_count; i++) {
> +		err = devm_request_irq(dev, pci_irq_vector(pdev, i),
> +				       vmd_irq, IRQF_NO_THREAD,
> +				       "vmd", &vmd->irqs[i]);
> +		if (err)
> +			return err;
> +	}
>  
>  	pci_restore_state(pdev);
>  	return 0;
> -- 
> 2.11.0
> 

[PATCH] PCI: vmd: Free up IRQs on suspend path

[Scott Bauer]

This patch frees up the IRQs we request on the suspend path,
and reallocates them on the resume path.

Fixes:
[  559.964386] CPU 111 disable failed: CPU has 9 vectors assigned and there are only 0 available.
[  559.966824] Error taking CPU111 down: -34
[  559.966825] Non-boot CPUs are not disabled
[  559.966826] Enabling non-boot CPUs ...

Signed-off-by: Scott Bauer <scott.bauer@intel.com>
---
 drivers/pci/host/vmd.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
index 6088c3083194..88bf0c754fb2 100644
--- a/drivers/pci/host/vmd.c
+++ b/drivers/pci/host/vmd.c
@@ -755,6 +755,11 @@ static void vmd_remove(struct pci_dev *dev)
 static int vmd_suspend(struct device *dev)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
+	struct vmd_dev *vmd = pci_get_drvdata(pdev);
+	int i;
+
+	for (i = 0; i < vmd->msix_count; i++)
+                devm_free_irq(dev, pci_irq_vector(pdev, i), &vmd->irqs[i]);
 
 	pci_save_state(pdev);
 	return 0;
@@ -763,7 +768,16 @@ static int vmd_suspend(struct device *dev)
 static int vmd_resume(struct device *dev)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
+	struct vmd_dev *vmd = pci_get_drvdata(pdev);
+	int err, i;
 
+	for (i = 0; i < vmd->msix_count; i++) {
+		err = devm_request_irq(dev, pci_irq_vector(pdev, i),
+				       vmd_irq, IRQF_NO_THREAD,
+				       "vmd", &vmd->irqs[i]);
+		if (err)
+			return err;
+	}
 	pci_restore_state(pdev);
 	return 0;
 }
-- 
2.11.0

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[Jon Derrick]

On 07/20/2017 02:28 PM, Scott Bauer wrote:
> This patch frees up the IRQs we request on the suspend path,
> and reallocates them on the resume path.
> 
> Fixes:
> [  559.964386] CPU 111 disable failed: CPU has 9 vectors assigned and there are only 0 available.
> [  559.966824] Error taking CPU111 down: -34
> [  559.966825] Non-boot CPUs are not disabled
> [  559.966826] Enabling non-boot CPUs ...
> 
> Signed-off-by: Scott Bauer <scott.bauer@intel.com>
> ---
>  drivers/pci/host/vmd.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
> index 6088c3083194..88bf0c754fb2 100644
> --- a/drivers/pci/host/vmd.c
> +++ b/drivers/pci/host/vmd.c
> @@ -755,6 +755,11 @@ static void vmd_remove(struct pci_dev *dev)
>  static int vmd_suspend(struct device *dev)
>  {
>  	struct pci_dev *pdev = to_pci_dev(dev);
> +	struct vmd_dev *vmd = pci_get_drvdata(pdev);
> +	int i;
> +
> +	for (i = 0; i < vmd->msix_count; i++)
> +                devm_free_irq(dev, pci_irq_vector(pdev, i), &vmd->irqs[i]);
>  
>  	pci_save_state(pdev);
>  	return 0;
> @@ -763,7 +768,16 @@ static int vmd_suspend(struct device *dev)
>  static int vmd_resume(struct device *dev)
>  {
>  	struct pci_dev *pdev = to_pci_dev(dev);
> +	struct vmd_dev *vmd = pci_get_drvdata(pdev);
> +	int err, i;
>  
> +	for (i = 0; i < vmd->msix_count; i++) {
> +		err = devm_request_irq(dev, pci_irq_vector(pdev, i),
> +				       vmd_irq, IRQF_NO_THREAD,
The flag here looks redundant to me because the fn calls into
request_threaded_irq with no thread_fn. But since it's a bit more
verbose, can you change the other devm_request_irq to use IRQF_NO_THREAD?

> +				       "vmd", &vmd->irqs[i]);
> +		if (err)
> +			return err;
> +	}
>  	pci_restore_state(pdev);
>  	return 0;
>  }
> 

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[Scott Bauer]

On Tue, Jul 25, 2017 at 08:15:32PM -0600, Jon Derrick wrote:
> On 07/20/2017 02:28 PM, Scott Bauer wrote:
> > This patch frees up the IRQs we request on the suspend path,
> > and reallocates them on the resume path.
> > 
> > Fixes:
> > [  559.964386] CPU 111 disable failed: CPU has 9 vectors assigned and there are only 0 available.
> > [  559.966824] Error taking CPU111 down: -34
> > [  559.966825] Non-boot CPUs are not disabled
> > [  559.966826] Enabling non-boot CPUs ...
> > 
> > Signed-off-by: Scott Bauer <scott.bauer@intel.com>
> > ---
> >  drivers/pci/host/vmd.c | 14 ++++++++++++++
> >  1 file changed, 14 insertions(+)
> > 
> > diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
> > index 6088c3083194..88bf0c754fb2 100644
> > --- a/drivers/pci/host/vmd.c
> > +++ b/drivers/pci/host/vmd.c
> > @@ -755,6 +755,11 @@ static void vmd_remove(struct pci_dev *dev)
> >  static int vmd_suspend(struct device *dev)
> >  {
> >  	struct pci_dev *pdev = to_pci_dev(dev);
> > +	struct vmd_dev *vmd = pci_get_drvdata(pdev);
> > +	int i;
> > +
> > +	for (i = 0; i < vmd->msix_count; i++)
> > +                devm_free_irq(dev, pci_irq_vector(pdev, i), &vmd->irqs[i]);
> >  
> >  	pci_save_state(pdev);
> >  	return 0;
> > @@ -763,7 +768,16 @@ static int vmd_suspend(struct device *dev)
> >  static int vmd_resume(struct device *dev)
> >  {
> >  	struct pci_dev *pdev = to_pci_dev(dev);
> > +	struct vmd_dev *vmd = pci_get_drvdata(pdev);
> > +	int err, i;
> >  
> > +	for (i = 0; i < vmd->msix_count; i++) {
> > +		err = devm_request_irq(dev, pci_irq_vector(pdev, i),
> > +				       vmd_irq, IRQF_NO_THREAD,
> The flag here looks redundant to me because the fn calls into
> request_threaded_irq with no thread_fn. But since it's a bit more
> verbose, can you change the other devm_request_irq to use IRQF_NO_THREAD?

I actually yanked the request irq from the other call. It was changed
to NO_THREAD in April:

(3eefa790c968) PCI: host: Mark PCIe/PCI (MSI) cascade ISR as IRQF_NO_THREAD

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[Jon Derrick]

On 07/20/2017 03:34 PM, Scott Bauer wrote:
[snip]
>>> +	for (i = 0; i < vmd->msix_count; i++) {
>>> +		err = devm_request_irq(dev, pci_irq_vector(pdev, i),
>>> +				       vmd_irq, IRQF_NO_THREAD,
>> The flag here looks redundant to me because the fn calls into
>> request_threaded_irq with no thread_fn. But since it's a bit more
>> verbose, can you change the other devm_request_irq to use IRQF_NO_THREAD?
> 
> I actually yanked the request irq from the other call. It was changed
> to NO_THREAD in April:
> 
> (3eefa790c968) PCI: host: Mark PCIe/PCI (MSI) cascade ISR as IRQF_NO_THREAD
> 

Ah so it was. Sorry was looking at v4.12


I think we'll need to do more than free the irq handlers because if an
interrupt occurs after that, I think it'll get kicked to handle_bad_irq.
I think we just need to add pci_disable_device after save and
pcim_enable_device after restore.

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[Scott Bauer]

On Tue, Jul 25, 2017 at 08:36:10PM -0600, Jon Derrick wrote:
> On 07/20/2017 03:34 PM, Scott Bauer wrote:
> [snip]
> >>> +	for (i = 0; i < vmd->msix_count; i++) {
> >>> +		err = devm_request_irq(dev, pci_irq_vector(pdev, i),
> >>> +				       vmd_irq, IRQF_NO_THREAD,
> >> The flag here looks redundant to me because the fn calls into
> >> request_threaded_irq with no thread_fn. But since it's a bit more
> >> verbose, can you change the other devm_request_irq to use IRQF_NO_THREAD?
> > 
> > I actually yanked the request irq from the other call. It was changed
> > to NO_THREAD in April:
> > 
> > (3eefa790c968) PCI: host: Mark PCIe/PCI (MSI) cascade ISR as IRQF_NO_THREAD
> > 
> 
> Ah so it was. Sorry was looking at v4.12
> 
> 
> I think we'll need to do more than free the irq handlers because if an
> interrupt occurs after that, I think it'll get kicked to handle_bad_irq.
> I think we just need to add pci_disable_device after save and
> pcim_enable_device after restore.

I orignally had this in the patch I was testing on SLES and took it out when
it seemingly didn't make a difference. I didnt test yanking a drive or anything
behind the domain while it was suspended, so I wouldn't have hit your scenario.

I'll add it back in. Also, I think I need to CC stable on this one too since
this is missing all the way since the original inception of this driver.

Re: [PATCH] PCI: vmd: Free up IRQs on suspend path

[Keith Busch]

On Thu, Jul 20, 2017 at 03:56:54PM -0600, Scott Bauer wrote:
> On Tue, Jul 25, 2017 at 08:36:10PM -0600, Jon Derrick wrote:
> > 
> > I think we'll need to do more than free the irq handlers because if an
> > interrupt occurs after that, I think it'll get kicked to handle_bad_irq.
> > I think we just need to add pci_disable_device after save and
> > pcim_enable_device after restore.
> 
> I orignally had this in the patch I was testing on SLES and took it out when
> it seemingly didn't make a difference. I didnt test yanking a drive or anything
> behind the domain while it was suspended, so I wouldn't have hit your scenario.
> 
> I'll add it back in. Also, I think I need to CC stable on this one too since
> this is missing all the way since the original inception of this driver.

I think your patch should be okay as-is. After freeing VMD's IRQs,
the kernel will automatically clear the MSI-x control bit to disable it
from firing.