The imx6q suspend/resume is broken on 3.16-rc due to PCIe

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

Hi Lucas,

I found that imx6q suspend/resume is broken on 3.16-rc, and it works
just fine if PCIe support is built out.  However the issue was not
exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
is part of the cause.

$ echo mem > /sys/power/state
PM: Syncing filesystems ... done.
Freezing user space processes ... (elapsed 0.002 seconds) done.
Freezing remaining freezable tasks ... (elapsed 0.002 seconds) done.
sd 0:0:0:0: [sda] Synchronizing SCSI cache
sd 0:0:0:0: [sda] Stopping disk
PM: suspend of devices complete after 85.695 msecs
PM: suspend devices took 0.100 seconds
PM: late suspend of devices complete after 6.786 msecs
PM: noirq suspend of devices complete after 6.228 msecs
Disabling non-boot CPUs ...
CPU1: shutdown
CPU2: shutdown
CPU3: shutdown
Enabling non-boot CPUs ...
CPU1: Booted secondary processor
CPU1 is up
CPU2: Booted secondary processor
CPU2 is up
CPU3: Booted secondary processor
CPU3 is up

The issue is actually with resume procedure.  I noticed that Richard
added the following change in FSL kernel tree, which seems to fix the
issue.

@@ -229,6 +232,22 @@ static int imx6_suspend_finish(unsigned long val)

 static int imx6_pm_enter(suspend_state_t state)
 {
+       struct regmap *gpr;
+
+       /*
+        * L2 can exit by 'reset' or Inband beacon (from remote EP)
+        * toggling phy_powerdown has same effect as 'inband beacon'
+        * So, toggle bit18 of GPR1, to fix errata
+        * "PCIe PCIe does not support L2 Power Down"
+        */
+       gpr = syscon_regmap_lookup_by_compatible("fsl,imx6q-iomuxc-gpr");
+       if (IS_ERR(gpr)) {
+               pr_err("failed to find fsl,imx6q-iomux-gpr regmap\n");
+               return PTR_ERR(gpr);
+       }
+       regmap_update_bits(gpr, IOMUXC_GPR1, IMX6Q_GPR1_PCIE_TEST_PD,
+                       IMX6Q_GPR1_PCIE_TEST_PD);
+
        switch (state) {
        case PM_SUSPEND_STANDBY:
                imx6_set_lpm(STOP_POWER_ON);
@@ -258,6 +277,15 @@ static int imx6_pm_enter(suspend_state_t state)
                return -EINVAL;
        }

+       /*
+        * L2 can exit by 'reset' or Inband beacon (from remote EP)
+        * toggling phy_powerdown has same effect as 'inband beacon'
+        * So, toggle bit18 of GPR1, to fix errata
+        * "PCIe PCIe does not support L2 Power Down"
+        */
+       regmap_update_bits(gpr, IOMUXC_GPR1, IMX6Q_GPR1_PCIE_TEST_PD,
+                       !IMX6Q_GPR1_PCIE_TEST_PD);
+
        return 0;
 }
 
And I copied the details of the errata as below.

ERR005723 PCIe: PCIe does not support L2 power down

Description:
When PCIe works as Root Complex, it can exit L2 mode only through reset.
Since PCIe does not have a dedicated reset control bit, it cannot exit
L2 mode.

Projected Impact:
PCIe does not support L2 power down.

Workarounds:
The PCIe can be put into PDDQ mode to save PCIe PHY power and wake up
only by the OOB (Out of Band) wakeup signal (since wakeup by a beacon
from link partner is not supported) driven from the link partner (End
Point). This signal could be used as a GPIO interrupt to exit this mode.
The limitation of this workaround is that the link partner cannot be put
into L2 mode.

As I have no knowledge of how PCIe works, I provided all the info I
found here to kick the discussion.  Hopefully, we can find a proper fix
to get suspend/resume back to work.

Shawn

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Hi Shawn,

Am Dienstag, den 24.06.2014, 23:36 +0800 schrieb Shawn Guo:
> Hi Lucas,
> 
> I found that imx6q suspend/resume is broken on 3.16-rc, and it works
> just fine if PCIe support is built out.  However the issue was not
> exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
> is part of the cause.
> 
I don't yet see why any of the changes to the designware/imx6 host
driver would cause such failures.

Which board are you doing your testing on? What devices are connected to
the PCIe bus? In what way does it break system resume, complete hang?

Regards,
Lucas

> $ echo mem > /sys/power/state
> PM: Syncing filesystems ... done.
> Freezing user space processes ... (elapsed 0.002 seconds) done.
> Freezing remaining freezable tasks ... (elapsed 0.002 seconds) done.
> sd 0:0:0:0: [sda] Synchronizing SCSI cache
> sd 0:0:0:0: [sda] Stopping disk
> PM: suspend of devices complete after 85.695 msecs
> PM: suspend devices took 0.100 seconds
> PM: late suspend of devices complete after 6.786 msecs
> PM: noirq suspend of devices complete after 6.228 msecs
> Disabling non-boot CPUs ...
> CPU1: shutdown
> CPU2: shutdown
> CPU3: shutdown
> Enabling non-boot CPUs ...
> CPU1: Booted secondary processor
> CPU1 is up
> CPU2: Booted secondary processor
> CPU2 is up
> CPU3: Booted secondary processor
> CPU3 is up
> 
> The issue is actually with resume procedure.  I noticed that Richard
> added the following change in FSL kernel tree, which seems to fix the
> issue.
> 
> @@ -229,6 +232,22 @@ static int imx6_suspend_finish(unsigned long val)
> 
>  static int imx6_pm_enter(suspend_state_t state)
>  {
> +       struct regmap *gpr;
> +
> +       /*
> +        * L2 can exit by 'reset' or Inband beacon (from remote EP)
> +        * toggling phy_powerdown has same effect as 'inband beacon'
> +        * So, toggle bit18 of GPR1, to fix errata
> +        * "PCIe PCIe does not support L2 Power Down"
> +        */
> +       gpr = syscon_regmap_lookup_by_compatible("fsl,imx6q-iomuxc-gpr");
> +       if (IS_ERR(gpr)) {
> +               pr_err("failed to find fsl,imx6q-iomux-gpr regmap\n");
> +               return PTR_ERR(gpr);
> +       }
> +       regmap_update_bits(gpr, IOMUXC_GPR1, IMX6Q_GPR1_PCIE_TEST_PD,
> +                       IMX6Q_GPR1_PCIE_TEST_PD);
> +
>         switch (state) {
>         case PM_SUSPEND_STANDBY:
>                 imx6_set_lpm(STOP_POWER_ON);
> @@ -258,6 +277,15 @@ static int imx6_pm_enter(suspend_state_t state)
>                 return -EINVAL;
>         }
> 
> +       /*
> +        * L2 can exit by 'reset' or Inband beacon (from remote EP)
> +        * toggling phy_powerdown has same effect as 'inband beacon'
> +        * So, toggle bit18 of GPR1, to fix errata
> +        * "PCIe PCIe does not support L2 Power Down"
> +        */
> +       regmap_update_bits(gpr, IOMUXC_GPR1, IMX6Q_GPR1_PCIE_TEST_PD,
> +                       !IMX6Q_GPR1_PCIE_TEST_PD);
> +
>         return 0;
>  }
>  
> And I copied the details of the errata as below.
> 
> ERR005723 PCIe: PCIe does not support L2 power down
> 
> Description:
> When PCIe works as Root Complex, it can exit L2 mode only through reset.
> Since PCIe does not have a dedicated reset control bit, it cannot exit
> L2 mode.
> 
> Projected Impact:
> PCIe does not support L2 power down.
> 
> Workarounds:
> The PCIe can be put into PDDQ mode to save PCIe PHY power and wake up
> only by the OOB (Out of Band) wakeup signal (since wakeup by a beacon
> from link partner is not supported) driven from the link partner (End
> Point). This signal could be used as a GPIO interrupt to exit this mode.
> The limitation of this workaround is that the link partner cannot be put
> into L2 mode.
> 
> As I have no knowledge of how PCIe works, I provided all the info I
> found here to kick the discussion.  Hopefully, we can find a proper fix
> to get suspend/resume back to work.
> 
> Shawn
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Tue, Jun 24, 2014 at 06:13:55PM +0200, Lucas Stach wrote:
> Hi Shawn,
> 
> Am Dienstag, den 24.06.2014, 23:36 +0800 schrieb Shawn Guo:
> > Hi Lucas,
> > 
> > I found that imx6q suspend/resume is broken on 3.16-rc, and it works
> > just fine if PCIe support is built out.  However the issue was not
> > exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
> > is part of the cause.
> > 
> I don't yet see why any of the changes to the designware/imx6 host
> driver would cause such failures.
> 
> Which board are you doing your testing on? What devices are connected to
> the PCIe bus? In what way does it break system resume, complete hang?

It's on imx6q-sabresd without any PCIe device connected.  Right, it's a
complete hang after enabling non-boot CPUs.

...
Enabling non-boot CPUs ...
CPU1: Booted secondary processor
CPU1 is up
CPU2: Booted secondary processor
CPU2 is up
CPU3: Booted secondary processor
CPU3 is up
<*** hang here ***>

Shawn

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Fabio Estevam]

On Tue, Jun 24, 2014 at 12:36 PM, Shawn Guo <shawn.guo@linaro.org> wrote:
> Hi Lucas,
>
> I found that imx6q suspend/resume is broken on 3.16-rc, and it works
> just fine if PCIe support is built out.  However the issue was not
> exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
> is part of the cause.

I have just tested this on linux-next and I see that suspend/resume
fails even if I build a kernel without PCI support.

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Tue, Jun 24, 2014 at 04:37:19PM -0300, Fabio Estevam wrote:
> On Tue, Jun 24, 2014 at 12:36 PM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > Hi Lucas,
> >
> > I found that imx6q suspend/resume is broken on 3.16-rc, and it works
> > just fine if PCIe support is built out.  However the issue was not
> > exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
> > is part of the cause.
> 
> I have just tested this on linux-next and I see that suspend/resume
> fails even if I build a kernel without PCI support.

Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
should be another issue.

Shawn

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Fabio Estevam]

On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:

> Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> should be another issue.

Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
driver removed.

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Am Mittwoch, den 25.06.2014, 07:50 -0300 schrieb Fabio Estevam:
> On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:
> 
> > Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> > should be another issue.
> 
> Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
> driver removed.

Can you please send me a dmesg from a working boot?

Regards,
Lucas
-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Wed, Jun 25, 2014 at 12:50:52PM +0200, Lucas Stach wrote:
> Am Mittwoch, den 25.06.2014, 07:50 -0300 schrieb Fabio Estevam:
> > On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > 
> > > Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> > > should be another issue.
> > 
> > Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
> > driver removed.
> 
> Can you please send me a dmesg from a working boot?

What do you mean by "working boot"?  Boot is always working.  What fails
is system resuming.

Shawn

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Am Mittwoch, den 25.06.2014, 20:53 +0800 schrieb Shawn Guo:
> On Wed, Jun 25, 2014 at 12:50:52PM +0200, Lucas Stach wrote:
> > Am Mittwoch, den 25.06.2014, 07:50 -0300 schrieb Fabio Estevam:
> > > On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > > 
> > > > Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> > > > should be another issue.
> > > 
> > > Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
> > > driver removed.
> > 
> > Can you please send me a dmesg from a working boot?
> 
> What do you mean by "working boot"?  Boot is always working.  What fails
> is system resuming.
> 
> Shawn
> 
Oops, what I meant is a dmesg after a working resume (from kernel 3.15).
I can see how failing to resume from L2 can cause the hang and we should
really implement the PHY PD toggling in the pcie driver, but I would
like to understand why you don't hit this on 3.15.

Regards,
Lucas

-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Thu, Jun 26, 2014 at 10:43:49AM +0200, Lucas Stach wrote:
> Am Mittwoch, den 25.06.2014, 20:53 +0800 schrieb Shawn Guo:
> > On Wed, Jun 25, 2014 at 12:50:52PM +0200, Lucas Stach wrote:
> > > Am Mittwoch, den 25.06.2014, 07:50 -0300 schrieb Fabio Estevam:
> > > > On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > > > 
> > > > > Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> > > > > should be another issue.
> > > > 
> > > > Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
> > > > driver removed.
> > > 
> > > Can you please send me a dmesg from a working boot?
> > 
> > What do you mean by "working boot"?  Boot is always working.  What fails
> > is system resuming.
> > 
> > Shawn
> > 
> Oops, what I meant is a dmesg after a working resume (from kernel 3.15).
> I can see how failing to resume from L2 can cause the hang and we should
> really implement the PHY PD toggling in the pcie driver, but I would
> like to understand why you don't hit this on 3.15.

Sorry, Lucas.  I just noticed that in my 3.15 testing PCIe is not
enabled in device tree at all.  So the issue should have already been
there before your changes to PCIe driver.

Shawn

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Am Samstag, den 28.06.2014, 22:37 +0800 schrieb Shawn Guo:
> On Thu, Jun 26, 2014 at 10:43:49AM +0200, Lucas Stach wrote:
> > Am Mittwoch, den 25.06.2014, 20:53 +0800 schrieb Shawn Guo:
> > > On Wed, Jun 25, 2014 at 12:50:52PM +0200, Lucas Stach wrote:
> > > > Am Mittwoch, den 25.06.2014, 07:50 -0300 schrieb Fabio Estevam:
> > > > > On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > > > > 
> > > > > > Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> > > > > > should be another issue.
> > > > > 
> > > > > Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
> > > > > driver removed.
> > > > 
> > > > Can you please send me a dmesg from a working boot?
> > > 
> > > What do you mean by "working boot"?  Boot is always working.  What fails
> > > is system resuming.
> > > 
> > > Shawn
> > > 
> > Oops, what I meant is a dmesg after a working resume (from kernel 3.15).
> > I can see how failing to resume from L2 can cause the hang and we should
> > really implement the PHY PD toggling in the pcie driver, but I would
> > like to understand why you don't hit this on 3.15.
> 
> Sorry, Lucas.  I just noticed that in my 3.15 testing PCIe is not
> enabled in device tree at all.  So the issue should have already been
> there before your changes to PCIe driver.
> 
Hi Shawn,

can you please test the attached patch on top of 3.16-rc* to see if it
helps? If it works for you I would like to try and get this into 3.16 as
a bugfix.

Regards,
Lucas

--------------------------->8-----------------------------
>From f3bfdc9c19249188fa153b1568c3129cd31af4f5 Mon Sep 17 00:00:00 2001
From: Lucas Stach <l.stach@pengutronix.de>
Date: Mon, 30 Jun 2014 15:33:04 +0200
Subject: [PATCH] PCI: imx6: add workaround for errata ERR005723

Fixes system hang after resume from suspend to mem.

Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
---
 drivers/pci/host/pci-imx6.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
index a568efaa331c..5c58a3ee5dbf 100644
--- a/drivers/pci/host/pci-imx6.c
+++ b/drivers/pci/host/pci-imx6.c
@@ -589,6 +589,38 @@ static int __init imx6_pcie_probe(struct platform_device *pdev)
 	return 0;
 }
 
+#ifdef CONFIG_PM_SLEEP
+static int imx6_pcie_suspend(struct device *dev)
+{
+	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
+
+	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
+			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
+
+	return 0;
+}
+
+static int imx6_pcie_resume(struct device *dev)
+{
+	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
+
+	/*
+	 * This is a workaround for
+	 * ERR005723: PCIe does not support L2 power down
+	 * Toggling the PHY PD bit around system L2 state switching seems to be
+	 * enough to wake the PCIe logic after a power down.
+	 */
+	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
+			IMX6Q_GPR1_PCIE_TEST_PD, 0 << 18);
+
+	return 0;
+}
+#endif
+
+static SIMPLE_DEV_PM_OPS(imx6_pcie_pm_ops,
+		imx6_pcie_suspend,
+		imx6_pcie_resume);
+
 static const struct of_device_id imx6_pcie_of_match[] = {
 	{ .compatible = "fsl,imx6q-pcie", },
 	{},
@@ -600,6 +632,7 @@ static struct platform_driver imx6_pcie_driver = {
 		.name	= "imx6q-pcie",
 		.owner	= THIS_MODULE,
 		.of_match_table = imx6_pcie_of_match,
+		.pm = &imx6_pcie_pm_ops,
 	},
 };
 
-- 
2.0.0


-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Mon, Jun 30, 2014 at 03:44:24PM +0200, Lucas Stach wrote:
> Hi Shawn,
> 
> can you please test the attached patch on top of 3.16-rc* to see if it
> helps? If it works for you I would like to try and get this into 3.16 as
> a bugfix.

Unfortunately, it doesn't work.  With your change, now it hangs in
suspend procedure.

$ echo mem > /sys/power/state
PM: Syncing filesystems ... done.
PM: Preparing system for mem sleep
Freezing user space processes ... (elapsed 0.005 seconds) done.
Freezing remaining freezable tasks ... (elapsed 0.003 seconds) done.
PM: Entering mem sleep
sd 0:0:0:0: [sda] Synchronizing SCSI cache
sd 0:0:0:0: [sda] Stopping disk
PM: suspend of devices complete after 102.992 msecs
PM: suspend devices took 0.110 seconds
PM: late suspend of devices complete after 12.808 msecs

Shawn

> 
> Regards,
> Lucas
> 
> --------------------------->8-----------------------------
> From f3bfdc9c19249188fa153b1568c3129cd31af4f5 Mon Sep 17 00:00:00 2001
> From: Lucas Stach <l.stach@pengutronix.de>
> Date: Mon, 30 Jun 2014 15:33:04 +0200
> Subject: [PATCH] PCI: imx6: add workaround for errata ERR005723
> 
> Fixes system hang after resume from suspend to mem.
> 
> Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
> ---
>  drivers/pci/host/pci-imx6.c | 33 +++++++++++++++++++++++++++++++++
>  1 file changed, 33 insertions(+)
> 
> diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
> index a568efaa331c..5c58a3ee5dbf 100644
> --- a/drivers/pci/host/pci-imx6.c
> +++ b/drivers/pci/host/pci-imx6.c
> @@ -589,6 +589,38 @@ static int __init imx6_pcie_probe(struct platform_device *pdev)
>  	return 0;
>  }
>  
> +#ifdef CONFIG_PM_SLEEP
> +static int imx6_pcie_suspend(struct device *dev)
> +{
> +	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
> +
> +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
> +			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
> +
> +	return 0;
> +}
> +
> +static int imx6_pcie_resume(struct device *dev)
> +{
> +	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
> +
> +	/*
> +	 * This is a workaround for
> +	 * ERR005723: PCIe does not support L2 power down
> +	 * Toggling the PHY PD bit around system L2 state switching seems to be
> +	 * enough to wake the PCIe logic after a power down.
> +	 */
> +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
> +			IMX6Q_GPR1_PCIE_TEST_PD, 0 << 18);
> +
> +	return 0;
> +}
> +#endif
> +
> +static SIMPLE_DEV_PM_OPS(imx6_pcie_pm_ops,
> +		imx6_pcie_suspend,
> +		imx6_pcie_resume);
> +
>  static const struct of_device_id imx6_pcie_of_match[] = {
>  	{ .compatible = "fsl,imx6q-pcie", },
>  	{},
> @@ -600,6 +632,7 @@ static struct platform_driver imx6_pcie_driver = {
>  		.name	= "imx6q-pcie",
>  		.owner	= THIS_MODULE,
>  		.of_match_table = imx6_pcie_of_match,
> +		.pm = &imx6_pcie_pm_ops,
>  	},
>  };
>  
> -- 
> 2.0.0
> 
> 
> -- 
> Pengutronix e.K.             | Lucas Stach                 |
> Industrial Linux Solutions   | http://www.pengutronix.de/  |
> 

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Hi Shawn,

Over the weekend I tried to reproduce your problem on a SabreSD board,
but wasn't able to trigger the issue. 3.16-rc3 with PCIe active works
just fine over a suspend and resume cycle for me.

One possibly relevant difference is that I've booted with NFSroot, while
it seems you are using a SATA connected device. Is this right? If so,
can you test if it works if you boot from SDcard or the like? This might
be relevant as PCIe and SATA share some clocks.

Regards,
Lucas

Am Dienstag, den 01.07.2014, 14:51 +0800 schrieb Shawn Guo:
> On Mon, Jun 30, 2014 at 03:44:24PM +0200, Lucas Stach wrote:
> > Hi Shawn,
> > 
> > can you please test the attached patch on top of 3.16-rc* to see if it
> > helps? If it works for you I would like to try and get this into 3.16 as
> > a bugfix.
> 
> Unfortunately, it doesn't work.  With your change, now it hangs in
> suspend procedure.
> 
> $ echo mem > /sys/power/state
> PM: Syncing filesystems ... done.
> PM: Preparing system for mem sleep
> Freezing user space processes ... (elapsed 0.005 seconds) done.
> Freezing remaining freezable tasks ... (elapsed 0.003 seconds) done.
> PM: Entering mem sleep
> sd 0:0:0:0: [sda] Synchronizing SCSI cache
> sd 0:0:0:0: [sda] Stopping disk
> PM: suspend of devices complete after 102.992 msecs
> PM: suspend devices took 0.110 seconds
> PM: late suspend of devices complete after 12.808 msecs
> 
> Shawn
> 
> > 
> > Regards,
> > Lucas
> > 
> > --------------------------->8-----------------------------
> > From f3bfdc9c19249188fa153b1568c3129cd31af4f5 Mon Sep 17 00:00:00 2001
> > From: Lucas Stach <l.stach@pengutronix.de>
> > Date: Mon, 30 Jun 2014 15:33:04 +0200
> > Subject: [PATCH] PCI: imx6: add workaround for errata ERR005723
> > 
> > Fixes system hang after resume from suspend to mem.
> > 
> > Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
> > ---
> >  drivers/pci/host/pci-imx6.c | 33 +++++++++++++++++++++++++++++++++
> >  1 file changed, 33 insertions(+)
> > 
> > diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
> > index a568efaa331c..5c58a3ee5dbf 100644
> > --- a/drivers/pci/host/pci-imx6.c
> > +++ b/drivers/pci/host/pci-imx6.c
> > @@ -589,6 +589,38 @@ static int __init imx6_pcie_probe(struct platform_device *pdev)
> >  	return 0;
> >  }
> >  
> > +#ifdef CONFIG_PM_SLEEP
> > +static int imx6_pcie_suspend(struct device *dev)
> > +{
> > +	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
> > +
> > +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
> > +			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
> > +
> > +	return 0;
> > +}
> > +
> > +static int imx6_pcie_resume(struct device *dev)
> > +{
> > +	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
> > +
> > +	/*
> > +	 * This is a workaround for
> > +	 * ERR005723: PCIe does not support L2 power down
> > +	 * Toggling the PHY PD bit around system L2 state switching seems to be
> > +	 * enough to wake the PCIe logic after a power down.
> > +	 */
> > +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
> > +			IMX6Q_GPR1_PCIE_TEST_PD, 0 << 18);
> > +
> > +	return 0;
> > +}
> > +#endif
> > +
> > +static SIMPLE_DEV_PM_OPS(imx6_pcie_pm_ops,
> > +		imx6_pcie_suspend,
> > +		imx6_pcie_resume);
> > +
> >  static const struct of_device_id imx6_pcie_of_match[] = {
> >  	{ .compatible = "fsl,imx6q-pcie", },
> >  	{},
> > @@ -600,6 +632,7 @@ static struct platform_driver imx6_pcie_driver = {
> >  		.name	= "imx6q-pcie",
> >  		.owner	= THIS_MODULE,
> >  		.of_match_table = imx6_pcie_of_match,
> > +		.pm = &imx6_pcie_pm_ops,
> >  	},
> >  };
> >  
> > -- 
> > 2.0.0
> > 
> > 
> > -- 
> > Pengutronix e.K.             | Lucas Stach                 |
> > Industrial Linux Solutions   | http://www.pengutronix.de/  |
> > 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Mon, Jul 07, 2014 at 11:10:51AM +0200, Lucas Stach wrote:
> Hi Shawn,
> 
> Over the weekend I tried to reproduce your problem on a SabreSD board,
> but wasn't able to trigger the issue. 3.16-rc3 with PCIe active works
> just fine over a suspend and resume cycle for me.

That's strange.  In my setup, PCIe support is enabled in kernel and DT,
but I do not have a PCIe device connected to the board.

> 
> One possibly relevant difference is that I've booted with NFSroot, while
> it seems you are using a SATA connected device. Is this right?

I have a SATA disk connected, but did boot with NFSroot.

> If so,
> can you test if it works if you boot from SDcard or the like? This might
> be relevant as PCIe and SATA share some clocks.

I tried to disable SATA support completely, but it doesn't help.

$ echo mem > /sys/power/state
[  410.052595] PM: Syncing filesystems ... done.
[  410.150033] PM: Preparing system for mem sleep
[  410.207963] Freezing user space processes ... (elapsed 0.004 seconds) done.
[  410.219796] Freezing remaining freezable tasks ... (elapsed 0.002 seconds) done.
[  410.230243] PM: Entering mem sleep
[  410.316574] PM: suspend of devices complete after 79.461 msecs
[  410.322498] PM: suspend devices took 0.090 seconds
[  410.332655] PM: late suspend of devices complete after 5.338 msecs


Shawn

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Mon, Jul 07, 2014 at 09:55:03PM +0800, Shawn Guo wrote:
> On Mon, Jul 07, 2014 at 11:10:51AM +0200, Lucas Stach wrote:
> > Hi Shawn,
> > 
> > Over the weekend I tried to reproduce your problem on a SabreSD board,
> > but wasn't able to trigger the issue. 3.16-rc3 with PCIe active works
> > just fine over a suspend and resume cycle for me.
> 
> That's strange.  In my setup, PCIe support is enabled in kernel and DT,
> but I do not have a PCIe device connected to the board.
> 
> > 
> > One possibly relevant difference is that I've booted with NFSroot, while
> > it seems you are using a SATA connected device. Is this right?
> 
> I have a SATA disk connected, but did boot with NFSroot.
> 
> > If so,
> > can you test if it works if you boot from SDcard or the like? This might
> > be relevant as PCIe and SATA share some clocks.
> 
> I tried to disable SATA support completely, but it doesn't help.

Lucas, any news on this?  Or should we just try to use Richard's patch
to solve the problem?

Shawn

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Hi Shawn,

Am Mittwoch, den 16.07.2014, 14:55 +0800 schrieb Shawn Guo:
> On Mon, Jul 07, 2014 at 09:55:03PM +0800, Shawn Guo wrote:
> > On Mon, Jul 07, 2014 at 11:10:51AM +0200, Lucas Stach wrote:
> > > Hi Shawn,
> > > 
> > > Over the weekend I tried to reproduce your problem on a SabreSD board,
> > > but wasn't able to trigger the issue. 3.16-rc3 with PCIe active works
> > > just fine over a suspend and resume cycle for me.
> > 
> > That's strange.  In my setup, PCIe support is enabled in kernel and DT,
> > but I do not have a PCIe device connected to the board.
> > 
> > > 
> > > One possibly relevant difference is that I've booted with NFSroot, while
> > > it seems you are using a SATA connected device. Is this right?
> > 
> > I have a SATA disk connected, but did boot with NFSroot.
> > 
> > > If so,
> > > can you test if it works if you boot from SDcard or the like? This might
> > > be relevant as PCIe and SATA share some clocks.
> > 
> > I tried to disable SATA support completely, but it doesn't help.
> 
> Lucas, any news on this?  Or should we just try to use Richard's patch
> to solve the problem?

I would like to understand the problem first before throwing "fixes" at
the issue. As you said this isn't a regression we are in no hurry and
should try to analyze the issue properly. I just retested and I'm still
not able to reproduce the issue on my SabreSD. Maybe there are board
revision that exhibit different behavior? My board has two sticks on
saying "Rev B3" and "Rev X3".

As you can see from the log suspend/resume is working fine for me with
kernel 3.16-rc5 + imx_v6_v7_defconfig.

root@XXX:~ echo mem > /sys/power/state
PM: Syncing filesystems ... done.
Freezing user space processes ... (elapsed 0.004 seconds) done.
Freezing remaining freezable tasks ... (elapsed 0.002 seconds) done.
Suspending console(s) (use no_console_suspend to debug)
PM: suspend of devices complete after 95.846 msecs
PM: suspend devices took 0.100 seconds
PM: late suspend of devices complete after 5.395 msecs
PM: noirq suspend of devices complete after 4.914 msecs
Disabling non-boot CPUs ...
CPU1: shutdown
CPU2: shutdown
CPU3: shutdown
Enabling non-boot CPUs ...
CPU1: Booted secondary processor
CPU1 is up
CPU2: Booted secondary processor
CPU2 is up
CPU3: Booted secondary processor
CPU3 is up
PM: noirq resume of devices complete after 1556.385 msecs
PM: early resume of devices complete after 3.134 msecs
PM: resume of devices complete after 153.568 msecs
PM: resume devices took 0.160 seconds
Restarting tasks ... done.
ata1: SATA link down (SStatus 0 SControl 300)
fec 2188000.ethernet eth0: Link is Down
fec 2188000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
root@XXX:~ echo mem > /sys/power/state
PM: Syncing filesystems ... done.
Freezing user space processes ... (elapsed 0.002 seconds) done.
Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
Suspending console(s) (use no_console_suspend to debug)
PM: suspend of devices complete after 87.638 msecs
PM: suspend devices took 0.090 seconds
PM: late suspend of devices complete after 4.528 msecs
PM: noirq suspend of devices complete after 5.031 msecs
Disabling non-boot CPUs ...
CPU1: shutdown
CPU2: shutdown
CPU3: shutdown
Enabling non-boot CPUs ...
CPU1: Booted secondary processor
CPU1 is up
CPU2: Booted secondary processor
CPU2 is up
CPU3: Booted secondary processor
CPU3 is up
PM: noirq resume of devices complete after 1425.302 msecs
PM: early resume of devices complete after 2.736 msecs
PM: resume of devices complete after 156.013 msecs
PM: resume devices took 0.160 seconds
Restarting tasks ... done.
ata1: SATA link down (SStatus 0 SControl 300)
fec 2188000.ethernet eth0: Link is Down
fec 2188000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
root@XXX:~

-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

RE: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Hong-Xing.Zhu]

PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBMdWNhcyBTdGFjaCBbbWFpbHRv
Omwuc3RhY2hAcGVuZ3V0cm9uaXguZGVdDQo+IFNlbnQ6IFRodXJzZGF5LCBKdWx5IDE3LCAyMDE0
IDk6NTUgUE0NCj4gVG86IEd1byBTaGF3bi1SNjUwNzMNCj4gQ2M6IFpodSBSaWNoYXJkLVI2NTAz
NzsgbGludXgtcGNpQHZnZXIua2VybmVsLm9yZzsgU2FzY2hhIEhhdWVyOyBCam9ybiBIZWxnYWFz
Ow0KPiBTaGF3biBHdW87IEZhYmlvIEVzdGV2YW07IGxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5m
cmFkZWFkLm9yZw0KPiBTdWJqZWN0OiBSZTogVGhlIGlteDZxIHN1c3BlbmQvcmVzdW1lIGlzIGJy
b2tlbiBvbiAzLjE2LXJjIGR1ZSB0byBQQ0llDQo+IA0KPiBIaSBTaGF3biwNCj4gDQo+IEFtIE1p
dHR3b2NoLCBkZW4gMTYuMDcuMjAxNCwgMTQ6NTUgKzA4MDAgc2NocmllYiBTaGF3biBHdW86DQo+
ID4gT24gTW9uLCBKdWwgMDcsIDIwMTQgYXQgMDk6NTU6MDNQTSArMDgwMCwgU2hhd24gR3VvIHdy
b3RlOg0KPiA+ID4gT24gTW9uLCBKdWwgMDcsIDIwMTQgYXQgMTE6MTA6NTFBTSArMDIwMCwgTHVj
YXMgU3RhY2ggd3JvdGU6DQo+ID4gPiA+IEhpIFNoYXduLA0KPiA+ID4gPg0KPiA+ID4gPiBPdmVy
IHRoZSB3ZWVrZW5kIEkgdHJpZWQgdG8gcmVwcm9kdWNlIHlvdXIgcHJvYmxlbSBvbiBhIFNhYnJl
U0QNCj4gPiA+ID4gYm9hcmQsIGJ1dCB3YXNuJ3QgYWJsZSB0byB0cmlnZ2VyIHRoZSBpc3N1ZS4g
My4xNi1yYzMgd2l0aCBQQ0llDQo+ID4gPiA+IGFjdGl2ZSB3b3JrcyBqdXN0IGZpbmUgb3ZlciBh
IHN1c3BlbmQgYW5kIHJlc3VtZSBjeWNsZSBmb3IgbWUuDQo+ID4gPg0KPiA+ID4gVGhhdCdzIHN0
cmFuZ2UuICBJbiBteSBzZXR1cCwgUENJZSBzdXBwb3J0IGlzIGVuYWJsZWQgaW4ga2VybmVsIGFu
ZA0KPiA+ID4gRFQsIGJ1dCBJIGRvIG5vdCBoYXZlIGEgUENJZSBkZXZpY2UgY29ubmVjdGVkIHRv
IHRoZSBib2FyZC4NCj4gPiA+DQo+ID4gPiA+DQo+ID4gPiA+IE9uZSBwb3NzaWJseSByZWxldmFu
dCBkaWZmZXJlbmNlIGlzIHRoYXQgSSd2ZSBib290ZWQgd2l0aCBORlNyb290LA0KPiA+ID4gPiB3
aGlsZSBpdCBzZWVtcyB5b3UgYXJlIHVzaW5nIGEgU0FUQSBjb25uZWN0ZWQgZGV2aWNlLiBJcyB0
aGlzIHJpZ2h0Pw0KPiA+ID4NCj4gPiA+IEkgaGF2ZSBhIFNBVEEgZGlzayBjb25uZWN0ZWQsIGJ1
dCBkaWQgYm9vdCB3aXRoIE5GU3Jvb3QuDQo+ID4gPg0KPiA+ID4gPiBJZiBzbywNCj4gPiA+ID4g
Y2FuIHlvdSB0ZXN0IGlmIGl0IHdvcmtzIGlmIHlvdSBib290IGZyb20gU0RjYXJkIG9yIHRoZSBs
aWtlPyBUaGlzDQo+ID4gPiA+IG1pZ2h0IGJlIHJlbGV2YW50IGFzIFBDSWUgYW5kIFNBVEEgc2hh
cmUgc29tZSBjbG9ja3MuDQo+ID4gPg0KPiA+ID4gSSB0cmllZCB0byBkaXNhYmxlIFNBVEEgc3Vw
cG9ydCBjb21wbGV0ZWx5LCBidXQgaXQgZG9lc24ndCBoZWxwLg0KPiA+DQo+ID4gTHVjYXMsIGFu
eSBuZXdzIG9uIHRoaXM/ICBPciBzaG91bGQgd2UganVzdCB0cnkgdG8gdXNlIFJpY2hhcmQncyBw
YXRjaA0KPiA+IHRvIHNvbHZlIHRoZSBwcm9ibGVtPw0KPiANCj4gSSB3b3VsZCBsaWtlIHRvIHVu
ZGVyc3RhbmQgdGhlIHByb2JsZW0gZmlyc3QgYmVmb3JlIHRocm93aW5nICJmaXhlcyIgYXQgdGhl
DQo+IGlzc3VlLiBBcyB5b3Ugc2FpZCB0aGlzIGlzbid0IGEgcmVncmVzc2lvbiB3ZSBhcmUgaW4g
bm8gaHVycnkgYW5kIHNob3VsZCB0cnkNCj4gdG8gYW5hbHl6ZSB0aGUgaXNzdWUgcHJvcGVybHku
IEkganVzdCByZXRlc3RlZCBhbmQgSSdtIHN0aWxsIG5vdCBhYmxlIHRvDQo+IHJlcHJvZHVjZSB0
aGUgaXNzdWUgb24gbXkgU2FicmVTRC4gTWF5YmUgdGhlcmUgYXJlIGJvYXJkIHJldmlzaW9uIHRo
YXQgZXhoaWJpdA0KPiBkaWZmZXJlbnQgYmVoYXZpb3I/IE15IGJvYXJkIGhhcyB0d28gc3RpY2tz
IG9uIHNheWluZyAiUmV2IEIzIiBhbmQgIlJldiBYMyIuDQo+IA0KPiBBcyB5b3UgY2FuIHNlZSBm
cm9tIHRoZSBsb2cgc3VzcGVuZC9yZXN1bWUgaXMgd29ya2luZyBmaW5lIGZvciBtZSB3aXRoIGtl
cm5lbA0KPiAzLjE2LXJjNSArIGlteF92Nl92N19kZWZjb25maWcuDQpbUmljaGFyZF0gQXMgSSBr
bm93IHRoYXQgaW14NiBwY2llIGlzIG5vdCBlbmFibGVkIGluIGlteF92Nl92N19kZWZjb25maWcg
aW4gZGVmYXVsdC4NCk1lbnUtY29uZmlnIGlzIHJlcXVpcmVkIGlmIHlvdSB3YW50IHRvIHRlc3Rz
IHN5c3RlbSBzdXNwZW5kL3Jlc3VtZSB3aXRoIHBjaWUgYnVpbHQtaW4uDQpKdXN0IGRvdWJsZSBj
b25maXJtLCBpcyB0aGUgcGNpZSBidWlsdC1pbiBhdCB5b3VyIHNpZGU/DQoNCj4gDQo+IHJvb3RA
WFhYOn4gZWNobyBtZW0gPiAvc3lzL3Bvd2VyL3N0YXRlDQo+IFBNOiBTeW5jaW5nIGZpbGVzeXN0
ZW1zIC4uLiBkb25lLg0KPiBGcmVlemluZyB1c2VyIHNwYWNlIHByb2Nlc3NlcyAuLi4gKGVsYXBz
ZWQgMC4wMDQgc2Vjb25kcykgZG9uZS4NCj4gRnJlZXppbmcgcmVtYWluaW5nIGZyZWV6YWJsZSB0
YXNrcyAuLi4gKGVsYXBzZWQgMC4wMDIgc2Vjb25kcykgZG9uZS4NCj4gU3VzcGVuZGluZyBjb25z
b2xlKHMpICh1c2Ugbm9fY29uc29sZV9zdXNwZW5kIHRvIGRlYnVnKQ0KPiBQTTogc3VzcGVuZCBv
ZiBkZXZpY2VzIGNvbXBsZXRlIGFmdGVyIDk1Ljg0NiBtc2Vjcw0KPiBQTTogc3VzcGVuZCBkZXZp
Y2VzIHRvb2sgMC4xMDAgc2Vjb25kcw0KPiBQTTogbGF0ZSBzdXNwZW5kIG9mIGRldmljZXMgY29t
cGxldGUgYWZ0ZXIgNS4zOTUgbXNlY3MNCj4gUE06IG5vaXJxIHN1c3BlbmQgb2YgZGV2aWNlcyBj
b21wbGV0ZSBhZnRlciA0LjkxNCBtc2VjcyBEaXNhYmxpbmcgbm9uLWJvb3QNCj4gQ1BVcyAuLi4N
Cj4gQ1BVMTogc2h1dGRvd24NCj4gQ1BVMjogc2h1dGRvd24NCj4gQ1BVMzogc2h1dGRvd24NCj4g
RW5hYmxpbmcgbm9uLWJvb3QgQ1BVcyAuLi4NCj4gQ1BVMTogQm9vdGVkIHNlY29uZGFyeSBwcm9j
ZXNzb3INCj4gQ1BVMSBpcyB1cA0KPiBDUFUyOiBCb290ZWQgc2Vjb25kYXJ5IHByb2Nlc3Nvcg0K
PiBDUFUyIGlzIHVwDQo+IENQVTM6IEJvb3RlZCBzZWNvbmRhcnkgcHJvY2Vzc29yDQo+IENQVTMg
aXMgdXANCj4gUE06IG5vaXJxIHJlc3VtZSBvZiBkZXZpY2VzIGNvbXBsZXRlIGFmdGVyIDE1NTYu
Mzg1IG1zZWNzDQo+IFBNOiBlYXJseSByZXN1bWUgb2YgZGV2aWNlcyBjb21wbGV0ZSBhZnRlciAz
LjEzNCBtc2Vjcw0KPiBQTTogcmVzdW1lIG9mIGRldmljZXMgY29tcGxldGUgYWZ0ZXIgMTUzLjU2
OCBtc2Vjcw0KPiBQTTogcmVzdW1lIGRldmljZXMgdG9vayAwLjE2MCBzZWNvbmRzDQo+IFJlc3Rh
cnRpbmcgdGFza3MgLi4uIGRvbmUuDQo+IGF0YTE6IFNBVEEgbGluayBkb3duIChTU3RhdHVzIDAg
U0NvbnRyb2wgMzAwKSBmZWMgMjE4ODAwMC5ldGhlcm5ldCBldGgwOiBMaW5rDQo+IGlzIERvd24g
ZmVjIDIxODgwMDAuZXRoZXJuZXQgZXRoMDogTGluayBpcyBVcCAtIDFHYnBzL0Z1bGwgLSBmbG93
IGNvbnRyb2wgb2ZmDQo+IHJvb3RAWFhYOn4gZWNobyBtZW0gPiAvc3lzL3Bvd2VyL3N0YXRlDQo+
IFBNOiBTeW5jaW5nIGZpbGVzeXN0ZW1zIC4uLiBkb25lLg0KPiBGcmVlemluZyB1c2VyIHNwYWNl
IHByb2Nlc3NlcyAuLi4gKGVsYXBzZWQgMC4wMDIgc2Vjb25kcykgZG9uZS4NCj4gRnJlZXppbmcg
cmVtYWluaW5nIGZyZWV6YWJsZSB0YXNrcyAuLi4gKGVsYXBzZWQgMC4wMDEgc2Vjb25kcykgZG9u
ZS4NCj4gU3VzcGVuZGluZyBjb25zb2xlKHMpICh1c2Ugbm9fY29uc29sZV9zdXNwZW5kIHRvIGRl
YnVnKQ0KPiBQTTogc3VzcGVuZCBvZiBkZXZpY2VzIGNvbXBsZXRlIGFmdGVyIDg3LjYzOCBtc2Vj
cw0KPiBQTTogc3VzcGVuZCBkZXZpY2VzIHRvb2sgMC4wOTAgc2Vjb25kcw0KPiBQTTogbGF0ZSBz
dXNwZW5kIG9mIGRldmljZXMgY29tcGxldGUgYWZ0ZXIgNC41MjggbXNlY3MNCj4gUE06IG5vaXJx
IHN1c3BlbmQgb2YgZGV2aWNlcyBjb21wbGV0ZSBhZnRlciA1LjAzMSBtc2VjcyBEaXNhYmxpbmcg
bm9uLWJvb3QNCj4gQ1BVcyAuLi4NCj4gQ1BVMTogc2h1dGRvd24NCj4gQ1BVMjogc2h1dGRvd24N
Cj4gQ1BVMzogc2h1dGRvd24NCj4gRW5hYmxpbmcgbm9uLWJvb3QgQ1BVcyAuLi4NCj4gQ1BVMTog
Qm9vdGVkIHNlY29uZGFyeSBwcm9jZXNzb3INCj4gQ1BVMSBpcyB1cA0KPiBDUFUyOiBCb290ZWQg
c2Vjb25kYXJ5IHByb2Nlc3Nvcg0KPiBDUFUyIGlzIHVwDQo+IENQVTM6IEJvb3RlZCBzZWNvbmRh
cnkgcHJvY2Vzc29yDQo+IENQVTMgaXMgdXANCj4gUE06IG5vaXJxIHJlc3VtZSBvZiBkZXZpY2Vz
IGNvbXBsZXRlIGFmdGVyIDE0MjUuMzAyIG1zZWNzDQo+IFBNOiBlYXJseSByZXN1bWUgb2YgZGV2
aWNlcyBjb21wbGV0ZSBhZnRlciAyLjczNiBtc2Vjcw0KPiBQTTogcmVzdW1lIG9mIGRldmljZXMg
Y29tcGxldGUgYWZ0ZXIgMTU2LjAxMyBtc2Vjcw0KPiBQTTogcmVzdW1lIGRldmljZXMgdG9vayAw
LjE2MCBzZWNvbmRzDQo+IFJlc3RhcnRpbmcgdGFza3MgLi4uIGRvbmUuDQo+IGF0YTE6IFNBVEEg
bGluayBkb3duIChTU3RhdHVzIDAgU0NvbnRyb2wgMzAwKSBmZWMgMjE4ODAwMC5ldGhlcm5ldCBl
dGgwOiBMaW5rDQo+IGlzIERvd24gZmVjIDIxODgwMDAuZXRoZXJuZXQgZXRoMDogTGluayBpcyBV
cCAtIDFHYnBzL0Z1bGwgLSBmbG93IGNvbnRyb2wgb2ZmDQo+IHJvb3RAWFhYOn4NCj4gDQo+IC0t
DQo+IFBlbmd1dHJvbml4IGUuSy4gICAgICAgICAgICAgfCBMdWNhcyBTdGFjaCAgICAgICAgICAg
ICAgICAgfA0KPiBJbmR1c3RyaWFsIExpbnV4IFNvbHV0aW9ucyAgIHwgaHR0cDovL3d3dy5wZW5n
dXRyb25peC5kZS8gIHwNCg0KDQpCZXN0IFJlZ2FyZHMNClJpY2hhcmQgWmh1DQoNCg==

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Am Freitag, den 18.07.2014, 02:11 +0000 schrieb
Hong-Xing.Zhu@freescale.com:
> > -----Original Message-----
> > From: Lucas Stach [mailto:l.stach@pengutronix.de]
> > Sent: Thursday, July 17, 2014 9:55 PM
> > To: Guo Shawn-R65073
> > Cc: Zhu Richard-R65037; linux-pci@vger.kernel.org; Sascha Hauer; Bjorn Helgaas;
> > Shawn Guo; Fabio Estevam; linux-arm-kernel@lists.infradead.org
> > Subject: Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe
> > 
> > Hi Shawn,
> > 
> > Am Mittwoch, den 16.07.2014, 14:55 +0800 schrieb Shawn Guo:
> > > On Mon, Jul 07, 2014 at 09:55:03PM +0800, Shawn Guo wrote:
> > > > On Mon, Jul 07, 2014 at 11:10:51AM +0200, Lucas Stach wrote:
> > > > > Hi Shawn,
> > > > >
> > > > > Over the weekend I tried to reproduce your problem on a SabreSD
> > > > > board, but wasn't able to trigger the issue. 3.16-rc3 with PCIe
> > > > > active works just fine over a suspend and resume cycle for me.
> > > >
> > > > That's strange.  In my setup, PCIe support is enabled in kernel and
> > > > DT, but I do not have a PCIe device connected to the board.
> > > >
> > > > >
> > > > > One possibly relevant difference is that I've booted with NFSroot,
> > > > > while it seems you are using a SATA connected device. Is this right?
> > > >
> > > > I have a SATA disk connected, but did boot with NFSroot.
> > > >
> > > > > If so,
> > > > > can you test if it works if you boot from SDcard or the like? This
> > > > > might be relevant as PCIe and SATA share some clocks.
> > > >
> > > > I tried to disable SATA support completely, but it doesn't help.
> > >
> > > Lucas, any news on this?  Or should we just try to use Richard's patch
> > > to solve the problem?
> > 
> > I would like to understand the problem first before throwing "fixes" at the
> > issue. As you said this isn't a regression we are in no hurry and should try
> > to analyze the issue properly. I just retested and I'm still not able to
> > reproduce the issue on my SabreSD. Maybe there are board revision that exhibit
> > different behavior? My board has two sticks on saying "Rev B3" and "Rev X3".
> > 
> > As you can see from the log suspend/resume is working fine for me with kernel
> > 3.16-rc5 + imx_v6_v7_defconfig.
> [Richard] As I know that imx6 pcie is not enabled in imx_v6_v7_defconfig in default.
> Menu-config is required if you want to tests system suspend/resume with pcie built-in.
> Just double confirm, is the pcie built-in at your side?

This is not right, PCI is enabled in imx_v6_v7_defconfig since
c0bea59ca58e30fb8fd29254569bdaae482398ad "ARM: imx_v6_v7_defconfig:
Select PCI support".

However I seem to have a board with rev 1.1 silicon and although the pci
driver starts up it never establishes a link. So my board may just work
by chance as I don't really know the differences between rev 1.1 silicon
and later revisions.

Regards,
Lucas
-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Fri, Jul 18, 2014 at 11:21:48AM +0200, Lucas Stach wrote:
> However I seem to have a board with rev 1.1 silicon and although the pci
> driver starts up it never establishes a link. So my board may just work
> by chance as I don't really know the differences between rev 1.1 silicon
> and later revisions.

Lucas, thanks for the info.

I just checked my board on which the issue is seen, and found it has a
rev 1.3 silicon.  I just tested on a board with rev 1.2 and did not even
see the issue.

Richard,

Is this an issue only with rev 1.3?

Shawn

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Fabio Estevam]

On Sun, Jul 20, 2014 at 11:50 PM, Shawn Guo <shawn.guo@freescale.com> wrote:
> On Fri, Jul 18, 2014 at 11:21:48AM +0200, Lucas Stach wrote:
>> However I seem to have a board with rev 1.1 silicon and although the pci
>> driver starts up it never establishes a link. So my board may just work
>> by chance as I don't really know the differences between rev 1.1 silicon
>> and later revisions.
>
> Lucas, thanks for the info.
>
> I just checked my board on which the issue is seen, and found it has a
> rev 1.3 silicon.  I just tested on a board with rev 1.2 and did not even
> see the issue.
>
> Richard,
>
> Is this an issue only with rev 1.3?

I could see this issue on my board with rev 1.2.

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Sun, Jul 20, 2014 at 11:55:42PM -0300, Fabio Estevam wrote:
> On Sun, Jul 20, 2014 at 11:50 PM, Shawn Guo <shawn.guo@freescale.com> wrote:
> > On Fri, Jul 18, 2014 at 11:21:48AM +0200, Lucas Stach wrote:
> >> However I seem to have a board with rev 1.1 silicon and although the pci
> >> driver starts up it never establishes a link. So my board may just work
> >> by chance as I don't really know the differences between rev 1.1 silicon
> >> and later revisions.
> >
> > Lucas, thanks for the info.
> >
> > I just checked my board on which the issue is seen, and found it has a
> > rev 1.3 silicon.  I just tested on a board with rev 1.2 and did not even
> > see the issue.
> >
> > Richard,
> >
> > Is this an issue only with rev 1.3?
> 
> I could see this issue on my board with rev 1.2.

Hmm, that's strange.  I tested two imx6q-sabresd boards with rev 1.2
silicon, and did not see this issue.  Hopefully, someone else can also
test.

Shawn

Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Wed, Jun 25, 2014 at 02:22:44PM +0800, Shawn Guo wrote:
> On Tue, Jun 24, 2014 at 04:37:19PM -0300, Fabio Estevam wrote:
> > On Tue, Jun 24, 2014 at 12:36 PM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > > Hi Lucas,
> > >
> > > I found that imx6q suspend/resume is broken on 3.16-rc, and it works
> > > just fine if PCIe support is built out.  However the issue was not
> > > exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
> > > is part of the cause.
> > 
> > I have just tested this on linux-next and I see that suspend/resume
> > fails even if I build a kernel without PCI support.
> 
> Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> should be another issue.

Yeah, the issue on linux-next is caused by commit 478850160636 (irq_work:
Implement remote queueing).  And Stephen reported the issue as below.

https://lkml.org/lkml/2014/6/24/765

Shawn