The imx6q suspend/resume is broken on 3.16-rc due to PCIe

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

Hi Lucas,

I found that imx6q suspend/resume is broken on 3.16-rc, and it works
just fine if PCIe support is built out.  However the issue was not
exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
is part of the cause.

$ echo mem > /sys/power/state
PM: Syncing filesystems ... done.
Freezing user space processes ... (elapsed 0.002 seconds) done.
Freezing remaining freezable tasks ... (elapsed 0.002 seconds) done.
sd 0:0:0:0: [sda] Synchronizing SCSI cache
sd 0:0:0:0: [sda] Stopping disk
PM: suspend of devices complete after 85.695 msecs
PM: suspend devices took 0.100 seconds
PM: late suspend of devices complete after 6.786 msecs
PM: noirq suspend of devices complete after 6.228 msecs
Disabling non-boot CPUs ...
CPU1: shutdown
CPU2: shutdown
CPU3: shutdown
Enabling non-boot CPUs ...
CPU1: Booted secondary processor
CPU1 is up
CPU2: Booted secondary processor
CPU2 is up
CPU3: Booted secondary processor
CPU3 is up

The issue is actually with resume procedure.  I noticed that Richard
added the following change in FSL kernel tree, which seems to fix the
issue.

@@ -229,6 +232,22 @@ static int imx6_suspend_finish(unsigned long val)

 static int imx6_pm_enter(suspend_state_t state)
 {
+       struct regmap *gpr;
+
+       /*
+        * L2 can exit by 'reset' or Inband beacon (from remote EP)
+        * toggling phy_powerdown has same effect as 'inband beacon'
+        * So, toggle bit18 of GPR1, to fix errata
+        * "PCIe PCIe does not support L2 Power Down"
+        */
+       gpr = syscon_regmap_lookup_by_compatible("fsl,imx6q-iomuxc-gpr");
+       if (IS_ERR(gpr)) {
+               pr_err("failed to find fsl,imx6q-iomux-gpr regmap\n");
+               return PTR_ERR(gpr);
+       }
+       regmap_update_bits(gpr, IOMUXC_GPR1, IMX6Q_GPR1_PCIE_TEST_PD,
+                       IMX6Q_GPR1_PCIE_TEST_PD);
+
        switch (state) {
        case PM_SUSPEND_STANDBY:
                imx6_set_lpm(STOP_POWER_ON);
@@ -258,6 +277,15 @@ static int imx6_pm_enter(suspend_state_t state)
                return -EINVAL;
        }

+       /*
+        * L2 can exit by 'reset' or Inband beacon (from remote EP)
+        * toggling phy_powerdown has same effect as 'inband beacon'
+        * So, toggle bit18 of GPR1, to fix errata
+        * "PCIe PCIe does not support L2 Power Down"
+        */
+       regmap_update_bits(gpr, IOMUXC_GPR1, IMX6Q_GPR1_PCIE_TEST_PD,
+                       !IMX6Q_GPR1_PCIE_TEST_PD);
+
        return 0;
 }
 
And I copied the details of the errata as below.

ERR005723 PCIe: PCIe does not support L2 power down

Description:
When PCIe works as Root Complex, it can exit L2 mode only through reset.
Since PCIe does not have a dedicated reset control bit, it cannot exit
L2 mode.

Projected Impact:
PCIe does not support L2 power down.

Workarounds:
The PCIe can be put into PDDQ mode to save PCIe PHY power and wake up
only by the OOB (Out of Band) wakeup signal (since wakeup by a beacon
from link partner is not supported) driven from the link partner (End
Point). This signal could be used as a GPIO interrupt to exit this mode.
The limitation of this workaround is that the link partner cannot be put
into L2 mode.

As I have no knowledge of how PCIe works, I provided all the info I
found here to kick the discussion.  Hopefully, we can find a proper fix
to get suspend/resume back to work.

Shawn

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Hi Shawn,

Am Dienstag, den 24.06.2014, 23:36 +0800 schrieb Shawn Guo:
> Hi Lucas,
> 
> I found that imx6q suspend/resume is broken on 3.16-rc, and it works
> just fine if PCIe support is built out.  However the issue was not
> exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
> is part of the cause.
> 
I don't yet see why any of the changes to the designware/imx6 host
driver would cause such failures.

Which board are you doing your testing on? What devices are connected to
the PCIe bus? In what way does it break system resume, complete hang?

Regards,
Lucas

> $ echo mem > /sys/power/state
> PM: Syncing filesystems ... done.
> Freezing user space processes ... (elapsed 0.002 seconds) done.
> Freezing remaining freezable tasks ... (elapsed 0.002 seconds) done.
> sd 0:0:0:0: [sda] Synchronizing SCSI cache
> sd 0:0:0:0: [sda] Stopping disk
> PM: suspend of devices complete after 85.695 msecs
> PM: suspend devices took 0.100 seconds
> PM: late suspend of devices complete after 6.786 msecs
> PM: noirq suspend of devices complete after 6.228 msecs
> Disabling non-boot CPUs ...
> CPU1: shutdown
> CPU2: shutdown
> CPU3: shutdown
> Enabling non-boot CPUs ...
> CPU1: Booted secondary processor
> CPU1 is up
> CPU2: Booted secondary processor
> CPU2 is up
> CPU3: Booted secondary processor
> CPU3 is up
> 
> The issue is actually with resume procedure.  I noticed that Richard
> added the following change in FSL kernel tree, which seems to fix the
> issue.
> 
> @@ -229,6 +232,22 @@ static int imx6_suspend_finish(unsigned long val)
> 
>  static int imx6_pm_enter(suspend_state_t state)
>  {
> +       struct regmap *gpr;
> +
> +       /*
> +        * L2 can exit by 'reset' or Inband beacon (from remote EP)
> +        * toggling phy_powerdown has same effect as 'inband beacon'
> +        * So, toggle bit18 of GPR1, to fix errata
> +        * "PCIe PCIe does not support L2 Power Down"
> +        */
> +       gpr = syscon_regmap_lookup_by_compatible("fsl,imx6q-iomuxc-gpr");
> +       if (IS_ERR(gpr)) {
> +               pr_err("failed to find fsl,imx6q-iomux-gpr regmap\n");
> +               return PTR_ERR(gpr);
> +       }
> +       regmap_update_bits(gpr, IOMUXC_GPR1, IMX6Q_GPR1_PCIE_TEST_PD,
> +                       IMX6Q_GPR1_PCIE_TEST_PD);
> +
>         switch (state) {
>         case PM_SUSPEND_STANDBY:
>                 imx6_set_lpm(STOP_POWER_ON);
> @@ -258,6 +277,15 @@ static int imx6_pm_enter(suspend_state_t state)
>                 return -EINVAL;
>         }
> 
> +       /*
> +        * L2 can exit by 'reset' or Inband beacon (from remote EP)
> +        * toggling phy_powerdown has same effect as 'inband beacon'
> +        * So, toggle bit18 of GPR1, to fix errata
> +        * "PCIe PCIe does not support L2 Power Down"
> +        */
> +       regmap_update_bits(gpr, IOMUXC_GPR1, IMX6Q_GPR1_PCIE_TEST_PD,
> +                       !IMX6Q_GPR1_PCIE_TEST_PD);
> +
>         return 0;
>  }
>  
> And I copied the details of the errata as below.
> 
> ERR005723 PCIe: PCIe does not support L2 power down
> 
> Description:
> When PCIe works as Root Complex, it can exit L2 mode only through reset.
> Since PCIe does not have a dedicated reset control bit, it cannot exit
> L2 mode.
> 
> Projected Impact:
> PCIe does not support L2 power down.
> 
> Workarounds:
> The PCIe can be put into PDDQ mode to save PCIe PHY power and wake up
> only by the OOB (Out of Band) wakeup signal (since wakeup by a beacon
> from link partner is not supported) driven from the link partner (End
> Point). This signal could be used as a GPIO interrupt to exit this mode.
> The limitation of this workaround is that the link partner cannot be put
> into L2 mode.
> 
> As I have no knowledge of how PCIe works, I provided all the info I
> found here to kick the discussion.  Hopefully, we can find a proper fix
> to get suspend/resume back to work.
> 
> Shawn
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Tue, Jun 24, 2014 at 06:13:55PM +0200, Lucas Stach wrote:
> Hi Shawn,
> 
> Am Dienstag, den 24.06.2014, 23:36 +0800 schrieb Shawn Guo:
> > Hi Lucas,
> > 
> > I found that imx6q suspend/resume is broken on 3.16-rc, and it works
> > just fine if PCIe support is built out.  However the issue was not
> > exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
> > is part of the cause.
> > 
> I don't yet see why any of the changes to the designware/imx6 host
> driver would cause such failures.
> 
> Which board are you doing your testing on? What devices are connected to
> the PCIe bus? In what way does it break system resume, complete hang?

It's on imx6q-sabresd without any PCIe device connected.  Right, it's a
complete hang after enabling non-boot CPUs.

...
Enabling non-boot CPUs ...
CPU1: Booted secondary processor
CPU1 is up
CPU2: Booted secondary processor
CPU2 is up
CPU3: Booted secondary processor
CPU3 is up
<*** hang here ***>

Shawn

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Fabio Estevam]

On Tue, Jun 24, 2014 at 12:36 PM, Shawn Guo <shawn.guo@linaro.org> wrote:
> Hi Lucas,
>
> I found that imx6q suspend/resume is broken on 3.16-rc, and it works
> just fine if PCIe support is built out.  However the issue was not
> exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
> is part of the cause.

I have just tested this on linux-next and I see that suspend/resume
fails even if I build a kernel without PCI support.

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Tue, Jun 24, 2014 at 04:37:19PM -0300, Fabio Estevam wrote:
> On Tue, Jun 24, 2014 at 12:36 PM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > Hi Lucas,
> >
> > I found that imx6q suspend/resume is broken on 3.16-rc, and it works
> > just fine if PCIe support is built out.  However the issue was not
> > exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
> > is part of the cause.
> 
> I have just tested this on linux-next and I see that suspend/resume
> fails even if I build a kernel without PCI support.

Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
should be another issue.

Shawn

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Fabio Estevam]

On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:

> Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> should be another issue.

Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
driver removed.

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Am Mittwoch, den 25.06.2014, 07:50 -0300 schrieb Fabio Estevam:
> On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:
> 
> > Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> > should be another issue.
> 
> Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
> driver removed.

Can you please send me a dmesg from a working boot?

Regards,
Lucas
-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Wed, Jun 25, 2014 at 12:50:52PM +0200, Lucas Stach wrote:
> Am Mittwoch, den 25.06.2014, 07:50 -0300 schrieb Fabio Estevam:
> > On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > 
> > > Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> > > should be another issue.
> > 
> > Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
> > driver removed.
> 
> Can you please send me a dmesg from a working boot?

What do you mean by "working boot"?  Boot is always working.  What fails
is system resuming.

Shawn

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Am Mittwoch, den 25.06.2014, 20:53 +0800 schrieb Shawn Guo:
> On Wed, Jun 25, 2014 at 12:50:52PM +0200, Lucas Stach wrote:
> > Am Mittwoch, den 25.06.2014, 07:50 -0300 schrieb Fabio Estevam:
> > > On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > > 
> > > > Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> > > > should be another issue.
> > > 
> > > Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
> > > driver removed.
> > 
> > Can you please send me a dmesg from a working boot?
> 
> What do you mean by "working boot"?  Boot is always working.  What fails
> is system resuming.
> 
> Shawn
> 
Oops, what I meant is a dmesg after a working resume (from kernel 3.15).
I can see how failing to resume from L2 can cause the hang and we should
really implement the PHY PD toggling in the pcie driver, but I would
like to understand why you don't hit this on 3.15.

Regards,
Lucas

-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Thu, Jun 26, 2014 at 10:43:49AM +0200, Lucas Stach wrote:
> Am Mittwoch, den 25.06.2014, 20:53 +0800 schrieb Shawn Guo:
> > On Wed, Jun 25, 2014 at 12:50:52PM +0200, Lucas Stach wrote:
> > > Am Mittwoch, den 25.06.2014, 07:50 -0300 schrieb Fabio Estevam:
> > > > On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > > > 
> > > > > Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> > > > > should be another issue.
> > > > 
> > > > Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
> > > > driver removed.
> > > 
> > > Can you please send me a dmesg from a working boot?
> > 
> > What do you mean by "working boot"?  Boot is always working.  What fails
> > is system resuming.
> > 
> > Shawn
> > 
> Oops, what I meant is a dmesg after a working resume (from kernel 3.15).
> I can see how failing to resume from L2 can cause the hang and we should
> really implement the PHY PD toggling in the pcie driver, but I would
> like to understand why you don't hit this on 3.15.

Sorry, Lucas.  I just noticed that in my 3.15 testing PCIe is not
enabled in device tree at all.  So the issue should have already been
there before your changes to PCIe driver.

Shawn

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Am Samstag, den 28.06.2014, 22:37 +0800 schrieb Shawn Guo:
> On Thu, Jun 26, 2014 at 10:43:49AM +0200, Lucas Stach wrote:
> > Am Mittwoch, den 25.06.2014, 20:53 +0800 schrieb Shawn Guo:
> > > On Wed, Jun 25, 2014 at 12:50:52PM +0200, Lucas Stach wrote:
> > > > Am Mittwoch, den 25.06.2014, 07:50 -0300 schrieb Fabio Estevam:
> > > > > On Wed, Jun 25, 2014 at 3:22 AM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > > > > 
> > > > > > Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> > > > > > should be another issue.
> > > > > 
> > > > > Yes, I see it now. Tested on -rc2 and suspend/resume works with PCI
> > > > > driver removed.
> > > > 
> > > > Can you please send me a dmesg from a working boot?
> > > 
> > > What do you mean by "working boot"?  Boot is always working.  What fails
> > > is system resuming.
> > > 
> > > Shawn
> > > 
> > Oops, what I meant is a dmesg after a working resume (from kernel 3.15).
> > I can see how failing to resume from L2 can cause the hang and we should
> > really implement the PHY PD toggling in the pcie driver, but I would
> > like to understand why you don't hit this on 3.15.
> 
> Sorry, Lucas.  I just noticed that in my 3.15 testing PCIe is not
> enabled in device tree at all.  So the issue should have already been
> there before your changes to PCIe driver.
> 
Hi Shawn,

can you please test the attached patch on top of 3.16-rc* to see if it
helps? If it works for you I would like to try and get this into 3.16 as
a bugfix.

Regards,
Lucas

--------------------------->8-----------------------------
>From f3bfdc9c19249188fa153b1568c3129cd31af4f5 Mon Sep 17 00:00:00 2001
From: Lucas Stach <l.stach@pengutronix.de>
Date: Mon, 30 Jun 2014 15:33:04 +0200
Subject: [PATCH] PCI: imx6: add workaround for errata ERR005723

Fixes system hang after resume from suspend to mem.

Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
---
 drivers/pci/host/pci-imx6.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
index a568efaa331c..5c58a3ee5dbf 100644
--- a/drivers/pci/host/pci-imx6.c
+++ b/drivers/pci/host/pci-imx6.c
@@ -589,6 +589,38 @@ static int __init imx6_pcie_probe(struct platform_device *pdev)
 	return 0;
 }
 
+#ifdef CONFIG_PM_SLEEP
+static int imx6_pcie_suspend(struct device *dev)
+{
+	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
+
+	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
+			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
+
+	return 0;
+}
+
+static int imx6_pcie_resume(struct device *dev)
+{
+	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
+
+	/*
+	 * This is a workaround for
+	 * ERR005723: PCIe does not support L2 power down
+	 * Toggling the PHY PD bit around system L2 state switching seems to be
+	 * enough to wake the PCIe logic after a power down.
+	 */
+	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
+			IMX6Q_GPR1_PCIE_TEST_PD, 0 << 18);
+
+	return 0;
+}
+#endif
+
+static SIMPLE_DEV_PM_OPS(imx6_pcie_pm_ops,
+		imx6_pcie_suspend,
+		imx6_pcie_resume);
+
 static const struct of_device_id imx6_pcie_of_match[] = {
 	{ .compatible = "fsl,imx6q-pcie", },
 	{},
@@ -600,6 +632,7 @@ static struct platform_driver imx6_pcie_driver = {
 		.name	= "imx6q-pcie",
 		.owner	= THIS_MODULE,
 		.of_match_table = imx6_pcie_of_match,
+		.pm = &imx6_pcie_pm_ops,
 	},
 };
 
-- 
2.0.0


-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Mon, Jun 30, 2014 at 03:44:24PM +0200, Lucas Stach wrote:
> Hi Shawn,
> 
> can you please test the attached patch on top of 3.16-rc* to see if it
> helps? If it works for you I would like to try and get this into 3.16 as
> a bugfix.

Unfortunately, it doesn't work.  With your change, now it hangs in
suspend procedure.

$ echo mem > /sys/power/state
PM: Syncing filesystems ... done.
PM: Preparing system for mem sleep
Freezing user space processes ... (elapsed 0.005 seconds) done.
Freezing remaining freezable tasks ... (elapsed 0.003 seconds) done.
PM: Entering mem sleep
sd 0:0:0:0: [sda] Synchronizing SCSI cache
sd 0:0:0:0: [sda] Stopping disk
PM: suspend of devices complete after 102.992 msecs
PM: suspend devices took 0.110 seconds
PM: late suspend of devices complete after 12.808 msecs

Shawn

> 
> Regards,
> Lucas
> 
> --------------------------->8-----------------------------
> From f3bfdc9c19249188fa153b1568c3129cd31af4f5 Mon Sep 17 00:00:00 2001
> From: Lucas Stach <l.stach@pengutronix.de>
> Date: Mon, 30 Jun 2014 15:33:04 +0200
> Subject: [PATCH] PCI: imx6: add workaround for errata ERR005723
> 
> Fixes system hang after resume from suspend to mem.
> 
> Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
> ---
>  drivers/pci/host/pci-imx6.c | 33 +++++++++++++++++++++++++++++++++
>  1 file changed, 33 insertions(+)
> 
> diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
> index a568efaa331c..5c58a3ee5dbf 100644
> --- a/drivers/pci/host/pci-imx6.c
> +++ b/drivers/pci/host/pci-imx6.c
> @@ -589,6 +589,38 @@ static int __init imx6_pcie_probe(struct platform_device *pdev)
>  	return 0;
>  }
>  
> +#ifdef CONFIG_PM_SLEEP
> +static int imx6_pcie_suspend(struct device *dev)
> +{
> +	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
> +
> +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
> +			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
> +
> +	return 0;
> +}
> +
> +static int imx6_pcie_resume(struct device *dev)
> +{
> +	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
> +
> +	/*
> +	 * This is a workaround for
> +	 * ERR005723: PCIe does not support L2 power down
> +	 * Toggling the PHY PD bit around system L2 state switching seems to be
> +	 * enough to wake the PCIe logic after a power down.
> +	 */
> +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
> +			IMX6Q_GPR1_PCIE_TEST_PD, 0 << 18);
> +
> +	return 0;
> +}
> +#endif
> +
> +static SIMPLE_DEV_PM_OPS(imx6_pcie_pm_ops,
> +		imx6_pcie_suspend,
> +		imx6_pcie_resume);
> +
>  static const struct of_device_id imx6_pcie_of_match[] = {
>  	{ .compatible = "fsl,imx6q-pcie", },
>  	{},
> @@ -600,6 +632,7 @@ static struct platform_driver imx6_pcie_driver = {
>  		.name	= "imx6q-pcie",
>  		.owner	= THIS_MODULE,
>  		.of_match_table = imx6_pcie_of_match,
> +		.pm = &imx6_pcie_pm_ops,
>  	},
>  };
>  
> -- 
> 2.0.0
> 
> 
> -- 
> Pengutronix e.K.             | Lucas Stach                 |
> Industrial Linux Solutions   | http://www.pengutronix.de/  |
> 

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Hi Shawn,

Over the weekend I tried to reproduce your problem on a SabreSD board,
but wasn't able to trigger the issue. 3.16-rc3 with PCIe active works
just fine over a suspend and resume cycle for me.

One possibly relevant difference is that I've booted with NFSroot, while
it seems you are using a SATA connected device. Is this right? If so,
can you test if it works if you boot from SDcard or the like? This might
be relevant as PCIe and SATA share some clocks.

Regards,
Lucas

Am Dienstag, den 01.07.2014, 14:51 +0800 schrieb Shawn Guo:
> On Mon, Jun 30, 2014 at 03:44:24PM +0200, Lucas Stach wrote:
> > Hi Shawn,
> > 
> > can you please test the attached patch on top of 3.16-rc* to see if it
> > helps? If it works for you I would like to try and get this into 3.16 as
> > a bugfix.
> 
> Unfortunately, it doesn't work.  With your change, now it hangs in
> suspend procedure.
> 
> $ echo mem > /sys/power/state
> PM: Syncing filesystems ... done.
> PM: Preparing system for mem sleep
> Freezing user space processes ... (elapsed 0.005 seconds) done.
> Freezing remaining freezable tasks ... (elapsed 0.003 seconds) done.
> PM: Entering mem sleep
> sd 0:0:0:0: [sda] Synchronizing SCSI cache
> sd 0:0:0:0: [sda] Stopping disk
> PM: suspend of devices complete after 102.992 msecs
> PM: suspend devices took 0.110 seconds
> PM: late suspend of devices complete after 12.808 msecs
> 
> Shawn
> 
> > 
> > Regards,
> > Lucas
> > 
> > --------------------------->8-----------------------------
> > From f3bfdc9c19249188fa153b1568c3129cd31af4f5 Mon Sep 17 00:00:00 2001
> > From: Lucas Stach <l.stach@pengutronix.de>
> > Date: Mon, 30 Jun 2014 15:33:04 +0200
> > Subject: [PATCH] PCI: imx6: add workaround for errata ERR005723
> > 
> > Fixes system hang after resume from suspend to mem.
> > 
> > Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
> > ---
> >  drivers/pci/host/pci-imx6.c | 33 +++++++++++++++++++++++++++++++++
> >  1 file changed, 33 insertions(+)
> > 
> > diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c
> > index a568efaa331c..5c58a3ee5dbf 100644
> > --- a/drivers/pci/host/pci-imx6.c
> > +++ b/drivers/pci/host/pci-imx6.c
> > @@ -589,6 +589,38 @@ static int __init imx6_pcie_probe(struct platform_device *pdev)
> >  	return 0;
> >  }
> >  
> > +#ifdef CONFIG_PM_SLEEP
> > +static int imx6_pcie_suspend(struct device *dev)
> > +{
> > +	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
> > +
> > +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
> > +			IMX6Q_GPR1_PCIE_TEST_PD, 1 << 18);
> > +
> > +	return 0;
> > +}
> > +
> > +static int imx6_pcie_resume(struct device *dev)
> > +{
> > +	struct imx6_pcie *imx6_pcie = dev_get_drvdata(dev);
> > +
> > +	/*
> > +	 * This is a workaround for
> > +	 * ERR005723: PCIe does not support L2 power down
> > +	 * Toggling the PHY PD bit around system L2 state switching seems to be
> > +	 * enough to wake the PCIe logic after a power down.
> > +	 */
> > +	regmap_update_bits(imx6_pcie->iomuxc_gpr, IOMUXC_GPR1,
> > +			IMX6Q_GPR1_PCIE_TEST_PD, 0 << 18);
> > +
> > +	return 0;
> > +}
> > +#endif
> > +
> > +static SIMPLE_DEV_PM_OPS(imx6_pcie_pm_ops,
> > +		imx6_pcie_suspend,
> > +		imx6_pcie_resume);
> > +
> >  static const struct of_device_id imx6_pcie_of_match[] = {
> >  	{ .compatible = "fsl,imx6q-pcie", },
> >  	{},
> > @@ -600,6 +632,7 @@ static struct platform_driver imx6_pcie_driver = {
> >  		.name	= "imx6q-pcie",
> >  		.owner	= THIS_MODULE,
> >  		.of_match_table = imx6_pcie_of_match,
> > +		.pm = &imx6_pcie_pm_ops,
> >  	},
> >  };
> >  
> > -- 
> > 2.0.0
> > 
> > 
> > -- 
> > Pengutronix e.K.             | Lucas Stach                 |
> > Industrial Linux Solutions   | http://www.pengutronix.de/  |
> > 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Mon, Jul 07, 2014 at 11:10:51AM +0200, Lucas Stach wrote:
> Hi Shawn,
> 
> Over the weekend I tried to reproduce your problem on a SabreSD board,
> but wasn't able to trigger the issue. 3.16-rc3 with PCIe active works
> just fine over a suspend and resume cycle for me.

That's strange.  In my setup, PCIe support is enabled in kernel and DT,
but I do not have a PCIe device connected to the board.

> 
> One possibly relevant difference is that I've booted with NFSroot, while
> it seems you are using a SATA connected device. Is this right?

I have a SATA disk connected, but did boot with NFSroot.

> If so,
> can you test if it works if you boot from SDcard or the like? This might
> be relevant as PCIe and SATA share some clocks.

I tried to disable SATA support completely, but it doesn't help.

$ echo mem > /sys/power/state
[  410.052595] PM: Syncing filesystems ... done.
[  410.150033] PM: Preparing system for mem sleep
[  410.207963] Freezing user space processes ... (elapsed 0.004 seconds) done.
[  410.219796] Freezing remaining freezable tasks ... (elapsed 0.002 seconds) done.
[  410.230243] PM: Entering mem sleep
[  410.316574] PM: suspend of devices complete after 79.461 msecs
[  410.322498] PM: suspend devices took 0.090 seconds
[  410.332655] PM: late suspend of devices complete after 5.338 msecs


Shawn

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Mon, Jul 07, 2014 at 09:55:03PM +0800, Shawn Guo wrote:
> On Mon, Jul 07, 2014 at 11:10:51AM +0200, Lucas Stach wrote:
> > Hi Shawn,
> > 
> > Over the weekend I tried to reproduce your problem on a SabreSD board,
> > but wasn't able to trigger the issue. 3.16-rc3 with PCIe active works
> > just fine over a suspend and resume cycle for me.
> 
> That's strange.  In my setup, PCIe support is enabled in kernel and DT,
> but I do not have a PCIe device connected to the board.
> 
> > 
> > One possibly relevant difference is that I've booted with NFSroot, while
> > it seems you are using a SATA connected device. Is this right?
> 
> I have a SATA disk connected, but did boot with NFSroot.
> 
> > If so,
> > can you test if it works if you boot from SDcard or the like? This might
> > be relevant as PCIe and SATA share some clocks.
> 
> I tried to disable SATA support completely, but it doesn't help.

Lucas, any news on this?  Or should we just try to use Richard's patch
to solve the problem?

Shawn

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Hi Shawn,

Am Mittwoch, den 16.07.2014, 14:55 +0800 schrieb Shawn Guo:
> On Mon, Jul 07, 2014 at 09:55:03PM +0800, Shawn Guo wrote:
> > On Mon, Jul 07, 2014 at 11:10:51AM +0200, Lucas Stach wrote:
> > > Hi Shawn,
> > > 
> > > Over the weekend I tried to reproduce your problem on a SabreSD board,
> > > but wasn't able to trigger the issue. 3.16-rc3 with PCIe active works
> > > just fine over a suspend and resume cycle for me.
> > 
> > That's strange.  In my setup, PCIe support is enabled in kernel and DT,
> > but I do not have a PCIe device connected to the board.
> > 
> > > 
> > > One possibly relevant difference is that I've booted with NFSroot, while
> > > it seems you are using a SATA connected device. Is this right?
> > 
> > I have a SATA disk connected, but did boot with NFSroot.
> > 
> > > If so,
> > > can you test if it works if you boot from SDcard or the like? This might
> > > be relevant as PCIe and SATA share some clocks.
> > 
> > I tried to disable SATA support completely, but it doesn't help.
> 
> Lucas, any news on this?  Or should we just try to use Richard's patch
> to solve the problem?

I would like to understand the problem first before throwing "fixes" at
the issue. As you said this isn't a regression we are in no hurry and
should try to analyze the issue properly. I just retested and I'm still
not able to reproduce the issue on my SabreSD. Maybe there are board
revision that exhibit different behavior? My board has two sticks on
saying "Rev B3" and "Rev X3".

As you can see from the log suspend/resume is working fine for me with
kernel 3.16-rc5 + imx_v6_v7_defconfig.

root at XXX:~ echo mem > /sys/power/state
PM: Syncing filesystems ... done.
Freezing user space processes ... (elapsed 0.004 seconds) done.
Freezing remaining freezable tasks ... (elapsed 0.002 seconds) done.
Suspending console(s) (use no_console_suspend to debug)
PM: suspend of devices complete after 95.846 msecs
PM: suspend devices took 0.100 seconds
PM: late suspend of devices complete after 5.395 msecs
PM: noirq suspend of devices complete after 4.914 msecs
Disabling non-boot CPUs ...
CPU1: shutdown
CPU2: shutdown
CPU3: shutdown
Enabling non-boot CPUs ...
CPU1: Booted secondary processor
CPU1 is up
CPU2: Booted secondary processor
CPU2 is up
CPU3: Booted secondary processor
CPU3 is up
PM: noirq resume of devices complete after 1556.385 msecs
PM: early resume of devices complete after 3.134 msecs
PM: resume of devices complete after 153.568 msecs
PM: resume devices took 0.160 seconds
Restarting tasks ... done.
ata1: SATA link down (SStatus 0 SControl 300)
fec 2188000.ethernet eth0: Link is Down
fec 2188000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
root at XXX:~ echo mem > /sys/power/state
PM: Syncing filesystems ... done.
Freezing user space processes ... (elapsed 0.002 seconds) done.
Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
Suspending console(s) (use no_console_suspend to debug)
PM: suspend of devices complete after 87.638 msecs
PM: suspend devices took 0.090 seconds
PM: late suspend of devices complete after 4.528 msecs
PM: noirq suspend of devices complete after 5.031 msecs
Disabling non-boot CPUs ...
CPU1: shutdown
CPU2: shutdown
CPU3: shutdown
Enabling non-boot CPUs ...
CPU1: Booted secondary processor
CPU1 is up
CPU2: Booted secondary processor
CPU2 is up
CPU3: Booted secondary processor
CPU3 is up
PM: noirq resume of devices complete after 1425.302 msecs
PM: early resume of devices complete after 2.736 msecs
PM: resume of devices complete after 156.013 msecs
PM: resume devices took 0.160 seconds
Restarting tasks ... done.
ata1: SATA link down (SStatus 0 SControl 300)
fec 2188000.ethernet eth0: Link is Down
fec 2188000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
root at XXX:~

-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Hong-Xing.Zhu at freescale.com]

> -----Original Message-----
> From: Lucas Stach [mailto:l.stach at pengutronix.de]
> Sent: Thursday, July 17, 2014 9:55 PM
> To: Guo Shawn-R65073
> Cc: Zhu Richard-R65037; linux-pci at vger.kernel.org; Sascha Hauer; Bjorn Helgaas;
> Shawn Guo; Fabio Estevam; linux-arm-kernel at lists.infradead.org
> Subject: Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe
> 
> Hi Shawn,
> 
> Am Mittwoch, den 16.07.2014, 14:55 +0800 schrieb Shawn Guo:
> > On Mon, Jul 07, 2014 at 09:55:03PM +0800, Shawn Guo wrote:
> > > On Mon, Jul 07, 2014 at 11:10:51AM +0200, Lucas Stach wrote:
> > > > Hi Shawn,
> > > >
> > > > Over the weekend I tried to reproduce your problem on a SabreSD
> > > > board, but wasn't able to trigger the issue. 3.16-rc3 with PCIe
> > > > active works just fine over a suspend and resume cycle for me.
> > >
> > > That's strange.  In my setup, PCIe support is enabled in kernel and
> > > DT, but I do not have a PCIe device connected to the board.
> > >
> > > >
> > > > One possibly relevant difference is that I've booted with NFSroot,
> > > > while it seems you are using a SATA connected device. Is this right?
> > >
> > > I have a SATA disk connected, but did boot with NFSroot.
> > >
> > > > If so,
> > > > can you test if it works if you boot from SDcard or the like? This
> > > > might be relevant as PCIe and SATA share some clocks.
> > >
> > > I tried to disable SATA support completely, but it doesn't help.
> >
> > Lucas, any news on this?  Or should we just try to use Richard's patch
> > to solve the problem?
> 
> I would like to understand the problem first before throwing "fixes" at the
> issue. As you said this isn't a regression we are in no hurry and should try
> to analyze the issue properly. I just retested and I'm still not able to
> reproduce the issue on my SabreSD. Maybe there are board revision that exhibit
> different behavior? My board has two sticks on saying "Rev B3" and "Rev X3".
> 
> As you can see from the log suspend/resume is working fine for me with kernel
> 3.16-rc5 + imx_v6_v7_defconfig.
[Richard] As I know that imx6 pcie is not enabled in imx_v6_v7_defconfig in default.
Menu-config is required if you want to tests system suspend/resume with pcie built-in.
Just double confirm, is the pcie built-in at your side?

> 
> root at XXX:~ echo mem > /sys/power/state
> PM: Syncing filesystems ... done.
> Freezing user space processes ... (elapsed 0.004 seconds) done.
> Freezing remaining freezable tasks ... (elapsed 0.002 seconds) done.
> Suspending console(s) (use no_console_suspend to debug)
> PM: suspend of devices complete after 95.846 msecs
> PM: suspend devices took 0.100 seconds
> PM: late suspend of devices complete after 5.395 msecs
> PM: noirq suspend of devices complete after 4.914 msecs Disabling non-boot
> CPUs ...
> CPU1: shutdown
> CPU2: shutdown
> CPU3: shutdown
> Enabling non-boot CPUs ...
> CPU1: Booted secondary processor
> CPU1 is up
> CPU2: Booted secondary processor
> CPU2 is up
> CPU3: Booted secondary processor
> CPU3 is up
> PM: noirq resume of devices complete after 1556.385 msecs
> PM: early resume of devices complete after 3.134 msecs
> PM: resume of devices complete after 153.568 msecs
> PM: resume devices took 0.160 seconds
> Restarting tasks ... done.
> ata1: SATA link down (SStatus 0 SControl 300) fec 2188000.ethernet eth0: Link
> is Down fec 2188000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
> root at XXX:~ echo mem > /sys/power/state
> PM: Syncing filesystems ... done.
> Freezing user space processes ... (elapsed 0.002 seconds) done.
> Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
> Suspending console(s) (use no_console_suspend to debug)
> PM: suspend of devices complete after 87.638 msecs
> PM: suspend devices took 0.090 seconds
> PM: late suspend of devices complete after 4.528 msecs
> PM: noirq suspend of devices complete after 5.031 msecs Disabling non-boot
> CPUs ...
> CPU1: shutdown
> CPU2: shutdown
> CPU3: shutdown
> Enabling non-boot CPUs ...
> CPU1: Booted secondary processor
> CPU1 is up
> CPU2: Booted secondary processor
> CPU2 is up
> CPU3: Booted secondary processor
> CPU3 is up
> PM: noirq resume of devices complete after 1425.302 msecs
> PM: early resume of devices complete after 2.736 msecs
> PM: resume of devices complete after 156.013 msecs
> PM: resume devices took 0.160 seconds
> Restarting tasks ... done.
> ata1: SATA link down (SStatus 0 SControl 300) fec 2188000.ethernet eth0: Link
> is Down fec 2188000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
> root at XXX:~
> 
> --
> Pengutronix e.K.             | Lucas Stach                 |
> Industrial Linux Solutions   | http://www.pengutronix.de/  |


Best Regards
Richard Zhu

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Lucas Stach]

Am Freitag, den 18.07.2014, 02:11 +0000 schrieb
Hong-Xing.Zhu at freescale.com:
> > -----Original Message-----
> > From: Lucas Stach [mailto:l.stach at pengutronix.de]
> > Sent: Thursday, July 17, 2014 9:55 PM
> > To: Guo Shawn-R65073
> > Cc: Zhu Richard-R65037; linux-pci at vger.kernel.org; Sascha Hauer; Bjorn Helgaas;
> > Shawn Guo; Fabio Estevam; linux-arm-kernel at lists.infradead.org
> > Subject: Re: The imx6q suspend/resume is broken on 3.16-rc due to PCIe
> > 
> > Hi Shawn,
> > 
> > Am Mittwoch, den 16.07.2014, 14:55 +0800 schrieb Shawn Guo:
> > > On Mon, Jul 07, 2014 at 09:55:03PM +0800, Shawn Guo wrote:
> > > > On Mon, Jul 07, 2014 at 11:10:51AM +0200, Lucas Stach wrote:
> > > > > Hi Shawn,
> > > > >
> > > > > Over the weekend I tried to reproduce your problem on a SabreSD
> > > > > board, but wasn't able to trigger the issue. 3.16-rc3 with PCIe
> > > > > active works just fine over a suspend and resume cycle for me.
> > > >
> > > > That's strange.  In my setup, PCIe support is enabled in kernel and
> > > > DT, but I do not have a PCIe device connected to the board.
> > > >
> > > > >
> > > > > One possibly relevant difference is that I've booted with NFSroot,
> > > > > while it seems you are using a SATA connected device. Is this right?
> > > >
> > > > I have a SATA disk connected, but did boot with NFSroot.
> > > >
> > > > > If so,
> > > > > can you test if it works if you boot from SDcard or the like? This
> > > > > might be relevant as PCIe and SATA share some clocks.
> > > >
> > > > I tried to disable SATA support completely, but it doesn't help.
> > >
> > > Lucas, any news on this?  Or should we just try to use Richard's patch
> > > to solve the problem?
> > 
> > I would like to understand the problem first before throwing "fixes" at the
> > issue. As you said this isn't a regression we are in no hurry and should try
> > to analyze the issue properly. I just retested and I'm still not able to
> > reproduce the issue on my SabreSD. Maybe there are board revision that exhibit
> > different behavior? My board has two sticks on saying "Rev B3" and "Rev X3".
> > 
> > As you can see from the log suspend/resume is working fine for me with kernel
> > 3.16-rc5 + imx_v6_v7_defconfig.
> [Richard] As I know that imx6 pcie is not enabled in imx_v6_v7_defconfig in default.
> Menu-config is required if you want to tests system suspend/resume with pcie built-in.
> Just double confirm, is the pcie built-in at your side?

This is not right, PCI is enabled in imx_v6_v7_defconfig since
c0bea59ca58e30fb8fd29254569bdaae482398ad "ARM: imx_v6_v7_defconfig:
Select PCI support".

However I seem to have a board with rev 1.1 silicon and although the pci
driver starts up it never establishes a link. So my board may just work
by chance as I don't really know the differences between rev 1.1 silicon
and later revisions.

Regards,
Lucas
-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Fri, Jul 18, 2014 at 11:21:48AM +0200, Lucas Stach wrote:
> However I seem to have a board with rev 1.1 silicon and although the pci
> driver starts up it never establishes a link. So my board may just work
> by chance as I don't really know the differences between rev 1.1 silicon
> and later revisions.

Lucas, thanks for the info.

I just checked my board on which the issue is seen, and found it has a
rev 1.3 silicon.  I just tested on a board with rev 1.2 and did not even
see the issue.

Richard,

Is this an issue only with rev 1.3?

Shawn

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Fabio Estevam]

On Sun, Jul 20, 2014 at 11:50 PM, Shawn Guo <shawn.guo@freescale.com> wrote:
> On Fri, Jul 18, 2014 at 11:21:48AM +0200, Lucas Stach wrote:
>> However I seem to have a board with rev 1.1 silicon and although the pci
>> driver starts up it never establishes a link. So my board may just work
>> by chance as I don't really know the differences between rev 1.1 silicon
>> and later revisions.
>
> Lucas, thanks for the info.
>
> I just checked my board on which the issue is seen, and found it has a
> rev 1.3 silicon.  I just tested on a board with rev 1.2 and did not even
> see the issue.
>
> Richard,
>
> Is this an issue only with rev 1.3?

I could see this issue on my board with rev 1.2.

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Sun, Jul 20, 2014 at 11:55:42PM -0300, Fabio Estevam wrote:
> On Sun, Jul 20, 2014 at 11:50 PM, Shawn Guo <shawn.guo@freescale.com> wrote:
> > On Fri, Jul 18, 2014 at 11:21:48AM +0200, Lucas Stach wrote:
> >> However I seem to have a board with rev 1.1 silicon and although the pci
> >> driver starts up it never establishes a link. So my board may just work
> >> by chance as I don't really know the differences between rev 1.1 silicon
> >> and later revisions.
> >
> > Lucas, thanks for the info.
> >
> > I just checked my board on which the issue is seen, and found it has a
> > rev 1.3 silicon.  I just tested on a board with rev 1.2 and did not even
> > see the issue.
> >
> > Richard,
> >
> > Is this an issue only with rev 1.3?
> 
> I could see this issue on my board with rev 1.2.

Hmm, that's strange.  I tested two imx6q-sabresd boards with rev 1.2
silicon, and did not see this issue.  Hopefully, someone else can also
test.

Shawn

The imx6q suspend/resume is broken on 3.16-rc due to PCIe

[Shawn Guo]

On Wed, Jun 25, 2014 at 02:22:44PM +0800, Shawn Guo wrote:
> On Tue, Jun 24, 2014 at 04:37:19PM -0300, Fabio Estevam wrote:
> > On Tue, Jun 24, 2014 at 12:36 PM, Shawn Guo <shawn.guo@linaro.org> wrote:
> > > Hi Lucas,
> > >
> > > I found that imx6q suspend/resume is broken on 3.16-rc, and it works
> > > just fine if PCIe support is built out.  However the issue was not
> > > exposed on 3.15, so I suspect the PCIe driver change merged in 3.16
> > > is part of the cause.
> > 
> > I have just tested this on linux-next and I see that suspend/resume
> > fails even if I build a kernel without PCI support.
> 
> Please test against v3.16-rc1 or -rc2.  The breakage on linux-next
> should be another issue.

Yeah, the issue on linux-next is caused by commit 478850160636 (irq_work:
Implement remote queueing).  And Stephen reported the issue as below.

https://lkml.org/lkml/2014/6/24/765

Shawn