[PATCH v3 0/5] Enabling the asynchronous threads for other phases

[PATCH v3 0/5] Enabling the asynchronous threads for other phases

[Chuansheng Liu]

Hello,

This patch series are for enabling the asynchronous threads for the phases
resume_noirq, resume_early, suspend_noirq and suspend_late.

Just like commit 5af84b82701a and 97df8c12995, with async threads it will
reduce the system suspending and resuming time significantly.

With these patches, in my test platform, it saved 80% time in resume_noirq
phase.

Best Regards,

---
V2:
 -- Based on Rafael's minor changes related to coding style, white space etc;
 -- Rafael pointed out the v1 series break the device parent-children
    suspending/resuming order when enabling asyncing, here using the
    dev completion to sync parent-children order;

V3:
 -- In patch v2 5/5, there is one missing "dpm_wait_for_children";

---

[PATCH v3 1/5] PM / sleep: Two flags for async suspend_noirq and
[PATCH v3 2/5] PM / sleep: Asynchronous threads for resume_noirq
[PATCH v3 3/5] PM / sleep: Asynchronous threads for resume_early
[PATCH v3 4/5] PM / sleep: Asynchronous threads for suspend_noirq
[PATCH v3 5/5] PM / sleep: Asynchronous threads for suspend_late

 drivers/base/power/main.c | 275 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------------
 include/linux/pm.h        |   2 ++
 2 files changed, 227 insertions(+), 50 deletions(-)

[PATCH v3 1/5] PM / sleep: Two flags for async suspend_noirq and suspend_late

[Chuansheng Liu]

The patch is a helper adding two new flags for implementing
async threads for suspend_noirq and suspend_late.

Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
---
 drivers/base/power/main.c | 24 ++++++++++++++++++++++--
 include/linux/pm.h        |  2 ++
 2 files changed, 24 insertions(+), 2 deletions(-)

diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
index 1b41fca..00c53eb 100644
--- a/drivers/base/power/main.c
+++ b/drivers/base/power/main.c
@@ -91,6 +91,8 @@ void device_pm_sleep_init(struct device *dev)
 {
 	dev->power.is_prepared = false;
 	dev->power.is_suspended = false;
+	dev->power.is_noirq_suspended = false;
+	dev->power.is_late_suspended = false;
 	init_completion(&dev->power.completion);
 	complete_all(&dev->power.completion);
 	dev->power.wakeup = NULL;
@@ -479,6 +481,9 @@ static int device_resume_noirq(struct device *dev, pm_message_t state)
 	if (dev->power.syscore)
 		goto Out;
 
+	if (!dev->power.is_noirq_suspended)
+		goto Out;
+
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
 		callback = pm_noirq_op(&dev->pm_domain->ops, state);
@@ -499,6 +504,7 @@ static int device_resume_noirq(struct device *dev, pm_message_t state)
 	}
 
 	error = dpm_run_callback(callback, dev, state, info);
+	dev->power.is_noirq_suspended = false;
 
  Out:
 	TRACE_RESUME(error);
@@ -561,6 +567,9 @@ static int device_resume_early(struct device *dev, pm_message_t state)
 	if (dev->power.syscore)
 		goto Out;
 
+	if (!dev->power.is_late_suspended)
+		goto Out;
+
 	if (dev->pm_domain) {
 		info = "early power domain ";
 		callback = pm_late_early_op(&dev->pm_domain->ops, state);
@@ -581,6 +590,7 @@ static int device_resume_early(struct device *dev, pm_message_t state)
 	}
 
 	error = dpm_run_callback(callback, dev, state, info);
+	dev->power.is_late_suspended = false;
 
  Out:
 	TRACE_RESUME(error);
@@ -917,6 +927,7 @@ static int device_suspend_noirq(struct device *dev, pm_message_t state)
 {
 	pm_callback_t callback = NULL;
 	char *info = NULL;
+	int error;
 
 	if (dev->power.syscore)
 		return 0;
@@ -940,7 +951,11 @@ static int device_suspend_noirq(struct device *dev, pm_message_t state)
 		callback = pm_noirq_op(dev->driver->pm, state);
 	}
 
-	return dpm_run_callback(callback, dev, state, info);
+	error = dpm_run_callback(callback, dev, state, info);
+	if (!error)
+		dev->power.is_noirq_suspended = true;
+
+	return error;
 }
 
 /**
@@ -1003,6 +1018,7 @@ static int device_suspend_late(struct device *dev, pm_message_t state)
 {
 	pm_callback_t callback = NULL;
 	char *info = NULL;
+	int error;
 
 	__pm_runtime_disable(dev, false);
 
@@ -1028,7 +1044,11 @@ static int device_suspend_late(struct device *dev, pm_message_t state)
 		callback = pm_late_early_op(dev->driver->pm, state);
 	}
 
-	return dpm_run_callback(callback, dev, state, info);
+	error = dpm_run_callback(callback, dev, state, info);
+	if (!error)
+		dev->power.is_late_suspended = true;
+
+	return error;
 }
 
 /**
diff --git a/include/linux/pm.h b/include/linux/pm.h
index 8c6583a..f23a4f1 100644
--- a/include/linux/pm.h
+++ b/include/linux/pm.h
@@ -542,6 +542,8 @@ struct dev_pm_info {
 	unsigned int		async_suspend:1;
 	bool			is_prepared:1;	/* Owned by the PM core */
 	bool			is_suspended:1;	/* Ditto */
+	bool			is_noirq_suspended:1;
+	bool			is_late_suspended:1;
 	bool			ignore_children:1;
 	bool			early_init:1;	/* Owned by the PM core */
 	spinlock_t		lock;
-- 
1.9.rc0

[PATCH v3 2/5] PM / sleep: Asynchronous threads for resume_noirq

[Chuansheng Liu]

In analogy with commits 5af84b82701a and 97df8c12995, using
asynchronous threads can improve the overall resume_noirq time
significantly.

One typical case is:
In resume_noirq phase and for the PCI devices, the function
pci_pm_resume_noirq() will be called, and there is one d3_delay
(10ms) at least.

With the way of asynchronous threads, we just need wait d3_delay
time once in parallel for each calling, which saves much time to
resume quickly.

Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
---
 drivers/base/power/main.c | 66 +++++++++++++++++++++++++++++++++++------------
 1 file changed, 50 insertions(+), 16 deletions(-)

diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
index 00c53eb..89172aa 100644
--- a/drivers/base/power/main.c
+++ b/drivers/base/power/main.c
@@ -469,7 +469,7 @@ static void dpm_watchdog_clear(struct dpm_watchdog *wd)
  * The driver of @dev will not receive interrupts while this function is being
  * executed.
  */
-static int device_resume_noirq(struct device *dev, pm_message_t state)
+static int device_resume_noirq(struct device *dev, pm_message_t state, bool async)
 {
 	pm_callback_t callback = NULL;
 	char *info = NULL;
@@ -481,6 +481,8 @@ static int device_resume_noirq(struct device *dev, pm_message_t state)
 	if (dev->power.syscore)
 		goto Out;
 
+	dpm_wait(dev->parent, async);
+
 	if (!dev->power.is_noirq_suspended)
 		goto Out;
 
@@ -507,10 +509,29 @@ static int device_resume_noirq(struct device *dev, pm_message_t state)
 	dev->power.is_noirq_suspended = false;
 
  Out:
+	complete_all(&dev->power.completion);
 	TRACE_RESUME(error);
 	return error;
 }
 
+static bool is_async(struct device *dev)
+{
+	return dev->power.async_suspend && pm_async_enabled
+		&& !pm_trace_is_enabled();
+}
+
+static void async_resume_noirq(void *data, async_cookie_t cookie)
+{
+	struct device *dev = (struct device *)data;
+	int error;
+
+	error = device_resume_noirq(dev, pm_transition, true);
+	if (error)
+		pm_dev_err(dev, pm_transition, " async", error);
+
+	put_device(dev);
+}
+
 /**
  * dpm_resume_noirq - Execute "noirq resume" callbacks for all devices.
  * @state: PM transition of the system being carried out.
@@ -520,29 +541,48 @@ static int device_resume_noirq(struct device *dev, pm_message_t state)
  */
 static void dpm_resume_noirq(pm_message_t state)
 {
+	struct device *dev;
 	ktime_t starttime = ktime_get();
 
 	mutex_lock(&dpm_list_mtx);
-	while (!list_empty(&dpm_noirq_list)) {
-		struct device *dev = to_device(dpm_noirq_list.next);
-		int error;
+	pm_transition = state;
+
+	/*
+	 * Advanced the async threads upfront,
+	 * in case the starting of async threads is
+	 * delayed by non-async resuming devices.
+	 */
+	list_for_each_entry(dev, &dpm_noirq_list, power.entry) {
+		reinit_completion(&dev->power.completion);
+		if (is_async(dev)) {
+			get_device(dev);
+			async_schedule(async_resume_noirq, dev);
+		}
+	}
 
+	while (!list_empty(&dpm_noirq_list)) {
+		dev = to_device(dpm_noirq_list.next);
 		get_device(dev);
 		list_move_tail(&dev->power.entry, &dpm_late_early_list);
 		mutex_unlock(&dpm_list_mtx);
 
-		error = device_resume_noirq(dev, state);
-		if (error) {
-			suspend_stats.failed_resume_noirq++;
-			dpm_save_failed_step(SUSPEND_RESUME_NOIRQ);
-			dpm_save_failed_dev(dev_name(dev));
-			pm_dev_err(dev, state, " noirq", error);
+		if (!is_async(dev)) {
+			int error;
+
+			error = device_resume_noirq(dev, state, false);
+			if (error) {
+				suspend_stats.failed_resume_noirq++;
+				dpm_save_failed_step(SUSPEND_RESUME_NOIRQ);
+				dpm_save_failed_dev(dev_name(dev));
+				pm_dev_err(dev, state, " noirq", error);
+			}
 		}
 
 		mutex_lock(&dpm_list_mtx);
 		put_device(dev);
 	}
 	mutex_unlock(&dpm_list_mtx);
+	async_synchronize_full();
 	dpm_show_time(starttime, state, "noirq");
 	resume_device_irqs();
 	cpuidle_resume();
@@ -742,12 +782,6 @@ static void async_resume(void *data, async_cookie_t cookie)
 	put_device(dev);
 }
 
-static bool is_async(struct device *dev)
-{
-	return dev->power.async_suspend && pm_async_enabled
-		&& !pm_trace_is_enabled();
-}
-
 /**
  * dpm_resume - Execute "resume" callbacks for non-sysdev devices.
  * @state: PM transition of the system being carried out.
-- 
1.9.rc0

Re: [PATCH v3 2/5] PM / sleep: Asynchronous threads for resume_noirq

[Rafael J. Wysocki]

On Monday, February 17, 2014 02:19:11 PM Chuansheng Liu wrote:
> In analogy with commits 5af84b82701a and 97df8c12995, using
> asynchronous threads can improve the overall resume_noirq time
> significantly.
> 
> One typical case is:
> In resume_noirq phase and for the PCI devices, the function
> pci_pm_resume_noirq() will be called, and there is one d3_delay
> (10ms) at least.
> 
> With the way of asynchronous threads, we just need wait d3_delay
> time once in parallel for each calling, which saves much time to
> resume quickly.
> 
> Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
> ---
>  drivers/base/power/main.c | 66 +++++++++++++++++++++++++++++++++++------------
>  1 file changed, 50 insertions(+), 16 deletions(-)
> 
> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> index 00c53eb..89172aa 100644
> --- a/drivers/base/power/main.c
> +++ b/drivers/base/power/main.c
> @@ -469,7 +469,7 @@ static void dpm_watchdog_clear(struct dpm_watchdog *wd)
>   * The driver of @dev will not receive interrupts while this function is being
>   * executed.
>   */
> -static int device_resume_noirq(struct device *dev, pm_message_t state)
> +static int device_resume_noirq(struct device *dev, pm_message_t state, bool async)
>  {
>  	pm_callback_t callback = NULL;
>  	char *info = NULL;
> @@ -481,6 +481,8 @@ static int device_resume_noirq(struct device *dev, pm_message_t state)
>  	if (dev->power.syscore)
>  		goto Out;
>  
> +	dpm_wait(dev->parent, async);
> +

That is too early to wait.  Please move it under the next check.

>  	if (!dev->power.is_noirq_suspended)
>  		goto Out;
>  
> @@ -507,10 +509,29 @@ static int device_resume_noirq(struct device *dev, pm_message_t state)
>  	dev->power.is_noirq_suspended = false;
>  
>   Out:
> +	complete_all(&dev->power.completion);
>  	TRACE_RESUME(error);
>  	return error;
>  }
>  
> +static bool is_async(struct device *dev)
> +{
> +	return dev->power.async_suspend && pm_async_enabled
> +		&& !pm_trace_is_enabled();
> +}
> +
> +static void async_resume_noirq(void *data, async_cookie_t cookie)
> +{
> +	struct device *dev = (struct device *)data;
> +	int error;
> +
> +	error = device_resume_noirq(dev, pm_transition, true);
> +	if (error)
> +		pm_dev_err(dev, pm_transition, " async", error);
> +
> +	put_device(dev);
> +}
> +
>  /**
>   * dpm_resume_noirq - Execute "noirq resume" callbacks for all devices.
>   * @state: PM transition of the system being carried out.
> @@ -520,29 +541,48 @@ static int device_resume_noirq(struct device *dev, pm_message_t state)
>   */
>  static void dpm_resume_noirq(pm_message_t state)
>  {
> +	struct device *dev;
>  	ktime_t starttime = ktime_get();
>  
>  	mutex_lock(&dpm_list_mtx);
> -	while (!list_empty(&dpm_noirq_list)) {
> -		struct device *dev = to_device(dpm_noirq_list.next);
> -		int error;
> +	pm_transition = state;
> +
> +	/*
> +	 * Advanced the async threads upfront,
> +	 * in case the starting of async threads is
> +	 * delayed by non-async resuming devices.
> +	 */
> +	list_for_each_entry(dev, &dpm_noirq_list, power.entry) {
> +		reinit_completion(&dev->power.completion);
> +		if (is_async(dev)) {
> +			get_device(dev);
> +			async_schedule(async_resume_noirq, dev);
> +		}
> +	}
>  
> +	while (!list_empty(&dpm_noirq_list)) {
> +		dev = to_device(dpm_noirq_list.next);
>  		get_device(dev);
>  		list_move_tail(&dev->power.entry, &dpm_late_early_list);
>  		mutex_unlock(&dpm_list_mtx);
>  
> -		error = device_resume_noirq(dev, state);
> -		if (error) {
> -			suspend_stats.failed_resume_noirq++;
> -			dpm_save_failed_step(SUSPEND_RESUME_NOIRQ);
> -			dpm_save_failed_dev(dev_name(dev));
> -			pm_dev_err(dev, state, " noirq", error);
> +		if (!is_async(dev)) {
> +			int error;
> +
> +			error = device_resume_noirq(dev, state, false);
> +			if (error) {
> +				suspend_stats.failed_resume_noirq++;
> +				dpm_save_failed_step(SUSPEND_RESUME_NOIRQ);
> +				dpm_save_failed_dev(dev_name(dev));
> +				pm_dev_err(dev, state, " noirq", error);
> +			}
>  		}
>  
>  		mutex_lock(&dpm_list_mtx);
>  		put_device(dev);
>  	}
>  	mutex_unlock(&dpm_list_mtx);
> +	async_synchronize_full();
>  	dpm_show_time(starttime, state, "noirq");
>  	resume_device_irqs();
>  	cpuidle_resume();
> @@ -742,12 +782,6 @@ static void async_resume(void *data, async_cookie_t cookie)
>  	put_device(dev);
>  }
>  
> -static bool is_async(struct device *dev)
> -{
> -	return dev->power.async_suspend && pm_async_enabled
> -		&& !pm_trace_is_enabled();
> -}
> -
>  /**
>   * dpm_resume - Execute "resume" callbacks for non-sysdev devices.
>   * @state: PM transition of the system being carried out.
> 

-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

[PATCH v3 3/5] PM / sleep: Asynchronous threads for resume_early

[Chuansheng Liu]

In analogy with commits 5af84b82701a and 97df8c12995, using
asynchronous threads can improve the overall resume_early
time significantly.

This patch is for resume_early phase.

Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
---
 drivers/base/power/main.c | 55 +++++++++++++++++++++++++++++++++++++----------
 1 file changed, 44 insertions(+), 11 deletions(-)

diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
index 89172aa..2f2d110 100644
--- a/drivers/base/power/main.c
+++ b/drivers/base/power/main.c
@@ -595,7 +595,7 @@ static void dpm_resume_noirq(pm_message_t state)
  *
  * Runtime PM is disabled for @dev while this function is being executed.
  */
-static int device_resume_early(struct device *dev, pm_message_t state)
+static int device_resume_early(struct device *dev, pm_message_t state, bool async)
 {
 	pm_callback_t callback = NULL;
 	char *info = NULL;
@@ -607,6 +607,8 @@ static int device_resume_early(struct device *dev, pm_message_t state)
 	if (dev->power.syscore)
 		goto Out;
 
+	dpm_wait(dev->parent, async);
+
 	if (!dev->power.is_late_suspended)
 		goto Out;
 
@@ -636,38 +638,69 @@ static int device_resume_early(struct device *dev, pm_message_t state)
 	TRACE_RESUME(error);
 
 	pm_runtime_enable(dev);
+	complete_all(&dev->power.completion);
 	return error;
 }
 
+static void async_resume_early(void *data, async_cookie_t cookie)
+{
+	struct device *dev = (struct device *)data;
+	int error;
+
+	error = device_resume_early(dev, pm_transition, true);
+	if (error)
+		pm_dev_err(dev, pm_transition, " async", error);
+
+	put_device(dev);
+}
+
 /**
  * dpm_resume_early - Execute "early resume" callbacks for all devices.
  * @state: PM transition of the system being carried out.
  */
 static void dpm_resume_early(pm_message_t state)
 {
+	struct device *dev;
 	ktime_t starttime = ktime_get();
 
 	mutex_lock(&dpm_list_mtx);
-	while (!list_empty(&dpm_late_early_list)) {
-		struct device *dev = to_device(dpm_late_early_list.next);
-		int error;
+	pm_transition = state;
+
+	/*
+	 * Advanced the async threads upfront,
+	 * in case the starting of async threads is
+	 * delayed by non-async resuming devices.
+	 */
+	list_for_each_entry(dev, &dpm_late_early_list, power.entry) {
+		reinit_completion(&dev->power.completion);
+		if (is_async(dev)) {
+			get_device(dev);
+			async_schedule(async_resume_early, dev);
+		}
+	}
 
+	while (!list_empty(&dpm_late_early_list)) {
+		dev = to_device(dpm_late_early_list.next);
 		get_device(dev);
 		list_move_tail(&dev->power.entry, &dpm_suspended_list);
 		mutex_unlock(&dpm_list_mtx);
 
-		error = device_resume_early(dev, state);
-		if (error) {
-			suspend_stats.failed_resume_early++;
-			dpm_save_failed_step(SUSPEND_RESUME_EARLY);
-			dpm_save_failed_dev(dev_name(dev));
-			pm_dev_err(dev, state, " early", error);
-		}
+		if (!is_async(dev)) {
+			int error;
 
+			error = device_resume_early(dev, state, false);
+			if (error) {
+				suspend_stats.failed_resume_early++;
+				dpm_save_failed_step(SUSPEND_RESUME_EARLY);
+				dpm_save_failed_dev(dev_name(dev));
+				pm_dev_err(dev, state, " early", error);
+			}
+		}
 		mutex_lock(&dpm_list_mtx);
 		put_device(dev);
 	}
 	mutex_unlock(&dpm_list_mtx);
+	async_synchronize_full();
 	dpm_show_time(starttime, state, "early");
 }
 
-- 
1.9.rc0

Re: [PATCH v3 3/5] PM / sleep: Asynchronous threads for resume_early

[Rafael J. Wysocki]

On Monday, February 17, 2014 02:19:12 PM Chuansheng Liu wrote:
> In analogy with commits 5af84b82701a and 97df8c12995, using
> asynchronous threads can improve the overall resume_early
> time significantly.
> 
> This patch is for resume_early phase.
> 
> Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
> ---
>  drivers/base/power/main.c | 55 +++++++++++++++++++++++++++++++++++++----------
>  1 file changed, 44 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> index 89172aa..2f2d110 100644
> --- a/drivers/base/power/main.c
> +++ b/drivers/base/power/main.c
> @@ -595,7 +595,7 @@ static void dpm_resume_noirq(pm_message_t state)
>   *
>   * Runtime PM is disabled for @dev while this function is being executed.
>   */
> -static int device_resume_early(struct device *dev, pm_message_t state)
> +static int device_resume_early(struct device *dev, pm_message_t state, bool async)
>  {
>  	pm_callback_t callback = NULL;
>  	char *info = NULL;
> @@ -607,6 +607,8 @@ static int device_resume_early(struct device *dev, pm_message_t state)
>  	if (dev->power.syscore)
>  		goto Out;
>  
> +	dpm_wait(dev->parent, async);
> +

If power.is_late_suspended is unset, you don't have to wait.

>  	if (!dev->power.is_late_suspended)
>  		goto Out;
>  
> @@ -636,38 +638,69 @@ static int device_resume_early(struct device *dev, pm_message_t state)
>  	TRACE_RESUME(error);
>  
>  	pm_runtime_enable(dev);
> +	complete_all(&dev->power.completion);
>  	return error;
>  }
>  
> +static void async_resume_early(void *data, async_cookie_t cookie)
> +{
> +	struct device *dev = (struct device *)data;
> +	int error;
> +
> +	error = device_resume_early(dev, pm_transition, true);
> +	if (error)
> +		pm_dev_err(dev, pm_transition, " async", error);
> +
> +	put_device(dev);
> +}
> +
>  /**
>   * dpm_resume_early - Execute "early resume" callbacks for all devices.
>   * @state: PM transition of the system being carried out.
>   */
>  static void dpm_resume_early(pm_message_t state)
>  {
> +	struct device *dev;
>  	ktime_t starttime = ktime_get();
>  
>  	mutex_lock(&dpm_list_mtx);
> -	while (!list_empty(&dpm_late_early_list)) {
> -		struct device *dev = to_device(dpm_late_early_list.next);
> -		int error;
> +	pm_transition = state;
> +
> +	/*
> +	 * Advanced the async threads upfront,
> +	 * in case the starting of async threads is
> +	 * delayed by non-async resuming devices.
> +	 */
> +	list_for_each_entry(dev, &dpm_late_early_list, power.entry) {
> +		reinit_completion(&dev->power.completion);
> +		if (is_async(dev)) {
> +			get_device(dev);
> +			async_schedule(async_resume_early, dev);
> +		}
> +	}
>  
> +	while (!list_empty(&dpm_late_early_list)) {
> +		dev = to_device(dpm_late_early_list.next);
>  		get_device(dev);
>  		list_move_tail(&dev->power.entry, &dpm_suspended_list);
>  		mutex_unlock(&dpm_list_mtx);
>  
> -		error = device_resume_early(dev, state);
> -		if (error) {
> -			suspend_stats.failed_resume_early++;
> -			dpm_save_failed_step(SUSPEND_RESUME_EARLY);
> -			dpm_save_failed_dev(dev_name(dev));
> -			pm_dev_err(dev, state, " early", error);
> -		}
> +		if (!is_async(dev)) {
> +			int error;
>  
> +			error = device_resume_early(dev, state, false);
> +			if (error) {
> +				suspend_stats.failed_resume_early++;
> +				dpm_save_failed_step(SUSPEND_RESUME_EARLY);
> +				dpm_save_failed_dev(dev_name(dev));
> +				pm_dev_err(dev, state, " early", error);
> +			}
> +		}
>  		mutex_lock(&dpm_list_mtx);
>  		put_device(dev);
>  	}
>  	mutex_unlock(&dpm_list_mtx);
> +	async_synchronize_full();
>  	dpm_show_time(starttime, state, "early");
>  }
>  
> 

-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

[PATCH v3 4/5] PM / sleep: Asynchronous threads for suspend_noirq

[Chuansheng Liu]

In analogy with commits 5af84b82701a and 97df8c12995, using
asynchronous threads can improve the overall suspend_noirq
time significantly.

This patch is for suspend_noirq phase.

Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
---
 drivers/base/power/main.c | 68 +++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 57 insertions(+), 11 deletions(-)

diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
index 2f2d110..72b4c9c 100644
--- a/drivers/base/power/main.c
+++ b/drivers/base/power/main.c
@@ -990,14 +990,24 @@ static pm_message_t resume_event(pm_message_t sleep_state)
  * The driver of @dev will not receive interrupts while this function is being
  * executed.
  */
-static int device_suspend_noirq(struct device *dev, pm_message_t state)
+static int __device_suspend_noirq(struct device *dev, pm_message_t state, bool async)
 {
 	pm_callback_t callback = NULL;
 	char *info = NULL;
-	int error;
+	int error = 0;
+
+	dpm_wait_for_children(dev, async);
+
+	if (async_error)
+		goto Complete;
+
+	if (pm_wakeup_pending()) {
+		async_error = -EBUSY;
+		goto Complete;
+	}
 
 	if (dev->power.syscore)
-		return 0;
+		goto Complete;
 
 	if (dev->pm_domain) {
 		info = "noirq power domain ";
@@ -1021,10 +1031,40 @@ static int device_suspend_noirq(struct device *dev, pm_message_t state)
 	error = dpm_run_callback(callback, dev, state, info);
 	if (!error)
 		dev->power.is_noirq_suspended = true;
+	else
+		async_error = error;
 
+Complete:
+	complete_all(&dev->power.completion);
 	return error;
 }
 
+static void async_suspend_noirq(void *data, async_cookie_t cookie)
+{
+	struct device *dev = (struct device *)data;
+	int error;
+
+	error = __device_suspend_noirq(dev, pm_transition, true);
+	if (error) {
+		dpm_save_failed_dev(dev_name(dev));
+		pm_dev_err(dev, pm_transition, " async", error);
+	}
+
+	put_device(dev);
+}
+
+static int device_suspend_noirq(struct device *dev)
+{
+	reinit_completion(&dev->power.completion);
+
+	if (pm_async_enabled && dev->power.async_suspend) {
+		get_device(dev);
+		async_schedule(async_suspend_noirq, dev);
+		return 0;
+	}
+	return __device_suspend_noirq(dev, pm_transition, false);
+}
+
 /**
  * dpm_suspend_noirq - Execute "noirq suspend" callbacks for all devices.
  * @state: PM transition of the system being carried out.
@@ -1040,19 +1080,20 @@ static int dpm_suspend_noirq(pm_message_t state)
 	cpuidle_pause();
 	suspend_device_irqs();
 	mutex_lock(&dpm_list_mtx);
+	pm_transition = state;
+	async_error = 0;
+
 	while (!list_empty(&dpm_late_early_list)) {
 		struct device *dev = to_device(dpm_late_early_list.prev);
 
 		get_device(dev);
 		mutex_unlock(&dpm_list_mtx);
 
-		error = device_suspend_noirq(dev, state);
+		error = device_suspend_noirq(dev);
 
 		mutex_lock(&dpm_list_mtx);
 		if (error) {
 			pm_dev_err(dev, state, " noirq", error);
-			suspend_stats.failed_suspend_noirq++;
-			dpm_save_failed_step(SUSPEND_SUSPEND_NOIRQ);
 			dpm_save_failed_dev(dev_name(dev));
 			put_device(dev);
 			break;
@@ -1061,16 +1102,21 @@ static int dpm_suspend_noirq(pm_message_t state)
 			list_move(&dev->power.entry, &dpm_noirq_list);
 		put_device(dev);
 
-		if (pm_wakeup_pending()) {
-			error = -EBUSY;
+		if (async_error)
 			break;
-		}
 	}
 	mutex_unlock(&dpm_list_mtx);
-	if (error)
+	async_synchronize_full();
+	if (!error)
+		error = async_error;
+
+	if (error) {
+		suspend_stats.failed_suspend_noirq++;
+		dpm_save_failed_step(SUSPEND_SUSPEND_NOIRQ);
 		dpm_resume_noirq(resume_event(state));
-	else
+	} else {
 		dpm_show_time(starttime, state, "noirq");
+	}
 	return error;
 }
 
-- 
1.9.rc0

Re: [PATCH v3 4/5] PM / sleep: Asynchronous threads for suspend_noirq

[Rafael J. Wysocki]

On Monday, February 17, 2014 02:19:13 PM Chuansheng Liu wrote:
> In analogy with commits 5af84b82701a and 97df8c12995, using
> asynchronous threads can improve the overall suspend_noirq
> time significantly.
> 
> This patch is for suspend_noirq phase.
> 
> Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
> ---
>  drivers/base/power/main.c | 68 +++++++++++++++++++++++++++++++++++++++--------
>  1 file changed, 57 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> index 2f2d110..72b4c9c 100644
> --- a/drivers/base/power/main.c
> +++ b/drivers/base/power/main.c
> @@ -990,14 +990,24 @@ static pm_message_t resume_event(pm_message_t sleep_state)
>   * The driver of @dev will not receive interrupts while this function is being
>   * executed.
>   */
> -static int device_suspend_noirq(struct device *dev, pm_message_t state)
> +static int __device_suspend_noirq(struct device *dev, pm_message_t state, bool async)
>  {
>  	pm_callback_t callback = NULL;
>  	char *info = NULL;
> -	int error;
> +	int error = 0;
> +
> +	dpm_wait_for_children(dev, async);

That is too early to wait.  All of the "goto Complete" statements can go before
that.

> +
> +	if (async_error)
> +		goto Complete;
> +
> +	if (pm_wakeup_pending()) {
> +		async_error = -EBUSY;
> +		goto Complete;
> +	}
>  
>  	if (dev->power.syscore)
> -		return 0;
> +		goto Complete;
>  
>  	if (dev->pm_domain) {
>  		info = "noirq power domain ";
> @@ -1021,10 +1031,40 @@ static int device_suspend_noirq(struct device *dev, pm_message_t state)
>  	error = dpm_run_callback(callback, dev, state, info);
>  	if (!error)
>  		dev->power.is_noirq_suspended = true;
> +	else
> +		async_error = error;
>  
> +Complete:
> +	complete_all(&dev->power.completion);
>  	return error;
>  }
>  
> +static void async_suspend_noirq(void *data, async_cookie_t cookie)
> +{
> +	struct device *dev = (struct device *)data;
> +	int error;
> +
> +	error = __device_suspend_noirq(dev, pm_transition, true);
> +	if (error) {
> +		dpm_save_failed_dev(dev_name(dev));
> +		pm_dev_err(dev, pm_transition, " async", error);
> +	}
> +
> +	put_device(dev);
> +}
> +
> +static int device_suspend_noirq(struct device *dev)
> +{
> +	reinit_completion(&dev->power.completion);
> +
> +	if (pm_async_enabled && dev->power.async_suspend) {
> +		get_device(dev);
> +		async_schedule(async_suspend_noirq, dev);
> +		return 0;
> +	}
> +	return __device_suspend_noirq(dev, pm_transition, false);
> +}
> +
>  /**
>   * dpm_suspend_noirq - Execute "noirq suspend" callbacks for all devices.
>   * @state: PM transition of the system being carried out.
> @@ -1040,19 +1080,20 @@ static int dpm_suspend_noirq(pm_message_t state)
>  	cpuidle_pause();
>  	suspend_device_irqs();
>  	mutex_lock(&dpm_list_mtx);
> +	pm_transition = state;
> +	async_error = 0;
> +
>  	while (!list_empty(&dpm_late_early_list)) {
>  		struct device *dev = to_device(dpm_late_early_list.prev);
>  
>  		get_device(dev);
>  		mutex_unlock(&dpm_list_mtx);
>  
> -		error = device_suspend_noirq(dev, state);
> +		error = device_suspend_noirq(dev);
>  
>  		mutex_lock(&dpm_list_mtx);
>  		if (error) {
>  			pm_dev_err(dev, state, " noirq", error);
> -			suspend_stats.failed_suspend_noirq++;
> -			dpm_save_failed_step(SUSPEND_SUSPEND_NOIRQ);
>  			dpm_save_failed_dev(dev_name(dev));
>  			put_device(dev);
>  			break;
> @@ -1061,16 +1102,21 @@ static int dpm_suspend_noirq(pm_message_t state)
>  			list_move(&dev->power.entry, &dpm_noirq_list);
>  		put_device(dev);
>  
> -		if (pm_wakeup_pending()) {
> -			error = -EBUSY;
> +		if (async_error)
>  			break;
> -		}
>  	}
>  	mutex_unlock(&dpm_list_mtx);
> -	if (error)
> +	async_synchronize_full();
> +	if (!error)
> +		error = async_error;
> +
> +	if (error) {
> +		suspend_stats.failed_suspend_noirq++;
> +		dpm_save_failed_step(SUSPEND_SUSPEND_NOIRQ);
>  		dpm_resume_noirq(resume_event(state));
> -	else
> +	} else {
>  		dpm_show_time(starttime, state, "noirq");
> +	}
>  	return error;
>  }
>  
> 

-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

[PATCH v3 5/5] PM / sleep: Asynchronous threads for suspend_late

[Chuansheng Liu]

In analogy with commits 5af84b82701a and 97df8c12995, using
asynchronous threads can improve the overall suspend_late
time significantly.

This patch is for suspend_late phase.

Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
---
 drivers/base/power/main.c | 66 ++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 54 insertions(+), 12 deletions(-)

diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
index 72b4c9c..c031050 100644
--- a/drivers/base/power/main.c
+++ b/drivers/base/power/main.c
@@ -1127,16 +1127,26 @@ static int dpm_suspend_noirq(pm_message_t state)
  *
  * Runtime PM is disabled for @dev while this function is being executed.
  */
-static int device_suspend_late(struct device *dev, pm_message_t state)
+static int __device_suspend_late(struct device *dev, pm_message_t state, bool async)
 {
 	pm_callback_t callback = NULL;
 	char *info = NULL;
-	int error;
+	int error = 0;
+
+	dpm_wait_for_children(dev, async);
 
 	__pm_runtime_disable(dev, false);
 
+	if (async_error)
+		goto Complete;
+
+	if (pm_wakeup_pending()) {
+		async_error = -EBUSY;
+		goto Complete;
+	}
+
 	if (dev->power.syscore)
-		return 0;
+		goto Complete;
 
 	if (dev->pm_domain) {
 		info = "late power domain ";
@@ -1160,10 +1170,40 @@ static int device_suspend_late(struct device *dev, pm_message_t state)
 	error = dpm_run_callback(callback, dev, state, info);
 	if (!error)
 		dev->power.is_late_suspended = true;
+	else
+		async_error = error;
 
+Complete:
+	complete_all(&dev->power.completion);
 	return error;
 }
 
+static void async_suspend_late(void *data, async_cookie_t cookie)
+{
+	struct device *dev = (struct device *)data;
+	int error;
+
+	error = __device_suspend_late(dev, pm_transition, true);
+	if (error) {
+		dpm_save_failed_dev(dev_name(dev));
+		pm_dev_err(dev, pm_transition, " async", error);
+	}
+	put_device(dev);
+}
+
+static int device_suspend_late(struct device *dev)
+{
+	reinit_completion(&dev->power.completion);
+
+	if (pm_async_enabled && dev->power.async_suspend) {
+		get_device(dev);
+		async_schedule(async_suspend_late, dev);
+		return 0;
+	}
+
+	return __device_suspend_late(dev, pm_transition, false);
+}
+
 /**
  * dpm_suspend_late - Execute "late suspend" callbacks for all devices.
  * @state: PM transition of the system being carried out.
@@ -1174,19 +1214,20 @@ static int dpm_suspend_late(pm_message_t state)
 	int error = 0;
 
 	mutex_lock(&dpm_list_mtx);
+	pm_transition = state;
+	async_error = 0;
+
 	while (!list_empty(&dpm_suspended_list)) {
 		struct device *dev = to_device(dpm_suspended_list.prev);
 
 		get_device(dev);
 		mutex_unlock(&dpm_list_mtx);
 
-		error = device_suspend_late(dev, state);
+		error = device_suspend_late(dev);
 
 		mutex_lock(&dpm_list_mtx);
 		if (error) {
 			pm_dev_err(dev, state, " late", error);
-			suspend_stats.failed_suspend_late++;
-			dpm_save_failed_step(SUSPEND_SUSPEND_LATE);
 			dpm_save_failed_dev(dev_name(dev));
 			put_device(dev);
 			break;
@@ -1195,17 +1236,18 @@ static int dpm_suspend_late(pm_message_t state)
 			list_move(&dev->power.entry, &dpm_late_early_list);
 		put_device(dev);
 
-		if (pm_wakeup_pending()) {
-			error = -EBUSY;
+		if (async_error)
 			break;
-		}
 	}
 	mutex_unlock(&dpm_list_mtx);
-	if (error)
+	async_synchronize_full();
+	if (error) {
+		suspend_stats.failed_suspend_late++;
+		dpm_save_failed_step(SUSPEND_SUSPEND_LATE);
 		dpm_resume_early(resume_event(state));
-	else
+	} else {
 		dpm_show_time(starttime, state, "late");
-
+	}
 	return error;
 }
 
-- 
1.9.rc0

Re: [PATCH v3 5/5] PM / sleep: Asynchronous threads for suspend_late

[Rafael J. Wysocki]

On Monday, February 17, 2014 02:19:14 PM Chuansheng Liu wrote:
> In analogy with commits 5af84b82701a and 97df8c12995, using
> asynchronous threads can improve the overall suspend_late
> time significantly.
> 
> This patch is for suspend_late phase.
> 
> Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
> ---
>  drivers/base/power/main.c | 66 ++++++++++++++++++++++++++++++++++++++---------
>  1 file changed, 54 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> index 72b4c9c..c031050 100644
> --- a/drivers/base/power/main.c
> +++ b/drivers/base/power/main.c
> @@ -1127,16 +1127,26 @@ static int dpm_suspend_noirq(pm_message_t state)
>   *
>   * Runtime PM is disabled for @dev while this function is being executed.
>   */
> -static int device_suspend_late(struct device *dev, pm_message_t state)
> +static int __device_suspend_late(struct device *dev, pm_message_t state, bool async)
>  {
>  	pm_callback_t callback = NULL;
>  	char *info = NULL;
> -	int error;
> +	int error = 0;
> +
> +	dpm_wait_for_children(dev, async);
>  

Like in patch [4/5], all of the "goto Complete" statements can go before
the waiting.

>  	__pm_runtime_disable(dev, false);
>  
> +	if (async_error)
> +		goto Complete;
> +
> +	if (pm_wakeup_pending()) {
> +		async_error = -EBUSY;
> +		goto Complete;
> +	}
> +
>  	if (dev->power.syscore)
> -		return 0;
> +		goto Complete;
>  
>  	if (dev->pm_domain) {
>  		info = "late power domain ";
> @@ -1160,10 +1170,40 @@ static int device_suspend_late(struct device *dev, pm_message_t state)
>  	error = dpm_run_callback(callback, dev, state, info);
>  	if (!error)
>  		dev->power.is_late_suspended = true;
> +	else
> +		async_error = error;
>  
> +Complete:
> +	complete_all(&dev->power.completion);
>  	return error;
>  }
>  
> +static void async_suspend_late(void *data, async_cookie_t cookie)
> +{
> +	struct device *dev = (struct device *)data;
> +	int error;
> +
> +	error = __device_suspend_late(dev, pm_transition, true);
> +	if (error) {
> +		dpm_save_failed_dev(dev_name(dev));
> +		pm_dev_err(dev, pm_transition, " async", error);
> +	}
> +	put_device(dev);
> +}
> +
> +static int device_suspend_late(struct device *dev)
> +{
> +	reinit_completion(&dev->power.completion);
> +
> +	if (pm_async_enabled && dev->power.async_suspend) {
> +		get_device(dev);
> +		async_schedule(async_suspend_late, dev);
> +		return 0;
> +	}
> +
> +	return __device_suspend_late(dev, pm_transition, false);
> +}
> +
>  /**
>   * dpm_suspend_late - Execute "late suspend" callbacks for all devices.
>   * @state: PM transition of the system being carried out.
> @@ -1174,19 +1214,20 @@ static int dpm_suspend_late(pm_message_t state)
>  	int error = 0;
>  
>  	mutex_lock(&dpm_list_mtx);
> +	pm_transition = state;
> +	async_error = 0;
> +
>  	while (!list_empty(&dpm_suspended_list)) {
>  		struct device *dev = to_device(dpm_suspended_list.prev);
>  
>  		get_device(dev);
>  		mutex_unlock(&dpm_list_mtx);
>  
> -		error = device_suspend_late(dev, state);
> +		error = device_suspend_late(dev);
>  
>  		mutex_lock(&dpm_list_mtx);
>  		if (error) {
>  			pm_dev_err(dev, state, " late", error);
> -			suspend_stats.failed_suspend_late++;
> -			dpm_save_failed_step(SUSPEND_SUSPEND_LATE);
>  			dpm_save_failed_dev(dev_name(dev));
>  			put_device(dev);
>  			break;
> @@ -1195,17 +1236,18 @@ static int dpm_suspend_late(pm_message_t state)
>  			list_move(&dev->power.entry, &dpm_late_early_list);
>  		put_device(dev);
>  
> -		if (pm_wakeup_pending()) {
> -			error = -EBUSY;
> +		if (async_error)
>  			break;
> -		}
>  	}
>  	mutex_unlock(&dpm_list_mtx);
> -	if (error)
> +	async_synchronize_full();
> +	if (error) {
> +		suspend_stats.failed_suspend_late++;
> +		dpm_save_failed_step(SUSPEND_SUSPEND_LATE);
>  		dpm_resume_early(resume_event(state));
> -	else
> +	} else {
>  		dpm_show_time(starttime, state, "late");
> -
> +	}
>  	return error;
>  }
>  
> 

-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

RE: [PATCH v3 5/5] PM / sleep: Asynchronous threads for suspend_late

[Liu, Chuansheng]

Hello Rafael,

> -----Original Message-----
> From: Rafael J. Wysocki [mailto:rjw@rjwysocki.net]
> Sent: Tuesday, February 18, 2014 1:28 AM
> To: Liu, Chuansheng
> Cc: gregkh@linuxfoundation.org; Brown, Len; pavel@ucw.cz;
> linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; Li, Zhuangzhi
> Subject: Re: [PATCH v3 5/5] PM / sleep: Asynchronous threads for suspend_late
> 
> On Monday, February 17, 2014 02:19:14 PM Chuansheng Liu wrote:
> > In analogy with commits 5af84b82701a and 97df8c12995, using
> > asynchronous threads can improve the overall suspend_late
> > time significantly.
> >
> > This patch is for suspend_late phase.
> >
> > Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
> > ---
> >  drivers/base/power/main.c | 66
> ++++++++++++++++++++++++++++++++++++++---------
> >  1 file changed, 54 insertions(+), 12 deletions(-)
> >
> > diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> > index 72b4c9c..c031050 100644
> > --- a/drivers/base/power/main.c
> > +++ b/drivers/base/power/main.c
> > @@ -1127,16 +1127,26 @@ static int dpm_suspend_noirq(pm_message_t
> state)
> >   *
> >   * Runtime PM is disabled for @dev while this function is being executed.
> >   */
> > -static int device_suspend_late(struct device *dev, pm_message_t state)
> > +static int __device_suspend_late(struct device *dev, pm_message_t state,
> bool async)
> >  {
> >  	pm_callback_t callback = NULL;
> >  	char *info = NULL;
> > -	int error;
> > +	int error = 0;
> > +
> > +	dpm_wait_for_children(dev, async);
> >
> 
> Like in patch [4/5], all of the "goto Complete" statements can go before
> the waiting.
> 
You are right, will do that in patch V4, thanks your reviewing.

Re: [PATCH v3 5/5] PM / sleep: Asynchronous threads for suspend_late

[Rafael J. Wysocki]

On Tuesday, February 18, 2014 12:33:11 AM Liu, Chuansheng wrote:
> Hello Rafael,
> 
> > -----Original Message-----
> > From: Rafael J. Wysocki [mailto:rjw@rjwysocki.net]
> > Sent: Tuesday, February 18, 2014 1:28 AM
> > To: Liu, Chuansheng
> > Cc: gregkh@linuxfoundation.org; Brown, Len; pavel@ucw.cz;
> > linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; Li, Zhuangzhi
> > Subject: Re: [PATCH v3 5/5] PM / sleep: Asynchronous threads for suspend_late
> > 
> > On Monday, February 17, 2014 02:19:14 PM Chuansheng Liu wrote:
> > > In analogy with commits 5af84b82701a and 97df8c12995, using
> > > asynchronous threads can improve the overall suspend_late
> > > time significantly.
> > >
> > > This patch is for suspend_late phase.
> > >
> > > Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
> > > ---
> > >  drivers/base/power/main.c | 66
> > ++++++++++++++++++++++++++++++++++++++---------
> > >  1 file changed, 54 insertions(+), 12 deletions(-)
> > >
> > > diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> > > index 72b4c9c..c031050 100644
> > > --- a/drivers/base/power/main.c
> > > +++ b/drivers/base/power/main.c
> > > @@ -1127,16 +1127,26 @@ static int dpm_suspend_noirq(pm_message_t
> > state)
> > >   *
> > >   * Runtime PM is disabled for @dev while this function is being executed.
> > >   */
> > > -static int device_suspend_late(struct device *dev, pm_message_t state)
> > > +static int __device_suspend_late(struct device *dev, pm_message_t state,
> > bool async)
> > >  {
> > >  	pm_callback_t callback = NULL;
> > >  	char *info = NULL;
> > > -	int error;
> > > +	int error = 0;
> > > +
> > > +	dpm_wait_for_children(dev, async);
> > >
> > 
> > Like in patch [4/5], all of the "goto Complete" statements can go before
> > the waiting.
> > 
> You are right, will do that in patch V4, thanks your reviewing.

I've queued up your last series:

http://marc.info/?l=linux-pm&m=139269109711745&w=4

for 3.15, but I've rebased it on some patches already in my queue, so please
double check the result in linux-pm.git/bleeding-edge.

Thanks!

-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

RE: [PATCH v3 5/5] PM / sleep: Asynchronous threads for suspend_late

[Liu, Chuansheng]

Hello Rafael,

> -----Original Message-----
> From: Rafael J. Wysocki [mailto:rjw@rjwysocki.net]
> Sent: Tuesday, February 18, 2014 11:29 PM
> To: Liu, Chuansheng
> Cc: gregkh@linuxfoundation.org; Brown, Len; pavel@ucw.cz;
> linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; Li, Zhuangzhi
> Subject: Re: [PATCH v3 5/5] PM / sleep: Asynchronous threads for suspend_late
> 
> On Tuesday, February 18, 2014 12:33:11 AM Liu, Chuansheng wrote:
> > Hello Rafael,
> >
> > > -----Original Message-----
> > > From: Rafael J. Wysocki [mailto:rjw@rjwysocki.net]
> > > Sent: Tuesday, February 18, 2014 1:28 AM
> > > To: Liu, Chuansheng
> > > Cc: gregkh@linuxfoundation.org; Brown, Len; pavel@ucw.cz;
> > > linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; Li, Zhuangzhi
> > > Subject: Re: [PATCH v3 5/5] PM / sleep: Asynchronous threads for
> suspend_late
> > >
> > > On Monday, February 17, 2014 02:19:14 PM Chuansheng Liu wrote:
> > > > In analogy with commits 5af84b82701a and 97df8c12995, using
> > > > asynchronous threads can improve the overall suspend_late
> > > > time significantly.
> > > >
> > > > This patch is for suspend_late phase.
> > > >
> > > > Signed-off-by: Chuansheng Liu <chuansheng.liu@intel.com>
> > > > ---
> > > >  drivers/base/power/main.c | 66
> > > ++++++++++++++++++++++++++++++++++++++---------
> > > >  1 file changed, 54 insertions(+), 12 deletions(-)
> > > >
> > > > diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
> > > > index 72b4c9c..c031050 100644
> > > > --- a/drivers/base/power/main.c
> > > > +++ b/drivers/base/power/main.c
> > > > @@ -1127,16 +1127,26 @@ static int
> dpm_suspend_noirq(pm_message_t
> > > state)
> > > >   *
> > > >   * Runtime PM is disabled for @dev while this function is being
> executed.
> > > >   */
> > > > -static int device_suspend_late(struct device *dev, pm_message_t state)
> > > > +static int __device_suspend_late(struct device *dev, pm_message_t
> state,
> > > bool async)
> > > >  {
> > > >  	pm_callback_t callback = NULL;
> > > >  	char *info = NULL;
> > > > -	int error;
> > > > +	int error = 0;
> > > > +
> > > > +	dpm_wait_for_children(dev, async);
> > > >
> > >
> > > Like in patch [4/5], all of the "goto Complete" statements can go before
> > > the waiting.
> > >
> > You are right, will do that in patch V4, thanks your reviewing.
> 
> I've queued up your last series:
> 
> http://marc.info/?l=linux-pm&m=139269109711745&w=4
> 
> for 3.15, but I've rebased it on some patches already in my queue, so please
> double check the result in linux-pm.git/bleeding-edge.
> 
Thanks. I have pulled the latest linux-pm.git/bleeding-edge branch, and checked
them which are OK.