[PATCH] scsi: fix scsi_error_handler vs. scsi_host_dev_release race

[PATCH] scsi: fix scsi_error_handler vs. scsi_host_dev_release race

[mhocko]

From: Michal Hocko <mhocko@suse.com>

b9d5c6b7ef57 ("[SCSI] cleanup setting task state in
scsi_error_handler()") has introduced a race between scsi_error_handler
and scsi_host_dev_release resulting in the hang when the device goes
away because scsi_error_handler might miss a wake up:

CPU0					CPU1
scsi_error_handler			scsi_host_dev_release
  					  kthread_stop()
  kthread_should_stop()
    test_bit(KTHREAD_SHOULD_STOP)
					    set_bit(KTHREAD_SHOULD_STOP)
					    wake_up_process()
					    wait_for_completion()

  set_current_state(TASK_INTERRUPTIBLE)
  schedule()

The most straightforward solution seems to be to invert the ordering of
the set_current_state and kthread_should_stop.

The issue has been noticed during reboot test on a 3.0 based kernel but
the current code seems to be affected in the same way.

Cc: stable # 3.6+
Reported-and-Debugged-by: Mike Mayer <Mike.Meyer@teradata.com>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 drivers/scsi/scsi_error.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index 6457a8a0db9c..2c0a817d5dbe 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -2169,8 +2169,11 @@ int scsi_error_handler(void *data)
 	 * We never actually get interrupted because kthread_run
 	 * disables signal delivery for the created thread.
 	 */
-	while (!kthread_should_stop()) {
+	while (true) {
 		set_current_state(TASK_INTERRUPTIBLE);
+		if (kthread_should_stop())
+			break;
+
 		if ((shost->host_failed == 0 && shost->host_eh_scheduled == 0) ||
 		    shost->host_failed != atomic_read(&shost->host_busy)) {
 			SCSI_LOG_ERROR_RECOVERY(1,
-- 
2.5.0

Re: [PATCH] scsi: fix scsi_error_handler vs. scsi_host_dev_release race

[Dan Williams]

On Thu, Aug 27, 2015 at 11:16 AM,  <mhocko@kernel.org> wrote:
> From: Michal Hocko <mhocko@suse.com>
>
> b9d5c6b7ef57 ("[SCSI] cleanup setting task state in
> scsi_error_handler()") has introduced a race between scsi_error_handler
> and scsi_host_dev_release resulting in the hang when the device goes
> away because scsi_error_handler might miss a wake up:
>
> CPU0                                    CPU1
> scsi_error_handler                      scsi_host_dev_release
>                                           kthread_stop()
>   kthread_should_stop()
>     test_bit(KTHREAD_SHOULD_STOP)
>                                             set_bit(KTHREAD_SHOULD_STOP)
>                                             wake_up_process()
>                                             wait_for_completion()
>
>   set_current_state(TASK_INTERRUPTIBLE)
>   schedule()
>
> The most straightforward solution seems to be to invert the ordering of
> the set_current_state and kthread_should_stop.
>
> The issue has been noticed during reboot test on a 3.0 based kernel but
> the current code seems to be affected in the same way.
>
> Cc: stable # 3.6+
> Reported-and-Debugged-by: Mike Mayer <Mike.Meyer@teradata.com>
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Acked-by: Dan Williams <dan.j.williams@intel.com>

Re: [PATCH] scsi: fix scsi_error_handler vs. scsi_host_dev_release race

[Hannes Reinecke]

On 08/27/2015 08:16 PM, mhocko@kernel.org wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> b9d5c6b7ef57 ("[SCSI] cleanup setting task state in
> scsi_error_handler()") has introduced a race between scsi_error_handler
> and scsi_host_dev_release resulting in the hang when the device goes
> away because scsi_error_handler might miss a wake up:
> 
> CPU0					CPU1
> scsi_error_handler			scsi_host_dev_release
>   					  kthread_stop()
>   kthread_should_stop()
>     test_bit(KTHREAD_SHOULD_STOP)
> 					    set_bit(KTHREAD_SHOULD_STOP)
> 					    wake_up_process()
> 					    wait_for_completion()
> 
>   set_current_state(TASK_INTERRUPTIBLE)
>   schedule()
> 
> The most straightforward solution seems to be to invert the ordering of
> the set_current_state and kthread_should_stop.
> 
> The issue has been noticed during reboot test on a 3.0 based kernel but
> the current code seems to be affected in the same way.
> 
> Cc: stable # 3.6+
> Reported-and-Debugged-by: Mike Mayer <Mike.Meyer@teradata.com>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
> ---
>  drivers/scsi/scsi_error.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
> index 6457a8a0db9c..2c0a817d5dbe 100644
> --- a/drivers/scsi/scsi_error.c
> +++ b/drivers/scsi/scsi_error.c
> @@ -2169,8 +2169,11 @@ int scsi_error_handler(void *data)
>  	 * We never actually get interrupted because kthread_run
>  	 * disables signal delivery for the created thread.
>  	 */
> -	while (!kthread_should_stop()) {
> +	while (true) {
>  		set_current_state(TASK_INTERRUPTIBLE);
> +		if (kthread_should_stop())
> +			break;
> +
>  		if ((shost->host_failed == 0 && shost->host_eh_scheduled == 0) ||
>  		    shost->host_failed != atomic_read(&shost->host_busy)) {
>  			SCSI_LOG_ERROR_RECOVERY(1,
> 
Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		               zSeries & Storage
hare@suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH] scsi: fix scsi_error_handler vs. scsi_host_dev_release race

[James Bottomley]

On Thu, 2015-08-27 at 20:16 +0200, mhocko@kernel.org wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> b9d5c6b7ef57 ("[SCSI] cleanup setting task state in

Heh, it's tempting to revert that.  The reason is we always need to be
interruptible state before we check the flag to avoid missing wakeups.

> scsi_error_handler()") has introduced a race between scsi_error_handler
> and scsi_host_dev_release resulting in the hang when the device goes
> away because scsi_error_handler might miss a wake up:
> 
> CPU0					CPU1
> scsi_error_handler			scsi_host_dev_release
>   					  kthread_stop()
>   kthread_should_stop()
>     test_bit(KTHREAD_SHOULD_STOP)
> 					    set_bit(KTHREAD_SHOULD_STOP)
> 					    wake_up_process()
> 					    wait_for_completion()
> 
>   set_current_state(TASK_INTERRUPTIBLE)
>   schedule()
> 
> The most straightforward solution seems to be to invert the ordering of
> the set_current_state and kthread_should_stop.
> 
> The issue has been noticed during reboot test on a 3.0 based kernel but
> the current code seems to be affected in the same way.
> 
> Cc: stable # 3.6+
> Reported-and-Debugged-by: Mike Mayer <Mike.Meyer@teradata.com>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
> ---
>  drivers/scsi/scsi_error.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
> index 6457a8a0db9c..2c0a817d5dbe 100644
> --- a/drivers/scsi/scsi_error.c
> +++ b/drivers/scsi/scsi_error.c
> @@ -2169,8 +2169,11 @@ int scsi_error_handler(void *data)
>  	 * We never actually get interrupted because kthread_run
>  	 * disables signal delivery for the created thread.
>  	 */
> -	while (!kthread_should_stop()) {
> +	while (true) {

Comment here, I think, please to avoid any other erroneous tidying
attempts.  How about

/* 
 * The sequence in kthread_stop() sets the stop flag first then 
 * wakes the process.  To avoid missed wakeups, the task should always
 * be in a non running state before the stop flag is checked
 */

Otherwise this looks fine.

James

>  		set_current_state(TASK_INTERRUPTIBLE);
> +		if (kthread_should_stop())
> +			break;
> +
>  		if ((shost->host_failed == 0 && shost->host_eh_scheduled == 0) ||
>  		    shost->host_failed != atomic_read(&shost->host_busy)) {
>  			SCSI_LOG_ERROR_RECOVERY(1,

Re: [PATCH] scsi: fix scsi_error_handler vs. scsi_host_dev_release race

[Michal Hocko]

On Fri 28-08-15 07:56:13, James Bottomley wrote:
[...]
> > diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
> > index 6457a8a0db9c..2c0a817d5dbe 100644
> > --- a/drivers/scsi/scsi_error.c
> > +++ b/drivers/scsi/scsi_error.c
> > @@ -2169,8 +2169,11 @@ int scsi_error_handler(void *data)
> >  	 * We never actually get interrupted because kthread_run
> >  	 * disables signal delivery for the created thread.
> >  	 */
> > -	while (!kthread_should_stop()) {
> > +	while (true) {
> 
> Comment here, I think, please to avoid any other erroneous tidying
> attempts.  How about
> 
> /* 
>  * The sequence in kthread_stop() sets the stop flag first then 
>  * wakes the process.  To avoid missed wakeups, the task should always
>  * be in a non running state before the stop flag is checked
>  */
> 
> Otherwise this looks fine.

I do not have objections to the added comment.

-- 
Michal Hocko
SUSE Labs