* (unknown),
@ 2013-10-09 8:32 sthumma
2013-10-09 8:32 ` Race condition in block layer runtime PM init and scsi disk driver sthumma
0 siblings, 1 reply; 5+ messages in thread
From: sthumma @ 2013-10-09 8:32 UTC (permalink / raw)
To: Aaron Lu, stern; +Cc: linux-scsi
Hi Aaron,
I found a race condition with the block layer runtime PM due to which
the q->nr_pending is decremented to less than zero (0xFFFF_FFFF (-1))
and hence the blk pre-runtime suspend always returns -EBUSY.
The issue is easily reproduced with a scsi disk with disabled tagged
command queuing
sd_probe_async() ->
add_disk() ->
disk_add_event() ->
schedule(disk_events_workfn)
sd_revalidate_disk()
blk_pm_runtime_init()
return;
Let's say the disk_events_workfn() calls sd_check_events() which tries
to send test_unit_ready() and because of sd_revalidate_disk() trying to
send another commands the test_unit_ready() might be re-queued as the
tagged command queuing is disabled.
So the race condition is -
Thread 1 | Thread 2
sd_revalidate_disk() | sd_check_events()
...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()
blk_runtime_pm_init() | blk_pm_requeue_request() ->
| nr_pending = -1 since
| q->dev != NULL
Do you have any suggestions on how to fix this issue?
--
Regards,
Sujit
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Race condition in block layer runtime PM init and scsi disk driver
2013-10-09 8:32 (unknown), sthumma
@ 2013-10-09 8:32 ` sthumma
2013-10-09 9:07 ` Aaron Lu
0 siblings, 1 reply; 5+ messages in thread
From: sthumma @ 2013-10-09 8:32 UTC (permalink / raw)
To: sthumma; +Cc: Aaron Lu, stern, linux-scsi
> Hi Aaron,
>
> I found a race condition with the block layer runtime PM due to which
> the q->nr_pending is decremented to less than zero (0xFFFF_FFFF (-1))
> and hence the blk pre-runtime suspend always returns -EBUSY.
>
>
> The issue is easily reproduced with a scsi disk with disabled tagged
> command queuing
>
> sd_probe_async() ->
> add_disk() ->
> disk_add_event() ->
> schedule(disk_events_workfn)
> sd_revalidate_disk()
> blk_pm_runtime_init()
> return;
>
> Let's say the disk_events_workfn() calls sd_check_events() which tries
> to send test_unit_ready() and because of sd_revalidate_disk() trying to
> send another commands the test_unit_ready() might be re-queued as the
> tagged command queuing is disabled.
>
> So the race condition is -
>
> Thread 1 | Thread 2
> sd_revalidate_disk() | sd_check_events()
> ...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()
> blk_runtime_pm_init() | blk_pm_requeue_request() ->
> | nr_pending = -1 since
> | q->dev != NULL
>
> Do you have any suggestions on how to fix this issue?
>
>
> --
> Regards,
> Sujit
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Race condition in block layer runtime PM init and scsi disk driver
2013-10-09 8:32 ` Race condition in block layer runtime PM init and scsi disk driver sthumma
@ 2013-10-09 9:07 ` Aaron Lu
2013-10-10 4:55 ` sthumma
0 siblings, 1 reply; 5+ messages in thread
From: Aaron Lu @ 2013-10-09 9:07 UTC (permalink / raw)
To: sthumma; +Cc: stern, linux-scsi
On 10/09/2013 04:32 PM, sthumma@codeaurora.org wrote:
>> Hi Aaron,
>>
>> I found a race condition with the block layer runtime PM due to which
>> the q->nr_pending is decremented to less than zero (0xFFFF_FFFF (-1))
>> and hence the blk pre-runtime suspend always returns -EBUSY.
>>
>>
>> The issue is easily reproduced with a scsi disk with disabled tagged
>> command queuing
>>
>> sd_probe_async() ->
>> add_disk() ->
>> disk_add_event() ->
>> schedule(disk_events_workfn)
>> sd_revalidate_disk()
>> blk_pm_runtime_init()
>> return;
>>
>> Let's say the disk_events_workfn() calls sd_check_events() which tries
>> to send test_unit_ready() and because of sd_revalidate_disk() trying to
>> send another commands the test_unit_ready() might be re-queued as the
>> tagged command queuing is disabled.
>>
>> So the race condition is -
>>
>> Thread 1 | Thread 2
>> sd_revalidate_disk() | sd_check_events()
>> ...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()
>> blk_runtime_pm_init() | blk_pm_requeue_request() ->
>> | nr_pending = -1 since
>> | q->dev != NULL
>>
>> Do you have any suggestions on how to fix this issue?
Thanks for the report. I wonder if the following patch helps?
Do the runtime init related work before add_disk, so that every request
is counted properly.
diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
index e62d17d..5693f6d 100644
--- a/drivers/scsi/sd.c
+++ b/drivers/scsi/sd.c
@@ -2854,6 +2854,7 @@ static void sd_probe_async(void *data, async_cookie_t cookie)
gd->events |= DISK_EVENT_MEDIA_CHANGE;
}
+ blk_pm_runtime_init(sdp->request_queue, dev);
add_disk(gd);
if (sdkp->capacity)
sd_dif_config_host(sdkp);
@@ -2862,7 +2863,6 @@ static void sd_probe_async(void *data, async_cookie_t cookie)
sd_printk(KERN_NOTICE, sdkp, "Attached SCSI %sdisk\n",
sdp->removable ? "removable " : "");
- blk_pm_runtime_init(sdp->request_queue, dev);
scsi_autopm_put_device(sdp);
put_device(&sdkp->dev);
}
Thanks,
Aaron
>>
>>
>> --
>> Regards,
>> Sujit
>>
>
>
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: Race condition in block layer runtime PM init and scsi disk driver
2013-10-09 9:07 ` Aaron Lu
@ 2013-10-10 4:55 ` sthumma
2013-10-10 5:22 ` [PATCH] [SCSI] sd: call blk_pm_runtime_init before add_disk Aaron Lu
0 siblings, 1 reply; 5+ messages in thread
From: sthumma @ 2013-10-10 4:55 UTC (permalink / raw)
To: Aaron Lu; +Cc: sthumma, stern, linux-scsi
> On 10/09/2013 04:32 PM, sthumma@codeaurora.org wrote:
>>> Hi Aaron,
>>>
>>> I found a race condition with the block layer runtime PM due to which
>>> the q->nr_pending is decremented to less than zero (0xFFFF_FFFF (-1))
>>> and hence the blk pre-runtime suspend always returns -EBUSY.
>>>
>>>
>>> The issue is easily reproduced with a scsi disk with disabled tagged
>>> command queuing
>>>
>>> sd_probe_async() ->
>>> add_disk() ->
>>> disk_add_event() ->
>>> schedule(disk_events_workfn)
>>> sd_revalidate_disk()
>>> blk_pm_runtime_init()
>>> return;
>>>
>>> Let's say the disk_events_workfn() calls sd_check_events() which tries
>>> to send test_unit_ready() and because of sd_revalidate_disk() trying to
>>> send another commands the test_unit_ready() might be re-queued as the
>>> tagged command queuing is disabled.
>>>
>>> So the race condition is -
>>>
>>> Thread 1 | Thread 2
>>> sd_revalidate_disk() | sd_check_events()
>>> ...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()
>>> blk_runtime_pm_init() | blk_pm_requeue_request() ->
>>> | nr_pending = -1 since
>>> | q->dev != NULL
>>>
>>> Do you have any suggestions on how to fix this issue?
>
> Thanks for the report. I wonder if the following patch helps?
Thanks it works. Would you like to send formal patch for this?
You can add my tested-by ack.
Tested-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
>
> Do the runtime init related work before add_disk, so that every request
> is counted properly.
>
> diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
> index e62d17d..5693f6d 100644
> --- a/drivers/scsi/sd.c
> +++ b/drivers/scsi/sd.c
> @@ -2854,6 +2854,7 @@ static void sd_probe_async(void *data,
> async_cookie_t cookie)
> gd->events |= DISK_EVENT_MEDIA_CHANGE;
> }
>
> + blk_pm_runtime_init(sdp->request_queue, dev);
> add_disk(gd);
> if (sdkp->capacity)
> sd_dif_config_host(sdkp);
> @@ -2862,7 +2863,6 @@ static void sd_probe_async(void *data,
> async_cookie_t cookie)
>
> sd_printk(KERN_NOTICE, sdkp, "Attached SCSI %sdisk\n",
> sdp->removable ? "removable " : "");
> - blk_pm_runtime_init(sdp->request_queue, dev);
> scsi_autopm_put_device(sdp);
> put_device(&sdkp->dev);
> }
>
> Thanks,
> Aaron
>
>>>
>>>
>>> --
>>> Regards,
>>> Sujit
>>>
>>
>>
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH] [SCSI] sd: call blk_pm_runtime_init before add_disk
2013-10-10 4:55 ` sthumma
@ 2013-10-10 5:22 ` Aaron Lu
0 siblings, 0 replies; 5+ messages in thread
From: Aaron Lu @ 2013-10-10 5:22 UTC (permalink / raw)
To: Sujit Reddy Thumma
Cc: Alan Stern, SCSI development list, 'James Bottomley'
Sujit has found a race condition that would make q->nr_pending
unbalanced, it occurs as Sujit explained:
"
sd_probe_async() ->
add_disk() ->
disk_add_event() ->
schedule(disk_events_workfn)
sd_revalidate_disk()
blk_pm_runtime_init()
return;
Let's say the disk_events_workfn() calls sd_check_events() which tries
to send test_unit_ready() and because of sd_revalidate_disk() trying to
send another commands the test_unit_ready() might be re-queued as the
tagged command queuing is disabled.
So the race condition is -
Thread 1 | Thread 2
sd_revalidate_disk() | sd_check_events()
...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()
blk_runtime_pm_init() | blk_pm_requeue_request() ->
| nr_pending = -1 since
| q->dev != NULL
"
The problem is, the test_unit_ready request doesn't get counted the
first time it is queued, so the later decrement of q->nr_pending in
blk_pm_requeue_request makes it unbalanced.
Fix this by calling blk_pm_runtime_init before add_disk so that all
requests initiated there will all be counted.
Signed-off-by: Aaron Lu <aaron.lu@intel.com>
Reported-and-tested-by: Sujit Reddy Thumma <sthumma@codeaurora.org>
Cc: stable@vger.kernel.org
---
drivers/scsi/sd.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
index e62d17d..5693f6d 100644
--- a/drivers/scsi/sd.c
+++ b/drivers/scsi/sd.c
@@ -2854,6 +2854,7 @@ static void sd_probe_async(void *data, async_cookie_t cookie)
gd->events |= DISK_EVENT_MEDIA_CHANGE;
}
+ blk_pm_runtime_init(sdp->request_queue, dev);
add_disk(gd);
if (sdkp->capacity)
sd_dif_config_host(sdkp);
@@ -2862,7 +2863,6 @@ static void sd_probe_async(void *data, async_cookie_t cookie)
sd_printk(KERN_NOTICE, sdkp, "Attached SCSI %sdisk\n",
sdp->removable ? "removable " : "");
- blk_pm_runtime_init(sdp->request_queue, dev);
scsi_autopm_put_device(sdp);
put_device(&sdkp->dev);
}
--
1.8.4.12.g2ea3df6
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2013-10-10 5:22 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-10-09 8:32 (unknown), sthumma
2013-10-09 8:32 ` Race condition in block layer runtime PM init and scsi disk driver sthumma
2013-10-09 9:07 ` Aaron Lu
2013-10-10 4:55 ` sthumma
2013-10-10 5:22 ` [PATCH] [SCSI] sd: call blk_pm_runtime_init before add_disk Aaron Lu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).