[PATCH] scsi: sd_zbc: Disable zoned block devices with scsi-mq

[PATCH] scsi: sd_zbc: Disable zoned block devices with scsi-mq

[Damien Le Moal]

blk-mq may not maintain write requests order at dispatch time. So host
managed drives will not reliably work. Worse, potential reordering of
write requests on requeue may cause the zone write locking code to
deadlock command dispatch to the disk. So for now, until the write
ordering issue is fixed, do not support zoned block devices with
scsi-mq by showing a 0 LBA capacity disk.

Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
---
 drivers/scsi/sd_zbc.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/scsi/sd_zbc.c b/drivers/scsi/sd_zbc.c
index 8aa54779aac1..07bd8511102a 100644
--- a/drivers/scsi/sd_zbc.c
+++ b/drivers/scsi/sd_zbc.c
@@ -561,6 +561,19 @@ int sd_zbc_read_zones(struct scsi_disk *sdkp,
 		 */
 		return 0;
 
+	/*
+	 * In the scsi-mq case, write ordering is not guaranteed so
+	 * host managed drives will not reliably work. Worse, zone write
+	 * locking can cause dispatch deadlocks. So for now, do not support
+	 * zoned block devices with scsi-mq by showing a 0 capacity disk.
+	 */
+	if (sdkp->disk->queue->mq_ops) {
+		if (sdkp->first_scan)
+			sd_printk(KERN_WARNING, sdkp,
+			  "Zoned block devices are not supported with scsi-mq\n");
+		ret = -ENODEV;
+		goto err;
+	}
 
 	/* Get zoned block device characteristics */
 	ret = sd_zbc_read_zoned_characteristics(sdkp, buf);
-- 
2.13.5

Re: [PATCH] scsi: sd_zbc: Disable zoned block devices with scsi-mq

[Christoph Hellwig]

On Thu, Aug 17, 2017 at 11:45:50AM +0900, Damien Le Moal wrote:
> blk-mq may not maintain write requests order at dispatch time. So host
> managed drives will not reliably work. Worse, potential reordering of
> write requests on requeue may cause the zone write locking code to
> deadlock command dispatch to the disk. So for now, until the write
> ordering issue is fixed, do not support zoned block devices with
> scsi-mq by showing a 0 LBA capacity disk.

NAK.  Please fix the issue instead of this.

Re: [PATCH] scsi: sd_zbc: Disable zoned block devices with scsi-mq

[Damien Le Moal]

On 8/17/17 16:59, Christoph Hellwig wrote:
> On Thu, Aug 17, 2017 at 11:45:50AM +0900, Damien Le Moal wrote:
>> blk-mq may not maintain write requests order at dispatch time. So host
>> managed drives will not reliably work. Worse, potential reordering of
>> write requests on requeue may cause the zone write locking code to
>> deadlock command dispatch to the disk. So for now, until the write
>> ordering issue is fixed, do not support zoned block devices with
>> scsi-mq by showing a 0 LBA capacity disk.
> 
> NAK.  Please fix the issue instead of this.

I do have a fix ready to send. All the fixes are at blk-mq level and
basically ensure that dispatch from the hctx->dispatch list are in order
for write commands (regular writes, write same and write zeroes).
It is well tested and at least fixes all the problems I have seen with
dm-zoned and f2fs.

But:
1) It is not small, so I strongly doubt it is RC material
2) We need Bart's patch for "always unprepare requests" which was reverted
3) Bart does not really like the solution I implemented...

I can still send it if you want to have a look.

Best.

-- 
Damien Le Moal,
Western Digital Research

Re: [PATCH] scsi: sd_zbc: Disable zoned block devices with scsi-mq

[Damien Le Moal]

On 8/17/17 21:19, Damien Le Moal wrote:
> 
> 
> On 8/17/17 16:59, Christoph Hellwig wrote:
>> On Thu, Aug 17, 2017 at 11:45:50AM +0900, Damien Le Moal wrote:
>>> blk-mq may not maintain write requests order at dispatch time. So host
>>> managed drives will not reliably work. Worse, potential reordering of
>>> write requests on requeue may cause the zone write locking code to
>>> deadlock command dispatch to the disk. So for now, until the write
>>> ordering issue is fixed, do not support zoned block devices with
>>> scsi-mq by showing a 0 LBA capacity disk.
>>
>> NAK.  Please fix the issue instead of this.
> 
> I do have a fix ready to send. All the fixes are at blk-mq level and
> basically ensure that dispatch from the hctx->dispatch list are in order
> for write commands (regular writes, write same and write zeroes).
> It is well tested and at least fixes all the problems I have seen with
> dm-zoned and f2fs.
> 
> But:
> 1) It is not small, so I strongly doubt it is RC material
> 2) We need Bart's patch for "always unprepare requests" which was reverted
> 3) Bart does not really like the solution I implemented...
> 
> I can still send it if you want to have a look.

Or were you thinking of just getting the deadlock fixed without
addressing the reordering problem ? In the end, ZBC disks would still
not be usable, so to me, disabling them is exactly equivalent...

-- 
Damien Le Moal,
Western Digital Research

Re: [PATCH] scsi: sd_zbc: Disable zoned block devices with scsi-mq

[Bart Van Assche]

On Thu, 2017-08-17 at 21:19 +0900, Damien Le Moal wrote:
> On 8/17/17 16:59, Christoph Hellwig wrote:
> > On Thu, Aug 17, 2017 at 11:45:50AM +0900, Damien Le Moal wrote:
> > > blk-mq may not maintain write requests order at dispatch time. So host
> > > managed drives will not reliably work. Worse, potential reordering of
> > > write requests on requeue may cause the zone write locking code to
> > > deadlock command dispatch to the disk. So for now, until the write
> > > ordering issue is fixed, do not support zoned block devices with
> > > scsi-mq by showing a 0 LBA capacity disk.
> > 
> > NAK.  Please fix the issue instead of this.
> 
> I do have a fix ready to send. All the fixes are at blk-mq level and
> basically ensure that dispatch from the hctx->dispatch list are in order
> for write commands (regular writes, write same and write zeroes).
> It is well tested and at least fixes all the problems I have seen with
> dm-zoned and f2fs.

Hello Damien,

Another possible approach is to fix the ipr driver and to resubmit the
"scsi-mq: Always unprepare before requeuing a request" and "scsi: sd_zbc:
Write unlock zone from sd_uninit_cmnd()" patches.

Bart.

Re: [PATCH] scsi: sd_zbc: Disable zoned block devices with scsi-mq

[Damien Le Moal]

On 8/18/17 00:01, Bart Van Assche wrote:
> On Thu, 2017-08-17 at 21:19 +0900, Damien Le Moal wrote:
>> On 8/17/17 16:59, Christoph Hellwig wrote:
>>> On Thu, Aug 17, 2017 at 11:45:50AM +0900, Damien Le Moal wrote:
>>>> blk-mq may not maintain write requests order at dispatch time. So host
>>>> managed drives will not reliably work. Worse, potential reordering of
>>>> write requests on requeue may cause the zone write locking code to
>>>> deadlock command dispatch to the disk. So for now, until the write
>>>> ordering issue is fixed, do not support zoned block devices with
>>>> scsi-mq by showing a 0 LBA capacity disk.
>>>
>>> NAK.  Please fix the issue instead of this.
>>
>> I do have a fix ready to send. All the fixes are at blk-mq level and
>> basically ensure that dispatch from the hctx->dispatch list are in order
>> for write commands (regular writes, write same and write zeroes).
>> It is well tested and at least fixes all the problems I have seen with
>> dm-zoned and f2fs.
> 
> Hello Damien,
> 
> Another possible approach is to fix the ipr driver and to resubmit the
> "scsi-mq: Always unprepare before requeuing a request" and "scsi: sd_zbc:
> Write unlock zone from sd_uninit_cmnd()" patches.

Bart,

Yes, getting back "scsi-mq: Always unprepare before requeuing a request"
in would be ideal. Right now, I do not have a clue how to otherwise
avoid the deadlock on mq with a solution that fits both sq and mq, that
is, one that will not get a NAK from Christoph :)
But since we already are at rc5, it is risky to wait for a fix for the
PowerPC boot problem that may not come in time. So still trying.

Best.

-- 
Damien Le Moal,
Western Digital

Re: [PATCH] scsi: sd_zbc: Disable zoned block devices with scsi-mq

[hch]

On Thu, Aug 17, 2017 at 03:01:51PM +0000, Bart Van Assche wrote:
> Another possible approach is to fix the ipr driver and to resubmit the
> "scsi-mq: Always unprepare before requeuing a request" and "scsi: sd_zbc:
> Write unlock zone from sd_uninit_cmnd()" patches.

We should absolutely take that broader approach.

But it seems like we need to refne the always unprepare patch first,
to make sure we only release resources, but don't clear the
scmd->retries field.  The NVMe driver already works that way, so
we have an example to follow.

Re: [PATCH] scsi: sd_zbc: Disable zoned block devices with scsi-mq

[Martin K. Petersen]

Christoph,

> But it seems like we need to refne the always unprepare patch first,
> to make sure we only release resources, but don't clear the
> scmd->retries field.  The NVMe driver already works that way, so
> we have an example to follow.

Agree!

-- 
Martin K. Petersen	Oracle Linux Engineering