Permanently change thin provisioning method from user space?

Permanently change thin provisioning method from user space?

[Andrei Borzenkov]

I'm facing storage system that while generally advertising support for
WRITE_SAME_16 with UNMAP in reality fails this request depending on
exact volume configuration. This configuration is done on storage side
and may happen after host initially discovered LUN.

I though about simply overriding kernel auto-detect with UNMAP using
sysfs scsi_disk/.../provisioning_mode attribute, but it appears kernel
may re-evaluate it in some cases, so changes do not stick. Is it
possible to permanently restrict it per-device (or per-vendor/device)?

TIA

-andrei

Re: Permanently change thin provisioning method from user space?

[Ewan D. Milne]

On Tue, 2017-06-27 at 14:02 +0300, Andrei Borzenkov wrote:
> I'm facing storage system that while generally advertising support for
> WRITE_SAME_16 with UNMAP in reality fails this request depending on
> exact volume configuration. This configuration is done on storage side
> and may happen after host initially discovered LUN.
> 
> I though about simply overriding kernel auto-detect with UNMAP using
> sysfs scsi_disk/.../provisioning_mode attribute, but it appears kernel
> may re-evaluate it in some cases, so changes do not stick. Is it
> possible to permanently restrict it per-device (or per-vendor/device)?
> 
> TIA
> 
> -andrei

sd_fops->revalidate_disk() will cause the properties that cause the
provisioning_mode to be evaluated to be re-read, and sd_config_discard()
to set the determined mode.  We might want to re-think this, since the
user overrode what was probed earlier.  However, we might also want to
automatically handle the storage capabilities changing, so I'm not sure.

I think we should figure out how best to handle devices/arrays that aren't
correctly handling WRITE SAME(16) w/UNMAP, I've seen a few cases of this
recently with a certain FC array and it looks to me as if the problem
there was that it was advertising MAXIMUM WRITE SAME LENGTH of zero
in the block limits VPD page, but a MAXIMUM UNMAP LBA COUNT of 65536.
My reading of SBC-4r13 6.6.4 is that a WRITE SAME(16) w/UNMAP has
a length limited by the MAXIMUM WRITE SAME LENGTH, and that is what
sd.c implements, but I'm suspicious that the array treated a WRITE SAME(16)
w/UNMAP of 2097152 blocks as an UNMAP and failed it w/ILLEGAL COMMAND,
INVALID FIELD IN CDB.

I don't actually have one of these arrays, I'm trying to gather more info.

-Ewan

Re: Permanently change thin provisioning method from user space?

[Martin K. Petersen]

Ewan,

> sd_fops->revalidate_disk() will cause the properties that cause the
> provisioning_mode to be evaluated to be re-read, and sd_config_discard()
> to set the determined mode.  We might want to re-think this, since the
> user overrode what was probed earlier.  However, we might also want to
> automatically handle the storage capabilities changing, so I'm not
> sure.

The intent was for people to use udev to override things. But I guess we
could entertain introducing a flag to distinguish between detected and
configured state.

> My reading of SBC-4r13 6.6.4 is that a WRITE SAME(16) w/UNMAP has
> a length limited by the MAXIMUM WRITE SAME LENGTH, and that is what
> sd.c implements, but I'm suspicious that the array treated a WRITE SAME(16)
> w/UNMAP of 2097152 blocks as an UNMAP and failed it w/ILLEGAL COMMAND,
> INVALID FIELD IN CDB.

Ugh :(

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: Permanently change thin provisioning method from user space?

[Andrei Borzenkov]

28.06.2017 00:01, Ewan D. Milne пишет:
> On Tue, 2017-06-27 at 14:02 +0300, Andrei Borzenkov wrote:
>> I'm facing storage system that while generally advertising support for
>> WRITE_SAME_16 with UNMAP in reality fails this request depending on
>> exact volume configuration. This configuration is done on storage side
>> and may happen after host initially discovered LUN.
>>
>> I though about simply overriding kernel auto-detect with UNMAP using
>> sysfs scsi_disk/.../provisioning_mode attribute, but it appears kernel
>> may re-evaluate it in some cases, so changes do not stick. Is it
>> possible to permanently restrict it per-device (or per-vendor/device)?
>>
>> TIA
>>
>> -andrei
> 
> sd_fops->revalidate_disk() will cause the properties that cause the
> provisioning_mode to be evaluated to be re-read, and sd_config_discard()
> to set the determined mode.  We might want to re-think this, since the
> user overrode what was probed earlier.  However, we might also want to
> automatically handle the storage capabilities changing, so I'm not sure.
> 

Will revalidate_disk() emit "change" event? If yes, it can be handled by
udev rule (although small race window probably will still exist).

> I think we should figure out how best to handle devices/arrays that aren't
> correctly handling WRITE SAME(16) w/UNMAP,


In this case it caused total multipath outage because it tried DISCARD
on every path and failed all of them in turn.