[PATCH 0/2][RFC] block: default to deadline for SMR devices

[PATCH 0/2][RFC] block: default to deadline for SMR devices

[Jeff Moyer]

Bryan Gurney reported I/O errors when using dm-zoned with a host-managed
SMR device.  It turns out he was using CFQ, which is the default.
Unfortunately, as of v4.16, only the deadline schedulers work well with
host-managed SMR devices.  This series aatempts to switch the elevator
to deadline for those devices.

NOTE: I'm not super happy with setting up one iosched and then
immediately tearing it down.  I'm open to suggestions on better ways
to accomplish this goal.

[PATCH 1/2] block: __elevator_change: add try_loading parameter
[PATCH 2/2] block: default to deadline for host-managed SMR devices

[PATCH 1/2] block: __elevator_change: add try_loading parameter

[Jeff Moyer]

The next patch will add a caller that can't trigger module loads.
Also export this function for that caller.

Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
---
 block/blk.h      | 2 ++
 block/elevator.c | 7 ++++---
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/block/blk.h b/block/blk.h
index b034fd2..f6c3cc1 100644
--- a/block/blk.h
+++ b/block/blk.h
@@ -233,6 +233,8 @@ static inline void elv_deactivate_rq(struct request_queue *q, struct request *rq
 
 int elv_register_queue(struct request_queue *q);
 void elv_unregister_queue(struct request_queue *q);
+int __elevator_change(struct request_queue *q, const char *name,
+		      bool try_loading);
 
 struct hd_struct *__disk_get_part(struct gendisk *disk, int partno);
 
diff --git a/block/elevator.c b/block/elevator.c
index e87e9b4..5a91a43 100644
--- a/block/elevator.c
+++ b/block/elevator.c
@@ -1075,7 +1075,8 @@ static int elevator_switch(struct request_queue *q, struct elevator_type *new_e)
 /*
  * Switch this queue to the given IO scheduler.
  */
-static int __elevator_change(struct request_queue *q, const char *name)
+int __elevator_change(struct request_queue *q, const char *name,
+		      bool try_loading)
 {
 	char elevator_name[ELV_NAME_MAX];
 	struct elevator_type *e;
@@ -1091,7 +1092,7 @@ static int __elevator_change(struct request_queue *q, const char *name)
 		return elevator_switch(q, NULL);
 
 	strlcpy(elevator_name, name, sizeof(elevator_name));
-	e = elevator_get(q, strstrip(elevator_name), true);
+	e = elevator_get(q, strstrip(elevator_name), try_loading);
 	if (!e)
 		return -EINVAL;
 
@@ -1119,7 +1120,7 @@ ssize_t elv_iosched_store(struct request_queue *q, const char *name,
 	if (!(q->mq_ops || q->request_fn) || !elv_support_iosched(q))
 		return count;
 
-	ret = __elevator_change(q, name);
+	ret = __elevator_change(q, name, true);
 	if (!ret)
 		return count;
 
-- 
2.8.2.335.g4bb51ae

[PATCH 2/2] block: default to deadline for host-managed SMR devices

[Jeff Moyer]

Bryan Gurney reported a series of I/O errors resulting from using
CFQ with a host-managed SMR disk.  After commit 39051dd85f28 ("scsi:
sd: Remove zone write locking), which was merged in 4.16, users
should use one of the deadline I/O schedulers (unless applications
are very careful about only submitting 1 I/O per zone).  Change our
defaults to provide a working configuration.

Reported-by: Bryan Gurney <bgurney@redhat.com>
Signed-off-by: Jeff Moyer <jmoyer@redhat.com>
---
 block/blk-sysfs.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index d00d1b0..ec2f29b 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -908,6 +908,30 @@ int blk_register_queue(struct gendisk *disk)
 			return ret;
 		}
 	}
+
+	/*
+	 * Temporary work-around for host-managed SMR devices: right now,
+	 * the deadline I/O schedulers are the only ones that are SMR-
+	 * aware.  Use of other schedulers will almost always result
+	 * in I/O errors due to out-of-order writes to zones.  For now,
+	 * force deadline if we can.  The eventual goal is to get rid
+	 * of this in favor of a dispatch layer that would work with
+	 * all I/O schedulers.
+	 */
+	if (blk_queue_zoned_model(q) == BLK_ZONED_HM) {
+		ret = __elevator_change(q, "deadline", false);
+		if (ret == 0)
+			printk("%s: switched to deadline I/O scheduler for "
+			       "host-managed SMR device %s\n", __func__,
+			       disk->disk_name);
+		else {
+			printk("%s: warning: unable to switch to SMR-aware "
+			       "deadline I/O scheduler for host-managed SMR "
+			       "device %s\n", __func__, disk->disk_name);
+			printk("Consider building deadline and mq-deadline "
+			       "into your kernel (not as modules)\n");
+		}
+	}
 	ret = 0;
 unlock:
 	mutex_unlock(&q->sysfs_lock);
-- 
2.8.2.335.g4bb51ae

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Jens Axboe]

On 5/25/18 3:14 PM, Jeff Moyer wrote:
> Bryan Gurney reported I/O errors when using dm-zoned with a host-managed
> SMR device.  It turns out he was using CFQ, which is the default.
> Unfortunately, as of v4.16, only the deadline schedulers work well with
> host-managed SMR devices.  This series aatempts to switch the elevator
> to deadline for those devices.
> 
> NOTE: I'm not super happy with setting up one iosched and then
> immediately tearing it down.  I'm open to suggestions on better ways
> to accomplish this goal.

Let's please not do this, a few years ago I finally managed to kill
drivers changing the scheduler manually. Why can't this go into a
udev (or similar) rule? That's where it belongs, imho.

-- 
Jens Axboe

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Jeff Moyer]

Hi, Jens,

Jens Axboe <axboe@kernel.dk> writes:

> On 5/25/18 3:14 PM, Jeff Moyer wrote:
>> Bryan Gurney reported I/O errors when using dm-zoned with a host-managed
>> SMR device.  It turns out he was using CFQ, which is the default.
>> Unfortunately, as of v4.16, only the deadline schedulers work well with
>> host-managed SMR devices.  This series aatempts to switch the elevator
>> to deadline for those devices.
>> 
>> NOTE: I'm not super happy with setting up one iosched and then
>> immediately tearing it down.  I'm open to suggestions on better ways
>> to accomplish this goal.
>
> Let's please not do this, a few years ago I finally managed to kill
> drivers changing the scheduler manually. Why can't this go into a
> udev (or similar) rule? That's where it belongs, imho.

We could do that.  The downside is that distros will have to pick up
udev rules, which they haven't done yet, and the udev rules will have to
be kernel version dependent.  And then later, when this restriction is
lifted, we'll have to update the udev rules.  That also sounds awful to
me.

I understand why you don't like this patch set, but I happen to think
the alternative is worse.  FYI, in Bryan's case, his system actually got
bricked (likely due to buggy firmware).

Ultimately, it's your call of course.

Cheers,
Jeff

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Jens Axboe]

On 5/25/18 4:18 PM, Jeff Moyer wrote:
> Hi, Jens,
> 
> Jens Axboe <axboe@kernel.dk> writes:
> 
>> On 5/25/18 3:14 PM, Jeff Moyer wrote:
>>> Bryan Gurney reported I/O errors when using dm-zoned with a host-managed
>>> SMR device.  It turns out he was using CFQ, which is the default.
>>> Unfortunately, as of v4.16, only the deadline schedulers work well with
>>> host-managed SMR devices.  This series aatempts to switch the elevator
>>> to deadline for those devices.
>>>
>>> NOTE: I'm not super happy with setting up one iosched and then
>>> immediately tearing it down.  I'm open to suggestions on better ways
>>> to accomplish this goal.
>>
>> Let's please not do this, a few years ago I finally managed to kill
>> drivers changing the scheduler manually. Why can't this go into a
>> udev (or similar) rule? That's where it belongs, imho.
> 
> We could do that.  The downside is that distros will have to pick up
> udev rules, which they haven't done yet, and the udev rules will have to
> be kernel version dependent.  And then later, when this restriction is
> lifted, we'll have to update the udev rules.  That also sounds awful to
> me.

They only have to be feature dependent, which isn't that hard. And if I
had to pick between a kernel upgrade and a udev rule package update, the
choice is pretty clear.

> I understand why you don't like this patch set, but I happen to think
> the alternative is worse.  FYI, in Bryan's case, his system actually got
> bricked (likely due to buggy firmware).

I disagree, I think the rule approach is much easier. If the wrong write
location bricked the drive, then I think that user has much larger
issues... That seems like a trivial issue that should have been caught
in basic testing, I would not trust that drive with any data if it
bricks that easily.

-- 
Jens Axboe

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Bryan Gurney]

On Sat, May 26, 2018 at 12:01 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 5/25/18 4:18 PM, Jeff Moyer wrote:
>> Hi, Jens,
>>
>> Jens Axboe <axboe@kernel.dk> writes:
>>
>>> On 5/25/18 3:14 PM, Jeff Moyer wrote:
>>>> Bryan Gurney reported I/O errors when using dm-zoned with a host-managed
>>>> SMR device.  It turns out he was using CFQ, which is the default.
>>>> Unfortunately, as of v4.16, only the deadline schedulers work well with
>>>> host-managed SMR devices.  This series aatempts to switch the elevator
>>>> to deadline for those devices.
>>>>
>>>> NOTE: I'm not super happy with setting up one iosched and then
>>>> immediately tearing it down.  I'm open to suggestions on better ways
>>>> to accomplish this goal.
>>>
>>> Let's please not do this, a few years ago I finally managed to kill
>>> drivers changing the scheduler manually. Why can't this go into a
>>> udev (or similar) rule? That's where it belongs, imho.
>>
>> We could do that.  The downside is that distros will have to pick up
>> udev rules, which they haven't done yet, and the udev rules will have to
>> be kernel version dependent.  And then later, when this restriction is
>> lifted, we'll have to update the udev rules.  That also sounds awful to
>> me.
>
> They only have to be feature dependent, which isn't that hard. And if I
> had to pick between a kernel upgrade and a udev rule package update, the
> choice is pretty clear.
>
>> I understand why you don't like this patch set, but I happen to think
>> the alternative is worse.  FYI, in Bryan's case, his system actually got
>> bricked (likely due to buggy firmware).
>
> I disagree, I think the rule approach is much easier. If the wrong write
> location bricked the drive, then I think that user has much larger
> issues... That seems like a trivial issue that should have been caught
> in basic testing, I would not trust that drive with any data if it
> bricks that easily.

To set the record straight, it wasn't the drive that "bricked"; it was
the system motherboard BIOS.

I set up a test to copy some data onto an SMR drive over a weekend
(one of two on the system), and when I came back on Monday morning, I
was greeted with "kernel BUG at block/bio.c:1720!" (which would be
"BUG_ON(atomic_read(&bio->__bi_remaining <= 0);").

After taking a picture of the kernel bug screen, I power cycled the
system, and it just sat there, with fans running, but no power light,
and no other activity (not even a blinking network activity light).

I transplanted the drive and HBA to another test system with identical
hardware, and saw that the SMR drives were running properly.  I
decided to reset the test, thinking, "That _can't_ happen again..."

It happened again.  Same kernel bug, same "fans running, but power
light is off" behavior.  I had two catatonic Xeon workstations in
front of me, which were thankfully both revived with the help of a
chip programmer writing the binary file of the latest BIOS update.

After reaching out to Damien regarding the large amount of "Aborted
command" senses that I was seeing prior to the kernel bug, he
recommended using the "deadline" scheduler for kernel 4.16 and
greater.


Thanks,

Bryan

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Damien Le Moal]

SmVucywNCg0KT24gNS8yNi8xOCAxMzowMSwgSmVucyBBeGJvZSB3cm90ZToNCj4gT24gNS8yNS8x
OCA0OjE4IFBNLCBKZWZmIE1veWVyIHdyb3RlOg0KPj4gSGksIEplbnMsDQo+Pg0KPj4gSmVucyBB
eGJvZSA8YXhib2VAa2VybmVsLmRrPiB3cml0ZXM6DQo+Pg0KPj4+IE9uIDUvMjUvMTggMzoxNCBQ
TSwgSmVmZiBNb3llciB3cm90ZToNCj4+Pj4gQnJ5YW4gR3VybmV5IHJlcG9ydGVkIEkvTyBlcnJv
cnMgd2hlbiB1c2luZyBkbS16b25lZCB3aXRoIGEgaG9zdC1tYW5hZ2VkDQo+Pj4+IFNNUiBkZXZp
Y2UuICBJdCB0dXJucyBvdXQgaGUgd2FzIHVzaW5nIENGUSwgd2hpY2ggaXMgdGhlIGRlZmF1bHQu
DQo+Pj4+IFVuZm9ydHVuYXRlbHksIGFzIG9mIHY0LjE2LCBvbmx5IHRoZSBkZWFkbGluZSBzY2hl
ZHVsZXJzIHdvcmsgd2VsbCB3aXRoDQo+Pj4+IGhvc3QtbWFuYWdlZCBTTVIgZGV2aWNlcy4gIFRo
aXMgc2VyaWVzIGFhdGVtcHRzIHRvIHN3aXRjaCB0aGUgZWxldmF0b3INCj4+Pj4gdG8gZGVhZGxp
bmUgZm9yIHRob3NlIGRldmljZXMuDQo+Pj4+DQo+Pj4+IE5PVEU6IEknbSBub3Qgc3VwZXIgaGFw
cHkgd2l0aCBzZXR0aW5nIHVwIG9uZSBpb3NjaGVkIGFuZCB0aGVuDQo+Pj4+IGltbWVkaWF0ZWx5
IHRlYXJpbmcgaXQgZG93bi4gIEknbSBvcGVuIHRvIHN1Z2dlc3Rpb25zIG9uIGJldHRlciB3YXlz
DQo+Pj4+IHRvIGFjY29tcGxpc2ggdGhpcyBnb2FsLg0KPj4+DQo+Pj4gTGV0J3MgcGxlYXNlIG5v
dCBkbyB0aGlzLCBhIGZldyB5ZWFycyBhZ28gSSBmaW5hbGx5IG1hbmFnZWQgdG8ga2lsbA0KPj4+
IGRyaXZlcnMgY2hhbmdpbmcgdGhlIHNjaGVkdWxlciBtYW51YWxseS4gV2h5IGNhbid0IHRoaXMg
Z28gaW50byBhDQo+Pj4gdWRldiAob3Igc2ltaWxhcikgcnVsZT8gVGhhdCdzIHdoZXJlIGl0IGJl
bG9uZ3MsIGltaG8uDQo+Pg0KPj4gV2UgY291bGQgZG8gdGhhdC4gIFRoZSBkb3duc2lkZSBpcyB0
aGF0IGRpc3Ryb3Mgd2lsbCBoYXZlIHRvIHBpY2sgdXANCj4+IHVkZXYgcnVsZXMsIHdoaWNoIHRo
ZXkgaGF2ZW4ndCBkb25lIHlldCwgYW5kIHRoZSB1ZGV2IHJ1bGVzIHdpbGwgaGF2ZSB0bw0KPj4g
YmUga2VybmVsIHZlcnNpb24gZGVwZW5kZW50LiAgQW5kIHRoZW4gbGF0ZXIsIHdoZW4gdGhpcyBy
ZXN0cmljdGlvbiBpcw0KPj4gbGlmdGVkLCB3ZSdsbCBoYXZlIHRvIHVwZGF0ZSB0aGUgdWRldiBy
dWxlcy4gIFRoYXQgYWxzbyBzb3VuZHMgYXdmdWwgdG8NCj4+IG1lLg0KPiANCj4gVGhleSBvbmx5
IGhhdmUgdG8gYmUgZmVhdHVyZSBkZXBlbmRlbnQsIHdoaWNoIGlzbid0IHRoYXQgaGFyZC4gQW5k
IGlmIEkNCj4gaGFkIHRvIHBpY2sgYmV0d2VlbiBhIGtlcm5lbCB1cGdyYWRlIGFuZCBhIHVkZXYg
cnVsZSBwYWNrYWdlIHVwZGF0ZSwgdGhlDQo+IGNob2ljZSBpcyBwcmV0dHkgY2xlYXIuDQoNCkN1
cnJlbnRseSwgYSBxdWV1ZSBlbGV2YXRvciBpcyBmaXJzdCBpbml0aWFsaXplZCB3aXRoIGEgY2Fs
bCB0bw0KZWxldmF0b3JfaW5pdCgpIGZyb20gZWl0aGVyIGJsa19tcV9pbml0X2FsbG9jYXRlZF9x
dWV1ZSgpIG9yDQpibGtfaW5pdF9hbGxvY2F0ZWRfcXVldWUoKS4gV2hlbiB0aGVzZSBhcmUgY2Fs
bGVkLCB0aGUgZGV2aWNlIGhhcyBub3QNCnlldCBiZWVuIHByb2JlZCBhbmQgbm8gaW5mb3JtYXRp
b24gaXMgYXZhaWxhYmxlLiBEbyB5b3UgdGhpbmsgaXQgaXMNCnJlYXNvbmFibGUgdG8gZGVsYXkg
dGhlIGRlZmF1bHQgZWxldmF0b3IgYnJpbmctdXAgdG8gYWZ0ZXIgc29tZQ0KaW5mb3JtYXRpb24g
YWJvdXQgdGhlIGRldmljZSBpcyBvYnRhaW5lZCA/DQoNClRoYXQgd291bGQgbmVjZXNzaXRhdGUg
c3BsaXR0aW5nIGVsZXZhdG9yX2luaXQoKSBpbnRvIGEgZ2VuZXJpYyBlbGV2YXRvcg0KaW5pdGlh
bGl6YXRpb24gZnVuY3Rpb24gc2V0dGluZyB1cCB0aGUgZWxldmF0b3IgcmVsYXRlZCBmaWVsZHMg
b2YgdGhlDQpyZXF1ZXN0IHF1ZXVlIGFuZCBhIHNlY29uZCBwYXJ0IHNldHRpbmcgdXAgdGhlIGRl
ZmF1bHQgZWxldmF0b3IgKGUuZy4NCmVsZXZhdG9yX3NldF9kZWZhdWx0KCkpLiBEb2luZyBzbywg
dGhlIGZ1bmN0aW9uIGVsZXZhdG9yX3NldF9kZWZhdWx0KCkNCmNvdWxkIGJlIGNhbGxlZCBsYXRl
ciBpbiB0aGUgZGV2aWNlIGluaXRpYWxpemF0aW9uIHNlcXVlbmNlLCBhZnRlcg0KaW5mb3JtYXRp
b24gZnJvbSB0aGUgZGV2aWNlIGhhcyBiZWVuIG9idGFpbmVkLiBJdCB3b3VsZCBtYWtlIGNob29z
aW5nIGENCnNhbmUgZGVmYXVsdCBlbGV2YXRvciBtdWNoIGNsZWFuZXIuDQoNCkdyYW50ZWQsIHRo
YXQgaXMgaW4gZWZmZWN0IHZlcnkgc2ltaWxhciB0byB3aGF0IEplZmYncyBwYXRjaGVzIGRvLiBC
dXQNCmluIGEgc2Vuc2UsIHRoZSBjdXJyZW50IGNvZGUgYWxyZWFkeSBjb25zaWRlciBzb21lIG9m
IHRoZSBkZXZpY2UNCmNoYXJhY3RlcmlzdGljcyB3aGVuIHNldHRpbmcgdGhlIGRlZmF1bHQgZWxl
dmF0b3IuIFRoYXQgaXMsIGlmIHRoZQ0KZGV2aWNlIGlzIG1xIG9yIG5vdC4gV291bGQgaXQgYmUg
YSBzdHJldGNoIHRvIGFkZCAiaXNfem9uZWQiIGFzIGFub3RoZXINCmNoYXJhY3RlcmlzdGljIHRo
YXQgaXMgbG9va2VkIGF0ID8NCg0KT24gYW5vdGhlciBub3RlLCBJIG5vdGljZWQgdGhhdCB0aGUg
b25seSB0d28gY2FsbGVycyBvZiBlbGV2YXRvcl9pbml0KCkNCmFyZSBibGtfbXFfaW5pdF9hbGxv
Y2F0ZWRfcXVldWUoKSBhbmQgYmxrX2luaXRfYWxsb2NhdGVkX3F1ZXVlKCkuIEJvdGgNCmNhbGwg
ZWxldmF0b3JfaW5pdCgpIHdpdGggYSBOVUxMIG5hbWUuIEkgd29uZGVyIHRoZSB1c2VmdWxuZXNz
IG9mIHRoYXQNCmFyZ3VtZW50LiBGdXJ0aGVybW9yZSwgZWxldmF0b3JfaW5pdCgpIGlzIGFsc28g
ZXhwb3J0ZWQgYnV0IEkgZG8gbm90IHNlZQ0KYW55IGV4dGVybmFsIG1vZHVsZSB1c2VyLiBJcyB0
aGlzIGluIHB1cnBvc2Ugb3IgaXMgaXQgdGhlIHJlbW5hbnQgb2YNCm9sZGVyIHVzZSBvZiBpdCA/
IEkgY2FuIHNlbmQgYSBwYXRjaCBjbGVhbmluZyB0aGF0IHVwIGlmIHlvdSBmZWVsIHRoYXQNCm5l
ZWRzIHNvIGNsZWFudXAuDQoNCj4gDQo+PiBJIHVuZGVyc3RhbmQgd2h5IHlvdSBkb24ndCBsaWtl
IHRoaXMgcGF0Y2ggc2V0LCBidXQgSSBoYXBwZW4gdG8gdGhpbmsNCj4+IHRoZSBhbHRlcm5hdGl2
ZSBpcyB3b3JzZS4gIEZZSSwgaW4gQnJ5YW4ncyBjYXNlLCBoaXMgc3lzdGVtIGFjdHVhbGx5IGdv
dA0KPj4gYnJpY2tlZCAobGlrZWx5IGR1ZSB0byBidWdneSBmaXJtd2FyZSkuDQo+IA0KPiBJIGRp
c2FncmVlLCBJIHRoaW5rIHRoZSBydWxlIGFwcHJvYWNoIGlzIG11Y2ggZWFzaWVyLiBJZiB0aGUg
d3Jvbmcgd3JpdGUNCj4gbG9jYXRpb24gYnJpY2tlZCB0aGUgZHJpdmUsIHRoZW4gSSB0aGluayB0
aGF0IHVzZXIgaGFzIG11Y2ggbGFyZ2VyDQo+IGlzc3Vlcy4uLiBUaGF0IHNlZW1zIGxpa2UgYSB0
cml2aWFsIGlzc3VlIHRoYXQgc2hvdWxkIGhhdmUgYmVlbiBjYXVnaHQNCj4gaW4gYmFzaWMgdGVz
dGluZywgSSB3b3VsZCBub3QgdHJ1c3QgdGhhdCBkcml2ZSB3aXRoIGFueSBkYXRhIGlmIGl0DQo+
IGJyaWNrcyB0aGF0IGVhc2lseS4NCg0KSSB0aGluayB0aGF0IHRoZSBidWcgaW4gdGhpcyBwYXJ0
aWN1bGFyIGNhc2Ugd2FzIHdpdGggdGhlIEJJT1MgKHByb2JhYmx5DQpVRUZJKSwgbm90IHRoZSBk
aXNrLiBEaXNrcyBvbmx5IGFuc3dlciB0byB0cml2aWFsIHJlcXVlc3RzIHdoZW4gcHJvYmVkDQpv
biBib290LCB1bmxlc3MgdGhlIEJJT1MgaXMgdG9vIHNtYXJ0IGZvciBpdHMgb3duIGdvb2QgYW5k
IGRvaW5nIHZlcnkNCndlaXJkIHRoaW5ncy4gRGlmZmVyZW50IGlzc3VlIGFueXdheSwgbm90IHJl
bGF0ZWQgdG8gdGhlIGtlcm5lbC4NCg0KQmVzdCByZWdhcmRzLg0KDQotLSANCkRhbWllbiBMZSBN
b2FsLA0KV2VzdGVybiBEaWdpdGFs

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Christoph Hellwig]

On Wed, May 30, 2018 at 06:22:04AM +0000, Damien Le Moal wrote:
> That would necessitate splitting elevator_init() into a generic elevator
> initialization function setting up the elevator related fields of the
> request queue and a second part setting up the default elevator (e.g.
> elevator_set_default()). Doing so, the function elevator_set_default()
> could be called later in the device initialization sequence, after
> information from the device has been obtained. It would make choosing a
> sane default elevator much cleaner.

For blk-mq this makes entirely sense, untested series here:

	http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/elevator_init

For the legacy case we always need at least some I/O scheduler,
so we can't just defer it.  But I think we could still switch to
deadline in blk_register_queue if we found a zoned device.

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Damien Le Moal]

Q2hyaXN0b3BoLA0KDQpPbiA1LzMwLzE4IDE3OjQ3LCBDaHJpc3RvcGggSGVsbHdpZyB3cm90ZToN
Cj4gT24gV2VkLCBNYXkgMzAsIDIwMTggYXQgMDY6MjI6MDRBTSArMDAwMCwgRGFtaWVuIExlIE1v
YWwgd3JvdGU6DQo+PiBUaGF0IHdvdWxkIG5lY2Vzc2l0YXRlIHNwbGl0dGluZyBlbGV2YXRvcl9p
bml0KCkgaW50byBhIGdlbmVyaWMgZWxldmF0b3INCj4+IGluaXRpYWxpemF0aW9uIGZ1bmN0aW9u
IHNldHRpbmcgdXAgdGhlIGVsZXZhdG9yIHJlbGF0ZWQgZmllbGRzIG9mIHRoZQ0KPj4gcmVxdWVz
dCBxdWV1ZSBhbmQgYSBzZWNvbmQgcGFydCBzZXR0aW5nIHVwIHRoZSBkZWZhdWx0IGVsZXZhdG9y
IChlLmcuDQo+PiBlbGV2YXRvcl9zZXRfZGVmYXVsdCgpKS4gRG9pbmcgc28sIHRoZSBmdW5jdGlv
biBlbGV2YXRvcl9zZXRfZGVmYXVsdCgpDQo+PiBjb3VsZCBiZSBjYWxsZWQgbGF0ZXIgaW4gdGhl
IGRldmljZSBpbml0aWFsaXphdGlvbiBzZXF1ZW5jZSwgYWZ0ZXINCj4+IGluZm9ybWF0aW9uIGZy
b20gdGhlIGRldmljZSBoYXMgYmVlbiBvYnRhaW5lZC4gSXQgd291bGQgbWFrZSBjaG9vc2luZyBh
DQo+PiBzYW5lIGRlZmF1bHQgZWxldmF0b3IgbXVjaCBjbGVhbmVyLg0KPiANCj4gRm9yIGJsay1t
cSB0aGlzIG1ha2VzIGVudGlyZWx5IHNlbnNlLCB1bnRlc3RlZCBzZXJpZXMgaGVyZToNCj4gDQo+
IAlodHRwOi8vZ2l0LmluZnJhZGVhZC5vcmcvdXNlcnMvaGNoL2Jsb2NrLmdpdC9zaG9ydGxvZy9y
ZWZzL2hlYWRzL2VsZXZhdG9yX2luaXQNCg0KSnVzdCByZXZpZXdlZCBhbmQgdHJpZWQgdGhlIHNl
cmllcy4gTG9va3MgT0sgdG8gbWUuDQpGZWVsIGZyZWUgdG8gcG9zdCBhZGRpbmc6DQpSZXZpZXdl
ZC1ieTogRGFtaWVuIExlIE1vYWwgPGRhbWllbi5sZW1vYWxAd2RjLmNvbT4NClRlc3RlZC1ieTog
RGFtaWVuIExlIE1vYWwgPGRhbWllbi5sZW1vYWxAd2RjLmNvbT4NCg0KPiBGb3IgdGhlIGxlZ2Fj
eSBjYXNlIHdlIGFsd2F5cyBuZWVkIGF0IGxlYXN0IHNvbWUgSS9PIHNjaGVkdWxlciwNCj4gc28g
d2UgY2FuJ3QganVzdCBkZWZlciBpdC4gIEJ1dCBJIHRoaW5rIHdlIGNvdWxkIHN0aWxsIHN3aXRj
aCB0bw0KPiBkZWFkbGluZSBpbiBibGtfcmVnaXN0ZXJfcXVldWUgaWYgd2UgZm91bmQgYSB6b25l
ZCBkZXZpY2UuDQoNCkFycmcuLi4gWWVzLCBJIGZvcmdvdCB0aGF0LiBBdCBsZWFzdCBub29wIHdv
dWxkIGJlIG5lZWRlZC4gU28gbm8gcmVhbA0KcG9pbnQgaW4gY2hhbmdpbmcgYW55dGhpbmcgSSBn
dWVzcy4NCg0KRm9yIHRoZSBzd2l0Y2ggdG8gZGVhZGxpbmUgb25jZSB0aGUgZGV2aWNlIGlzIGlk
ZW50aWZpZWQgYXMgem9uZWQsIEkNCndvdWxkIGxpa2UgdG8gc2VlIHRoYXQgYWRkZWQuIEplbnMg
ZGVjaXNpb24gb24gdGhpcyB0aG91Z2guDQoNCkJlc3QuDQoNCi0tIA0KRGFtaWVuIExlIE1vYWws
DQpXZXN0ZXJuIERpZ2l0YWw=

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Bryan Gurney]

On Wed, May 30, 2018 at 2:22 AM, Damien Le Moal <Damien.LeMoal@wdc.com> wrote:
> Jens,
>
> On 5/26/18 13:01, Jens Axboe wrote:
>> On 5/25/18 4:18 PM, Jeff Moyer wrote:
>>> Hi, Jens,
>>>
>>> Jens Axboe <axboe@kernel.dk> writes:
>>>
>>>> On 5/25/18 3:14 PM, Jeff Moyer wrote:
>>>>> Bryan Gurney reported I/O errors when using dm-zoned with a host-managed
>>>>> SMR device.  It turns out he was using CFQ, which is the default.
>>>>> Unfortunately, as of v4.16, only the deadline schedulers work well with
>>>>> host-managed SMR devices.  This series aatempts to switch the elevator
>>>>> to deadline for those devices.
>>>>>
>>>>> NOTE: I'm not super happy with setting up one iosched and then
>>>>> immediately tearing it down.  I'm open to suggestions on better ways
>>>>> to accomplish this goal.
>>>>
>>>> Let's please not do this, a few years ago I finally managed to kill
>>>> drivers changing the scheduler manually. Why can't this go into a
>>>> udev (or similar) rule? That's where it belongs, imho.
>>>
>>> We could do that.  The downside is that distros will have to pick up
>>> udev rules, which they haven't done yet, and the udev rules will have to
>>> be kernel version dependent.  And then later, when this restriction is
>>> lifted, we'll have to update the udev rules.  That also sounds awful to
>>> me.
>>
>> They only have to be feature dependent, which isn't that hard. And if I
>> had to pick between a kernel upgrade and a udev rule package update, the
>> choice is pretty clear.
>
> Currently, a queue elevator is first initialized with a call to
> elevator_init() from either blk_mq_init_allocated_queue() or
> blk_init_allocated_queue(). When these are called, the device has not
> yet been probed and no information is available. Do you think it is
> reasonable to delay the default elevator bring-up to after some
> information about the device is obtained ?
>
> That would necessitate splitting elevator_init() into a generic elevator
> initialization function setting up the elevator related fields of the
> request queue and a second part setting up the default elevator (e.g.
> elevator_set_default()). Doing so, the function elevator_set_default()
> could be called later in the device initialization sequence, after
> information from the device has been obtained. It would make choosing a
> sane default elevator much cleaner.
>
> Granted, that is in effect very similar to what Jeff's patches do. But
> in a sense, the current code already consider some of the device
> characteristics when setting the default elevator. That is, if the
> device is mq or not. Would it be a stretch to add "is_zoned" as another
> characteristic that is looked at ?
>
> On another note, I noticed that the only two callers of elevator_init()
> are blk_mq_init_allocated_queue() and blk_init_allocated_queue(). Both
> call elevator_init() with a NULL name. I wonder the usefulness of that
> argument. Furthermore, elevator_init() is also exported but I do not see
> any external module user. Is this in purpose or is it the remnant of
> older use of it ? I can send a patch cleaning that up if you feel that
> needs so cleanup.
>
>>
>>> I understand why you don't like this patch set, but I happen to think
>>> the alternative is worse.  FYI, in Bryan's case, his system actually got
>>> bricked (likely due to buggy firmware).
>>
>> I disagree, I think the rule approach is much easier. If the wrong write
>> location bricked the drive, then I think that user has much larger
>> issues... That seems like a trivial issue that should have been caught
>> in basic testing, I would not trust that drive with any data if it
>> bricks that easily.
>
> I think that the bug in this particular case was with the BIOS (probably
> UEFI), not the disk. Disks only answer to trivial requests when probed
> on boot, unless the BIOS is too smart for its own good and doing very
> weird things. Different issue anyway, not related to the kernel.

Actually, the BIOS boot mode was configured to "Legacy" mode (i.e.:
non-UEFI) at the time of the bricking.  I'm pretty confident of this,
because after the BIOS reflash, I can see a different rendering mode
in the GRUB screen of Fedora 28, and when I checked back in the BIOS
setup utility, I saw that the default for the new BIOS image was UEFI.

On the test systems in general, I prefer to use legacy BIOS if I don't
need UEFI.  I'm not sure if this means that I walked into a latent
BIOS bug, though.


Thanks,

Bryan

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Christoph Hellwig]

While I really don't want drivers to change the I/O schedule themselves
we have a class of devices (zoned) that don't work at all with certain
I/O schedulers.  The kernel not chosing something sane and requiring
user workarounds is just silly.

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Jens Axboe]

On 5/30/18 2:49 AM, Christoph Hellwig wrote:
> While I really don't want drivers to change the I/O schedule themselves
> we have a class of devices (zoned) that don't work at all with certain
> I/O schedulers.  The kernel not chosing something sane and requiring
> user workarounds is just silly.

They work just fine for probing and reading purposes. There's absolutely
no reason why we can't handle these special snowflakes with a udev rule.

-- 
Jens Axboe

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Jeff Moyer]

Hi, Jens,

Jens Axboe <axboe@kernel.dk> writes:

> On 5/30/18 2:49 AM, Christoph Hellwig wrote:
>> While I really don't want drivers to change the I/O schedule themselves
>> we have a class of devices (zoned) that don't work at all with certain
>> I/O schedulers.  The kernel not chosing something sane and requiring
>> user workarounds is just silly.
>
> They work just fine for probing and reading purposes. There's absolutely
> no reason why we can't handle these special snowflakes with a udev rule.

udev rules aren't shipped with the kernel, so it makes it hard to keep
them in sync.  In this instance, I'm not sure anyone made an effort to
notify distributions that a udev rule was even necessary.  (Is there a
way of notifying distributions about kernel changes that require new
udev rules, other than emailing each list individually?)

>From a technical standpoint, I totally agree with you, Jens.  However, I
think the user experience sucks.  4.15 worked by default, 4.16 doesn't.
The result will be bug reports from users (to the drive vendors,
distribution bugzillas, here, etc.).

Moving on, assuming your mind is made up...

I'm not sure how much logic should go into the udev rule.  As mentioned,
this limitation was introduced in 4.16, and Damien has plans to lift the
restriction in future kernels.  Because distributions tend to cherry
pick changes, making decisions on whether a feature exists based solely
on kernel version is usually not a great thing.  My inclination would be
to just always force deadline for host-managed SMR drives.  These drives
aren't that popular, after all.  Any opinions on this?

Thanks!
Jeff

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Jens Axboe]

On 5/30/18 9:06 AM, Jeff Moyer wrote:
> Hi, Jens,
> 
> Jens Axboe <axboe@kernel.dk> writes:
> 
>> On 5/30/18 2:49 AM, Christoph Hellwig wrote:
>>> While I really don't want drivers to change the I/O schedule themselves
>>> we have a class of devices (zoned) that don't work at all with certain
>>> I/O schedulers.  The kernel not chosing something sane and requiring
>>> user workarounds is just silly.
>>
>> They work just fine for probing and reading purposes. There's absolutely
>> no reason why we can't handle these special snowflakes with a udev rule.
> 
> udev rules aren't shipped with the kernel, so it makes it hard to keep
> them in sync.  In this instance, I'm not sure anyone made an effort to
> notify distributions that a udev rule was even necessary.  (Is there a
> way of notifying distributions about kernel changes that require new
> udev rules, other than emailing each list individually?)
> 
> From a technical standpoint, I totally agree with you, Jens.  However, I
> think the user experience sucks.  4.15 worked by default, 4.16 doesn't.
> The result will be bug reports from users (to the drive vendors,
> distribution bugzillas, here, etc.).

I would imagine that most folks get their updates from a distro of some
sort, in which case there's absolutely nothing stopping the distro from
shipping updated rules for the 4.16 kernel update.

But what's the regression? 4.15 had no zone write locking at all.

> Moving on, assuming your mind is made up...
> 
> I'm not sure how much logic should go into the udev rule.  As mentioned,
> this limitation was introduced in 4.16, and Damien has plans to lift the
> restriction in future kernels.  Because distributions tend to cherry
> pick changes, making decisions on whether a feature exists based solely
> on kernel version is usually not a great thing.  My inclination would be
> to just always force deadline for host-managed SMR drives.  These drives
> aren't that popular, after all.  Any opinions on this?

The problem is that it's tied to an IO scheduler, which ends up causing
issues like this, since users are free to select a different scheduler.
Then things break. Granted, in this case, some extraordinarily shitty
hardware even broke. That is on the hardware, not the kernel, that
kind of breakage should not occur.

So now we're stuck with this temporary situation which needs a work-around.
I don't think it's a terrible idea to have a rule that just sets
deadline/mq-deadline for an SMR device regardless of what kernel it is
running on. It'll probably never be a bad default.

-- 
Jens Axboe

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Jeff Moyer]

Jens Axboe <axboe@kernel.dk> writes:

> On 5/30/18 9:06 AM, Jeff Moyer wrote:
>> Hi, Jens,
>> 
>> Jens Axboe <axboe@kernel.dk> writes:
>> 
>>> On 5/30/18 2:49 AM, Christoph Hellwig wrote:
>>>> While I really don't want drivers to change the I/O schedule themselves
>>>> we have a class of devices (zoned) that don't work at all with certain
>>>> I/O schedulers.  The kernel not chosing something sane and requiring
>>>> user workarounds is just silly.
>>>
>>> They work just fine for probing and reading purposes. There's absolutely
>>> no reason why we can't handle these special snowflakes with a udev rule.
>> 
>> udev rules aren't shipped with the kernel, so it makes it hard to keep
>> them in sync.  In this instance, I'm not sure anyone made an effort to
>> notify distributions that a udev rule was even necessary.  (Is there a
>> way of notifying distributions about kernel changes that require new
>> udev rules, other than emailing each list individually?)
>> 
>> From a technical standpoint, I totally agree with you, Jens.  However, I
>> think the user experience sucks.  4.15 worked by default, 4.16 doesn't.
>> The result will be bug reports from users (to the drive vendors,
>> distribution bugzillas, here, etc.).
>
> I would imagine that most folks get their updates from a distro of some
> sort, in which case there's absolutely nothing stopping the distro from
> shipping updated rules for the 4.16 kernel update.

The problem is distros have already shipped that kernel.  :)

> But what's the regression? 4.15 had no zone write locking at all.

The zone write locking was done in the sd driver prior to 4.16.  See
commit 39051dd85f287 ("scsi: sd: Remove zone write locking") for where
it was removed.  That means these devices "just worked" with all I/O
schedulers.

>> Moving on, assuming your mind is made up...
>> 
>> I'm not sure how much logic should go into the udev rule.  As mentioned,
>> this limitation was introduced in 4.16, and Damien has plans to lift the
>> restriction in future kernels.  Because distributions tend to cherry
>> pick changes, making decisions on whether a feature exists based solely
>> on kernel version is usually not a great thing.  My inclination would be
>> to just always force deadline for host-managed SMR drives.  These drives
>> aren't that popular, after all.  Any opinions on this?
>
> The problem is that it's tied to an IO scheduler, which ends up causing
> issues like this, since users are free to select a different scheduler.
> Then things break. Granted, in this case, some extraordinarily shitty
> hardware even broke. That is on the hardware, not the kernel, that
> kind of breakage should not occur.

If the firmware problem was widespread, I think we'd try to avoid it.  I
have no reason to believe that is the case, though.

Damien made the argument that the user should be able to select an I/O
scheduler that doesn't perform the write locking, because a well-behaved
application could theoretically make use of it.  I think this is a weak
argument, given that dm-zoned doesn't even support such a mode.

I definitely see this udev rule as a temporary workaround.

> So now we're stuck with this temporary situation which needs a work-around.
> I don't think it's a terrible idea to have a rule that just sets
> deadline/mq-deadline for an SMR device regardless of what kernel it is
> running on. It'll probably never be a bad default.

OK.  Barring future input to the contrary, I'll work to get updates into
fedora, at least.  I've CC'd Colin and Hannes.  I'm not sure who else to
include.

FYI, below is the udev rule Damien had provided to Bryan.  I'm not sure
about the KERNEL=="sd[a-z]" bit, that may need modification.  Note: I'm
no udev expert.

Cheers,
Jeff

ACTION=="add|change", KERNEL=="sd[a-z]",
ATTRS{queue/zoned}=="host-managed", ATTR{queue/scheduler}="deadline"

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Jens Axboe]

On 5/30/18 9:45 AM, Jeff Moyer wrote:
> Jens Axboe <axboe@kernel.dk> writes:
> 
>> On 5/30/18 9:06 AM, Jeff Moyer wrote:
>>> Hi, Jens,
>>>
>>> Jens Axboe <axboe@kernel.dk> writes:
>>>
>>>> On 5/30/18 2:49 AM, Christoph Hellwig wrote:
>>>>> While I really don't want drivers to change the I/O schedule themselves
>>>>> we have a class of devices (zoned) that don't work at all with certain
>>>>> I/O schedulers.  The kernel not chosing something sane and requiring
>>>>> user workarounds is just silly.
>>>>
>>>> They work just fine for probing and reading purposes. There's absolutely
>>>> no reason why we can't handle these special snowflakes with a udev rule.
>>>
>>> udev rules aren't shipped with the kernel, so it makes it hard to keep
>>> them in sync.  In this instance, I'm not sure anyone made an effort to
>>> notify distributions that a udev rule was even necessary.  (Is there a
>>> way of notifying distributions about kernel changes that require new
>>> udev rules, other than emailing each list individually?)
>>>
>>> From a technical standpoint, I totally agree with you, Jens.  However, I
>>> think the user experience sucks.  4.15 worked by default, 4.16 doesn't.
>>> The result will be bug reports from users (to the drive vendors,
>>> distribution bugzillas, here, etc.).
>>
>> I would imagine that most folks get their updates from a distro of some
>> sort, in which case there's absolutely nothing stopping the distro from
>> shipping updated rules for the 4.16 kernel update.
> 
> The problem is distros have already shipped that kernel.  :)

Ship an update, then! I'm sure that most people would prefer a simple
rule update over a kernel update. And you have to do one of them, to
resolve this anyway.

>> But what's the regression? 4.15 had no zone write locking at all.
> 
> The zone write locking was done in the sd driver prior to 4.16.  See
> commit 39051dd85f287 ("scsi: sd: Remove zone write locking") for where
> it was removed.  That means these devices "just worked" with all I/O
> schedulers.

Gotcha, makes sense.

>>> Moving on, assuming your mind is made up...
>>>
>>> I'm not sure how much logic should go into the udev rule.  As mentioned,
>>> this limitation was introduced in 4.16, and Damien has plans to lift the
>>> restriction in future kernels.  Because distributions tend to cherry
>>> pick changes, making decisions on whether a feature exists based solely
>>> on kernel version is usually not a great thing.  My inclination would be
>>> to just always force deadline for host-managed SMR drives.  These drives
>>> aren't that popular, after all.  Any opinions on this?
>>
>> The problem is that it's tied to an IO scheduler, which ends up causing
>> issues like this, since users are free to select a different scheduler.
>> Then things break. Granted, in this case, some extraordinarily shitty
>> hardware even broke. That is on the hardware, not the kernel, that
>> kind of breakage should not occur.
> 
> If the firmware problem was widespread, I think we'd try to avoid it.  I
> have no reason to believe that is the case, though.
> 
> Damien made the argument that the user should be able to select an I/O
> scheduler that doesn't perform the write locking, because a well-behaved
> application could theoretically make use of it.  I think this is a weak
> argument, given that dm-zoned doesn't even support such a mode.

Sure, the user should be able to select whatever they want. Maybe they
are strictly using it through bsg or a similar interface, in which
case no scheduler or kernel support is really neeeded to drive it.

>> So now we're stuck with this temporary situation which needs a work-around.
>> I don't think it's a terrible idea to have a rule that just sets
>> deadline/mq-deadline for an SMR device regardless of what kernel it is
>> running on. It'll probably never be a bad default.
> 
> OK.  Barring future input to the contrary, I'll work to get updates into
> fedora, at least.  I've CC'd Colin and Hannes.  I'm not sure who else to
> include.
> 
> FYI, below is the udev rule Damien had provided to Bryan.  I'm not sure
> about the KERNEL=="sd[a-z]" bit, that may need modification.  Note: I'm
> no udev expert.

Rule looks fine to me, but I'm also no udev expert.

-- 
Jens Axboe

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Damien Le Moal]

DQoNCk9uIDUvMzEvMTggMDA6NTIsIEplbnMgQXhib2Ugd3JvdGU6DQo+IE9uIDUvMzAvMTggOTo0
NSBBTSwgSmVmZiBNb3llciB3cm90ZToNCj4+IEplbnMgQXhib2UgPGF4Ym9lQGtlcm5lbC5kaz4g
d3JpdGVzOg0KPj4NCj4+PiBPbiA1LzMwLzE4IDk6MDYgQU0sIEplZmYgTW95ZXIgd3JvdGU6DQo+
Pj4+IEhpLCBKZW5zLA0KPj4+Pg0KPj4+PiBKZW5zIEF4Ym9lIDxheGJvZUBrZXJuZWwuZGs+IHdy
aXRlczoNCj4+Pj4NCj4+Pj4+IE9uIDUvMzAvMTggMjo0OSBBTSwgQ2hyaXN0b3BoIEhlbGx3aWcg
d3JvdGU6DQo+Pj4+Pj4gV2hpbGUgSSByZWFsbHkgZG9uJ3Qgd2FudCBkcml2ZXJzIHRvIGNoYW5n
ZSB0aGUgSS9PIHNjaGVkdWxlIHRoZW1zZWx2ZXMNCj4+Pj4+PiB3ZSBoYXZlIGEgY2xhc3Mgb2Yg
ZGV2aWNlcyAoem9uZWQpIHRoYXQgZG9uJ3Qgd29yayBhdCBhbGwgd2l0aCBjZXJ0YWluDQo+Pj4+
Pj4gSS9PIHNjaGVkdWxlcnMuICBUaGUga2VybmVsIG5vdCBjaG9zaW5nIHNvbWV0aGluZyBzYW5l
IGFuZCByZXF1aXJpbmcNCj4+Pj4+PiB1c2VyIHdvcmthcm91bmRzIGlzIGp1c3Qgc2lsbHkuDQo+
Pj4+Pg0KPj4+Pj4gVGhleSB3b3JrIGp1c3QgZmluZSBmb3IgcHJvYmluZyBhbmQgcmVhZGluZyBw
dXJwb3Nlcy4gVGhlcmUncyBhYnNvbHV0ZWx5DQo+Pj4+PiBubyByZWFzb24gd2h5IHdlIGNhbid0
IGhhbmRsZSB0aGVzZSBzcGVjaWFsIHNub3dmbGFrZXMgd2l0aCBhIHVkZXYgcnVsZS4NCj4+Pj4N
Cj4+Pj4gdWRldiBydWxlcyBhcmVuJ3Qgc2hpcHBlZCB3aXRoIHRoZSBrZXJuZWwsIHNvIGl0IG1h
a2VzIGl0IGhhcmQgdG8ga2VlcA0KPj4+PiB0aGVtIGluIHN5bmMuICBJbiB0aGlzIGluc3RhbmNl
LCBJJ20gbm90IHN1cmUgYW55b25lIG1hZGUgYW4gZWZmb3J0IHRvDQo+Pj4+IG5vdGlmeSBkaXN0
cmlidXRpb25zIHRoYXQgYSB1ZGV2IHJ1bGUgd2FzIGV2ZW4gbmVjZXNzYXJ5LiAgKElzIHRoZXJl
IGENCj4+Pj4gd2F5IG9mIG5vdGlmeWluZyBkaXN0cmlidXRpb25zIGFib3V0IGtlcm5lbCBjaGFu
Z2VzIHRoYXQgcmVxdWlyZSBuZXcNCj4+Pj4gdWRldiBydWxlcywgb3RoZXIgdGhhbiBlbWFpbGlu
ZyBlYWNoIGxpc3QgaW5kaXZpZHVhbGx5PykNCj4+Pj4NCj4+Pj4gRnJvbSBhIHRlY2huaWNhbCBz
dGFuZHBvaW50LCBJIHRvdGFsbHkgYWdyZWUgd2l0aCB5b3UsIEplbnMuICBIb3dldmVyLCBJDQo+
Pj4+IHRoaW5rIHRoZSB1c2VyIGV4cGVyaWVuY2Ugc3Vja3MuICA0LjE1IHdvcmtlZCBieSBkZWZh
dWx0LCA0LjE2IGRvZXNuJ3QuDQo+Pj4+IFRoZSByZXN1bHQgd2lsbCBiZSBidWcgcmVwb3J0cyBm
cm9tIHVzZXJzICh0byB0aGUgZHJpdmUgdmVuZG9ycywNCj4+Pj4gZGlzdHJpYnV0aW9uIGJ1Z3pp
bGxhcywgaGVyZSwgZXRjLikuDQo+Pj4NCj4+PiBJIHdvdWxkIGltYWdpbmUgdGhhdCBtb3N0IGZv
bGtzIGdldCB0aGVpciB1cGRhdGVzIGZyb20gYSBkaXN0cm8gb2Ygc29tZQ0KPj4+IHNvcnQsIGlu
IHdoaWNoIGNhc2UgdGhlcmUncyBhYnNvbHV0ZWx5IG5vdGhpbmcgc3RvcHBpbmcgdGhlIGRpc3Ry
byBmcm9tDQo+Pj4gc2hpcHBpbmcgdXBkYXRlZCBydWxlcyBmb3IgdGhlIDQuMTYga2VybmVsIHVw
ZGF0ZS4NCj4+DQo+PiBUaGUgcHJvYmxlbSBpcyBkaXN0cm9zIGhhdmUgYWxyZWFkeSBzaGlwcGVk
IHRoYXQga2VybmVsLiAgOikNCj4gDQo+IFNoaXAgYW4gdXBkYXRlLCB0aGVuISBJJ20gc3VyZSB0
aGF0IG1vc3QgcGVvcGxlIHdvdWxkIHByZWZlciBhIHNpbXBsZQ0KPiBydWxlIHVwZGF0ZSBvdmVy
IGEga2VybmVsIHVwZGF0ZS4gQW5kIHlvdSBoYXZlIHRvIGRvIG9uZSBvZiB0aGVtLCB0bw0KPiBy
ZXNvbHZlIHRoaXMgYW55d2F5Lg0KPiANCj4+PiBCdXQgd2hhdCdzIHRoZSByZWdyZXNzaW9uPyA0
LjE1IGhhZCBubyB6b25lIHdyaXRlIGxvY2tpbmcgYXQgYWxsLg0KPj4NCj4+IFRoZSB6b25lIHdy
aXRlIGxvY2tpbmcgd2FzIGRvbmUgaW4gdGhlIHNkIGRyaXZlciBwcmlvciB0byA0LjE2LiAgU2Vl
DQo+PiBjb21taXQgMzkwNTFkZDg1ZjI4NyAoInNjc2k6IHNkOiBSZW1vdmUgem9uZSB3cml0ZSBs
b2NraW5nIikgZm9yIHdoZXJlDQo+PiBpdCB3YXMgcmVtb3ZlZC4gIFRoYXQgbWVhbnMgdGhlc2Ug
ZGV2aWNlcyAianVzdCB3b3JrZWQiIHdpdGggYWxsIEkvTw0KPj4gc2NoZWR1bGVycy4NCj4gDQo+
IEdvdGNoYSwgbWFrZXMgc2Vuc2UuDQo+IA0KPj4+PiBNb3Zpbmcgb24sIGFzc3VtaW5nIHlvdXIg
bWluZCBpcyBtYWRlIHVwLi4uDQo+Pj4+DQo+Pj4+IEknbSBub3Qgc3VyZSBob3cgbXVjaCBsb2dp
YyBzaG91bGQgZ28gaW50byB0aGUgdWRldiBydWxlLiAgQXMgbWVudGlvbmVkLA0KPj4+PiB0aGlz
IGxpbWl0YXRpb24gd2FzIGludHJvZHVjZWQgaW4gNC4xNiwgYW5kIERhbWllbiBoYXMgcGxhbnMg
dG8gbGlmdCB0aGUNCj4+Pj4gcmVzdHJpY3Rpb24gaW4gZnV0dXJlIGtlcm5lbHMuICBCZWNhdXNl
IGRpc3RyaWJ1dGlvbnMgdGVuZCB0byBjaGVycnkNCj4+Pj4gcGljayBjaGFuZ2VzLCBtYWtpbmcg
ZGVjaXNpb25zIG9uIHdoZXRoZXIgYSBmZWF0dXJlIGV4aXN0cyBiYXNlZCBzb2xlbHkNCj4+Pj4g
b24ga2VybmVsIHZlcnNpb24gaXMgdXN1YWxseSBub3QgYSBncmVhdCB0aGluZy4gIE15IGluY2xp
bmF0aW9uIHdvdWxkIGJlDQo+Pj4+IHRvIGp1c3QgYWx3YXlzIGZvcmNlIGRlYWRsaW5lIGZvciBo
b3N0LW1hbmFnZWQgU01SIGRyaXZlcy4gIFRoZXNlIGRyaXZlcw0KPj4+PiBhcmVuJ3QgdGhhdCBw
b3B1bGFyLCBhZnRlciBhbGwuICBBbnkgb3BpbmlvbnMgb24gdGhpcz8NCj4+Pg0KPj4+IFRoZSBw
cm9ibGVtIGlzIHRoYXQgaXQncyB0aWVkIHRvIGFuIElPIHNjaGVkdWxlciwgd2hpY2ggZW5kcyB1
cCBjYXVzaW5nDQo+Pj4gaXNzdWVzIGxpa2UgdGhpcywgc2luY2UgdXNlcnMgYXJlIGZyZWUgdG8g
c2VsZWN0IGEgZGlmZmVyZW50IHNjaGVkdWxlci4NCj4+PiBUaGVuIHRoaW5ncyBicmVhay4gR3Jh
bnRlZCwgaW4gdGhpcyBjYXNlLCBzb21lIGV4dHJhb3JkaW5hcmlseSBzaGl0dHkNCj4+PiBoYXJk
d2FyZSBldmVuIGJyb2tlLiBUaGF0IGlzIG9uIHRoZSBoYXJkd2FyZSwgbm90IHRoZSBrZXJuZWws
IHRoYXQNCj4+PiBraW5kIG9mIGJyZWFrYWdlIHNob3VsZCBub3Qgb2NjdXIuDQo+Pg0KPj4gSWYg
dGhlIGZpcm13YXJlIHByb2JsZW0gd2FzIHdpZGVzcHJlYWQsIEkgdGhpbmsgd2UnZCB0cnkgdG8g
YXZvaWQgaXQuICBJDQo+PiBoYXZlIG5vIHJlYXNvbiB0byBiZWxpZXZlIHRoYXQgaXMgdGhlIGNh
c2UsIHRob3VnaC4NCj4+DQo+PiBEYW1pZW4gbWFkZSB0aGUgYXJndW1lbnQgdGhhdCB0aGUgdXNl
ciBzaG91bGQgYmUgYWJsZSB0byBzZWxlY3QgYW4gSS9PDQo+PiBzY2hlZHVsZXIgdGhhdCBkb2Vz
bid0IHBlcmZvcm0gdGhlIHdyaXRlIGxvY2tpbmcsIGJlY2F1c2UgYSB3ZWxsLWJlaGF2ZWQNCj4+
IGFwcGxpY2F0aW9uIGNvdWxkIHRoZW9yZXRpY2FsbHkgbWFrZSB1c2Ugb2YgaXQuICBJIHRoaW5r
IHRoaXMgaXMgYSB3ZWFrDQo+PiBhcmd1bWVudCwgZ2l2ZW4gdGhhdCBkbS16b25lZCBkb2Vzbid0
IGV2ZW4gc3VwcG9ydCBzdWNoIGEgbW9kZS4NCj4gDQo+IFN1cmUsIHRoZSB1c2VyIHNob3VsZCBi
ZSBhYmxlIHRvIHNlbGVjdCB3aGF0ZXZlciB0aGV5IHdhbnQuIE1heWJlIHRoZXkNCj4gYXJlIHN0
cmljdGx5IHVzaW5nIGl0IHRocm91Z2ggYnNnIG9yIGEgc2ltaWxhciBpbnRlcmZhY2UsIGluIHdo
aWNoDQo+IGNhc2Ugbm8gc2NoZWR1bGVyIG9yIGtlcm5lbCBzdXBwb3J0IGlzIHJlYWxseSBuZWVl
ZGVkIHRvIGRyaXZlIGl0Lg0KDQpkbS16b25lZCwgZjJmcyAobW9yZSBGU2VzIGNvbWluZykgdXNl
IHRoZSBkcml2ZXMgdGhyb3VnaCB0aGUgcmVndWxhcg0KYmxvY2svYmlvIHN0YWNrLiBUaGUgc2No
ZWR1bGVyIGlzIGludm9sdmVkIGFuZCBuZWVkcyB0byBiZSBjb3JyZWN0bHkNCnNldC4gRm9yIGFw
cGxpY2F0aW9ucyBub3QgcmVseWluZyBvbiB0aGVzZSBjb21wb25lbnRzIGFuZCBkb2luZyByYXcg
ZGlzaw0KSS9PcywgSSBzZWUgMiBjYW1wcyBpbiB0aGUgZmllbGQ6IHRoZSBzZy9ic2cgY2FtcCBh
bmQgdGhlIHJlZ3VsYXIgUE9TSVgNCnN5c3RlbSBjYWxscyBjYW1wLiBGb3IgdGhlIGZvcm1lciwg
c2luY2UgdGhlIGtlcm5lbCB3aWxsIGhhdmUgbWluaW1hbA0KaW50ZXJhY3Rpb24gd2l0aCB0aGUg
Y29tbWFuZHMsIHRoZSBhcHBsaWNhdGlvbiBpcyBvbiBpdHMgb3duLiBDb250cm9sDQpvdmVyIHdy
aXRlIG9yZGVyaW5nIGhhcyB0byBiZSBjb2RlZCBpbi4gQnV0IGZvciB0aGUgbGF0dGVyIGNhc2Us
IHdoaWNoDQppcyBvcmRlcnMgb2YgbWFnbml0dWRlcyBlYXNpZXIgdG8gdXNlLCB0aGUgc2NoZWR1
bGVyIG5lZWRzIHRvIGJlIHNldA0KY29ycmVjdGx5IG9yIHRoZSBzYW1lIHByZXNzdXJlIGlzIG9u
IHRoZSBhcHBsaWNhdGlvbiB0byAiZG8gdGhlIHJpZ2h0DQp0aGluZyIuIFRoaXMgY29uZmxpY3Rz
IHdpdGggdGhlIGJlbmVmaXRzIG9mIHRoaXMgYWNjZXNzIHBhdGggY2hvaWNlDQooc2ltcGxpY2l0
eSkuDQoNCkluIHRoZSBlbmQsIGlmIHRoZSBkcml2ZXMgYXJlIHVzZWQgZGlyZWN0bHkgZnJvbSBh
cHBsaWNhdGlvbnMsIEkgdGhpbmsNCml0IGlzIE9LIHRvIG9ubHkgZXhwZWN0IGEgY29ycmVjdCBz
eXN0ZW0gc2V0dGluZyBpZiBkZWFkbGluZSBpcw0KcmVxdWlyZWQuIFNvIHVkZXYgaXMgZmluZS4g
QnV0IGZvciB0aGUga2VybmVsIGNvbXBvbmVudHMgbGlrZSBkbS16b25lZCwNCmEgc2FuZSBkZWZh
dWx0IGJlaW5nIHNldCBmcm9tIHRoZSBzdGFydCBpcyBteSBwcmVmZXJyZWQgY2hvaWNlLg0KDQpC
ZXN0IHJlZ2FyZHMuDQoNCi0tIA0KRGFtaWVuIExlIE1vYWwsDQpXZXN0ZXJuIERpZ2l0YWw=

Re: [PATCH 0/2][RFC] block: default to deadline for SMR devices

[Damien Le Moal]

DQpPbiA1LzMxLzE4IDAwOjQ1LCBKZWZmIE1veWVyIHdyb3RlOg0KPiBKZW5zIEF4Ym9lIDxheGJv
ZUBrZXJuZWwuZGs+IHdyaXRlczoNCj4+IEJ1dCB3aGF0J3MgdGhlIHJlZ3Jlc3Npb24/IDQuMTUg
aGFkIG5vIHpvbmUgd3JpdGUgbG9ja2luZyBhdCBhbGwuDQo+IA0KPiBUaGUgem9uZSB3cml0ZSBs
b2NraW5nIHdhcyBkb25lIGluIHRoZSBzZCBkcml2ZXIgcHJpb3IgdG8gNC4xNi4gIFNlZQ0KPiBj
b21taXQgMzkwNTFkZDg1ZjI4NyAoInNjc2k6IHNkOiBSZW1vdmUgem9uZSB3cml0ZSBsb2NraW5n
IikgZm9yIHdoZXJlDQo+IGl0IHdhcyByZW1vdmVkLiAgVGhhdCBtZWFucyB0aGVzZSBkZXZpY2Vz
ICJqdXN0IHdvcmtlZCIgd2l0aCBhbGwgSS9PDQo+IHNjaGVkdWxlcnMuDQoNClllcyB0aGV5IGRp
ZCAianVzdCB3b3JrIiwgYnV0IHRoYXQgd2FzIG5vdCBhbiBpZGVhbCBzb2x1dGlvbiBlaXRoZXIN
CmJlY2F1c2Ugb2YgdGhlIHBlcmZvcm1hbmNlIGltcGxpY2F0aW9uczogc2VxdWVudGlhbCB3cml0
ZXMgdG8gYSBzaW5nbGUNCnpvbmUgd2hlcmUgc3RhbGxpbmcgdGhlIGRpc3BhdGNoIHF1ZXVlIHdh
aXRpbmcgZm9yIHRoZSBkaXNwYXRjaGVkIHdyaXRlDQp0byB0aGUgbG9ja2VkIHpvbmUgdG8gY29t
cGxldGUuIFRoYXQgd2FzIG5vdCBvcHRpbWFsIGF0IGFsbCAoc3VyZSwgdGhlDQpkcml2ZSBzaWRl
IHdyaXRlIGNhY2hpbmcgd2FzIGhpZGluZyB0aGlzIGEgYml0LCBidXQgc3RpbGwpDQoNCj4+PiBN
b3Zpbmcgb24sIGFzc3VtaW5nIHlvdXIgbWluZCBpcyBtYWRlIHVwLi4uDQo+Pj4NCj4+PiBJJ20g
bm90IHN1cmUgaG93IG11Y2ggbG9naWMgc2hvdWxkIGdvIGludG8gdGhlIHVkZXYgcnVsZS4gIEFz
IG1lbnRpb25lZCwNCj4+PiB0aGlzIGxpbWl0YXRpb24gd2FzIGludHJvZHVjZWQgaW4gNC4xNiwg
YW5kIERhbWllbiBoYXMgcGxhbnMgdG8gbGlmdCB0aGUNCj4+PiByZXN0cmljdGlvbiBpbiBmdXR1
cmUga2VybmVscy4gIEJlY2F1c2UgZGlzdHJpYnV0aW9ucyB0ZW5kIHRvIGNoZXJyeQ0KPj4+IHBp
Y2sgY2hhbmdlcywgbWFraW5nIGRlY2lzaW9ucyBvbiB3aGV0aGVyIGEgZmVhdHVyZSBleGlzdHMg
YmFzZWQgc29sZWx5DQo+Pj4gb24ga2VybmVsIHZlcnNpb24gaXMgdXN1YWxseSBub3QgYSBncmVh
dCB0aGluZy4gIE15IGluY2xpbmF0aW9uIHdvdWxkIGJlDQo+Pj4gdG8ganVzdCBhbHdheXMgZm9y
Y2UgZGVhZGxpbmUgZm9yIGhvc3QtbWFuYWdlZCBTTVIgZHJpdmVzLiAgVGhlc2UgZHJpdmVzDQo+
Pj4gYXJlbid0IHRoYXQgcG9wdWxhciwgYWZ0ZXIgYWxsLiAgQW55IG9waW5pb25zIG9uIHRoaXM/
DQo+Pg0KPj4gVGhlIHByb2JsZW0gaXMgdGhhdCBpdCdzIHRpZWQgdG8gYW4gSU8gc2NoZWR1bGVy
LCB3aGljaCBlbmRzIHVwIGNhdXNpbmcNCj4+IGlzc3VlcyBsaWtlIHRoaXMsIHNpbmNlIHVzZXJz
IGFyZSBmcmVlIHRvIHNlbGVjdCBhIGRpZmZlcmVudCBzY2hlZHVsZXIuDQo+PiBUaGVuIHRoaW5n
cyBicmVhay4gR3JhbnRlZCwgaW4gdGhpcyBjYXNlLCBzb21lIGV4dHJhb3JkaW5hcmlseSBzaGl0
dHkNCj4+IGhhcmR3YXJlIGV2ZW4gYnJva2UuIFRoYXQgaXMgb24gdGhlIGhhcmR3YXJlLCBub3Qg
dGhlIGtlcm5lbCwgdGhhdA0KPj4ga2luZCBvZiBicmVha2FnZSBzaG91bGQgbm90IG9jY3VyLg0K
PiANCj4gSWYgdGhlIGZpcm13YXJlIHByb2JsZW0gd2FzIHdpZGVzcHJlYWQsIEkgdGhpbmsgd2Un
ZCB0cnkgdG8gYXZvaWQgaXQuICBJDQo+IGhhdmUgbm8gcmVhc29uIHRvIGJlbGlldmUgdGhhdCBp
cyB0aGUgY2FzZSwgdGhvdWdoLg0KDQpGaXJzdCB0aW1lIGluIG15IGNhcmVlciB0aGF0IEkgaGVh
cmQgb2YgYSBkaXNrIGJyZWFraW5nIGEgc3lzdGVtIEJJT1MuIEkNCndpbGwgbm90aWZ5IG91ciBz
eXN0ZW0gdGVzdCBsYWIgdG8gaW52ZXN0aWdhdGUgdGhpcy4gSmVmZiwgbGV0J3MgdGFrZQ0KdGhp
cyBkaXNjdXNzaW9uIG9mZi1saXN0IHNpbmNlIHRoYXQgaXMgbm90IGtlcm5lbCByZWxhdGVkLg0K
DQo+IERhbWllbiBtYWRlIHRoZSBhcmd1bWVudCB0aGF0IHRoZSB1c2VyIHNob3VsZCBiZSBhYmxl
IHRvIHNlbGVjdCBhbiBJL08NCj4gc2NoZWR1bGVyIHRoYXQgZG9lc24ndCBwZXJmb3JtIHRoZSB3
cml0ZSBsb2NraW5nLCBiZWNhdXNlIGEgd2VsbC1iZWhhdmVkDQo+IGFwcGxpY2F0aW9uIGNvdWxk
IHRoZW9yZXRpY2FsbHkgbWFrZSB1c2Ugb2YgaXQuICBJIHRoaW5rIHRoaXMgaXMgYSB3ZWFrDQo+
IGFyZ3VtZW50LCBnaXZlbiB0aGF0IGRtLXpvbmVkIGRvZXNuJ3QgZXZlbiBzdXBwb3J0IHN1Y2gg
YSBtb2RlLg0KDQpZZXMsIGEgbGl0dGxlIHdlZWsuIFRoYXQgaXMgZGVmaW5pdGVseSBub3QgdGhl
IG1haW4gdXNlIGNhc2UgSSBhbSBzZWVpbmcNCndpdGggY3VzdG9tZXJzLiBUaGF0IHNhaWQsIHRo
ZXNlIGRyaXZlcyBhcmUgc3RhcnRpbmcgdG8gYmUgdXNlZCB3aXRoDQpvdGhlciBmZWF0dXJlIHNl
dHMgYmVpbmcgZW5hYmxlZCBsaWtlIEkvTyBwcmlvcml0aWVzLiBDb25zaWRlcmluZyB0aGVyZQ0K
c2l6ZSwgdGhpcyBpcyBhIHZlcnkgaW50ZXJlc3RpbmcgZmVhdHVyZSB0byBjb250cm9sIGFjY2Vz
cyBsYXRlbmN5Lg0KRGVhZGxpbmUgYW5kIG1xLWRlYWRsaW5lIHdpbGwgbm90IGFjdCBvbiBJL08g
cHJpb3JpdGllcywgY2ZxIChhbmQgYmZxID8pDQp3aWxsLiBQb3RlbnRpYWxseSBiZXR0ZXIgcmVz
dWx0cyBjYW4gYmUgYWNoaWV2ZWQsIGJ1dCBhcyB0aGVzZQ0Kc2NoZWR1bGVycyBkbyBub3Qgc3Vw
cG9ydCB6b25lIHdyaXRlIGxvY2tpbmcsIHRoZSBhcHBsaWNhdGlvbiBuZWVkcyB0bw0KYmUgY2Fy
ZWZ1bCB3aXRoIGl0cyBwZXItem9uZSB3cml0ZSBxdWV1ZSBkZXB0aCBhbmQgZG9pbmcgc28gYXZv
aWQNCnRyaXBwaW5nIG92ZXIga2VybmVsIGxldmVsIGNvbW1hbmQgdW5pbnRlbmRlZCByZW9yZGVy
aW5nLiBJIHNlZSB0aGlzIGFzDQphIHZhbGlkIGVub3VnaCB1c2UgY2FzZSB0byBub3QgImxvY2st
ZG93biIgdGhlIHNjaGVkdWxlciB0byBkZWFkbGluZQ0Kb25seSBhbmQgYWxsb3cgb3RoZXJzIHNj
aGVkdWxlcnMgdG9vLiBZZXQsIGRlYWRsaW5lIHNob3VsZCBiZSB0aGUNCmRlZmF1bHQgdW50aWwg
YW4gYXBwbGljYXRpb24gYXNrcyBmb3Igc29tZXRoaW5nIGVsc2UgaWYgbmVlZGVkLg0KDQpkbS16
b25lZCBvciBmMmZzIChidHJmcyBpbiB0aGUgbGFiIHRvbykgImFzc3VtZSIgdGhhdCB0aGUgdW5k
ZXJseWluZw0Kc3RhY2sgZG9lcyB0aGUgcmlnaHQgdGhpbmcuIFRoYXQgaXMgb2YgY291cnNlIHRy
dWUgKGZvciBub3cpIG9ubHkgaWYgdGhlDQpkZWFkbGluZSBzY2hlZHVsZXIgaXMgZW5hYmxlZC4g
QSBzYW5lIGRlZmF1bHQgc2V0IG9uIGRldmljZQ0KaW5pdGlhbGl6YXRpb24gd291bGQgYmUgbmlj
ZSB0byBoYXZlIGFuZCBhdm9pZCBwb3RlbnRpYWwgaGVhZGFjaGVzIHdpdGgNCnJ1bGUgb3JkZXJp
bmcgd2l0aCByZWdhcmQgdG8gY29tcG9uZW50cyBpbml0aWFsaXphdGlvbiAobm90IHRvIG1lbnRp
b24NCnRoYXQgdGhpcyB3b3VsZCBtYWtlIGJvb3RpbmcgZnJvbSB0aGVzZSBkaXNrcyBwb3NzaWJs
ZSkuDQoNCj4gSSBkZWZpbml0ZWx5IHNlZSB0aGlzIHVkZXYgcnVsZSBhcyBhIHRlbXBvcmFyeSB3
b3JrYXJvdW5kLg0KDQpJIGFncmVlLiBJbiBmYWN0IEkgc2VlIHRoZSBkZWFkbGluZSBiYXNlZCB6
b25lIHdyaXRlIGxvY2tpbmcgaXRzZWxmIGFzIGENCnRlbXBvcmFyeSB3b3JrYXJvdW5kLiBGb3Ig
bm93LCBJIGRvIG5vdCBzZWUgYW55IG90aGVyIGNsZWFuIG1ldGhvZCB0aGF0DQpjb3ZlcnMgYm90
aCBtcSBhbmQgbGVnYWN5IHBhdGguIENvbnNpZGVyaW5nIG9ubHkgbXEsIHdlIGRpc2N1c3NlZA0K
aW50ZXJlc3RpbmcgcG9zc2liaWxpdGllcyBhdCBMU0ZNTSB1c2luZyBkZWRpY2F0ZWQgd3JpdGUg
cXVldWVzLiBUaGF0DQpjb3VsZCBiZSBoYW5kbGVkIGdlbmVyaWNhbGx5IGFuZCByZW1vdmUgdGhl
IGRlcGVuZGVuY3kgb24gdGhlIHNjaGVkdWxlcg0Kd2hpbGUgYWxzbyBpbmNyZWFzaW5nIGNvdmVy
YWdlIG9mIHRoZSBzdXBwb3J0IHRvIG9wZW4gY2hhbm5lbCBTU0RzIGFzIHdlbGwuDQoNCk15IGd1
ZXNzIGlzIHRoYXQgbm8gbWFqb3IgY2hhbmdlIGZvciB0aGlzIHdyaXRlIGxvY2tpbmcgd2lsbCBo
YXBwZW4gb24NCnRoZSBsZWdhY3kgcGF0aCwgd2hpY2ggaG9wZWZ1bGx5IHdpbGwgZ28gYXdheSBz
b29uICg/KS4gQnV0IHRoZXJlIGFyZQ0Kb3B0aW9ucyBmb3J3YXJkIHdpdGggYmxrLW1xLg0KDQo+
PiBTbyBub3cgd2UncmUgc3R1Y2sgd2l0aCB0aGlzIHRlbXBvcmFyeSBzaXR1YXRpb24gd2hpY2gg
bmVlZHMgYSB3b3JrLWFyb3VuZC4NCj4+IEkgZG9uJ3QgdGhpbmsgaXQncyBhIHRlcnJpYmxlIGlk
ZWEgdG8gaGF2ZSBhIHJ1bGUgdGhhdCBqdXN0IHNldHMNCj4+IGRlYWRsaW5lL21xLWRlYWRsaW5l
IGZvciBhbiBTTVIgZGV2aWNlIHJlZ2FyZGxlc3Mgb2Ygd2hhdCBrZXJuZWwgaXQgaXMNCj4+IHJ1
bm5pbmcgb24uIEl0J2xsIHByb2JhYmx5IG5ldmVyIGJlIGEgYmFkIGRlZmF1bHQuDQoNCkkgYWdy
ZWUuIEJ1dCBzaW5jZSB0aGVyZSBhcmUgb3RoZXIga2VybmVsIGNvbXBvbmVudHMgKGRtLXpvbmVk
LCBGU2VzIGFuZA0KdGhlIGVudGlyZSBmcy9ibG9jay1kZXYuYyBkaXJlY3QgSS9PIHdyaXRlIHBh
dGgpIGRlcGVuZGluZyBvbiB0aGUNCnNjaGVkdWxlciB0byBiZSBzZXQgdG8gc29tZXRoaW5nIHNh
bmUsIHNldHRpbmcgdGhhdCBlYXJseSBvbiBpbiB0aGUNCmRldmljZSBpbml0aWFsaXphdGlvbiBi
ZWZvcmUgaXQgaXMgZ3JhYmJlZCBieSBhbiBGUyBvciBhIGRldmljZSBtYXBwZXINCndvdWxkIGRl
ZmluaXRlbHkgYmUgbmljZSB0byBoYXZlLg0KDQo+IA0KPiBPSy4gIEJhcnJpbmcgZnV0dXJlIGlu
cHV0IHRvIHRoZSBjb250cmFyeSwgSSdsbCB3b3JrIHRvIGdldCB1cGRhdGVzIGludG8NCj4gZmVk
b3JhLCBhdCBsZWFzdC4gIEkndmUgQ0MnZCBDb2xpbiBhbmQgSGFubmVzLiAgSSdtIG5vdCBzdXJl
IHdobyBlbHNlIHRvDQo+IGluY2x1ZGUuDQo+IA0KPiBGWUksIGJlbG93IGlzIHRoZSB1ZGV2IHJ1
bGUgRGFtaWVuIGhhZCBwcm92aWRlZCB0byBCcnlhbi4gIEknbSBub3Qgc3VyZQ0KPiBhYm91dCB0
aGUgS0VSTkVMPT0ic2RbYS16XSIgYml0LCB0aGF0IG1heSBuZWVkIG1vZGlmaWNhdGlvbi4gIE5v
dGU6IEknbQ0KPiBubyB1ZGV2IGV4cGVydC4NCg0KSXQgcHJvYmFibHkgbmVlZHMgdG8gYmUgc29t
ZXRoaW5nIGxpa2UgS0VSTkVMPT0ic2QqIiB0byBhbGxvdyBtb3JlIHRoYW4NCjI2IGRyaXZlcy4N
Cg0KQmVzdCByZWdhcmRzLg0KDQo+IA0KPiBDaGVlcnMsDQo+IEplZmYNCj4gDQo+IEFDVElPTj09
ImFkZHxjaGFuZ2UiLCBLRVJORUw9PSJzZFthLXpdIiwNCj4gQVRUUlN7cXVldWUvem9uZWR9PT0i
aG9zdC1tYW5hZ2VkIiwgQVRUUntxdWV1ZS9zY2hlZHVsZXJ9PSJkZWFkbGluZSINCj4gDQoNCi0t
IA0KRGFtaWVuIExlIE1vYWwsDQpXZXN0ZXJuIERpZ2l0YWw=