[PATCH v3] NFS: Fix writeback performance issue on cache invalidation

[PATCH v3] NFS: Fix writeback performance issue on cache invalidation

[Trond Myklebust]

If a cache invalidation is triggered, and we happen to have a lot of
writebacks cached at the time, then the call to invalidate_inode_pages2()
will end up calling ->launder_page() on each and every dirty page in order
to sync its contents to disk, thus defeating write coalescing.
The following patch ensures that we try to sync the inode to disk before
calling invalidate_inode_pages2() so that we do the writeback as efficiently
as possible.

Reported-by: William Dauchy <william@gandi.net>
Reported-by: Pascal Bouchareine <pascal@gandi.net>
Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
Tested-by: William Dauchy <william@gandi.net>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
---
v2: Add check for regular file as per Jeff Layton's suggestion.
v3: Minor cleanup and add Jeff as a reviewer

 fs/nfs/inode.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index af6e806..3ea4f64 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -963,9 +963,15 @@ EXPORT_SYMBOL_GPL(nfs_revalidate_inode);
 static int nfs_invalidate_mapping(struct inode *inode, struct address_space *mapping)
 {
 	struct nfs_inode *nfsi = NFS_I(inode);
-	
+	int ret;
+
 	if (mapping->nrpages != 0) {
-		int ret = invalidate_inode_pages2(mapping);
+		if (S_ISREG(inode->i_mode)) {
+			ret = nfs_sync_mapping(mapping);
+			if (ret < 0)
+				return ret;
+		}
+		ret = invalidate_inode_pages2(mapping);
 		if (ret < 0)
 			return ret;
 	}
-- 
1.8.3.1

Re: [PATCH v3] NFS: Fix writeback performance issue on cache invalidation

[Jeff Layton]

On Wed, 7 Aug 2013 17:14:24 -0400
Trond Myklebust <Trond.Myklebust@netapp.com> wrote:

> If a cache invalidation is triggered, and we happen to have a lot of
> writebacks cached at the time, then the call to invalidate_inode_pages2()
> will end up calling ->launder_page() on each and every dirty page in order
> to sync its contents to disk, thus defeating write coalescing.
> The following patch ensures that we try to sync the inode to disk before
> calling invalidate_inode_pages2() so that we do the writeback as efficiently
> as possible.
> 
> Reported-by: William Dauchy <william@gandi.net>
> Reported-by: Pascal Bouchareine <pascal@gandi.net>
> Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
> Tested-by: William Dauchy <william@gandi.net>
> Reviewed-by: Jeff Layton <jlayton@redhat.com>
> ---
> v2: Add check for regular file as per Jeff Layton's suggestion.
> v3: Minor cleanup and add Jeff as a reviewer
> 
>  fs/nfs/inode.c | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
> index af6e806..3ea4f64 100644
> --- a/fs/nfs/inode.c
> +++ b/fs/nfs/inode.c
> @@ -963,9 +963,15 @@ EXPORT_SYMBOL_GPL(nfs_revalidate_inode);
>  static int nfs_invalidate_mapping(struct inode *inode, struct address_space *mapping)
>  {
>  	struct nfs_inode *nfsi = NFS_I(inode);
> -	
> +	int ret;
> +
>  	if (mapping->nrpages != 0) {
> -		int ret = invalidate_inode_pages2(mapping);
> +		if (S_ISREG(inode->i_mode)) {
> +			ret = nfs_sync_mapping(mapping);
> +			if (ret < 0)
> +				return ret;
> +		}
> +		ret = invalidate_inode_pages2(mapping);
>  		if (ret < 0)
>  			return ret;
>  	}

It occurs to me that we have several places that call nfs_sync_mapping
without checking S_ISREG. Are they potentially problematic?

Might it make more sense to move the S_ISREG test inside of
nfs_sync_mapping and just have it "return 0" when it's not a regular
file?

-- 
Jeff Layton <jlayton@redhat.com>

Re: [PATCH v3] NFS: Fix writeback performance issue on cache invalidation

[Myklebust, Trond]

T24gVGh1LCAyMDEzLTA4LTA4IGF0IDE0OjExIC0wNDAwLCBKZWZmIExheXRvbiB3cm90ZToNCj4g
T24gV2VkLCA3IEF1ZyAyMDEzIDE3OjE0OjI0IC0wNDAwDQo+IFRyb25kIE15a2xlYnVzdCA8VHJv
bmQuTXlrbGVidXN0QG5ldGFwcC5jb20+IHdyb3RlOg0KPiANCj4gPiBJZiBhIGNhY2hlIGludmFs
aWRhdGlvbiBpcyB0cmlnZ2VyZWQsIGFuZCB3ZSBoYXBwZW4gdG8gaGF2ZSBhIGxvdCBvZg0KPiA+
IHdyaXRlYmFja3MgY2FjaGVkIGF0IHRoZSB0aW1lLCB0aGVuIHRoZSBjYWxsIHRvIGludmFsaWRh
dGVfaW5vZGVfcGFnZXMyKCkNCj4gPiB3aWxsIGVuZCB1cCBjYWxsaW5nIC0+bGF1bmRlcl9wYWdl
KCkgb24gZWFjaCBhbmQgZXZlcnkgZGlydHkgcGFnZSBpbiBvcmRlcg0KPiA+IHRvIHN5bmMgaXRz
IGNvbnRlbnRzIHRvIGRpc2ssIHRodXMgZGVmZWF0aW5nIHdyaXRlIGNvYWxlc2NpbmcuDQo+ID4g
VGhlIGZvbGxvd2luZyBwYXRjaCBlbnN1cmVzIHRoYXQgd2UgdHJ5IHRvIHN5bmMgdGhlIGlub2Rl
IHRvIGRpc2sgYmVmb3JlDQo+ID4gY2FsbGluZyBpbnZhbGlkYXRlX2lub2RlX3BhZ2VzMigpIHNv
IHRoYXQgd2UgZG8gdGhlIHdyaXRlYmFjayBhcyBlZmZpY2llbnRseQ0KPiA+IGFzIHBvc3NpYmxl
Lg0KPiA+IA0KPiA+IFJlcG9ydGVkLWJ5OiBXaWxsaWFtIERhdWNoeSA8d2lsbGlhbUBnYW5kaS5u
ZXQ+DQo+ID4gUmVwb3J0ZWQtYnk6IFBhc2NhbCBCb3VjaGFyZWluZSA8cGFzY2FsQGdhbmRpLm5l
dD4NCj4gPiBTaWduZWQtb2ZmLWJ5OiBUcm9uZCBNeWtsZWJ1c3QgPFRyb25kLk15a2xlYnVzdEBu
ZXRhcHAuY29tPg0KPiA+IFRlc3RlZC1ieTogV2lsbGlhbSBEYXVjaHkgPHdpbGxpYW1AZ2FuZGku
bmV0Pg0KPiA+IFJldmlld2VkLWJ5OiBKZWZmIExheXRvbiA8amxheXRvbkByZWRoYXQuY29tPg0K
PiA+IC0tLQ0KPiA+IHYyOiBBZGQgY2hlY2sgZm9yIHJlZ3VsYXIgZmlsZSBhcyBwZXIgSmVmZiBM
YXl0b24ncyBzdWdnZXN0aW9uLg0KPiA+IHYzOiBNaW5vciBjbGVhbnVwIGFuZCBhZGQgSmVmZiBh
cyBhIHJldmlld2VyDQo+ID4gDQo+ID4gIGZzL25mcy9pbm9kZS5jIHwgMTAgKysrKysrKystLQ0K
PiA+ICAxIGZpbGUgY2hhbmdlZCwgOCBpbnNlcnRpb25zKCspLCAyIGRlbGV0aW9ucygtKQ0KPiA+
IA0KPiA+IGRpZmYgLS1naXQgYS9mcy9uZnMvaW5vZGUuYyBiL2ZzL25mcy9pbm9kZS5jDQo+ID4g
aW5kZXggYWY2ZTgwNi4uM2VhNGY2NCAxMDA2NDQNCj4gPiAtLS0gYS9mcy9uZnMvaW5vZGUuYw0K
PiA+ICsrKyBiL2ZzL25mcy9pbm9kZS5jDQo+ID4gQEAgLTk2Myw5ICs5NjMsMTUgQEAgRVhQT1JU
X1NZTUJPTF9HUEwobmZzX3JldmFsaWRhdGVfaW5vZGUpOw0KPiA+ICBzdGF0aWMgaW50IG5mc19p
bnZhbGlkYXRlX21hcHBpbmcoc3RydWN0IGlub2RlICppbm9kZSwgc3RydWN0IGFkZHJlc3Nfc3Bh
Y2UgKm1hcHBpbmcpDQo+ID4gIHsNCj4gPiAgCXN0cnVjdCBuZnNfaW5vZGUgKm5mc2kgPSBORlNf
SShpbm9kZSk7DQo+ID4gLQkNCj4gPiArCWludCByZXQ7DQo+ID4gKw0KPiA+ICAJaWYgKG1hcHBp
bmctPm5ycGFnZXMgIT0gMCkgew0KPiA+IC0JCWludCByZXQgPSBpbnZhbGlkYXRlX2lub2RlX3Bh
Z2VzMihtYXBwaW5nKTsNCj4gPiArCQlpZiAoU19JU1JFRyhpbm9kZS0+aV9tb2RlKSkgew0KPiA+
ICsJCQlyZXQgPSBuZnNfc3luY19tYXBwaW5nKG1hcHBpbmcpOw0KPiA+ICsJCQlpZiAocmV0IDwg
MCkNCj4gPiArCQkJCXJldHVybiByZXQ7DQo+ID4gKwkJfQ0KPiA+ICsJCXJldCA9IGludmFsaWRh
dGVfaW5vZGVfcGFnZXMyKG1hcHBpbmcpOw0KPiA+ICAJCWlmIChyZXQgPCAwKQ0KPiA+ICAJCQly
ZXR1cm4gcmV0Ow0KPiA+ICAJfQ0KPiANCj4gSXQgb2NjdXJzIHRvIG1lIHRoYXQgd2UgaGF2ZSBz
ZXZlcmFsIHBsYWNlcyB0aGF0IGNhbGwgbmZzX3N5bmNfbWFwcGluZw0KPiB3aXRob3V0IGNoZWNr
aW5nIFNfSVNSRUcuIEFyZSB0aGV5IHBvdGVudGlhbGx5IHByb2JsZW1hdGljPw0KPiANCj4gTWln
aHQgaXQgbWFrZSBtb3JlIHNlbnNlIHRvIG1vdmUgdGhlIFNfSVNSRUcgdGVzdCBpbnNpZGUgb2YN
Cj4gbmZzX3N5bmNfbWFwcGluZyBhbmQganVzdCBoYXZlIGl0ICJyZXR1cm4gMCIgd2hlbiBpdCdz
IG5vdCBhIHJlZ3VsYXINCj4gZmlsZT8NCg0KSSBzZWUgNSBjYWxsZXJzIG9mIG5mc19zeW5jX21h
cHBpbmcoKSBhc2lkZSBmcm9tIHRoZSBhYm92ZTogMiBhcmUgaW4gdGhlDQpPX0RJUkVDVCBjb2Rl
LCB0aGUgb3RoZXIgMyBhcmUgYWxsIGluIHRoZSBmaWxlIGxvY2tpbmcgY29kZS4gQUZBSUNTLA0K
bm9uZSBvZiB0aG9zZSBjYW4gZXZlciBiZSBmZWQgdG8gbm9uLXJlZ3VsYXIgZmlsZXMuDQoNCkFt
IEkgbWlzc2luZyBhbnl0aGluZz8NCg0KQ2hlZXJzDQogIFRyb25kDQoNCi0tIA0KVHJvbmQgTXlr
bGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWts
ZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5jb20NCg==

Re: [PATCH v3] NFS: Fix writeback performance issue on cache invalidation

[Jeff Layton]

On Thu, 8 Aug 2013 18:21:35 +0000
"Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote:

> On Thu, 2013-08-08 at 14:11 -0400, Jeff Layton wrote:
> > On Wed, 7 Aug 2013 17:14:24 -0400
> > Trond Myklebust <Trond.Myklebust@netapp.com> wrote:
> > 
> > > If a cache invalidation is triggered, and we happen to have a lot of
> > > writebacks cached at the time, then the call to invalidate_inode_pages2()
> > > will end up calling ->launder_page() on each and every dirty page in order
> > > to sync its contents to disk, thus defeating write coalescing.
> > > The following patch ensures that we try to sync the inode to disk before
> > > calling invalidate_inode_pages2() so that we do the writeback as efficiently
> > > as possible.
> > > 
> > > Reported-by: William Dauchy <william@gandi.net>
> > > Reported-by: Pascal Bouchareine <pascal@gandi.net>
> > > Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
> > > Tested-by: William Dauchy <william@gandi.net>
> > > Reviewed-by: Jeff Layton <jlayton@redhat.com>
> > > ---
> > > v2: Add check for regular file as per Jeff Layton's suggestion.
> > > v3: Minor cleanup and add Jeff as a reviewer
> > > 
> > >  fs/nfs/inode.c | 10 ++++++++--
> > >  1 file changed, 8 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
> > > index af6e806..3ea4f64 100644
> > > --- a/fs/nfs/inode.c
> > > +++ b/fs/nfs/inode.c
> > > @@ -963,9 +963,15 @@ EXPORT_SYMBOL_GPL(nfs_revalidate_inode);
> > >  static int nfs_invalidate_mapping(struct inode *inode, struct address_space *mapping)
> > >  {
> > >  	struct nfs_inode *nfsi = NFS_I(inode);
> > > -	
> > > +	int ret;
> > > +
> > >  	if (mapping->nrpages != 0) {
> > > -		int ret = invalidate_inode_pages2(mapping);
> > > +		if (S_ISREG(inode->i_mode)) {
> > > +			ret = nfs_sync_mapping(mapping);
> > > +			if (ret < 0)
> > > +				return ret;
> > > +		}
> > > +		ret = invalidate_inode_pages2(mapping);
> > >  		if (ret < 0)
> > >  			return ret;
> > >  	}
> > 
> > It occurs to me that we have several places that call nfs_sync_mapping
> > without checking S_ISREG. Are they potentially problematic?
> > 
> > Might it make more sense to move the S_ISREG test inside of
> > nfs_sync_mapping and just have it "return 0" when it's not a regular
> > file?
> 
> I see 5 callers of nfs_sync_mapping() aside from the above: 2 are in the
> O_DIRECT code, the other 3 are all in the file locking code. AFAICS,
> none of those can ever be fed to non-regular files.
> 
> Am I missing anything?
> 

You can lock a directory or device special file though, right?

In practice I don't think there's any way to end up with dirty pages on
a !S_ISREG inode, but in that case, the S_ISREG check here would be
superfluous (though checking it might be a reasonable optimization).

-- 
Jeff Layton <jlayton@redhat.com>

Re: [PATCH v3] NFS: Fix writeback performance issue on cache invalidation

[Myklebust, Trond]

T24gVGh1LCAyMDEzLTA4LTA4IGF0IDE0OjQyIC0wNDAwLCBKZWZmIExheXRvbiB3cm90ZToNCj4g
T24gVGh1LCA4IEF1ZyAyMDEzIDE4OjIxOjM1ICswMDAwDQo+ICJNeWtsZWJ1c3QsIFRyb25kIiA8
VHJvbmQuTXlrbGVidXN0QG5ldGFwcC5jb20+IHdyb3RlOg0KPiANCj4gPiBPbiBUaHUsIDIwMTMt
MDgtMDggYXQgMTQ6MTEgLTA0MDAsIEplZmYgTGF5dG9uIHdyb3RlOg0KPiA+ID4gT24gV2VkLCA3
IEF1ZyAyMDEzIDE3OjE0OjI0IC0wNDAwDQo+ID4gPiBUcm9uZCBNeWtsZWJ1c3QgPFRyb25kLk15
a2xlYnVzdEBuZXRhcHAuY29tPiB3cm90ZToNCj4gPiA+IA0KPiA+ID4gPiBJZiBhIGNhY2hlIGlu
dmFsaWRhdGlvbiBpcyB0cmlnZ2VyZWQsIGFuZCB3ZSBoYXBwZW4gdG8gaGF2ZSBhIGxvdCBvZg0K
PiA+ID4gPiB3cml0ZWJhY2tzIGNhY2hlZCBhdCB0aGUgdGltZSwgdGhlbiB0aGUgY2FsbCB0byBp
bnZhbGlkYXRlX2lub2RlX3BhZ2VzMigpDQo+ID4gPiA+IHdpbGwgZW5kIHVwIGNhbGxpbmcgLT5s
YXVuZGVyX3BhZ2UoKSBvbiBlYWNoIGFuZCBldmVyeSBkaXJ0eSBwYWdlIGluIG9yZGVyDQo+ID4g
PiA+IHRvIHN5bmMgaXRzIGNvbnRlbnRzIHRvIGRpc2ssIHRodXMgZGVmZWF0aW5nIHdyaXRlIGNv
YWxlc2NpbmcuDQo+ID4gPiA+IFRoZSBmb2xsb3dpbmcgcGF0Y2ggZW5zdXJlcyB0aGF0IHdlIHRy
eSB0byBzeW5jIHRoZSBpbm9kZSB0byBkaXNrIGJlZm9yZQ0KPiA+ID4gPiBjYWxsaW5nIGludmFs
aWRhdGVfaW5vZGVfcGFnZXMyKCkgc28gdGhhdCB3ZSBkbyB0aGUgd3JpdGViYWNrIGFzIGVmZmlj
aWVudGx5DQo+ID4gPiA+IGFzIHBvc3NpYmxlLg0KPiA+ID4gPiANCj4gPiA+ID4gUmVwb3J0ZWQt
Ynk6IFdpbGxpYW0gRGF1Y2h5IDx3aWxsaWFtQGdhbmRpLm5ldD4NCj4gPiA+ID4gUmVwb3J0ZWQt
Ynk6IFBhc2NhbCBCb3VjaGFyZWluZSA8cGFzY2FsQGdhbmRpLm5ldD4NCj4gPiA+ID4gU2lnbmVk
LW9mZi1ieTogVHJvbmQgTXlrbGVidXN0IDxUcm9uZC5NeWtsZWJ1c3RAbmV0YXBwLmNvbT4NCj4g
PiA+ID4gVGVzdGVkLWJ5OiBXaWxsaWFtIERhdWNoeSA8d2lsbGlhbUBnYW5kaS5uZXQ+DQo+ID4g
PiA+IFJldmlld2VkLWJ5OiBKZWZmIExheXRvbiA8amxheXRvbkByZWRoYXQuY29tPg0KPiA+ID4g
PiAtLS0NCj4gPiA+ID4gdjI6IEFkZCBjaGVjayBmb3IgcmVndWxhciBmaWxlIGFzIHBlciBKZWZm
IExheXRvbidzIHN1Z2dlc3Rpb24uDQo+ID4gPiA+IHYzOiBNaW5vciBjbGVhbnVwIGFuZCBhZGQg
SmVmZiBhcyBhIHJldmlld2VyDQo+ID4gPiA+IA0KPiA+ID4gPiAgZnMvbmZzL2lub2RlLmMgfCAx
MCArKysrKysrKy0tDQo+ID4gPiA+ICAxIGZpbGUgY2hhbmdlZCwgOCBpbnNlcnRpb25zKCspLCAy
IGRlbGV0aW9ucygtKQ0KPiA+ID4gPiANCj4gPiA+ID4gZGlmZiAtLWdpdCBhL2ZzL25mcy9pbm9k
ZS5jIGIvZnMvbmZzL2lub2RlLmMNCj4gPiA+ID4gaW5kZXggYWY2ZTgwNi4uM2VhNGY2NCAxMDA2
NDQNCj4gPiA+ID4gLS0tIGEvZnMvbmZzL2lub2RlLmMNCj4gPiA+ID4gKysrIGIvZnMvbmZzL2lu
b2RlLmMNCj4gPiA+ID4gQEAgLTk2Myw5ICs5NjMsMTUgQEAgRVhQT1JUX1NZTUJPTF9HUEwobmZz
X3JldmFsaWRhdGVfaW5vZGUpOw0KPiA+ID4gPiAgc3RhdGljIGludCBuZnNfaW52YWxpZGF0ZV9t
YXBwaW5nKHN0cnVjdCBpbm9kZSAqaW5vZGUsIHN0cnVjdCBhZGRyZXNzX3NwYWNlICptYXBwaW5n
KQ0KPiA+ID4gPiAgew0KPiA+ID4gPiAgCXN0cnVjdCBuZnNfaW5vZGUgKm5mc2kgPSBORlNfSShp
bm9kZSk7DQo+ID4gPiA+IC0JDQo+ID4gPiA+ICsJaW50IHJldDsNCj4gPiA+ID4gKw0KPiA+ID4g
PiAgCWlmIChtYXBwaW5nLT5ucnBhZ2VzICE9IDApIHsNCj4gPiA+ID4gLQkJaW50IHJldCA9IGlu
dmFsaWRhdGVfaW5vZGVfcGFnZXMyKG1hcHBpbmcpOw0KPiA+ID4gPiArCQlpZiAoU19JU1JFRyhp
bm9kZS0+aV9tb2RlKSkgew0KPiA+ID4gPiArCQkJcmV0ID0gbmZzX3N5bmNfbWFwcGluZyhtYXBw
aW5nKTsNCj4gPiA+ID4gKwkJCWlmIChyZXQgPCAwKQ0KPiA+ID4gPiArCQkJCXJldHVybiByZXQ7
DQo+ID4gPiA+ICsJCX0NCj4gPiA+ID4gKwkJcmV0ID0gaW52YWxpZGF0ZV9pbm9kZV9wYWdlczIo
bWFwcGluZyk7DQo+ID4gPiA+ICAJCWlmIChyZXQgPCAwKQ0KPiA+ID4gPiAgCQkJcmV0dXJuIHJl
dDsNCj4gPiA+ID4gIAl9DQo+ID4gPiANCj4gPiA+IEl0IG9jY3VycyB0byBtZSB0aGF0IHdlIGhh
dmUgc2V2ZXJhbCBwbGFjZXMgdGhhdCBjYWxsIG5mc19zeW5jX21hcHBpbmcNCj4gPiA+IHdpdGhv
dXQgY2hlY2tpbmcgU19JU1JFRy4gQXJlIHRoZXkgcG90ZW50aWFsbHkgcHJvYmxlbWF0aWM/DQo+
ID4gPiANCj4gPiA+IE1pZ2h0IGl0IG1ha2UgbW9yZSBzZW5zZSB0byBtb3ZlIHRoZSBTX0lTUkVH
IHRlc3QgaW5zaWRlIG9mDQo+ID4gPiBuZnNfc3luY19tYXBwaW5nIGFuZCBqdXN0IGhhdmUgaXQg
InJldHVybiAwIiB3aGVuIGl0J3Mgbm90IGEgcmVndWxhcg0KPiA+ID4gZmlsZT8NCj4gPiANCj4g
PiBJIHNlZSA1IGNhbGxlcnMgb2YgbmZzX3N5bmNfbWFwcGluZygpIGFzaWRlIGZyb20gdGhlIGFi
b3ZlOiAyIGFyZSBpbiB0aGUNCj4gPiBPX0RJUkVDVCBjb2RlLCB0aGUgb3RoZXIgMyBhcmUgYWxs
IGluIHRoZSBmaWxlIGxvY2tpbmcgY29kZS4gQUZBSUNTLA0KPiA+IG5vbmUgb2YgdGhvc2UgY2Fu
IGV2ZXIgYmUgZmVkIHRvIG5vbi1yZWd1bGFyIGZpbGVzLg0KPiA+IA0KPiA+IEFtIEkgbWlzc2lu
ZyBhbnl0aGluZz8NCj4gPiANCj4gDQo+IFlvdSBjYW4gbG9jayBhIGRpcmVjdG9yeSBvciBkZXZp
Y2Ugc3BlY2lhbCBmaWxlIHRob3VnaCwgcmlnaHQ/DQoNCk5vLiBpbm9kZS0+aV9mb3AgaXMgc3Bl
Y2lmaWMgdG8gdGhlIHJlZ3VsYXIgZmlsZXMuIE5GU3Y0IHdvbid0IHN1cHBvcnQNCmxvY2tpbmcg
b24gYW55dGhpbmcgd2hpY2ggeW91IGNhbid0IE9QRU4uDQoNClBPU0lYIGl0c2VsZiBkb2VzIGFw
cGVhciB0byBhbGxvdyBsb2NraW5nIG9uIGRpcmVjdG9yaWVzLCBidXQgc2luY2UgeW91DQpjYW4g
b25seSBpbiBwcmFjdGljZSBkbyByZWFkIGxvY2tzIChBRkFJSyB5b3UgY2FuJ3Qgb3BlbiBhIGRp
cmVjdG9yeSBmb3INCndyaXRpbmcpIHRoZW4gSSdtIG5vdCBsb3NpbmcgYW55IHNsZWVwIG92ZXIg
aXQuDQoNCj4gSW4gcHJhY3RpY2UgSSBkb24ndCB0aGluayB0aGVyZSdzIGFueSB3YXkgdG8gZW5k
IHVwIHdpdGggZGlydHkgcGFnZXMgb24NCj4gYSAhU19JU1JFRyBpbm9kZSwgYnV0IGluIHRoYXQg
Y2FzZSwgdGhlIFNfSVNSRUcgY2hlY2sgaGVyZSB3b3VsZCBiZQ0KPiBzdXBlcmZsdW91cyAodGhv
dWdoIGNoZWNraW5nIGl0IG1pZ2h0IGJlIGEgcmVhc29uYWJsZSBvcHRpbWl6YXRpb24pLg0KDQpB
RkFJQ1IsIG5mc19zeW5jX21hcHBpbmcoKSBkb2VzIGFjdHVhbGx5IGJsb3cgdXAgaWYgeW91IGNh
bGwgaXQgb24gYQ0Kbm9uLXJlZ3VsYXIgZmlsZSwgYnV0IGFzIEkgc2FpZCwgdGhlIG9ubHkgcGxh
Y2Ugd2hlcmUgdGhhdCBhcHBlYXJzIHRvIGJlDQpwb3NzaWJsZSBpcyBuZnNfaW52YWxpZGF0ZV9t
YXBwaW5nKCkuDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50
YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWtsZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5j
b20NCg==