From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexandre DERUMIER Subject: Re: ceph osd commit latency increase over time, until restart Date: Mon, 4 Feb 2019 15:17:13 +0100 (CET) Message-ID: <229754897.167048.1549289833437.JavaMail.zimbra@oxygem.tv> References: <395511117.2665.1548405853447.JavaMail.zimbra@oxygem.tv> <1548181710.219518.1548833599717.JavaMail.zimbra@oxygem.tv> <317086845.245472.1548856741512.JavaMail.zimbra@oxygem.tv> <85320911-75f8-0e9d-af71-151391839153@profihost.ag> <1814646360.255765.1548874695212.JavaMail.zimbra@oxygem.tv> <494474215.139609.1549269491013.JavaMail.zimbra@oxygem.tv> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <494474215.139609.1549269491013.JavaMail.zimbra-M8QNeUgB6UTyG1zEObXtfA@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org Sender: "ceph-users" To: "Stefan Priebe, Profihost AG" , Mark Nelson Cc: ceph-users , ceph-devel List-Id: ceph-devel.vger.kernel.org SGkgYWdhaW4sCgpJIHNwZWFrIHRvbyBmYXN0LCB0aGUgcHJvYmxlbSBoYXMgb2NjdXJlZCBhZ2Fp biwgc28gaXQncyBub3QgdGNtYWxsb2MgY2FjaGUgc2l6ZSByZWxhdGVkLgoKCkkgaGF2ZSBub3Rp Y2Ugc29tZXRoaW5nIHVzaW5nIGEgc2ltcGxlICJwZXJmIHRvcCIsCgplYWNoIHRpbWUgSSBoYXZl IHRoaXMgcHJvYmxlbSAoSSBoYXZlIHNlZW4gZXhhY3RseSA0IHRpbWVzIHRoZSBzYW1lIGJlaGF2 aW91ciksCgp3aGVuIGxhdGVuY3kgaXMgYmFkLCBwZXJmIHRvcCBnaXZlIG1lIDogCgpTdHVwaWRB bGxvY2F0b3I6Ol9hbGlnbmVkX2xlbgphbmQKYnRyZWU6OmJ0cmVlX2l0ZXJhdG9yPGJ0cmVlOjpi dHJlZV9ub2RlPGJ0cmVlOjpidHJlZV9tYXBfcGFyYW1zPHVuc2lnbmVkIGxvbmcsIHVuc2lnbmVk IGxvbmcsIHN0ZDo6bGVzczx1bnNpZ25lZCBsb25nPiwgbWVtcG9vCmw6OnBvb2xfYWxsb2NhdG9y PChtZW1wb29sOjpwb29sX2luZGV4X3QpMSwgc3RkOjpwYWlyPHVuc2lnbmVkIGxvbmcgY29uc3Qs IHVuc2lnbmVkIGxvbmc+ID4sIDI1Nj4gPiwgc3RkOjpwYWlyPHVuc2lnbmVkIGxvbmcgY29uc3Qs IHVuc2lnbmVkIGxvbmc+Jiwgc3RkOjpwYWlyPHVuc2lnbmVkIGxvbmcKY29uc3QsIHVuc2lnbmVk IGxvbmc+Kj46OmluY3JlbWVudF9zbG93KCkKCihhcm91bmQgMTAtMjAlIHRpbWUgZm9yIGJvdGgp CgoKd2hlbiBsYXRlbmN5IGlzIGdvb2QsIEkgZG9uJ3Qgc2VlIHRoZW0gYXQgYWxsLgoKCkkgaGF2 ZSB1c2VkIHRoZSBNYXJrIHdhbGxjbG9jayBwcm9maWxlciwgaGVyZSB0aGUgcmVzdWx0czoKCmh0 dHA6Ly9vZGlzb3dlYjEub2Rpc28ubmV0L2dkYnBtcC1vay50eHQKCmh0dHA6Ly9vZGlzb3dlYjEu b2Rpc28ubmV0L2dkYnBtcC1iYWQudHh0CgoKaGVyZSBhbiBleHRyYWN0IG9mIHRoZSB0aHJlYWQg d2l0aCBidHJlZTo6YnRyZWVfaXRlcmF0b3IgJiYgU3R1cGlkQWxsb2NhdG9yOjpfYWxpZ25lZF9s ZW4KCgorIDEwMC4wMCUgY2xvbmUKICArIDEwMC4wMCUgc3RhcnRfdGhyZWFkCiAgICArIDEwMC4w MCUgU2hhcmRlZFRocmVhZFBvb2w6OldvcmtUaHJlYWRTaGFyZGVkOjplbnRyeSgpCiAgICAgICsg MTAwLjAwJSBTaGFyZGVkVGhyZWFkUG9vbDo6c2hhcmRlZHRocmVhZHBvb2xfd29ya2VyKHVuc2ln bmVkIGludCkKICAgICAgICArIDEwMC4wMCUgT1NEOjpTaGFyZGVkT3BXUTo6X3Byb2Nlc3ModW5z aWduZWQgaW50LCBjZXBoOjpoZWFydGJlYXRfaGFuZGxlX2QqKQogICAgICAgICAgKyA3MC4wMCUg UEdPcEl0ZW06OnJ1bihPU0QqLCBPU0RTaGFyZCosIGJvb3N0OjppbnRydXNpdmVfcHRyPFBHPiYs IFRocmVhZFBvb2w6OlRQSGFuZGxlJikKICAgICAgICAgIHwgKyA3MC4wMCUgT1NEOjpkZXF1ZXVl X29wKGJvb3N0OjppbnRydXNpdmVfcHRyPFBHPiwgYm9vc3Q6OmludHJ1c2l2ZV9wdHI8T3BSZXF1 ZXN0PiwgVGhyZWFkUG9vbDo6VFBIYW5kbGUmKQogICAgICAgICAgfCAgICsgNzAuMDAlIFByaW1h cnlMb2dQRzo6ZG9fcmVxdWVzdChib29zdDo6aW50cnVzaXZlX3B0cjxPcFJlcXVlc3Q+JiwgVGhy ZWFkUG9vbDo6VFBIYW5kbGUmKQogICAgICAgICAgfCAgICAgKyA2OC4wMCUgUEdCYWNrZW5kOjpo YW5kbGVfbWVzc2FnZShib29zdDo6aW50cnVzaXZlX3B0cjxPcFJlcXVlc3Q+KQogICAgICAgICAg fCAgICAgfCArIDY4LjAwJSBSZXBsaWNhdGVkQmFja2VuZDo6X2hhbmRsZV9tZXNzYWdlKGJvb3N0 OjppbnRydXNpdmVfcHRyPE9wUmVxdWVzdD4pCiAgICAgICAgICB8ICAgICB8ICAgKyA2OC4wMCUg UmVwbGljYXRlZEJhY2tlbmQ6OmRvX3JlcG9wKGJvb3N0OjppbnRydXNpdmVfcHRyPE9wUmVxdWVz dD4pCiAgICAgICAgICB8ICAgICB8ICAgICArIDY3LjAwJSBub24tdmlydHVhbCB0aHVuayB0byBQ cmltYXJ5TG9nUEc6OnF1ZXVlX3RyYW5zYWN0aW9ucyhzdGQ6OnZlY3RvcjxPYmplY3RTdG9yZTo6 VHJhbnNhY3Rpb24sIHN0ZDo6YWxsb2NhdG9yPE9iamVjdFN0b3JlOjpUcmFuc2FjdGlvbj4gPiYs IGJvb3N0OjppbnRydXNpdmVfcHRyPE9wUmVxdWVzdD4pCiAgICAgICAgICB8ICAgICB8ICAgICB8 ICsgNjcuMDAlIEJsdWVTdG9yZTo6cXVldWVfdHJhbnNhY3Rpb25zKGJvb3N0OjppbnRydXNpdmVf cHRyPE9iamVjdFN0b3JlOjpDb2xsZWN0aW9uSW1wbD4mLCBzdGQ6OnZlY3RvcjxPYmplY3RTdG9y ZTo6VHJhbnNhY3Rpb24sIHN0ZDo6YWxsb2NhdG9yPE9iamVjdFN0b3JlOjpUcmFuc2FjdGlvbj4g PiYsIGJvb3N0OjppbnRydXNpdmVfcHRyPFRyYWNrZWRPcD4sIFRocmVhZFBvb2w6OlRQSGFuZGxl KikKICAgICAgICAgIHwgICAgIHwgICAgIHwgICArIDY2LjAwJSBCbHVlU3RvcmU6Ol90eGNfYWRk X3RyYW5zYWN0aW9uKEJsdWVTdG9yZTo6VHJhbnNDb250ZXh0KiwgT2JqZWN0U3RvcmU6OlRyYW5z YWN0aW9uKikKICAgICAgICAgIHwgICAgIHwgICAgIHwgICB8ICsgNjYuMDAlIEJsdWVTdG9yZTo6 X3dyaXRlKEJsdWVTdG9yZTo6VHJhbnNDb250ZXh0KiwgYm9vc3Q6OmludHJ1c2l2ZV9wdHI8Qmx1 ZVN0b3JlOjpDb2xsZWN0aW9uPiYsIGJvb3N0OjppbnRydXNpdmVfcHRyPEJsdWVTdG9yZTo6T25v ZGU+JiwgdW5zaWduZWQgbG9uZywgdW5zaWduZWQgbG9uZywgY2VwaDo6YnVmZmVyOjpsaXN0Jiwg dW5zaWduZWQgaW50KQogICAgICAgICAgfCAgICAgfCAgICAgfCAgIHwgICArIDY2LjAwJSBCbHVl U3RvcmU6Ol9kb193cml0ZShCbHVlU3RvcmU6OlRyYW5zQ29udGV4dCosIGJvb3N0OjppbnRydXNp dmVfcHRyPEJsdWVTdG9yZTo6Q29sbGVjdGlvbj4mLCBib29zdDo6aW50cnVzaXZlX3B0cjxCbHVl U3RvcmU6Ok9ub2RlPiwgdW5zaWduZWQgbG9uZywgdW5zaWduZWQgbG9uZywgY2VwaDo6YnVmZmVy OjpsaXN0JiwgdW5zaWduZWQgaW50KQogICAgICAgICAgfCAgICAgfCAgICAgfCAgIHwgICAgICsg NjUuMDAlIEJsdWVTdG9yZTo6X2RvX2FsbG9jX3dyaXRlKEJsdWVTdG9yZTo6VHJhbnNDb250ZXh0 KiwgYm9vc3Q6OmludHJ1c2l2ZV9wdHI8Qmx1ZVN0b3JlOjpDb2xsZWN0aW9uPiwgYm9vc3Q6Omlu dHJ1c2l2ZV9wdHI8Qmx1ZVN0b3JlOjpPbm9kZT4sIEJsdWVTdG9yZTo6V3JpdGVDb250ZXh0KikK ICAgICAgICAgIHwgICAgIHwgICAgIHwgICB8ICAgICB8ICsgNjQuMDAlIFN0dXBpZEFsbG9jYXRv cjo6YWxsb2NhdGUodW5zaWduZWQgbG9uZywgdW5zaWduZWQgbG9uZywgdW5zaWduZWQgbG9uZywg bG9uZywgc3RkOjp2ZWN0b3I8Ymx1ZXN0b3JlX3BleHRlbnRfdCwgbWVtcG9vbDo6cG9vbF9hbGxv Y2F0b3I8KG1lbXBvb2w6OnBvb2xfaW5kZXhfdCk0LCBibHVlc3RvcmVfcGV4dGVudF90PiA+KikK ICAgICAgICAgIHwgICAgIHwgICAgIHwgICB8ICAgICB8IHwgKyA2NC4wMCUgU3R1cGlkQWxsb2Nh dG9yOjphbGxvY2F0ZV9pbnQodW5zaWduZWQgbG9uZywgdW5zaWduZWQgbG9uZywgbG9uZywgdW5z aWduZWQgbG9uZyosIHVuc2lnbmVkIGludCopCiAgICAgICAgICB8ICAgICB8ICAgICB8ICAgfCAg ICAgfCB8ICAgKyAzNC4wMCUgYnRyZWU6OmJ0cmVlX2l0ZXJhdG9yPGJ0cmVlOjpidHJlZV9ub2Rl PGJ0cmVlOjpidHJlZV9tYXBfcGFyYW1zPHVuc2lnbmVkIGxvbmcsIHVuc2lnbmVkIGxvbmcsIHN0 ZDo6bGVzczx1bnNpZ25lZCBsb25nPiwgbWVtcG9vbDo6cG9vbF9hbGxvY2F0b3I8KG1lbXBvb2w6 OnBvb2xfaW5kZXhfdCkxLCBzdGQ6OnBhaXI8dW5zaWduZWQgbG9uZyBjb25zdCwgdW5zaWduZWQg bG9uZz4gPiwgMjU2PiA+LCBzdGQ6OnBhaXI8dW5zaWduZWQgbG9uZyBjb25zdCwgdW5zaWduZWQg bG9uZz4mLCBzdGQ6OnBhaXI8dW5zaWduZWQgbG9uZyBjb25zdCwgdW5zaWduZWQgbG9uZz4qPjo6 aW5jcmVtZW50X3Nsb3coKQogICAgICAgICAgfCAgICAgfCAgICAgfCAgIHwgICAgIHwgfCAgICsg MjYuMDAlIFN0dXBpZEFsbG9jYXRvcjo6X2FsaWduZWRfbGVuKGludGVydmFsX3NldDx1bnNpZ25l ZCBsb25nLCBidHJlZTo6YnRyZWVfbWFwPHVuc2lnbmVkIGxvbmcsIHVuc2lnbmVkIGxvbmcsIHN0 ZDo6bGVzczx1bnNpZ25lZCBsb25nPiwgbWVtcG9vbDo6cG9vbF9hbGxvY2F0b3I8KG1lbXBvb2w6 OnBvb2xfaW5kZXhfdCkxLCBzdGQ6OnBhaXI8dW5zaWduZWQgbG9uZyBjb25zdCwgdW5zaWduZWQg bG9uZz4gPiwgMjU2PiA+OjppdGVyYXRvciwgdW5zaWduZWQgbG9uZykKCgoKLS0tLS0gTWFpbCBv cmlnaW5hbCAtLS0tLQpEZTogIkFsZXhhbmRyZSBEZXJ1bWllciIgPGFkZXJ1bWllckBvZGlzby5j b20+CsOAOiAiU3RlZmFuIFByaWViZSwgUHJvZmlob3N0IEFHIiA8cy5wcmllYmVAcHJvZmlob3N0 LmFnPgpDYzogIlNhZ2UgV2VpbCIgPHNhZ2VAbmV3ZHJlYW0ubmV0PiwgImNlcGgtdXNlcnMiIDxj ZXBoLXVzZXJzQGxpc3RzLmNlcGguY29tPiwgImNlcGgtZGV2ZWwiIDxjZXBoLWRldmVsQHZnZXIu a2VybmVsLm9yZz4KRW52b3nDqTogTHVuZGkgNCBGw6l2cmllciAyMDE5IDA5OjM4OjExCk9iamV0 OiBSZTogW2NlcGgtdXNlcnNdIGNlcGggb3NkIGNvbW1pdCBsYXRlbmN5IGluY3JlYXNlIG92ZXIg dGltZSwgdW50aWwgcmVzdGFydAoKSGksIAoKc29tZSBuZXdzOiAKCkkgaGF2ZSB0cmllZCB3aXRo IGRpZmZlcmVudCB0cmFuc3BhcmVudCBodWdlcGFnZSB2YWx1ZXMgKG1hZHZpc2UsIG5ldmVyKSA6 IG5vIGNoYW5nZSAKCkkgaGF2ZSB0cmllZCB0byBpbmNyZWFzZSBibHVlc3RvcmVfY2FjaGVfc2l6 ZV9zc2QgdG8gOEc6IG5vIGNoYW5nZSAKCkkgaGF2ZSB0cmllZCB0byBpbmNyZWFzZSBUQ01BTExP Q19NQVhfVE9UQUxfVEhSRUFEX0NBQ0hFX0JZVEVTIHRvIDI1Nm1iIDogaXQgc2VlbSB0byBoZWxw LCBhZnRlciAyNGggSSdtIHN0aWxsIGFyb3VuZCAxLDVtcy4gKG5lZWQgdG8gd2FpdCBzb21lIG1v cmUgZGF5cyB0byBiZSBzdXJlKSAKCgpOb3RlIHRoYXQgdGhpcyBiZWhhdmlvdXIgc2VlbSB0byBo YXBwZW4gcmVhbGx5IGZhc3RlciAoPCAyIGRheXMpIG9uIG15IGJpZyBudm1lIGRyaXZlcyAoNlRC KSwgCm15IG90aGVycyBjbHVzdGVycyB1c2VyIDEsNlRCIHNzZC4gCgpDdXJyZW50bHkgSSdtIHVz aW5nIG9ubHkgMSBvc2QgYnkgbnZtZSAoSSBkb24ndCBoYXZlIG1vcmUgdGhhbiA1MDAwaW9wcyBi eSBvc2QpLCBidXQgSSdsbCB0cnkgdGhpcyB3ZWVrIHdpdGggMm9zZCBieSBudm1lLCB0byBzZWUg aWYgaXQncyBoZWxwaW5nLiAKCgpCVFcsIGRvZXMgc29tZWJvZHkgaGF2ZSBhbHJlYWR5IHRlc3Rl ZCBjZXBoIHdpdGhvdXQgdGNtYWxsb2MsIHdpdGggZ2xpYmMgPj0gMi4yNiAod2hpY2ggaGF2ZSBh bHNvIHRocmVhZCBjYWNoZSkgPyAKCgpSZWdhcmRzLCAKCkFsZXhhbmRyZSAKCgotLS0tLSBNYWls IG9yaWdpbmFsIC0tLS0tIApEZTogImFkZXJ1bWllciIgPGFkZXJ1bWllckBvZGlzby5jb20+IArD gDogIlN0ZWZhbiBQcmllYmUsIFByb2ZpaG9zdCBBRyIgPHMucHJpZWJlQHByb2ZpaG9zdC5hZz4g CkNjOiAiU2FnZSBXZWlsIiA8c2FnZUBuZXdkcmVhbS5uZXQ+LCAiY2VwaC11c2VycyIgPGNlcGgt dXNlcnNAbGlzdHMuY2VwaC5jb20+LCAiY2VwaC1kZXZlbCIgPGNlcGgtZGV2ZWxAdmdlci5rZXJu ZWwub3JnPiAKRW52b3nDqTogTWVyY3JlZGkgMzAgSmFudmllciAyMDE5IDE5OjU4OjE1IApPYmpl dDogUmU6IFtjZXBoLXVzZXJzXSBjZXBoIG9zZCBjb21taXQgbGF0ZW5jeSBpbmNyZWFzZSBvdmVy IHRpbWUsIHVudGlsIHJlc3RhcnQgCgo+PlRoYW5rcy4gSXMgdGhlcmUgYW55IHJlYXNvbiB5b3Ug bW9uaXRvciBvcF93X2xhdGVuY3kgYnV0IG5vdCAKPj5vcF9yX2xhdGVuY3kgYnV0IGluc3RlYWQg b3BfbGF0ZW5jeT8gCj4+IAo+PkFsc28gd2h5IGRvIHlvdSBtb25pdG9yIG9wX3dfcHJvY2Vzc19s YXRlbmN5PyBidXQgbm90IG9wX3JfcHJvY2Vzc19sYXRlbmN5PyAKCkkgbW9uaXRvciByZWFkIHRv by4gKEkgaGF2ZSBhbGwgbWV0cmljcyBmb3Igb3NkIHNvY2tldHMsIGFuZCBhIGxvdCBvZiBncmFw aHMpLiAKCkkganVzdCBkb24ndCBzZWUgbGF0ZW5jeSBkaWZmZXJlbmNlIG9uIHJlYWRzLiAob3Ig dGhleSBhcmUgdmVyeSB2ZXJ5IHNtYWxsIHZzIHRoZSB3cml0ZSBsYXRlbmN5IGluY3JlYXNlKSAK CgoKLS0tLS0gTWFpbCBvcmlnaW5hbCAtLS0tLSAKRGU6ICJTdGVmYW4gUHJpZWJlLCBQcm9maWhv c3QgQUciIDxzLnByaWViZUBwcm9maWhvc3QuYWc+IArDgDogImFkZXJ1bWllciIgPGFkZXJ1bWll ckBvZGlzby5jb20+IApDYzogIlNhZ2UgV2VpbCIgPHNhZ2VAbmV3ZHJlYW0ubmV0PiwgImNlcGgt dXNlcnMiIDxjZXBoLXVzZXJzQGxpc3RzLmNlcGguY29tPiwgImNlcGgtZGV2ZWwiIDxjZXBoLWRl dmVsQHZnZXIua2VybmVsLm9yZz4gCkVudm95w6k6IE1lcmNyZWRpIDMwIEphbnZpZXIgMjAxOSAx OTo1MDoyMCAKT2JqZXQ6IFJlOiBbY2VwaC11c2Vyc10gY2VwaCBvc2QgY29tbWl0IGxhdGVuY3kg aW5jcmVhc2Ugb3ZlciB0aW1lLCB1bnRpbCByZXN0YXJ0IAoKSGksIAoKQW0gMzAuMDEuMTkgdW0g MTQ6NTkgc2NocmllYiBBbGV4YW5kcmUgREVSVU1JRVI6IAo+IEhpIFN0ZWZhbiwgCj4gCj4+PiBj dXJyZW50bHkgaSdtIGluIHRoZSBwcm9jZXNzIG9mIHN3aXRjaGluZyBiYWNrIGZyb20gamVtYWxs b2MgdG8gdGNtYWxsb2MgCj4+PiBsaWtlIHN1Z2dlc3RlZC4gVGhpcyByZXBvcnQgbWFrZXMgbWUg YSBsaXR0bGUgbmVydm91cyBhYm91dCBteSBjaGFuZ2UuIAo+IFdlbGwsSSdtIHJlYWxseSBub3Qg c3VyZSB0aGF0IGl0J3MgYSB0Y21hbGxvYyBidWcuIAo+IG1heWJlIGJsdWVzdG9yZSByZWxhdGVk IChkb24ndCBoYXZlIGZpbGVzdG9yZSBhbnltb3JlIHRvIGNvbXBhcmUpIAo+IEkgbmVlZCB0byBj b21wYXJlIHdpdGggYmlnZ2VyIGxhdGVuY2llcyAKPiAKPiBoZXJlIGFuIGV4YW1wbGUsIHdoZW4g YWxsIG9zZCBhdCAyMC01MG1zIGJlZm9yZSByZXN0YXJ0LCB0aGVuIGFmdGVyIHJlc3RhcnQgKGF0 IDIxOjE1KSwgMW1zIAo+IGh0dHA6Ly9vZGlzb3dlYjEub2Rpc28ubmV0L2xhdGVuY3liYWQucG5n IAo+IAo+IEkgb2JzZXJ2ZSB0aGUgbGF0ZW5jeSBpbiBteSBndWVzdCB2bSB0b28sIG9uIGRpc2tz IGlvd2FpdC4gCj4gCj4gaHR0cDovL29kaXNvd2ViMS5vZGlzby5uZXQvbGF0ZW5jeWJhZHZtLnBu ZyAKPiAKPj4+IEFsc28gaSdtIGN1cnJlbnRseSBvbmx5IG1vbml0b3JpbmcgbGF0ZW5jeSBmb3Ig ZmlsZXN0b3JlIG9zZHMuIFdoaWNoIAo+Pj4gZXhhY3QgdmFsdWVzIG91dCBvZiB0aGUgZGFlbW9u IGRvIHlvdSB1c2UgZm9yIGJsdWVzdG9yZT8gCj4gCj4gaGVyZSBteSBpbmZsdXhkYiBxdWVyaWVz OiAKPiAKPiBJdCB0YWtlIG9wX2xhdGVuY3kuc3VtL29wX2xhdGVuY3kuYXZnY291bnQgb24gbGFz dCBzZWNvbmQuIAo+IAo+IAo+IFNFTEVDVCBub25fbmVnYXRpdmVfZGVyaXZhdGl2ZShmaXJzdCgi b3BfbGF0ZW5jeS5zdW0iKSwgMXMpL25vbl9uZWdhdGl2ZV9kZXJpdmF0aXZlKGZpcnN0KCJvcF9s YXRlbmN5LmF2Z2NvdW50IiksMXMpIEZST00gImNlcGgiIFdIRVJFICJob3N0IiA9fiAvXihbW2hv c3RdXSkkLyBBTkQgImlkIiA9fiAvXihbW29zZF1dKSQvIEFORCAkdGltZUZpbHRlciBHUk9VUCBC WSB0aW1lKCRpbnRlcnZhbCksICJob3N0IiwgImlkIiBmaWxsKHByZXZpb3VzKSAKPiAKPiAKPiBT RUxFQ1Qgbm9uX25lZ2F0aXZlX2Rlcml2YXRpdmUoZmlyc3QoIm9wX3dfbGF0ZW5jeS5zdW0iKSwg MXMpL25vbl9uZWdhdGl2ZV9kZXJpdmF0aXZlKGZpcnN0KCJvcF93X2xhdGVuY3kuYXZnY291bnQi KSwxcykgRlJPTSAiY2VwaCIgV0hFUkUgImhvc3QiID1+IC9eKFtbaG9zdF1dKSQvIEFORCBjb2xs ZWN0aW9uPSdvc2QnIEFORCAiaWQiID1+IC9eKFtbb3NkXV0pJC8gQU5EICR0aW1lRmlsdGVyIEdS T1VQIEJZIHRpbWUoJGludGVydmFsKSwgImhvc3QiLCAiaWQiIGZpbGwocHJldmlvdXMpIAo+IAo+ IAo+IFNFTEVDVCBub25fbmVnYXRpdmVfZGVyaXZhdGl2ZShmaXJzdCgib3Bfd19wcm9jZXNzX2xh dGVuY3kuc3VtIiksIDFzKS9ub25fbmVnYXRpdmVfZGVyaXZhdGl2ZShmaXJzdCgib3Bfd19wcm9j ZXNzX2xhdGVuY3kuYXZnY291bnQiKSwxcykgRlJPTSAiY2VwaCIgV0hFUkUgImhvc3QiID1+IC9e KFtbaG9zdF1dKSQvIEFORCBjb2xsZWN0aW9uPSdvc2QnIEFORCAiaWQiID1+IC9eKFtbb3NkXV0p JC8gQU5EICR0aW1lRmlsdGVyIEdST1VQIEJZIHRpbWUoJGludGVydmFsKSwgImhvc3QiLCAiaWQi IGZpbGwocHJldmlvdXMpIAoKVGhhbmtzLiBJcyB0aGVyZSBhbnkgcmVhc29uIHlvdSBtb25pdG9y IG9wX3dfbGF0ZW5jeSBidXQgbm90IApvcF9yX2xhdGVuY3kgYnV0IGluc3RlYWQgb3BfbGF0ZW5j eT8gCgpBbHNvIHdoeSBkbyB5b3UgbW9uaXRvciBvcF93X3Byb2Nlc3NfbGF0ZW5jeT8gYnV0IG5v dCBvcF9yX3Byb2Nlc3NfbGF0ZW5jeT8gCgpncmVldHMsIApTdGVmYW4gCgo+IAo+IAo+IAo+IAo+ IAo+IC0tLS0tIE1haWwgb3JpZ2luYWwgLS0tLS0gCj4gRGU6ICJTdGVmYW4gUHJpZWJlLCBQcm9m aWhvc3QgQUciIDxzLnByaWViZUBwcm9maWhvc3QuYWc+IAo+IMOAOiAiYWRlcnVtaWVyIiA8YWRl cnVtaWVyQG9kaXNvLmNvbT4sICJTYWdlIFdlaWwiIDxzYWdlQG5ld2RyZWFtLm5ldD4gCj4gQ2M6 ICJjZXBoLXVzZXJzIiA8Y2VwaC11c2Vyc0BsaXN0cy5jZXBoLmNvbT4sICJjZXBoLWRldmVsIiA8 Y2VwaC1kZXZlbEB2Z2VyLmtlcm5lbC5vcmc+IAo+IEVudm95w6k6IE1lcmNyZWRpIDMwIEphbnZp ZXIgMjAxOSAwODo0NTozMyAKPiBPYmpldDogUmU6IFtjZXBoLXVzZXJzXSBjZXBoIG9zZCBjb21t aXQgbGF0ZW5jeSBpbmNyZWFzZSBvdmVyIHRpbWUsIHVudGlsIHJlc3RhcnQgCj4gCj4gSGksIAo+ IAo+IEFtIDMwLjAxLjE5IHVtIDA4OjMzIHNjaHJpZWIgQWxleGFuZHJlIERFUlVNSUVSOiAKPj4g SGksIAo+PiAKPj4gaGVyZSBzb21lIG5ldyByZXN1bHRzLCAKPj4gZGlmZmVyZW50IG9zZC8gZGlm ZmVyZW50IGNsdXN0ZXIgCj4+IAo+PiBiZWZvcmUgb3NkIHJlc3RhcnQgbGF0ZW5jeSB3YXMgYmV0 d2VlbiAyLTVtcyAKPj4gYWZ0ZXIgb3NkIHJlc3RhcnQgaXMgYXJvdW5kIDEtMS41bXMgCj4+IAo+ PiBodHRwOi8vb2Rpc293ZWIxLm9kaXNvLm5ldC9jZXBocGVyZjIvYmFkLnR4dCAoMi01bXMpIAo+ PiBodHRwOi8vb2Rpc293ZWIxLm9kaXNvLm5ldC9jZXBocGVyZjIvb2sudHh0ICgxLTEuNW1zKSAK Pj4gaHR0cDovL29kaXNvd2ViMS5vZGlzby5uZXQvY2VwaHBlcmYyL2RpZmYudHh0IAo+PiAKPj4g RnJvbSB3aGF0IEkgc2VlIGluIGRpZmYsIHRoZSBiaWdnZXN0IGRpZmZlcmVuY2UgaXMgaW4gdGNt YWxsb2MsIGJ1dCBtYXliZSBJJ20gd3JvbmcuIAo+PiAoSSdtIHVzaW5nIHRjbWFsbG9jIDIuNS0y LjIpIAo+IAo+IGN1cnJlbnRseSBpJ20gaW4gdGhlIHByb2Nlc3Mgb2Ygc3dpdGNoaW5nIGJhY2sg ZnJvbSBqZW1hbGxvYyB0byB0Y21hbGxvYyAKPiBsaWtlIHN1Z2dlc3RlZC4gVGhpcyByZXBvcnQg bWFrZXMgbWUgYSBsaXR0bGUgbmVydm91cyBhYm91dCBteSBjaGFuZ2UuIAo+IAo+IEFsc28gaSdt IGN1cnJlbnRseSBvbmx5IG1vbml0b3JpbmcgbGF0ZW5jeSBmb3IgZmlsZXN0b3JlIG9zZHMuIFdo aWNoIAo+IGV4YWN0IHZhbHVlcyBvdXQgb2YgdGhlIGRhZW1vbiBkbyB5b3UgdXNlIGZvciBibHVl c3RvcmU/IAo+IAo+IEkgd291bGQgbGlrZSB0byBjaGVjayBpZiBpIHNlZSB0aGUgc2FtZSBiZWhh dmlvdXIuIAo+IAo+IEdyZWV0cywgCj4gU3RlZmFuIAo+IAo+PiAKPj4gLS0tLS0gTWFpbCBvcmln aW5hbCAtLS0tLSAKPj4gRGU6ICJTYWdlIFdlaWwiIDxzYWdlQG5ld2RyZWFtLm5ldD4gCj4+IMOA OiAiYWRlcnVtaWVyIiA8YWRlcnVtaWVyQG9kaXNvLmNvbT4gCj4+IENjOiAiY2VwaC11c2VycyIg PGNlcGgtdXNlcnNAbGlzdHMuY2VwaC5jb20+LCAiY2VwaC1kZXZlbCIgPGNlcGgtZGV2ZWxAdmdl ci5rZXJuZWwub3JnPiAKPj4gRW52b3nDqTogVmVuZHJlZGkgMjUgSmFudmllciAyMDE5IDEwOjQ5 OjAyIAo+PiBPYmpldDogUmU6IGNlcGggb3NkIGNvbW1pdCBsYXRlbmN5IGluY3JlYXNlIG92ZXIg dGltZSwgdW50aWwgcmVzdGFydCAKPj4gCj4+IENhbiB5b3UgY2FwdHVyZSBhIHBlcmYgdG9wIG9y IHBlcmYgcmVjb3JkIHRvIHNlZSB3aGVyZSB0ZWggQ1BVIHRpbWUgaXMgCj4+IGdvaW5nIG9uIG9u ZSBvZiB0aGUgT1NEcyB3dGggYSBoaWdoIGxhdGVuY3k/IAo+PiAKPj4gVGhhbmtzISAKPj4gc2Fn ZSAKPj4gCj4+IAo+PiBPbiBGcmksIDI1IEphbiAyMDE5LCBBbGV4YW5kcmUgREVSVU1JRVIgd3Jv dGU6IAo+PiAKPj4+IAo+Pj4gSGksIAo+Pj4gCj4+PiBJIGhhdmUgYSBzdHJhbmdlIGJlaGF2aW91 ciBvZiBteSBvc2QsIG9uIG11bHRpcGxlIGNsdXN0ZXJzLCAKPj4+IAo+Pj4gQWxsIGNsdXN0ZXIg YXJlIHJ1bm5pbmcgbWltaWMgMTMuMi4xLGJsdWVzdG9yZSwgd2l0aCBzc2Qgb3IgbnZtZSBkcml2 ZXJzLCAKPj4+IHdvcmtsb2FkIGlzIHJiZCBvbmx5LCB3aXRoIHFlbXUta3ZtIHZtcyBydW5uaW5n IHdpdGggbGlicmJkICsgc25hcHNob3QvcmJkIGV4cG9ydC1kaWZmL3NuYXBzaG90ZGVsZXRlIGVh Y2ggZGF5IGZvciBiYWNrdXAgCj4+PiAKPj4+IFdoZW4gdGhlIG9zZCBhcmUgcmVmcmVzaGx5IHN0 YXJ0ZWQsIHRoZSBjb21taXQgbGF0ZW5jeSBpcyBiZXR3ZWVuIDAsNS0xbXMuIAo+Pj4gCj4+PiBC dXQgb3ZlcnRpbWUsIHRoaXMgbGF0ZW5jeSBpbmNyZWFzZSBzbG93bHkgKG1heWJlIGFyb3VuZCAx bXMgYnkgZGF5KSwgdW50aWwgcmVhY2hpbmcgY3JhenkgCj4+PiB2YWx1ZXMgbGlrZSAyMC0yMDBt cy4gCj4+PiAKPj4+IFNvbWUgZXhhbXBsZSBncmFwaHM6IAo+Pj4gCj4+PiBodHRwOi8vb2Rpc293 ZWIxLm9kaXNvLm5ldC9vc2RsYXRlbmN5MS5wbmcgCj4+PiBodHRwOi8vb2Rpc293ZWIxLm9kaXNv Lm5ldC9vc2RsYXRlbmN5Mi5wbmcgCj4+PiAKPj4+IEFsbCBvc2RzIGhhdmUgdGhpcyBiZWhhdmlv dXIsIGluIGFsbCBjbHVzdGVycy4gCj4+PiAKPj4+IFRoZSBsYXRlbmN5IG9mIHBoeXNpY2FsIGRp c2tzIGlzIG9rLiAoQ2x1c3RlcnMgYXJlIGZhciB0byBiZSBmdWxsIGxvYWRlZCkgCj4+PiAKPj4+ IEFuZCBpZiBJIHJlc3RhcnQgdGhlIG9zZCwgdGhlIGxhdGVuY3kgY29tZSBiYWNrIHRvIDAsNS0x bXMuIAo+Pj4gCj4+PiBUaGF0J3MgcmVtZW1iZXIgbWUgb2xkIHRjbWFsbG9jIGJ1ZywgYnV0IG1h eWJlIGNvdWxkIGl0IGJlIGEgYmx1ZXN0b3JlIG1lbW9yeSBidWcgPyAKPj4+IAo+Pj4gQW55IEhp bnRzIGZvciBjb3VudGVycy9sb2dzIHRvIGNoZWNrID8gCj4+PiAKPj4+IAo+Pj4gUmVnYXJkcywg Cj4+PiAKPj4+IEFsZXhhbmRyZSAKPj4+IAo+Pj4gCj4+IAo+PiBfX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fXyAKPj4gY2VwaC11c2VycyBtYWlsaW5nIGxpc3Qg Cj4+IGNlcGgtdXNlcnNAbGlzdHMuY2VwaC5jb20gCj4+IGh0dHA6Ly9saXN0cy5jZXBoLmNvbS9s aXN0aW5mby5jZ2kvY2VwaC11c2Vycy1jZXBoLmNvbSAKPj4gCj4gCgpfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpjZXBoLXVzZXJzIG1haWxpbmcgbGlzdApj ZXBoLXVzZXJzQGxpc3RzLmNlcGguY29tCmh0dHA6Ly9saXN0cy5jZXBoLmNvbS9saXN0aW5mby5j Z2kvY2VwaC11c2Vycy1jZXBoLmNvbQo=