From mboxrd@z Thu Jan 1 00:00:00 1970 From: Potnuri Bharat Teja Subject: Re: SQ overflow seen running isert traffic with high block sizes Date: Thu, 18 Jan 2018 23:23:17 +0530 Message-ID: <20180118175316.GA11338@chelsio.com> References: <210f538f-8c98-f480-64fc-f2124ed6d01b@grimberg.me> <1499970579.7987.8.camel@haakon3.daterainc.com> <20171006224025.GA23364@ssaleem-MOBL4.amr.corp.intel.com> <1515992195.24576.156.camel@haakon3.daterainc.com> <20180115152236.GA15484@ssaleem-MOBL4.amr.corp.intel.com> <1516269522.24576.274.camel@haakon3.daterainc.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <1516269522.24576.274.camel@haakon3.daterainc.com> Sender: target-devel-owner@vger.kernel.org To: "Nicholas A. Bellinger" Cc: Shiraz Saleem , "Kalderon, Michal" , "Amrani, Ram" , Sagi Grimberg , "linux-rdma@vger.kernel.org" , "Elior, Ariel" , target-devel List-Id: linux-rdma@vger.kernel.org Hi Nicholas, thanks for the suggestions. Comments below. On Thursday, January 01/18/18, 2018 at 15:28:42 +0530, Nicholas A. Bellinger wrote: > Hi Shiraz, Michal & Co, > > Thanks for the feedback. Comments below. > > On Mon, 2018-01-15 at 09:22 -0600, Shiraz Saleem wrote: > > On Mon, Jan 15, 2018 at 03:12:36AM -0700, Kalderon, Michal wrote: > > > > From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma- > > > > owner@vger.kernel.org] On Behalf Of Nicholas A. Bellinger > > > > Sent: Monday, January 15, 2018 6:57 AM > > > > To: Shiraz Saleem > > > > Cc: Amrani, Ram ; Sagi Grimberg > > > > ; linux-rdma@vger.kernel.org; Elior, Ariel > > > > ; target-devel ; > > > > Potnuri Bharat Teja > > > > Subject: Re: SQ overflow seen running isert traffic with high block sizes > > > > > > > > Hi Shiraz, Ram, Ariel, & Potnuri, > > > > > > > > Following up on this old thread, as it relates to Potnuri's recent fix for a iser- > > > > target queue-full memory leak: > > > > > > > > https://www.spinics.net/lists/target-devel/msg16282.html > > > > > > > > Just curious how frequent this happens in practice with sustained large block > > > > workloads, as it appears to effect at least three different iwarp RNICS (i40iw, > > > > qedr and iw_cxgb4)..? > > > > > > > > Is there anything else from an iser-target consumer level that should be > > > > changed for iwarp to avoid repeated ib_post_send() failures..? > > > > > > > Would like to mention, that although we are an iWARP RNIC as well, we've hit this > > > Issue when running RoCE. It's not iWARP related. > > > This is easily reproduced within seconds with IO size of 5121K > > > Using 5 Targets with 2 Ram Disk each and 5 targets with FileIO Disks each. > > > > > > IO Command used: > > > maim -b512k -T32 -t2 -Q8 -M0 -o -u -n -m17 -ftargets.dat -d1 > > > > > > thanks, > > > Michal > > > > Its seen with block size >= 2M on a single target 1 RAM disk config. And similar to Michals report; > > rather quickly, in a matter of seconds. > > > > fio --rw=read --bs=2048k --numjobs=1 --iodepth=128 --runtime=30 --size=20g --loops=1 --ioengine=libaio > > --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --exitall --filename=/dev/sdb --name=sdb > > > > A couple of thoughts. > > First, would it be helpful to limit maximum payload size per I/O for > consumers based on number of iser-target sq hw sges..? yes, I think HW num sge needs to be propagated to iscsi target. > > That is, if rdma_rw_ctx_post() -> ib_post_send() failures are related to > maximum payload size per I/O being too large there is an existing Yes they are IO size specific, I observed SQ overflow with fio for IO sizes above 256k and for READ tests only with chelsio(iw_cxgb4) adapters. > target_core_fabric_ops mechanism for limiting using SCSI residuals, > originally utilized by qla2xxx here: > > target/qla2xxx: Honor max_data_sg_nents I/O transfer limit > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8f9b565482c537821588444e09ff732c7d65ed6e > > Note this patch also will return a smaller Block Limits VPD (0x86) > MAXIMUM TRANSFER LENGTH based on max_data_sg_nents * PAGE_SIZE, which > means for modern SCSI initiators honoring MAXIMUM TRANSFER LENGTH will > automatically limit maximum outgoing payload transfer length, and avoid > SCSI residual logic. > > As-is, iser-target doesn't a propagate max_data_sg_ents limit into > iscsi-target, but you can try testing with a smaller value to see if > it's useful. Eg: > > diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configf > index 0ebc481..d8a4cc5 100644 > --- a/drivers/target/iscsi/iscsi_target_configfs.c > +++ b/drivers/target/iscsi/iscsi_target_configfs.c > @@ -1553,6 +1553,7 @@ static void lio_release_cmd(struct se_cmd *se_cmd) > .module = THIS_MODULE, > .name = "iscsi", > .node_acl_size = sizeof(struct iscsi_node_acl), > + .max_data_sg_nents = 32, /* 32 * PAGE_SIZE = MAXIMUM TRANSFER LENGTH */ > .get_fabric_name = iscsi_get_fabric_name, > .tpg_get_wwn = lio_tpg_get_endpoint_wwn, > .tpg_get_tag = lio_tpg_get_tag, > With above change, SQ overflow isn't observed. I started of with max_data_sg_nents = 16. > Second, if the failures are not SCSI transfer length specific, another > option would be to limit the total command sequence number depth (CmdSN) > per session. > > This is controlled at runtime by default_cmdsn_depth TPG attribute: > > /sys/kernel/config/target/iscsi/$TARGET_IQN/$TPG/attrib/default_cmdsn_depth > > and on per initiator context with cmdsn_depth NodeACL attribute: > > /sys/kernel/config/target/iscsi/$TARGET_IQN/$TPG/acls/$ACL_IQN/cmdsn_depth > > Note these default to 64, and can be changed at build time via > include/target/iscsi/iscsi_target_core.h:TA_DEFAULT_CMDSN_DEPTH. > > That said, Sagi, any further comments as what else iser-target should be > doing to avoid repeated queue-fulls with limited hw sges..? > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Potnuri Bharat Teja Date: Thu, 18 Jan 2018 17:53:47 +0000 Subject: Re: SQ overflow seen running isert traffic with high block sizes Message-Id: <20180118175316.GA11338@chelsio.com> List-Id: References: <210f538f-8c98-f480-64fc-f2124ed6d01b@grimberg.me> <1499970579.7987.8.camel@haakon3.daterainc.com> <20171006224025.GA23364@ssaleem-MOBL4.amr.corp.intel.com> <1515992195.24576.156.camel@haakon3.daterainc.com> <20180115152236.GA15484@ssaleem-MOBL4.amr.corp.intel.com> <1516269522.24576.274.camel@haakon3.daterainc.com> In-Reply-To: <1516269522.24576.274.camel@haakon3.daterainc.com> MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1257" Content-Transfer-Encoding: base64 To: "Nicholas A. Bellinger" Cc: Shiraz Saleem , "Kalderon, Michal" , "Amrani, Ram" , Sagi Grimberg , "linux-rdma@vger.kernel.org" , "Elior, Ariel" , target-devel SGkgTmljaG9sYXMsIAp0aGFua3MgZm9yIHRoZSBzdWdnZXN0aW9ucy4gQ29tbWVudHMgYmVsb3cu CgpPbiBUaHVyc2RheSwgSmFudWFyeSAwMS8xOC8xOCwgMjAxOCBhdCAxNToyODo0MiArMDUzMCwg TmljaG9sYXMgQS4gQmVsbGluZ2VyIHdyb3RlOgo+IEhpIFNoaXJheiwgTWljaGFsICYgQ28sCj4g Cj4gVGhhbmtzIGZvciB0aGUgZmVlZGJhY2suICBDb21tZW50cyBiZWxvdy4KPiAKPiBPbiBNb24s IDIwMTgtMDEtMTUgYXQgMDk6MjIgLTA2MDAsIFNoaXJheiBTYWxlZW0gd3JvdGU6Cj4gPiBPbiBN b24sIEphbiAxNSwgMjAxOCBhdCAwMzoxMjozNkFNIC0wNzAwLCBLYWxkZXJvbiwgTWljaGFsIHdy b3RlOgo+ID4gPiA+IEZyb206IGxpbnV4LXJkbWEtb3duZXJAdmdlci5rZXJuZWwub3JnIFttYWls dG86bGludXgtcmRtYS0KPiA+ID4gPiBvd25lckB2Z2VyLmtlcm5lbC5vcmddIE9uIEJlaGFsZiBP ZiBOaWNob2xhcyBBLiBCZWxsaW5nZXIKPiA+ID4gPiBTZW50OiBNb25kYXksIEphbnVhcnkgMTUs IDIwMTggNjo1NyBBTQo+ID4gPiA+IFRvOiBTaGlyYXogU2FsZWVtIDxzaGlyYXouc2FsZWVtQGlu dGVsLmNvbT4KPiA+ID4gPiBDYzogQW1yYW5pLCBSYW0gPFJhbS5BbXJhbmlAY2F2aXVtLmNvbT47 IFNhZ2kgR3JpbWJlcmcKPiA+ID4gPiA8c2FnaUBncmltYmVyZy5tZT47IGxpbnV4LXJkbWFAdmdl ci5rZXJuZWwub3JnOyBFbGlvciwgQXJpZWwKPiA+ID4gPiA8QXJpZWwuRWxpb3JAY2F2aXVtLmNv bT47IHRhcmdldC1kZXZlbCA8dGFyZ2V0LWRldmVsQHZnZXIua2VybmVsLm9yZz47Cj4gPiA+ID4g UG90bnVyaSBCaGFyYXQgVGVqYSA8YmhhcmF0QGNoZWxzaW8uY29tPgo+ID4gPiA+IFN1YmplY3Q6 IFJlOiBTUSBvdmVyZmxvdyBzZWVuIHJ1bm5pbmcgaXNlcnQgdHJhZmZpYyB3aXRoIGhpZ2ggYmxv Y2sgc2l6ZXMKPiA+ID4gPiAKPiA+ID4gPiBIaSBTaGlyYXosIFJhbSwgQXJpZWwsICYgUG90bnVy aSwKPiA+ID4gPiAKPiA+ID4gPiBGb2xsb3dpbmcgdXAgb24gdGhpcyBvbGQgdGhyZWFkLCBhcyBp dCByZWxhdGVzIHRvIFBvdG51cmkncyByZWNlbnQgZml4IGZvciBhIGlzZXItCj4gPiA+ID4gdGFy Z2V0IHF1ZXVlLWZ1bGwgbWVtb3J5IGxlYWs6Cj4gPiA+ID4gCj4gPiA+ID4gaHR0cHM6Ly93d3cu c3Bpbmljcy5uZXQvbGlzdHMvdGFyZ2V0LWRldmVsL21zZzE2MjgyLmh0bWwKPiA+ID4gPiAKPiA+ ID4gPiBKdXN0IGN1cmlvdXMgaG93IGZyZXF1ZW50IHRoaXMgaGFwcGVucyBpbiBwcmFjdGljZSB3 aXRoIHN1c3RhaW5lZCBsYXJnZSBibG9jawo+ID4gPiA+IHdvcmtsb2FkcywgYXMgaXQgYXBwZWFy cyB0byBlZmZlY3QgYXQgbGVhc3QgdGhyZWUgZGlmZmVyZW50IGl3YXJwIFJOSUNTIChpNDBpdywK PiA+ID4gPiBxZWRyIGFuZCBpd19jeGdiNCkuLj8KPiA+ID4gPiAKPiA+ID4gPiBJcyB0aGVyZSBh bnl0aGluZyBlbHNlIGZyb20gYW4gaXNlci10YXJnZXQgY29uc3VtZXIgbGV2ZWwgdGhhdCBzaG91 bGQgYmUKPiA+ID4gPiBjaGFuZ2VkIGZvciBpd2FycCB0byBhdm9pZCByZXBlYXRlZCBpYl9wb3N0 X3NlbmQoKSBmYWlsdXJlcy4uPwo+ID4gPiA+IAo+ID4gPiBXb3VsZCBsaWtlIHRvIG1lbnRpb24s IHRoYXQgYWx0aG91Z2ggd2UgYXJlIGFuIGlXQVJQIFJOSUMgYXMgd2VsbCwgd2UndmUgaGl0IHRo aXMKPiA+ID4gSXNzdWUgd2hlbiBydW5uaW5nIFJvQ0UuIEl0J3Mgbm90IGlXQVJQIHJlbGF0ZWQu IAo+ID4gPiBUaGlzIGlzIGVhc2lseSByZXByb2R1Y2VkIHdpdGhpbiBzZWNvbmRzIHdpdGggSU8g c2l6ZSBvZiA1MTIxSwo+ID4gPiBVc2luZyA1IFRhcmdldHMgd2l0aCAyIFJhbSBEaXNrIGVhY2gg YW5kIDUgdGFyZ2V0cyB3aXRoIEZpbGVJTyBEaXNrcyBlYWNoLgo+ID4gPiAKPiA+ID4gSU8gQ29t bWFuZCB1c2VkOgo+ID4gPiBtYWltIC1iNTEyayAtVDMyIC10MiAtUTggLU0wIC1vIC11IC1uIC1t MTcgLWZ0YXJnZXRzLmRhdCAtZDEKPiA+ID4gCj4gPiA+IHRoYW5rcywKPiA+ID4gTWljaGFsCj4g PiAKPiA+IEl0cyBzZWVuIHdpdGggYmxvY2sgc2l6ZSA+PSAyTSBvbiBhIHNpbmdsZSB0YXJnZXQg MSBSQU0gZGlzayBjb25maWcuIEFuZCBzaW1pbGFyIHRvIE1pY2hhbHMgcmVwb3J0Owo+ID4gcmF0 aGVyIHF1aWNrbHksIGluIGEgbWF0dGVyIG9mIHNlY29uZHMuCj4gPiAKPiA+IGZpbyAtLXJ3PXJl YWQgLS1icyA0OGsgLS1udW1qb2JzPTEgLS1pb2RlcHRoEjggLS1ydW50aW1lMCAtLXNpemUgZyAt LWxvb3BzPTEgLS1pb2VuZ2luZT1saWJhaW8gCj4gPiAtLWRpcmVjdD0xIC0taW52YWxpZGF0ZT0x IC0tZnN5bmNfb25fY2xvc2U9MSAtLW5vcmFuZG9tbWFwIC0tZXhpdGFsbCAtLWZpbGVuYW1lPS9k ZXYvc2RiIC0tbmFtZT1zZGIgCj4gPiAKPiAKPiBBIGNvdXBsZSBvZiB0aG91Z2h0cy4KPiAKPiBG aXJzdCwgd291bGQgaXQgYmUgaGVscGZ1bCB0byBsaW1pdCBtYXhpbXVtIHBheWxvYWQgc2l6ZSBw ZXIgSS9PIGZvcgo+IGNvbnN1bWVycyBiYXNlZCBvbiBudW1iZXIgb2YgaXNlci10YXJnZXQgc3Eg aHcgc2dlcy4uPwp5ZXMsIEkgdGhpbmsgSFcgbnVtIHNnZSBuZWVkcyB0byBiZSBwcm9wYWdhdGVk IHRvIGlzY3NpIHRhcmdldC4KPiAKPiBUaGF0IGlzLCBpZiByZG1hX3J3X2N0eF9wb3N0KCkgLT4g aWJfcG9zdF9zZW5kKCkgZmFpbHVyZXMgYXJlIHJlbGF0ZWQgdG8KPiBtYXhpbXVtIHBheWxvYWQg c2l6ZSBwZXIgSS9PIGJlaW5nIHRvbyBsYXJnZSB0aGVyZSBpcyBhbiBleGlzdGluZwoKWWVzIHRo ZXkgYXJlIElPIHNpemUgc3BlY2lmaWMsIEkgb2JzZXJ2ZWQgU1Egb3ZlcmZsb3cgd2l0aCBmaW8g Zm9yIElPIHNpemVzIGFib3ZlCjI1NmsgYW5kIGZvciBSRUFEIHRlc3RzIG9ubHkgd2l0aCBjaGVs c2lvKGl3X2N4Z2I0KSBhZGFwdGVycy4KPiB0YXJnZXRfY29yZV9mYWJyaWNfb3BzIG1lY2hhbmlz bSBmb3IgbGltaXRpbmcgdXNpbmcgU0NTSSByZXNpZHVhbHMsCj4gb3JpZ2luYWxseSB1dGlsaXpl ZCBieSBxbGEyeHh4IGhlcmU6Cj4gCj4gdGFyZ2V0L3FsYTJ4eHg6IEhvbm9yIG1heF9kYXRhX3Nn X25lbnRzIEkvTyB0cmFuc2ZlciBsaW1pdAo+IGh0dHBzOi8vZ2l0Lmtlcm5lbC5vcmcvcHViL3Nj bS9saW51eC9rZXJuZWwvZ2l0L3RvcnZhbGRzL2xpbnV4LmdpdC9jb21taXQvP2lkjzliNTY1NDgy YzUzNzgyMTU4ODQ0NGUwOWZmNzMyYzdkNjVlZDZlCj4gCj4gTm90ZSB0aGlzIHBhdGNoIGFsc28g d2lsbCByZXR1cm4gYSBzbWFsbGVyIEJsb2NrIExpbWl0cyBWUEQgKDB4ODYpCj4gTUFYSU1VTSBU UkFOU0ZFUiBMRU5HVEggYmFzZWQgb24gbWF4X2RhdGFfc2dfbmVudHMgKiBQQUdFX1NJWkUsIHdo aWNoCj4gbWVhbnMgZm9yIG1vZGVybiBTQ1NJIGluaXRpYXRvcnMgaG9ub3JpbmcgTUFYSU1VTSBU UkFOU0ZFUiBMRU5HVEggd2lsbAo+IGF1dG9tYXRpY2FsbHkgbGltaXQgbWF4aW11bSBvdXRnb2lu ZyBwYXlsb2FkIHRyYW5zZmVyIGxlbmd0aCwgYW5kIGF2b2lkCj4gU0NTSSByZXNpZHVhbCBsb2dp Yy4KPiAKPiBBcy1pcywgaXNlci10YXJnZXQgZG9lc24ndCBhIHByb3BhZ2F0ZSBtYXhfZGF0YV9z Z19lbnRzIGxpbWl0IGludG8KPiBpc2NzaS10YXJnZXQsIGJ1dCB5b3UgY2FuIHRyeSB0ZXN0aW5n IHdpdGggYSBzbWFsbGVyIHZhbHVlIHRvIHNlZSBpZgo+IGl0J3MgdXNlZnVsLiAgRWc6Cj4gCj4g ZGlmZiAtLWdpdCBhL2RyaXZlcnMvdGFyZ2V0L2lzY3NpL2lzY3NpX3RhcmdldF9jb25maWdmcy5j IGIvZHJpdmVycy90YXJnZXQvaXNjc2kvaXNjc2lfdGFyZ2V0X2NvbmZpZ2YKPiBpbmRleCAwZWJj NDgxLi5kOGE0Y2M1IDEwMDY0NAo+IC0tLSBhL2RyaXZlcnMvdGFyZ2V0L2lzY3NpL2lzY3NpX3Rh cmdldF9jb25maWdmcy5jCj4gKysrIGIvZHJpdmVycy90YXJnZXQvaXNjc2kvaXNjc2lfdGFyZ2V0 X2NvbmZpZ2ZzLmMKPiBAQCAtMTU1Myw2ICsxNTUzLDcgQEAgc3RhdGljIHZvaWQgbGlvX3JlbGVh c2VfY21kKHN0cnVjdCBzZV9jbWQgKnNlX2NtZCkKPiAgICAgICAgIC5tb2R1bGUgICAgICAgICAg ICAgICAgICAgICAgICAgPSBUSElTX01PRFVMRSwKPiAgICAgICAgIC5uYW1lICAgICAgICAgICAg ICAgICAgICAgICAgICAgPSAiaXNjc2kiLAo+ICAgICAgICAgLm5vZGVfYWNsX3NpemUgICAgICAg ICAgICAgICAgICA9IHNpemVvZihzdHJ1Y3QgaXNjc2lfbm9kZV9hY2wpLAo+ICsgICAgICAgLm1h eF9kYXRhX3NnX25lbnRzICAgICAgICAgICAgICA9IDMyLCAvKiAzMiAqIFBBR0VfU0laRSA9IE1B WElNVU0gVFJBTlNGRVIgTEVOR1RIICovCj4gICAgICAgICAuZ2V0X2ZhYnJpY19uYW1lICAgICAg ICAgICAgICAgID0gaXNjc2lfZ2V0X2ZhYnJpY19uYW1lLAo+ICAgICAgICAgLnRwZ19nZXRfd3du ICAgICAgICAgICAgICAgICAgICA9IGxpb190cGdfZ2V0X2VuZHBvaW50X3d3biwKPiAgICAgICAg IC50cGdfZ2V0X3RhZyAgICAgICAgICAgICAgICAgICAgPSBsaW9fdHBnX2dldF90YWcsCj4gCldp dGggYWJvdmUgY2hhbmdlLCBTUSBvdmVyZmxvdyBpc24ndCBvYnNlcnZlZC4gSSBzdGFydGVkIG9m IHdpdGggbWF4X2RhdGFfc2dfbmVudHMgPSAxNi4KPiBTZWNvbmQsIGlmIHRoZSBmYWlsdXJlcyBh cmUgbm90IFNDU0kgdHJhbnNmZXIgbGVuZ3RoIHNwZWNpZmljLCBhbm90aGVyCj4gb3B0aW9uIHdv dWxkIGJlIHRvIGxpbWl0IHRoZSB0b3RhbCBjb21tYW5kIHNlcXVlbmNlIG51bWJlciBkZXB0aCAo Q21kU04pCj4gcGVyIHNlc3Npb24uCj4gCj4gVGhpcyBpcyBjb250cm9sbGVkIGF0IHJ1bnRpbWUg YnkgZGVmYXVsdF9jbWRzbl9kZXB0aCBUUEcgYXR0cmlidXRlOgo+IAo+IC9zeXMva2VybmVsL2Nv bmZpZy90YXJnZXQvaXNjc2kvJFRBUkdFVF9JUU4vJFRQRy9hdHRyaWIvZGVmYXVsdF9jbWRzbl9k ZXB0aAo+IAo+IGFuZCBvbiBwZXIgaW5pdGlhdG9yIGNvbnRleHQgd2l0aCBjbWRzbl9kZXB0aCBO b2RlQUNMIGF0dHJpYnV0ZToKPiAKPiAvc3lzL2tlcm5lbC9jb25maWcvdGFyZ2V0L2lzY3NpLyRU QVJHRVRfSVFOLyRUUEcvYWNscy8kQUNMX0lRTi9jbWRzbl9kZXB0aAo+IAo+IE5vdGUgdGhlc2Ug ZGVmYXVsdCB0byA2NCwgYW5kIGNhbiBiZSBjaGFuZ2VkIGF0IGJ1aWxkIHRpbWUgdmlhCj4gaW5j bHVkZS90YXJnZXQvaXNjc2kvaXNjc2lfdGFyZ2V0X2NvcmUuaDpUQV9ERUZBVUxUX0NNRFNOX0RF UFRILgo+IAo+IFRoYXQgc2FpZCwgU2FnaSwgYW55IGZ1cnRoZXIgY29tbWVudHMgYXMgd2hhdCBl bHNlIGlzZXItdGFyZ2V0IHNob3VsZCBiZQo+IGRvaW5nIHRvIGF2b2lkIHJlcGVhdGVkIHF1ZXVl LWZ1bGxzIHdpdGggbGltaXRlZCBodyBzZ2VzLi4/Cj4gCi0tClRvIHVuc3Vic2NyaWJlIGZyb20g dGhpcyBsaXN0OiBzZW5kIHRoZSBsaW5lICJ1bnN1YnNjcmliZSB0YXJnZXQtZGV2ZWwiIGluCnRo ZSBib2R5IG9mIGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJuZWwub3JnCk1vcmUgbWFq b3Jkb21vIGluZm8gYXQgIGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRt bA==