From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8432582498476632980==" MIME-Version: 1.0 From: Jonathan Richardson Subject: [SPDK] spdk/vpp performance Date: Tue, 10 Sep 2019 12:42:13 -0700 Message-ID: List-ID: To: spdk@lists.01.org --===============8432582498476632980== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi, I'm using vpp 19.04.2 and spdk 19.07. I have a couple questions about the architecture of the session API with nvmeotcp. I attached my perf results for a 4k random read. I have 3 cores for vpp workers with 3 RSS queues and the other 5 cores are for spdk. The memcpy is the bottleneck taking 31% cpu usage on the spdk threads when it writes the completion into shared memory. The vpp thread memcpy is 54% when it pulls the data out for tx. You can flip those results around for a random write. L3 cache miss is bad at 60+%. With RDMA it's ~10%. This gives rather poor performance. Memory is more of a bottleneck than cpu on my ARM system, so the spdk design worked out well being zero copy. It would be better if the copying in and out of the shmem were avoided though I'm not sure it's feasible and need to study the code further. Are there any plans to change the session API or the way it uses memory? Spdk and vpp both use dpdk independently with different memory regions (--file-prefix). But the shared memory in the session API uses a libc malloc'd buffer. Would using a dpdk mempool be better for cache efficiency and synchronization with other dpdk memory usage? Any thoughts on shared dpdk mempools between vpp and spdk instead of copying into the session API's shared memory? It looks like there are 2 copies for 1 read, the nvme completion into shmem, and the vpp copy out (haven't looked where session_tx_fifo_peek_and_snd is copying to). Even getting rid of one of the memcpy's could help. We are looking into possible optimizations so any guidance is appreciated since we want any changes merged into the mainline. Thanks, Jon --===============8432582498476632980== Content-Type: text/plain MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="vpp-performance.txt" LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQpTUERLIENP UkUNCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KDQor ICAgOTguMjMlICAgICAwLjAwJSAgcmVhY3Rvcl82ICAgICAgICBsaWJjLTIuMjcuc28gICAgICAg IFsuXSB0aHJlYWRfc3RhcnQNCisgICA5OC4yMyUgICAgIDAuMDAlICByZWFjdG9yXzYgICAgICAg IGxpYnB0aHJlYWQtMi4yNy5zbyAgWy5dIHN0YXJ0X3RocmVhZA0KKyAgIDk4LjIzJSAgICAgMC4w MCUgIHJlYWN0b3JfNiAgICAgICAgbnZtZl90Z3QgICAgICAgICAgICBbLl0gZWFsX3RocmVhZF9s b29wDQorICAgOTcuNTIlICAgICAwLjM4JSAgcmVhY3Rvcl82ICAgICAgICBudm1mX3RndCAgICAg ICAgICAgIFsuXSBfc3Bka19yZWFjdG9yX3J1bg0KKyAgIDk2LjQ5JSAgICAgMS4zNiUgIHJlYWN0 b3JfNiAgICAgICAgbnZtZl90Z3QgICAgICAgICAgICBbLl0gc3Bka190aHJlYWRfcG9sbA0KKyAg IDY5LjU2JSAgICAgMC42MSUgIHJlYWN0b3JfNiAgICAgICAgbnZtZl90Z3QgICAgICAgICAgICBb Ll0gYmRldl9udm1lX3BvbGwNCisgICA2OC44NSUgICAgIDMuNDklICByZWFjdG9yXzYgICAgICAg IG52bWZfdGd0ICAgICAgICAgICAgWy5dIHNwZGtfbnZtZV9xcGFpcl9wcm9jZXNzX2NvbXBsZXRp b25zDQorICAgNjUuNzklICAgICA0LjMyJSAgcmVhY3Rvcl82ICAgICAgICBudm1mX3RndCAgICAg ICAgICAgIFsuXSBzcGRrX252bWZfdGNwX3JlcV9wcm9jZXNzDQorICAgNjUuNDMlICAgICA0LjIx JSAgcmVhY3Rvcl82ICAgICAgICBudm1mX3RndCAgICAgICAgICAgIFsuXSBudm1lX3BjaWVfcXBh aXJfcHJvY2Vzc19jb21wbGV0aW9ucw0KKyAgIDYxLjE4JSAgICAgMC4zNCUgIHJlYWN0b3JfNiAg ICAgICAgbnZtZl90Z3QgICAgICAgICAgICBbLl0gbnZtZV9wY2llX3FwYWlyX2NvbXBsZXRlX3Ry YWNrZXINCisgICA1OC4wNyUgICAgIDEuMjclICByZWFjdG9yXzYgICAgICAgIG52bWZfdGd0ICAg ICAgICAgICAgWy5dIG52bWZfYmRldl9jdHJscl9jb21wbGV0ZV9jbWQNCisgICA1Ni41MSUgICAg IDAuODMlICByZWFjdG9yXzYgICAgICAgIG52bWZfdGd0ICAgICAgICAgICAgWy5dIHNwZGtfbnZt Zl9yZXF1ZXN0X2NvbXBsZXRlDQorICAgNTQuNjYlICAgICAwLjAzJSAgcmVhY3Rvcl82ICAgICAg ICBudm1mX3RndCAgICAgICAgICAgIFsuXSBzcGRrX252bWZfdGNwX3JlcV9jb21wbGV0ZQ0KKyAg IDUzLjY2JSAgICAgMC44NCUgIHJlYWN0b3JfNiAgICAgICAgbnZtZl90Z3QgICAgICAgICAgICBb Ll0gc3Bka19udm1mX3RjcF9zZW5kX2MyaF9kYXRhDQorICAgNTEuOTIlICAgICAwLjA2JSAgcmVh Y3Rvcl82ICAgICAgICBudm1mX3RndCAgICAgICAgICAgIFsuXSBzcGRrX252bWZfdGNwX3FwYWly X2ZsdXNoX3BkdXMNCisgICA1MS42OSUgICAgIDAuNzElICByZWFjdG9yXzYgICAgICAgIG52bWZf dGd0ICAgICAgICAgICAgWy5dIHNwZGtfbnZtZl90Y3BfcXBhaXJfZmx1c2hfcGR1c19pbnRlcm5h bA0KKyAgIDQ5LjI4JSAgICAgMy4wOCUgIHJlYWN0b3JfNiAgICAgICAgbnZtZl90Z3QgICAgICAg ICAgICBbLl0gc3Bka192cHBfc29ja193cml0ZXYNCi0gICAzMC45NCUgICAgMzAuOTAlICByZWFj dG9yXzYgICAgICAgIGxpYmMtMi4yNy5zbyAgICAgICAgWy5dIF9fbWVtY3B5X2dlbmVyaWMNCiAg ICAgMzAuOTAlIHRocmVhZF9zdGFydA0KICAgICAgICBzdGFydF90aHJlYWQNCiAgICAgICAgZWFs X3RocmVhZF9sb29wDQogICAgICAgIF9zcGRrX3JlYWN0b3JfcnVuDQogICAgICAtIHNwZGtfdGhy ZWFkX3BvbGwNCiAgICAgICAgIC0gMjkuNjIlIGJkZXZfbnZtZV9wb2xsDQogICAgICAgICAgICAg IHNwZGtfbnZtZV9xcGFpcl9wcm9jZXNzX2NvbXBsZXRpb25zDQogICAgICAgICAgICAgIG52bWVf cGNpZV9xcGFpcl9wcm9jZXNzX2NvbXBsZXRpb25zDQogICAgICAgICAgICAgIG52bWVfcGNpZV9x cGFpcl9jb21wbGV0ZV90cmFja2VyDQogICAgICAgICAgICAgIG52bWZfYmRldl9jdHJscl9jb21w bGV0ZV9jbWQNCiAgICAgICAgICAgICAgc3Bka19udm1mX3JlcXVlc3RfY29tcGxldGUNCiAgICAg ICAgICAgICAgc3Bka19udm1mX3RjcF9yZXFfY29tcGxldGUNCiAgICAgICAgICAgICAgc3Bka19u dm1mX3RjcF9yZXFfcHJvY2Vzcw0KICAgICAgICAgICAgICBzcGRrX252bWZfdGNwX3NlbmRfYzJo X2RhdGENCiAgICAgICAgICAgICAgc3Bka19udm1mX3RjcF9xcGFpcl9mbHVzaF9wZHVzDQogICAg ICAgICAgICAgIHNwZGtfbnZtZl90Y3BfcXBhaXJfZmx1c2hfcGR1c19pbnRlcm5hbA0KICAgICAg ICAgICAgLSBzcGRrX3ZwcF9zb2NrX3dyaXRldg0KICAgICAgICAgICAgICAgICAyOS42MSUgX19t ZW1jcHlfZ2VuZXJpYw0KICAgICAgICAgKyAxLjI4JSBzcGRrX252bWZfcG9sbF9ncm91cF9wb2xs DQorICAgMjUuMzElICAgICAwLjE0JSAgcmVhY3Rvcl82ICAgICAgICBudm1mX3RndCAgICAgICAg ICAgIFsuXSBzcGRrX252bWZfcG9sbF9ncm91cF9wb2xsDQorICAgMjUuMTglICAgICAwLjI3JSAg cmVhY3Rvcl82ICAgICAgICBudm1mX3RndCAgICAgICAgICAgIFsuXSBzcGRrX252bWZfdGNwX3Bv bGxfZ3JvdXBfcG9sbA0KKyAgIDE3LjYyJSAgICAgMC43OSUgIHJlYWN0b3JfNiAgICAgICAgbnZt Zl90Z3QgICAgICAgICAgICBbLl0gc3Bka19zb2NrX2dyb3VwX3BvbGxfY291bnQNCi0gICAxNS4x MyUgICAgMTUuMTElICByZWFjdG9yXzYgICAgICAgIGxpYnN2bS5zby4xOS4wNC4yICAgWy5dIHN2 bV9maWZvX2VucXVldWVfbm93YWl0X21hDQogICAgIDE1LjExJSB0aHJlYWRfc3RhcnQNCiAgICAg ICAgc3RhcnRfdGhyZWFkDQogICAgICAgIGVhbF90aHJlYWRfbG9vcA0KICAgICAgICBfc3Bka19y ZWFjdG9yX3J1bg0KICAgICAgICBzcGRrX3RocmVhZF9wb2xsDQogICAgICAgIGJkZXZfbnZtZV9w b2xsDQogICAgICAgIHNwZGtfbnZtZV9xcGFpcl9wcm9jZXNzX2NvbXBsZXRpb25zDQogICAgICAg IG52bWVfcGNpZV9xcGFpcl9wcm9jZXNzX2NvbXBsZXRpb25zDQogICAgICAgIG52bWVfcGNpZV9x cGFpcl9jb21wbGV0ZV90cmFja2VyDQogICAgICAgIG52bWZfYmRldl9jdHJscl9jb21wbGV0ZV9j bWQNCiAgICAgICAgc3Bka19udm1mX3JlcXVlc3RfY29tcGxldGUNCiAgICAgICAgc3Bka19udm1m X3RjcF9yZXFfY29tcGxldGUNCiAgICAgICAgc3Bka19udm1mX3RjcF9yZXFfcHJvY2Vzcw0KICAg ICAgICBzcGRrX252bWZfdGNwX3NlbmRfYzJoX2RhdGENCiAgICAgICAgc3Bka19udm1mX3RjcF9x cGFpcl9mbHVzaF9wZHVzDQogICAgICAtIHNwZGtfbnZtZl90Y3BfcXBhaXJfZmx1c2hfcGR1c19p bnRlcm5hbA0KICAgICAgICAgLSAxNS4xMSUgc3Bka192cHBfc29ja193cml0ZXYNCiAgICAgICAg ICAgICAgc3ZtX2ZpZm9fZW5xdWV1ZV9ub3dhaXRfbWENCisgICAxMi4wMiUgICAgIDAuNDMlICBy ZWFjdG9yXzYgICAgICAgIG52bWZfdGd0ICAgICAgICAgICAgWy5dIHNwZGtfbnZtZl90Y3Bfc29j a19jYg0KKyAgICA3LjE1JSAgICAgMC40OCUgIHJlYWN0b3JfNiAgICAgICAgbnZtZl90Z3QgICAg ICAgICAgICBbLl0gc3Bka19udm1mX3JlcXVlc3RfZXhlYw0KLSAgICA3LjAyJSAgICAgNy4wMSUg IHJlYWN0b3JfNiAgICAgICAgbnZtZl90Z3QgICAgICAgICAgICBbLl0gc3Bka19udm1mX3RjcF9y ZXFfc2V0X3N0YXRlDQogICAgIDcuMDElIHRocmVhZF9zdGFydA0KICAgICAgICBzdGFydF90aHJl YWQNCiAgICAgICAgZWFsX3RocmVhZF9sb29wDQogICAgICAgIF9zcGRrX3JlYWN0b3JfcnVuDQog ICAgICAtIHNwZGtfdGhyZWFkX3BvbGwNCiAgICAgICAgIC0gNS4zMSUgc3Bka19udm1mX3BvbGxf Z3JvdXBfcG9sbA0KICAgICAgICAgICAgLSBzcGRrX252bWZfdGNwX3BvbGxfZ3JvdXBfcG9sbA0K ICAgICAgICAgICAgICAgLSA1LjEwJSBzcGRrX3NvY2tfZ3JvdXBfcG9sbF9jb3VudA0KICAgICAg ICAgICAgICAgICAgICA0Ljc1JSBzcGRrX252bWZfdGNwX3JlcV9zZXRfc3RhdGUNCiAgICAgICAg ICsgMS43MCUgYmRldl9udm1lX3BvbGwNCisgICAgNi4yOCUgICAgIDAuMTIlICByZWFjdG9yXzYg ICAgICAgIG52bWZfdGd0ICAgICAgICAgICAgWy5dIHNwZGtfbnZtZl9iZGV2X2N0cmxyX3JlYWRf Y21kDQorICAgIDYuMTclICAgICAwLjAyJSAgcmVhY3Rvcl82ICAgICAgICBudm1mX3RndCAgICAg ICAgICAgIFsuXSBzcGRrX2JkZXZfcmVhZHZfYmxvY2tzDQorICAgIDUuNDklICAgICAwLjI2JSAg cmVhY3Rvcl82ICAgICAgICBudm1mX3RndCAgICAgICAgICAgIFsuXSBfc3Bka19iZGV2X3JlYWR2 X2Jsb2Nrc193aXRoX21kDQorICAgIDQuOTglICAgICAwLjAzJSAgcmVhY3Rvcl82ICAgICAgICBu dm1mX3RndCAgICAgICAgICAgIFsuXSBudm1lX3RjcF9yZWFkX2RhdGENCisgICAgNC44NyUgICAg IDAuMjMlICByZWFjdG9yXzYgICAgICAgIG52bWZfdGd0ICAgICAgICAgICAgWy5dIHNwZGtfdnBw X3NvY2tfcmVjdg0KKyAgICA0LjYxJSAgICAgMC4zMyUgIHJlYWN0b3JfNiAgICAgICAgbnZtZl90 Z3QgICAgICAgICAgICBbLl0gc3Bka19iZGV2X2lvX3N1Ym1pdA0KKyAgICA0LjA1JSAgICAgMC4x NSUgIHJlYWN0b3JfNiAgICAgICAgbnZtZl90Z3QgICAgICAgICAgICBbLl0gYmRldl9udm1lX2dl dF9idWZfY2INCisgICAgMy44OSUgICAgIDEuNzYlICByZWFjdG9yXzYgICAgICAgIG52bWZfdGd0 ICAgICAgICAgICAgWy5dIHNwZGtfbnZtZV9uc19jbWRfcmVhZHZfd2l0aF9tZA0KKyAgICAzLjIy JSAgICAgMy4yMiUgIHJlYWN0b3JfNiAgICAgICAgbGlic3ZtLnNvLjE5LjA0LjIgICBbLl0gc3Zt X2ZpZm9fZGVxdWV1ZV9ub3dhaXRfbWENCisgICAgMi43NCUgICAgIDIuNzQlICByZWFjdG9yXzYg ICAgICAgIG52bWZfdGd0ICAgICAgICAgICAgWy5dIHNwZGtfYmRldl9pb19jb21wbGV0ZQ0KDQot LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NClZQUCBDT1JF DQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCg0KKyAg MTAwLjAwJSAgICAgMC4wMCUgIHZwcF93a18xICBsaWJ2cHBpbmZyYS5zby4xOS4wNC4yICBbLl0g Y2xpYl9jYWxsam1wDQorICAgOTkuOTglICAgICAwLjM5JSAgdnBwX3drXzEgIGxpYnZsaWIuc28u MTkuMDQuMiAgICAgIFsuXSB2bGliX3dvcmtlcl9sb29wDQorICAgNjIuNzYlICAgICAwLjcxJSAg dnBwX3drXzEgIGxpYnZuZXQuc28uMTkuMDQuMiAgICAgIFsuXSBzZXNzaW9uX3F1ZXVlX25vZGVf Zm4NCisgICA1OS44MCUgICAgIDMuODIlICB2cHBfd2tfMSAgbGlidm5ldC5zby4xOS4wNC4yICAg ICAgWy5dIHNlc3Npb25fdHhfZmlmb19wZWVrX2FuZF9zbmQNCi0gICA1NC41OSUgICAgNTQuNTQl ICB2cHBfd2tfMSAgbGliYy0yLjI3LnNvICAgICAgICAgICAgWy5dIF9fbWVtY3B5X2dlbmVyaWMN CiAgICAgNTQuNTQlIGNsaWJfY2FsbGptcA0KICAgICAgLSB2bGliX3dvcmtlcl9sb29wDQogICAg ICAgICAtIDUyLjE0JSBzZXNzaW9uX3F1ZXVlX25vZGVfZm4NCiAgICAgICAgICAgIC0gNTIuMDUl IHNlc3Npb25fdHhfZmlmb19wZWVrX2FuZF9zbmQNCiAgICAgICAgICAgICAgICAgX19tZW1jcHlf Z2VuZXJpYw0KICAgICAgICAgLSAyLjM5JSBkaXNwYXRjaF9wZW5kaW5nX25vZGUNCiAgICAgICAg ICAgIC0gMi4zNSUgdGNwNF9lc3RhYmxpc2hlZF9ub2RlX2ZuDQogICAgICAgICAgICAgICAgIHRj cF9zZWdtZW50X3Jjdg0KICAgICAgICAgICAgICAgICBzZXNzaW9uX2VucXVldWVfc3RyZWFtX2Nv bm5lY3Rpb24NCiAgICAgICAgICAgICAgICAgX19tZW1jcHlfZ2VuZXJpYw0KKyAgIDM0LjA0JSAg ICAgMC43NSUgIHZwcF93a18xICBsaWJ2bGliLnNvLjE5LjA0LjIgICAgICBbLl0gZGlzcGF0Y2hf cGVuZGluZ19ub2RlDQorICAgMTQuMTQlICAgICAzLjk5JSAgdnBwX3drXzEgIGRwZGtfcGx1Z2lu LnNvICAgICAgICAgIFsuXSBkcGRrX2RldmljZV9jbGFzc190eF9mbg0KKyAgIDEwLjE0JSAgICAg Mi41OCUgIHZwcF93a18xICBkcGRrX3BsdWdpbi5zbyAgICAgICAgICBbLl0gYm54dF94bWl0X3Br dHMNCi0gICAgNy4yOCUgICAgIDYuNDklICB2cHBfd2tfMSAgZHBka19wbHVnaW4uc28gICAgICAg ICAgWy5dIGJueHRfaGFuZGxlX3R4X2NwDQogICAtIDYuNDklIGNsaWJfY2FsbGptcA0KICAgICAg ICB2bGliX3dvcmtlcl9sb29wDQogICAgICAgIGRpc3BhdGNoX3BlbmRpbmdfbm9kZQ0KICAgICAg LSBkcGRrX2RldmljZV9jbGFzc190eF9mbg0KICAgICAgICAgLSA2LjQ4JSBibnh0X3htaXRfcGt0 cw0KICAgICAgICAgICAgICBibnh0X2hhbmRsZV90eF9jcA0KICAgKyAwLjc5JSBibnh0X2hhbmRs ZV90eF9jcA0KKyAgICA2LjkzJSAgICAgMC42NiUgIHZwcF93a18xICBsaWJ2bmV0LnNvLjE5LjA0 LjIgICAgICBbLl0gdGNwNF9lc3RhYmxpc2hlZF9ub2RlX2ZuDQorICAgIDMuODQlICAgICAwLjA1 JSAgdnBwX3drXzEgIGxpYnZuZXQuc28uMTkuMDQuMiAgICAgIFsuXSB0Y3Bfc2VnbWVudF9yY3YN CisgICAgMy42OCUgICAgIDAuMTElICB2cHBfd2tfMSAgbGlidm5ldC5zby4xOS4wNC4yICAgICAg Wy5dIHNlc3Npb25fZW5xdWV1ZV9zdHJlYW1fY29ubmVjdGlvbg0KKyAgICAzLjU3JSAgICAgMy4z MCUgIHZwcF93a18xICBsaWJ2bmV0LnNvLjE5LjA0LjIgICAgICBbLl0gdGNwNF9vdXRwdXRfbm9k ZV9mbg0KKyAgICAyLjg1JSAgICAgMi44MCUgIHZwcF93a18xICBsaWJ2bmV0LnNvLjE5LjA0LjIg ICAgICBbLl0gaXA0X3Jld3JpdGVfbm9kZV9mbg0KKyAgICAyLjMxJSAgICAgMC42MyUgIHZwcF93 a18xICBkcGRrX3BsdWdpbi5zbyAgICAgICAgICBbLl0gZHBka19pbnB1dF9ub2RlX2ZuDQorICAg IDIuMTYlICAgICAxLjk2JSAgdnBwX3drXzEgIGxpYnZuZXQuc28uMTkuMDQuMiAgICAgIFsuXSBp cDRfbG9va3VwX25vZGVfZm4NCisgICAgMS42NCUgICAgIDEuNjQlICB2cHBfd2tfMSAgbGlic3Zt LnNvLjE5LjA0LjIgICAgICAgWy5dIHN2bV9maWZvX3BlZWtfbWENCisgICAgMS40OSUgICAgIDEu NDklICB2cHBfd2tfMSAgZHBka19wbHVnaW4uc28gICAgICAgICAgWy5dIGJueHRfcmVjdl9wa3Rz DQorICAgIDEuMzElICAgICAxLjMwJSAgdnBwX3drXzEgIGxpYnZuZXQuc28uMTkuMDQuMiAgICAg IFsuXSB0Y3Bfc2Vzc2lvbl9wdXNoX2hlYWRlcg0KKyAgICAxLjIwJSAgICAgMS4yMCUgIHZwcF93 a18xICBsaWJzdm0uc28uMTkuMDQuMiAgICAgICBbLl0gc3ZtX2ZpZm9fZW5xdWV1ZV9ub3dhaXRf bWENCisgICAgMS4wNiUgICAgIDAuMDglICB2cHBfd2tfMSAgbGlidm5ldC5zby4xOS4wNC4yICAg ICAgWy5dIHRyYW5zcG9ydF91cGRhdGVfdGltZQ0KKyAgICAwLjg0JSAgICAgMC44NCUgIHZwcF93 a18xICBsaWJzdm0uc28uMTkuMDQuMiAgICAgICBbLl0gc3ZtX2ZpZm9fZGVxdWV1ZV9kcm9wDQor ICAgIDAuODQlICAgICAwLjc4JSAgdnBwX3drXzEgIGxpYnZuZXQuc28uMTkuMDQuMiAgICAgIFsu XSBldGhlcm5ldF9pbnB1dF9ub2RlX2ZuDQorICAgIDAuNzklICAgICAwLjI2JSAgdnBwX3drXzEg IGxpYnZuZXQuc28uMTkuMDQuMiAgICAgIFsuXSB0Y3A0X2lucHV0X25vZGVfZm4NCisgICAgMC43 NiUgICAgIDAuNjklICB2cHBfd2tfMSAgbGlidm5ldC5zby4xOS4wNC4yICAgICAgWy5dIGlwNF9p bnB1dF9ub19jaGVja3N1bV9ub2RlX2ZuDQorICAgIDAuNjklICAgICAwLjY5JSAgdnBwX3drXzEg IGRwZGtfcGx1Z2luLnNvICAgICAgICAgIFsuXSBkcGRrX29wc192cHBfZW5xdWV1ZQ0KKyAgICAw LjY3JSAgICAgMC4wMSUgIHZwcF93a18xICBsaWJ2bmV0LnNvLjE5LjA0LjIgICAgICBbLl0gdGNw X3VwZGF0ZV90aW1lDQorICAgIDAuNjYlICAgICAwLjY2JSAgdnBwX3drXzEgIGxpYnZsaWIuc28u MTkuMDQuMiAgICAgIFsuXSB2bGliX2dldF9uZXh0X2ZyYW1lX2ludGVybmFsDQo= --===============8432582498476632980==--