From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2152143023620865128==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] Callback passed to spdk_nvme_ns_cmd_read not being called sometimes Date: Wed, 06 Jul 2016 15:35:09 +0000 Message-ID: <1467819307.5999.181.camel@intel.com> In-Reply-To: 370981E9-A83E-44B9-B3B7-E9AAEE3E2184@xeograph.com List-ID: To: spdk@lists.01.org --===============2152143023620865128== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Will, Since I can't see the code for your application I'd like to try and reprodu= ce the problem with code that I have some visibility into. Are you able to = reproduce the problem using our perf tool (examples/nvme/perf)? If you aren= 't, this is likely a problem with your test application and not SPDK. Based on the symptoms, my best guess is that your memory pool ran out of re= quest objects. The first thing to check is whether spdk_nvme_ns_cmd_read fa= iled. If it fails, it won't call the callback. You can check for failure by= looking at the return value - see the documentation here. Your applicatio= n allocates this memory pool up front - all of our examples allocate 8k req= uests (see line 1097 in examples/nvme/perf/perf.c) You need to allocate a l= arge enough pool to handle the maximum number of outstanding requests you p= lan to have. We recently added a "hello_world" style example for the NVMe d= river at https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world= with tons of comments. One of the comments explains this memory pool in de= tail. That memory pool allocation is a bit of a wart on our otherwise clean API. = We're looking at different strategies to clean that up. Let me know what th= e result of the debugging is and I'll shoot you some more ideas to try if n= ecessary. Thanks, Ben On Tue, 2016-07-05 at 21:03 +0000, Will Del Genio wrote: Hello, We have written a test application that is utilizing the spdk library to be= nchmark a set of 3 Intel P3700 drives and a single 750 drive (concurrently)= . We=E2=80=99ve done some testing using fio and the kernel nvme drivers an= d have had no problem achieving the claimed IOPs (4k random read) of all dr= ives on our system. What we have found during our testing is that spdk will sometimes start to = silently fail to call the callback passed to spdk_nvme_ns_cmd_read in the f= ollowing situations: 1. Testing a single drive and passing in 0 for max_completions to spd= k_nvme_qpair_process_completions(). We haven=E2=80=99t seen any issues wit= h single drive testing when max_completions was > 0. 2. Testing all four drives at once will result in one drive failing t= o receive callbacks, seemingly regardless of what number we pass for max_co= mpletions (1 through 128). Here are other observations we=E2=80=99ve made -When the callbacks fail to be called for a drive, they fail to be called f= or the remaining duration of the test. -The drive that =E2=80=98fails=E2=80=99 when testing 4 drives concurrently = varies from test to test. -=E2=80=98failure=E2=80=99 of a drive seems to be correlated with the numbe= r of outstanding read operations, though it is not a strict correlation. Our system is a dual socket E5-2630 v3. One drive is on a PCI slot for CP= U 0 and the other 3 are on PCI slots on CPU 1. The master/slave threads ar= e on the the same cpu socket as the nvme device they are talking to. We=E2=80=99d like to know what is causing this issue and what we can do to = help investigate the problem. What other information can we provide? Is t= here some part of the spdk code that we can look at to help determine the c= ause? Thanks, Will _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk --===============2152143023620865128== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4 bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l dGEgbmFtZT0iVGl0bGUiIGNvbnRlbnQ9IiI+DQo8bWV0YSBuYW1lPSJLZXl3b3JkcyIgY29udGVu dD0iIj4NCjxtZXRhIG5hbWU9IkdlbmVyYXRvciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUg KGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxlPjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8N CkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MCAw IDAgMCAwIDAgMCAwIDAgMDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJ cGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAyIDQ7fQ0KLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8N CnAuTXNvTm9ybWFsLCBsaS5Nc29Ob3JtYWwsIGRpdi5Nc29Ob3JtYWwNCgl7bWFyZ2luOjBpbjsN CgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJZm9udC1zaXplOjEyLjBwdDsNCglmb250LWZhbWls eToiVGltZXMgTmV3IFJvbWFuIjt9DQphOmxpbmssIHNwYW4uTXNvSHlwZXJsaW5rDQoJe21zby1z dHlsZS1wcmlvcml0eTo5OTsNCgljb2xvcjojMDU2M0MxOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRl cmxpbmU7fQ0KYTp2aXNpdGVkLCBzcGFuLk1zb0h5cGVybGlua0ZvbGxvd2VkDQoJe21zby1zdHls ZS1wcmlvcml0eTo5OTsNCgljb2xvcjojOTU0RjcyOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxp bmU7fQ0KcA0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJbWFyZ2luOjBpbjsNCgltYXJnaW4t Ym90dG9tOi4wMDAxcHQ7DQoJZm9udC1zaXplOjEyLjBwdDsNCglmb250LWZhbWlseToiVGltZXMg TmV3IFJvbWFuIjt9DQpzcGFuLkVtYWlsU3R5bGUxOA0KCXttc28tc3R5bGUtdHlwZTpwZXJzb25h bC1yZXBseTsNCglmb250LWZhbWlseTpDYWxpYnJpOw0KCWNvbG9yOndpbmRvd3RleHQ7fQ0Kc3Bh bi5tc29JbnMNCgl7bXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJbXNvLXN0eWxlLW5hbWU6 IiI7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTsNCgljb2xvcjp0ZWFsO30NCi5Nc29DaHBE ZWZhdWx0DQoJe21zby1zdHlsZS10eXBlOmV4cG9ydC1vbmx5Ow0KCWZvbnQtc2l6ZToxMC4wcHQ7 fQ0KQHBhZ2UgV29yZFNlY3Rpb24xDQoJe3NpemU6OC41aW4gMTEuMGluOw0KCW1hcmdpbjoxLjBp biAxLjBpbiAxLjBpbiAxLjBpbjt9DQpkaXYuV29yZFNlY3Rpb24xDQoJe3BhZ2U6V29yZFNlY3Rp b24xO30NCi0tPjwvc3R5bGU+DQo8L2hlYWQ+DQo8Ym9keSBsYW5nPSJFTi1VUyI+DQo8ZGl2Pkhp IFdpbGwsPC9kaXY+DQo8ZGl2Pjxicj4NCjwvZGl2Pg0KPGRpdj5TaW5jZSBJIGNhbid0IHNlZSB0 aGUgY29kZSBmb3IgeW91ciBhcHBsaWNhdGlvbiBJJ2QgbGlrZSB0byB0cnkgYW5kIHJlcHJvZHVj ZSB0aGUgcHJvYmxlbSB3aXRoIGNvZGUgdGhhdCBJIGhhdmUgc29tZSB2aXNpYmlsaXR5IGludG8u IEFyZSB5b3UgYWJsZSB0byByZXByb2R1Y2UgdGhlIHByb2JsZW0gdXNpbmcgb3VyIHBlcmYgdG9v bCAoZXhhbXBsZXMvbnZtZS9wZXJmKT8gSWYgeW91IGFyZW4ndCwgdGhpcyBpcyBsaWtlbHkgYSBw cm9ibGVtDQogd2l0aCB5b3VyIHRlc3QgYXBwbGljYXRpb24gYW5kIG5vdCBTUERLLiZuYnNwOzwv ZGl2Pg0KPGRpdj48YnI+DQo8L2Rpdj4NCjxkaXY+QmFzZWQgb24gdGhlIHN5bXB0b21zLCBteSBi ZXN0IGd1ZXNzIGlzIHRoYXQgeW91ciBtZW1vcnkgcG9vbCByYW4gb3V0IG9mIHJlcXVlc3Qgb2Jq ZWN0cy4gVGhlIGZpcnN0IHRoaW5nIHRvIGNoZWNrIGlzIHdoZXRoZXIgc3Bka19udm1lX25zX2Nt ZF9yZWFkIGZhaWxlZC4gSWYgaXQgZmFpbHMsIGl0IHdvbid0IGNhbGwgdGhlIGNhbGxiYWNrLiBZ b3UgY2FuIGNoZWNrIGZvciBmYWlsdXJlIGJ5IGxvb2tpbmcgYXQgdGhlIHJldHVybiB2YWx1ZQ0K IC0gc2VlIHRoZSA8YSBocmVmPSJodHRwOi8vd3d3LnNwZGsuaW8vc3Bkay9kb2MvbnZtZV84aC5o dG1sI2EwODRjNmVjYjUzYmQ4MTBmYmI1MDUxMTAwYjc5YmVjNSI+DQpkb2N1bWVudGF0aW9uIGhl cmU8L2E+LiBZb3VyIGFwcGxpY2F0aW9uIGFsbG9jYXRlcyB0aGlzIG1lbW9yeSBwb29sIHVwIGZy b250IC0gYWxsIG9mIG91ciBleGFtcGxlcyBhbGxvY2F0ZSA4ayByZXF1ZXN0cyAoc2VlIGxpbmUg MTA5NyBpbiBleGFtcGxlcy9udm1lL3BlcmYvcGVyZi5jKSBZb3UgbmVlZCB0byBhbGxvY2F0ZSBh IGxhcmdlIGVub3VnaCBwb29sIHRvIGhhbmRsZSB0aGUgbWF4aW11bSBudW1iZXIgb2Ygb3V0c3Rh bmRpbmcgcmVxdWVzdHMNCiB5b3UgcGxhbiB0byBoYXZlLiBXZSByZWNlbnRseSBhZGRlZCBhICZx dW90O2hlbGxvX3dvcmxkJnF1b3Q7IHN0eWxlIGV4YW1wbGUgZm9yIHRoZSBOVk1lIGRyaXZlciBh dCZuYnNwOzxhIGhyZWY9Imh0dHBzOi8vZ2l0aHViLmNvbS9zcGRrL3NwZGsvdHJlZS9tYXN0ZXIv ZXhhbXBsZXMvbnZtZS9oZWxsb193b3JsZCI+aHR0cHM6Ly9naXRodWIuY29tL3NwZGsvc3Bkay90 cmVlL21hc3Rlci9leGFtcGxlcy9udm1lL2hlbGxvX3dvcmxkPC9hPiZuYnNwO3dpdGggdG9ucyBv ZiBjb21tZW50cy4NCiBPbmUgb2YgdGhlIGNvbW1lbnRzIGV4cGxhaW5zIHRoaXMgbWVtb3J5IHBv b2wgaW4gZGV0YWlsLjwvZGl2Pg0KPGRpdj48YnI+DQo8L2Rpdj4NCjxkaXY+VGhhdCBtZW1vcnkg cG9vbCBhbGxvY2F0aW9uIGlzIGEgYml0IG9mIGEgd2FydCBvbiBvdXIgb3RoZXJ3aXNlIGNsZWFu IEFQSS4gV2UncmUgbG9va2luZyBhdCBkaWZmZXJlbnQgc3RyYXRlZ2llcyB0byBjbGVhbiB0aGF0 IHVwLiBMZXQgbWUga25vdyB3aGF0IHRoZSByZXN1bHQgb2YgdGhlIGRlYnVnZ2luZyBpcyBhbmQg SSdsbCBzaG9vdCB5b3Ugc29tZSBtb3JlIGlkZWFzIHRvIHRyeSBpZiBuZWNlc3NhcnkuPC9kaXY+ DQo8ZGl2Pjxicj4NCjwvZGl2Pg0KPGRpdj5UaGFua3MsPC9kaXY+DQo8ZGl2PkJlbjwvZGl2Pg0K PGRpdj48YnI+DQo8L2Rpdj4NCjxkaXY+T24gVHVlLCAyMDE2LTA3LTA1IGF0IDIxOjAzICYjNDM7 MDAwMCwgV2lsbCBEZWwgR2VuaW8gd3JvdGU6PC9kaXY+DQo8YmxvY2txdW90ZSB0eXBlPSJjaXRl Ij4NCjxkaXYgY2xhc3M9IldvcmRTZWN0aW9uMSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibXNvLW1hcmdpbi10b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG87YmFj a2dyb3VuZDp3aGl0ZSI+DQo8c3BhbiBzdHlsZT0iZm9udC1zaXplOjE1LjBwdDtjb2xvcjpibGFj ayI+SGVsbG8sPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3Nw YW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDph dXRvO21zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvO2JhY2tncm91bmQ6d2hpdGUiPg0KPHNwYW4g c3R5bGU9ImZvbnQtc2l6ZToxNS4wcHQ7Y29sb3I6YmxhY2siPldlIGhhdmUgd3JpdHRlbiBhIHRl c3QgYXBwbGljYXRpb24gdGhhdCBpcyB1dGlsaXppbmcgdGhlIHNwZGsgbGlicmFyeSB0byBiZW5j aG1hcmsgYSBzZXQgb2YgMyBJbnRlbCBQMzcwMCBkcml2ZXMgYW5kIGEgc2luZ2xlIDc1MCBkcml2 ZSAoY29uY3VycmVudGx5KS4mbmJzcDsgV2XigJl2ZSBkb25lIHNvbWUgdGVzdGluZyB1c2luZyBm aW8gYW5kIHRoZSBrZXJuZWwgbnZtZSBkcml2ZXJzDQogYW5kIGhhdmUgaGFkIG5vIHByb2JsZW0g YWNoaWV2aW5nIHRoZSBjbGFpbWVkIElPUHMgKDRrIHJhbmRvbSByZWFkKSBvZiBhbGwgZHJpdmVz IG9uIG91ciBzeXN0ZW0uPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286 cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9w LWFsdDphdXRvO21zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvO2JhY2tncm91bmQ6d2hpdGUiPg0K PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxNS4wcHQ7Y29sb3I6YmxhY2siPiZuYnNwOzwvc3Bhbj48 c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttc28tbWFyZ2luLWJv dHRvbS1hbHQ6YXV0bztiYWNrZ3JvdW5kOndoaXRlIj4NCjxzcGFuIHN0eWxlPSJmb250LXNpemU6 MTUuMHB0O2NvbG9yOmJsYWNrIj5XaGF0IHdlIGhhdmUgZm91bmQgZHVyaW5nIG91ciB0ZXN0aW5n IGlzIHRoYXQgc3BkayB3aWxsIHNvbWV0aW1lcyBzdGFydCB0byBzaWxlbnRseSBmYWlsIHRvIGNh bGwgdGhlIGNhbGxiYWNrIHBhc3NlZCB0byBzcGRrX252bWVfbnNfY21kX3JlYWQgaW4gdGhlIGZv bGxvd2luZyBzaXR1YXRpb25zOjwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2lu LXRvcC1hbHQ6YXV0bzttc28tbWFyZ2luLWJvdHRvbS1hbHQ6YXV0bztiYWNrZ3JvdW5kOndoaXRl Ij4NCjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTUuMHB0O2NvbG9yOmJsYWNrIj4xLjwvc3Bhbj48 c3BhbiBzdHlsZT0iZm9udC1zaXplOjkuMHB0O2NvbG9yOmJsYWNrIj4mbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsNCjwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjE1LjBw dDtjb2xvcjpibGFjayI+VGVzdGluZyBhIHNpbmdsZSBkcml2ZSBhbmQgcGFzc2luZyBpbiAwIGZv ciBtYXhfY29tcGxldGlvbnMgdG8gc3Bka19udm1lX3FwYWlyX3Byb2Nlc3NfY29tcGxldGlvbnMo KS4gJm5ic3A7V2UgaGF2ZW7igJl0IHNlZW4gYW55IGlzc3VlcyB3aXRoIHNpbmdsZSBkcml2ZSB0 ZXN0aW5nIHdoZW4gbWF4X2NvbXBsZXRpb25zIHdhcyAmZ3Q7IDAuPC9zcGFuPjxzcGFuIHN0eWxl PSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1h bCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDphdXRvO21zby1tYXJnaW4tYm90dG9tLWFsdDph dXRvO2JhY2tncm91bmQ6d2hpdGUiPg0KPHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxNS4wcHQ7Y29s b3I6YmxhY2siPjIuPC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LXNpemU6OS4wcHQ7Y29sb3I6Ymxh Y2siPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOw0KPC9zcGFuPjxzcGFuIHN0 eWxlPSJmb250LXNpemU6MTUuMHB0O2NvbG9yOmJsYWNrIj5UZXN0aW5nIGFsbCBmb3VyIGRyaXZl cyBhdCBvbmNlIHdpbGwgcmVzdWx0IGluIG9uZSBkcml2ZSBmYWlsaW5nIHRvIHJlY2VpdmUgY2Fs bGJhY2tzLCBzZWVtaW5nbHkgcmVnYXJkbGVzcyBvZiB3aGF0IG51bWJlciB3ZSBwYXNzIGZvciBt YXhfY29tcGxldGlvbnMgKDEgdGhyb3VnaCAxMjgpLjwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6 YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxl PSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttc28tbWFyZ2luLWJvdHRvbS1hbHQ6YXV0bztiYWNr Z3JvdW5kOndoaXRlIj4NCjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTUuMHB0O2NvbG9yOmJsYWNr Ij4mbmJzcDs8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bh bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OmF1 dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG87YmFja2dyb3VuZDp3aGl0ZSI+DQo8c3BhbiBz dHlsZT0iZm9udC1zaXplOjE1LjBwdDtjb2xvcjpibGFjayI+SGVyZSBhcmUgb3RoZXIgb2JzZXJ2 YXRpb25zIHdl4oCZdmUgbWFkZTwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2lu LXRvcC1hbHQ6YXV0bzttc28tbWFyZ2luLWJvdHRvbS1hbHQ6YXV0bzttYXJnaW4tbGVmdDo0OC4w cHQ7dGV4dC1pbmRlbnQ6LTQ4LjBwdDtiYWNrZ3JvdW5kOndoaXRlO3RleHQtYXV0b3NwYWNlOm5v bmUiPg0KPHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxNS4wcHQ7Y29sb3I6YmxhY2siPi1XaGVuIHRo ZSBjYWxsYmFja3MgZmFpbCB0byBiZSBjYWxsZWQgZm9yIGEgZHJpdmUsIHRoZXkgZmFpbCB0byBi ZSBjYWxsZWQgZm9yIHRoZSByZW1haW5pbmcgZHVyYXRpb24gb2YgdGhlIHRlc3QuPC9zcGFuPjxz cGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDphdXRvO21zby1tYXJnaW4tYm90 dG9tLWFsdDphdXRvO21hcmdpbi1sZWZ0OjQ4LjBwdDt0ZXh0LWluZGVudDotNDguMHB0O2JhY2tn cm91bmQ6d2hpdGU7dGV4dC1hdXRvc3BhY2U6bm9uZSI+DQo8c3BhbiBzdHlsZT0iZm9udC1zaXpl OjE1LjBwdDtjb2xvcjpibGFjayI+LVRoZSBkcml2ZSB0aGF0IOKAmGZhaWxz4oCZIHdoZW4gdGVz dGluZyA0IGRyaXZlcyBjb25jdXJyZW50bHkgdmFyaWVzIGZyb20gdGVzdCB0byB0ZXN0Ljwvc3Bh bj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNs YXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttc28tbWFyZ2lu LWJvdHRvbS1hbHQ6YXV0bzttYXJnaW4tbGVmdDo0OC4wcHQ7dGV4dC1pbmRlbnQ6LTQ4LjBwdDti YWNrZ3JvdW5kOndoaXRlO3RleHQtYXV0b3NwYWNlOm5vbmUiPg0KPHNwYW4gc3R5bGU9ImZvbnQt c2l6ZToxNS4wcHQ7Y29sb3I6YmxhY2siPi3igJhmYWlsdXJl4oCZIG9mIGEgZHJpdmUgc2VlbXMg dG8gYmUgY29ycmVsYXRlZCB3aXRoIHRoZSBudW1iZXIgb2Ygb3V0c3RhbmRpbmcgcmVhZCBvcGVy YXRpb25zLCB0aG91Z2ggaXQgaXMgbm90IGEgc3RyaWN0IGNvcnJlbGF0aW9uLjwvc3Bhbj48c3Bh biBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttc28tbWFyZ2luLWJvdHRv bS1hbHQ6YXV0bztiYWNrZ3JvdW5kOndoaXRlIj4NCjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTUu MHB0O2NvbG9yOmJsYWNrIj4mbmJzcDs8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48 bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1h cmdpbi10b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG87YmFja2dyb3VuZDp3 aGl0ZSI+DQo8c3BhbiBzdHlsZT0iZm9udC1zaXplOjE1LjBwdDtjb2xvcjpibGFjayI+T3VyIHN5 c3RlbSBpcyBhIGR1YWwgc29ja2V0Jm5ic3A7IEU1LTI2MzAgdjMuJm5ic3A7IE9uZSBkcml2ZSBp cyBvbiBhIFBDSSBzbG90IGZvciBDUFUgMCBhbmQgdGhlIG90aGVyIDMgYXJlIG9uIFBDSSBzbG90 cyBvbiBDUFUgMS4mbmJzcDsgVGhlIG1hc3Rlci9zbGF2ZSB0aHJlYWRzIGFyZSBvbiB0aGUgdGhl IHNhbWUgY3B1IHNvY2tldCBhcyB0aGUgbnZtZSBkZXZpY2UgdGhleSBhcmUgdGFsa2luZw0KIHRv Ljwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttc28t bWFyZ2luLWJvdHRvbS1hbHQ6YXV0bztiYWNrZ3JvdW5kOndoaXRlIj4NCjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTUuMHB0O2NvbG9yOmJsYWNrIj4mbmJzcDs8L3NwYW4+PHNwYW4gc3R5bGU9ImNv bG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBz dHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG87 YmFja2dyb3VuZDp3aGl0ZSI+DQo8c3BhbiBzdHlsZT0iZm9udC1zaXplOjE1LjBwdDtjb2xvcjpi bGFjayI+V2XigJlkIGxpa2UgdG8ga25vdyB3aGF0IGlzIGNhdXNpbmcgdGhpcyBpc3N1ZSBhbmQg d2hhdCB3ZSBjYW4gZG8gdG8gaGVscCBpbnZlc3RpZ2F0ZSB0aGUgcHJvYmxlbS4mbmJzcDsgV2hh dCBvdGhlciBpbmZvcm1hdGlvbiBjYW4gd2UgcHJvdmlkZT8mbmJzcDsgSXMgdGhlcmUgc29tZSBw YXJ0IG9mIHRoZSBzcGRrIGNvZGUgdGhhdCB3ZSBjYW4gbG9vayBhdCB0byBoZWxwIGRldGVybWlu ZSB0aGUNCiBjYXVzZT88L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpw Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3At YWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG87YmFja2dyb3VuZDp3aGl0ZSI+DQo8 c3BhbiBzdHlsZT0iZm9udC1zaXplOjE1LjBwdDtjb2xvcjpibGFjayI+Jm5ic3A7PC9zcGFuPjxz cGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDphdXRvO21zby1tYXJnaW4tYm90 dG9tLWFsdDphdXRvO2JhY2tncm91bmQ6d2hpdGUiPg0KPHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTox NS4wcHQ7Y29sb3I6YmxhY2siPlRoYW5rcyw8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNr Ij48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNv LW1hcmdpbi10b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG87YmFja2dyb3Vu ZDp3aGl0ZSI+DQo8c3BhbiBzdHlsZT0iZm9udC1zaXplOjE1LjBwdDtjb2xvcjpibGFjayI+V2ls bDwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJiYWNrZ3JvdW5kOndoaXRlIj48c3BhbiBzdHls ZT0iZm9udC1mYW1pbHk6Q2FsaWJyaTtjb2xvcjpibGFjayI+PG86cD4mbmJzcDs8L286cD48L3Nw YW4+PC9wPg0KPC9kaXY+DQo8cHJlPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fDQpTUERLIG1haWxpbmcgbGlzdA0KPGEgaHJlZj0ibWFpbHRvOlNQREtAbGlz dHMuMDEub3JnIj5TUERLQGxpc3RzLjAxLm9yZzwvYT4NCjxhIGhyZWY9Imh0dHBzOi8vbGlzdHMu MDEub3JnL21haWxtYW4vbGlzdGluZm8vc3BkayI+aHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFpbG1h bi9saXN0aW5mby9zcGRrPC9hPg0KPC9wcmU+DQo8L2Jsb2NrcXVvdGU+DQo8L2JvZHk+DQo8L2h0 bWw+DQo= --===============2152143023620865128==--