From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============3863136022233690908==" MIME-Version: 1.0 From: Sreeni (Sreenivasa) Busam (Stellus) Subject: Re: [SPDK] Request for comments regarding latency measurements Date: Tue, 14 Nov 2017 18:27:53 +0000 Message-ID: <2138115ff2844bc58be2aecdfcceab1a@stellus.com> In-Reply-To: 30F890FD-0AC9-4390-9B35-A6DCA41E6C4E@intel.com List-ID: To: spdk@lists.01.org --===============3863136022233690908== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Jim, I am not an SPDK driver expert. As I have been working on the same problem = in the last 2 weeks, in my investigation, I concluded the same as you have = written in the email, option 2 looked like easy and good solution. The bdev interfaces have been well developed for NVMe devices and it looks = similar to the Linux kernel Storage driver except for the polling and inter= rupt differences to get the status. These are just my comments based on wha= t I understood so far. I have already been working on collecting the I/O statistics for the applic= ations on NVMe layer. For any applications using the bdev layer, I would li= ke to use your changes. When you are planning to submit the patch to reposi= tory ? Please let me know when you complete your investigation. Thanks, Sreeni From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R Sent: Tuesday, November 14, 2017 7:43 AM To: Storage Performance Development Kit Subject: Re: [SPDK] Request for comments regarding latency measurements Agreed - #2 is the way to go. While we=E2=80=99re on the topic - should we be resetting the stats when sp= dk_bdev_get_io_stat() is called? This eliminates the ability for two separ= ate applications to monitor stats in parallel =E2=80=93 or even a separate = application plus some future yet-to-be-written internal load-balancing moni= tor. I=E2=80=99m thinking that bdev should just keep the running total and= let the caller own tracking differences from the last time it called spdk_= bdev_get_io_stat(). -Jim From: SPDK = > on behalf of Nathan Marushak > Reply-To: Storage Performance Development Kit > Date: Tuesday, November 14, 2017 at 7:29 AM To: Storage Performance Development Kit > Subject: Re: [SPDK] Request for comments regarding latency measurements So, not being the technical expert so to speak =E2=98=BA, please take this = with a grain of salt. Agree with option #2. Seems like this would allow for= keeping stats across the different device types e.g. NVMe, NVML, NVMe-oF (= although that has a transport within NVMe, so it might be covered for eithe= r option). From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E Sent: Tuesday, November 14, 2017 7:57 AM To: Storage Performance Development Kit > Subject: Re: [SPDK] Request for comments regarding latency measurements FWIW option 2 sounds like the most appropriate to me=E2=80=A6 Thx Paul From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Paul Von-Stamw= itz Sent: Monday, November 13, 2017 7:29 PM To: spdk(a)lists.01.org Subject: [SPDK] Request for comments regarding latency measurements Background: Bdev.c currently maintains a count of reads/writes/bytes_read/bytes_written= in channel->stat. This information is retrieved (and reset) via spdk_bdev_= get_io_stat. Proposal: Add latency information to channel->stat and enable the option to provide a= histogram. We can measure the latency of each IO and keep a running total on a read/wr= ite basis. We can also use the measured latency to keep a running count of = reads/writes in their associated histogram =E2=80=9Cbuckets=E2=80=9D. The question is, how do we measure latency? Option 1: Measure latency at the NVMe tracker . Currently, we already timestamp every command placed on the tracker if the = abort timeout callback is registered. When we remove a completed IO from th= e tracker, we can timestamp it again and calculate the latency. There are several issues here that need to be considered. We need to get the latency information back up to the bdev layer, most like= ly through a callback argument, but this would require a change the NVMe AP= I. If the bdev breaks down a request into smaller IOs, it can add up the la= tencies of the child IOs for io_stat and histogram purposes. Also, this method does not take into account any latency added by the spdk = bdev/nvme layers (except the poller.) If a request was queued before being = placed on the tracker, then the time it was queued is not factored into the= latency calculation. Option 2: Measure latency at the bdev layer. We can timestamp at submit and again at completion. This would keep all io_= stat information local to bdev and would take into account the overhead of = most queued operations. Any applications written directly to the NVMe layer= would have to calculate their own latencies, but that is currently true fo= r all io_stats. I=E2=80=99m sure that there are other issues I am missing, but I would appr= eciate any comments on how best to move forward on this. Thanks, Paul --===============3863136022233690908== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6 V2luZ2RpbmdzOw0KCXBhbm9zZS0xOjUgMCAwIDAgMCAwIDAgMCAwIDA7fQ0KQGZvbnQtZmFjZQ0K CXtmb250LWZhbWlseToiQ2FtYnJpYSBNYXRoIjsNCglwYW5vc2UtMToyIDQgNSAzIDUgNCA2IDMg MiA0O30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6Ik1hbGd1biBHb3RoaWMiOw0KCXBhbm9z ZS0xOjIgMTEgNSAzIDIgMCAwIDIgMCA0O30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6Q2Fs aWJyaTsNCglwYW5vc2UtMToyIDE1IDUgMiAyIDIgNCAzIDIgNDt9DQpAZm9udC1mYWNlDQoJe2Zv bnQtZmFtaWx5OiJcQE1hbGd1biBHb3RoaWMiOw0KCXBhbm9zZS0xOjIgMTEgNSAzIDIgMCAwIDIg MCA0O30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6IkludGVsIENsZWFyIjt9DQovKiBTdHls ZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3JtYWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1h bA0KCXttYXJnaW46MGluOw0KCW1hcmdpbi1ib3R0b206LjAwMDFwdDsNCglmb250LXNpemU6MTEu MHB0Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixzYW5zLXNlcmlmO30NCmE6bGluaywgc3Bhbi5N c29IeXBlcmxpbmsNCgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCWNvbG9yOmJsdWU7DQoJdGV4 dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQphOnZpc2l0ZWQsIHNwYW4uTXNvSHlwZXJsaW5rRm9s bG93ZWQNCgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCWNvbG9yOnB1cnBsZTsNCgl0ZXh0LWRl Y29yYXRpb246dW5kZXJsaW5lO30NCnAuTXNvTGlzdFBhcmFncmFwaCwgbGkuTXNvTGlzdFBhcmFn cmFwaCwgZGl2Lk1zb0xpc3RQYXJhZ3JhcGgNCgl7bXNvLXN0eWxlLXByaW9yaXR5OjM0Ow0KCW1h cmdpbi10b3A6MGluOw0KCW1hcmdpbi1yaWdodDowaW47DQoJbWFyZ2luLWJvdHRvbTowaW47DQoJ bWFyZ2luLWxlZnQ6LjVpbjsNCgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJZm9udC1zaXplOjEx LjBwdDsNCglmb250LWZhbWlseToiQ2FsaWJyaSIsc2Fucy1zZXJpZjt9DQpzcGFuLkVtYWlsU3R5 bGUxOA0KCXttc28tc3R5bGUtdHlwZTpwZXJzb25hbDsNCglmb250LWZhbWlseToiQ2FsaWJyaSIs c2Fucy1zZXJpZjsNCgljb2xvcjp3aW5kb3d0ZXh0O30NCnNwYW4uRW1haWxTdHlsZTE5DQoJe21z by1zdHlsZS10eXBlOnBlcnNvbmFsOw0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixzYW5zLXNlcmlm Ow0KCWNvbG9yOiMxRjQ5N0Q7fQ0Kc3Bhbi5FbWFpbFN0eWxlMjANCgl7bXNvLXN0eWxlLXR5cGU6 cGVyc29uYWw7DQoJZm9udC1mYW1pbHk6IkludGVsIENsZWFyIjsNCgljb2xvcjojOTkwMDAwOw0K CWZvbnQtd2VpZ2h0Om5vcm1hbDsNCglmb250LXN0eWxlOm5vcm1hbDsNCgl0ZXh0LWRlY29yYXRp b246bm9uZSBub25lO30NCnNwYW4uRW1haWxTdHlsZTIxDQoJe21zby1zdHlsZS10eXBlOnBlcnNv bmFsOw0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixzYW5zLXNlcmlmOw0KCWNvbG9yOndpbmRvd3Rl eHQ7fQ0Kc3Bhbi5FbWFpbFN0eWxlMjINCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtcmVwbHk7 DQoJZm9udC1mYW1pbHk6IkNhbGlicmkiLHNhbnMtc2VyaWY7DQoJY29sb3I6IzFGNDk3RDt9DQou TXNvQ2hwRGVmYXVsdA0KCXttc28tc3R5bGUtdHlwZTpleHBvcnQtb25seTsNCglmb250LXNpemU6 MTAuMHB0O30NCkBwYWdlIFdvcmRTZWN0aW9uMQ0KCXtzaXplOjguNWluIDExLjBpbjsNCgltYXJn aW46MS4waW4gMS4waW4gMS4waW4gMS4waW47fQ0KZGl2LldvcmRTZWN0aW9uMQ0KCXtwYWdlOldv cmRTZWN0aW9uMTt9DQotLT48L3N0eWxlPjwhLS1baWYgZ3RlIG1zbyA5XT48eG1sPg0KPG86c2hh cGVkZWZhdWx0cyB2OmV4dD0iZWRpdCIgc3BpZG1heD0iMTAyNiIgLz4NCjwveG1sPjwhW2VuZGlm XS0tPjwhLS1baWYgZ3RlIG1zbyA5XT48eG1sPg0KPG86c2hhcGVsYXlvdXQgdjpleHQ9ImVkaXQi Pg0KPG86aWRtYXAgdjpleHQ9ImVkaXQiIGRhdGE9IjEiIC8+DQo8L286c2hhcGVsYXlvdXQ+PC94 bWw+PCFbZW5kaWZdLS0+DQo8L2hlYWQ+DQo8Ym9keSBiZ2NvbG9yPSJ3aGl0ZSIgbGFuZz0iRU4t VVMiIGxpbms9ImJsdWUiIHZsaW5rPSJwdXJwbGUiPg0KPGRpdiBjbGFzcz0iV29yZFNlY3Rpb24x Ij4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj5IaSBK aW0sPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImNvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj5JIGFtIG5vdCBhbiBTUERL IGRyaXZlciBleHBlcnQuIEFzIEkgaGF2ZSBiZWVuIHdvcmtpbmcgb24gdGhlIHNhbWUgcHJvYmxl bSBpbiB0aGUgbGFzdCAyIHdlZWtzLCBpbiBteSBpbnZlc3RpZ2F0aW9uLCBJIGNvbmNsdWRlZCB0 aGUgc2FtZSBhcyB5b3UgaGF2ZSB3cml0dGVuIGluIHRoZSBlbWFpbCwgb3B0aW9uIDIgbG9va2Vk IGxpa2UgZWFzeSBhbmQgZ29vZCBzb2x1dGlvbi4NCjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj5UaGUgYmRldiBp bnRlcmZhY2VzIGhhdmUgYmVlbiB3ZWxsIGRldmVsb3BlZCBmb3IgTlZNZSBkZXZpY2VzIGFuZCBp dCBsb29rcyBzaW1pbGFyIHRvIHRoZSBMaW51eCBrZXJuZWwgU3RvcmFnZSBkcml2ZXIgZXhjZXB0 IGZvciB0aGUgcG9sbGluZyBhbmQgaW50ZXJydXB0IGRpZmZlcmVuY2VzIHRvIGdldCB0aGUgc3Rh dHVzLiBUaGVzZSBhcmUganVzdCBteSBjb21tZW50cw0KIGJhc2VkIG9uIHdoYXQgSSB1bmRlcnN0 b29kIHNvIGZhci48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+SSBoYXZlIGFscmVhZHkgYmVlbiB3b3JraW5nIG9u IGNvbGxlY3RpbmcgdGhlIEkvTyBzdGF0aXN0aWNzIGZvciB0aGUgYXBwbGljYXRpb25zIG9uIE5W TWUgbGF5ZXIuIEZvciBhbnkgYXBwbGljYXRpb25zIHVzaW5nIHRoZSBiZGV2IGxheWVyLCBJIHdv dWxkIGxpa2UgdG8gdXNlIHlvdXIgY2hhbmdlcy4gV2hlbiB5b3UgYXJlIHBsYW5uaW5nIHRvIHN1 Ym1pdCB0aGUNCiBwYXRjaCB0byByZXBvc2l0b3J5ID8gPG86cD48L286cD48L3NwYW4+PC9wPg0K PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImNvbG9yOiMxRjQ5N0QiPlBsZWFzZSBs ZXQgbWUga25vdyB3aGVuIHlvdSBjb21wbGV0ZSB5b3VyIGludmVzdGlnYXRpb24uDQo8bzpwPjwv bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6 IzFGNDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1h bCI+PHNwYW4gc3R5bGU9ImNvbG9yOiMxRjQ5N0QiPlRoYW5rcyw8bzpwPjwvbzpwPjwvc3Bhbj48 L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+U3Jl ZW5pPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PGEgbmFtZT0i X01haWxFbmRDb21wb3NlIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8 L286cD48L3NwYW4+PC9hPjwvcD4NCjxkaXY+DQo8ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3Jk ZXItdG9wOnNvbGlkICNFMUUxRTEgMS4wcHQ7cGFkZGluZzozLjBwdCAwaW4gMGluIDBpbiI+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48Yj5Gcm9tOjwvYj4gU1BESyBbbWFpbHRvOnNwZGstYm91bmNl c0BsaXN0cy4wMS5vcmddIDxiPk9uIEJlaGFsZiBPZg0KPC9iPkhhcnJpcywgSmFtZXMgUjxicj4N CjxiPlNlbnQ6PC9iPiBUdWVzZGF5LCBOb3ZlbWJlciAxNCwgMjAxNyA3OjQzIEFNPGJyPg0KPGI+ VG86PC9iPiBTdG9yYWdlIFBlcmZvcm1hbmNlIERldmVsb3BtZW50IEtpdCAmbHQ7c3Bka0BsaXN0 cy4wMS5vcmcmZ3Q7PGJyPg0KPGI+U3ViamVjdDo8L2I+IFJlOiBbU1BES10gUmVxdWVzdCBmb3Ig Y29tbWVudHMgcmVnYXJkaW5nIGxhdGVuY3kgbWVhc3VyZW1lbnRzPG86cD48L286cD48L3A+DQo8 L2Rpdj4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+ DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5BZ3JlZWQgLSAjMiBpcyB0aGUgd2F5IHRvIGdvLjxvOnA+ PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj5XaGlsZSB3ZeKAmXJlIG9uIHRoZSB0b3BpYyAtIHNob3VsZCB3 ZSBiZSByZXNldHRpbmcgdGhlIHN0YXRzIHdoZW4gc3Bka19iZGV2X2dldF9pb19zdGF0KCkgaXMg Y2FsbGVkPyZuYnNwOyBUaGlzIGVsaW1pbmF0ZXMgdGhlIGFiaWxpdHkgZm9yIHR3byBzZXBhcmF0 ZSBhcHBsaWNhdGlvbnMgdG8gbW9uaXRvciBzdGF0cyBpbiBwYXJhbGxlbCDigJMgb3IgZXZlbiBh IHNlcGFyYXRlIGFwcGxpY2F0aW9uIHBsdXMgc29tZSBmdXR1cmUNCiB5ZXQtdG8tYmUtd3JpdHRl biBpbnRlcm5hbCBsb2FkLWJhbGFuY2luZyBtb25pdG9yLiZuYnNwOyBJ4oCZbSB0aGlua2luZyB0 aGF0IGJkZXYgc2hvdWxkIGp1c3Qga2VlcCB0aGUgcnVubmluZyB0b3RhbCBhbmQgbGV0IHRoZSBj YWxsZXIgb3duIHRyYWNraW5nIGRpZmZlcmVuY2VzIGZyb20gdGhlIGxhc3QgdGltZSBpdCBjYWxs ZWQgc3Bka19iZGV2X2dldF9pb19zdGF0KCkuPG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPi1KaW08 bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9w Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8ZGl2IHN0eWxl PSJib3JkZXI6bm9uZTtib3JkZXItdG9wOnNvbGlkICNCNUM0REYgMS4wcHQ7cGFkZGluZzozLjBw dCAwaW4gMGluIDBpbiI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6 LjVpbiI+PGI+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5Gcm9tOiA8L3NwYW4+DQo8L2I+PHNw YW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5TUERLICZsdDs8YSBocmVmPSJtYWlsdG86c3Bkay1ib3Vu Y2VzQGxpc3RzLjAxLm9yZyI+c3Bkay1ib3VuY2VzQGxpc3RzLjAxLm9yZzwvYT4mZ3Q7IG9uIGJl aGFsZiBvZiBOYXRoYW4gTWFydXNoYWsgJmx0OzxhIGhyZWY9Im1haWx0bzpuYXRoYW4ubWFydXNo YWtAaW50ZWwuY29tIj5uYXRoYW4ubWFydXNoYWtAaW50ZWwuY29tPC9hPiZndDs8YnI+DQo8Yj5S ZXBseS1UbzogPC9iPlN0b3JhZ2UgUGVyZm9ybWFuY2UgRGV2ZWxvcG1lbnQgS2l0ICZsdDs8YSBo cmVmPSJtYWlsdG86c3Bka0BsaXN0cy4wMS5vcmciPnNwZGtAbGlzdHMuMDEub3JnPC9hPiZndDs8 YnI+DQo8Yj5EYXRlOiA8L2I+VHVlc2RheSwgTm92ZW1iZXIgMTQsIDIwMTcgYXQgNzoyOSBBTTxi cj4NCjxiPlRvOiA8L2I+U3RvcmFnZSBQZXJmb3JtYW5jZSBEZXZlbG9wbWVudCBLaXQgJmx0Ozxh IGhyZWY9Im1haWx0bzpzcGRrQGxpc3RzLjAxLm9yZyI+c3Bka0BsaXN0cy4wMS5vcmc8L2E+Jmd0 Ozxicj4NCjxiPlN1YmplY3Q6IDwvYj5SZTogW1NQREtdIFJlcXVlc3QgZm9yIGNvbW1lbnRzIHJl Z2FyZGluZyBsYXRlbmN5IG1lYXN1cmVtZW50czwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXpl OjEyLjBwdDtjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2 Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPjxzcGFuIHN0 eWxlPSJmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDssc2VyaWYiPjxvOnA+ Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5 bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtJbnRl bCBDbGVhciZxdW90Oztjb2xvcjojOTkwMDAwIj5Tbywgbm90IGJlaW5nIHRoZSB0ZWNobmljYWwg ZXhwZXJ0IHNvIHRvIHNwZWFrDQo8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5Oldpbmdk aW5ncztjb2xvcjojOTkwMDAwIj5KPC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVv dDtJbnRlbCBDbGVhciZxdW90Oztjb2xvcjojOTkwMDAwIj4sIHBsZWFzZSB0YWtlIHRoaXMgd2l0 aCBhIGdyYWluIG9mIHNhbHQuIEFncmVlIHdpdGggb3B0aW9uICMyLiBTZWVtcyBsaWtlIHRoaXMg d291bGQgYWxsb3cgZm9yIGtlZXBpbmcgc3RhdHMgYWNyb3NzIHRoZSBkaWZmZXJlbnQgZGV2aWNl IHR5cGVzDQogZS5nLiBOVk1lLCBOVk1MLCBOVk1lLW9GIChhbHRob3VnaCB0aGF0IGhhcyBhIHRy YW5zcG9ydCB3aXRoaW4gTlZNZSwgc28gaXQgbWlnaHQgYmUgY292ZXJlZCBmb3IgZWl0aGVyIG9w dGlvbikuPC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9 Im1hcmdpbi1sZWZ0Oi41aW4iPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtJbnRlbCBD bGVhciZxdW90Oztjb2xvcjojOTkwMDAwIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8 ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpzb2xpZCBibHVlIDEuNXB0O3BhZGRp bmc6MGluIDBpbiAwaW4gNC4wcHQiPg0KPGRpdj4NCjxkaXYgc3R5bGU9ImJvcmRlcjpub25lO2Jv cmRlci10b3A6c29saWQgI0UxRTFFMSAxLjBwdDtwYWRkaW5nOjMuMHB0IDBpbiAwaW4gMGluIj4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj48Yj5Gcm9tOjwv Yj4gU1BESyBbPGEgaHJlZj0ibWFpbHRvOnNwZGstYm91bmNlc0BsaXN0cy4wMS5vcmciPm1haWx0 bzpzcGRrLWJvdW5jZXNAbGlzdHMuMDEub3JnPC9hPl0NCjxiPk9uIEJlaGFsZiBPZiA8L2I+THVz ZSwgUGF1bCBFPGJyPg0KPGI+U2VudDo8L2I+IFR1ZXNkYXksIE5vdmVtYmVyIDE0LCAyMDE3IDc6 NTcgQU08YnI+DQo8Yj5Ubzo8L2I+IFN0b3JhZ2UgUGVyZm9ybWFuY2UgRGV2ZWxvcG1lbnQgS2l0 ICZsdDs8YSBocmVmPSJtYWlsdG86c3Bka0BsaXN0cy4wMS5vcmciPnNwZGtAbGlzdHMuMDEub3Jn PC9hPiZndDs8YnI+DQo8Yj5TdWJqZWN0OjwvYj4gUmU6IFtTUERLXSBSZXF1ZXN0IGZvciBjb21t ZW50cyByZWdhcmRpbmcgbGF0ZW5jeSBtZWFzdXJlbWVudHM8bzpwPjwvbzpwPjwvcD4NCjwvZGl2 Pg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+ Jm5ic3A7PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2lu LWxlZnQ6LjVpbiI+PHNwYW4gc3R5bGU9ImNvbG9yOiMxRjQ5N0QiPkZXSVcgb3B0aW9uIDIgc291 bmRzIGxpa2UgdGhlIG1vc3QgYXBwcm9wcmlhdGUgdG8gbWXigKY8L3NwYW4+PG86cD48L286cD48 L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+PHNwYW4g c3R5bGU9ImNvbG9yOiMxRjQ5N0QiPiZuYnNwOzwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjxwIGNs YXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj48c3BhbiBzdHlsZT0iY29s b3I6IzFGNDk3RCI+VGh4PC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1h bCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj5Q YXVsPC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1h cmdpbi1sZWZ0Oi41aW4iPjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj4mbmJzcDs8L3NwYW4+ PG86cD48L286cD48L3A+DQo8ZGl2Pg0KPGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLXRv cDpzb2xpZCAjRTFFMUUxIDEuMHB0O3BhZGRpbmc6My4wcHQgMGluIDBpbiAwaW4iPg0KPHAgY2xh c3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPjxhIG5hbWU9Il9fX19fcmVw bHlzZXBhcmF0b3IiPjwvYT48Yj5Gcm9tOjwvYj4gU1BESyBbPGEgaHJlZj0ibWFpbHRvOnNwZGst Ym91bmNlc0BsaXN0cy4wMS5vcmciPm1haWx0bzpzcGRrLWJvdW5jZXNAbGlzdHMuMDEub3JnPC9h Pl0NCjxiPk9uIEJlaGFsZiBPZiA8L2I+UGF1bCBWb24tU3RhbXdpdHo8YnI+DQo8Yj5TZW50Ojwv Yj4gTW9uZGF5LCBOb3ZlbWJlciAxMywgMjAxNyA3OjI5IFBNPGJyPg0KPGI+VG86PC9iPiA8YSBo cmVmPSJtYWlsdG86c3Bka0BsaXN0cy4wMS5vcmciPnNwZGtAbGlzdHMuMDEub3JnPC9hPjxicj4N CjxiPlN1YmplY3Q6PC9iPiBbU1BES10gUmVxdWVzdCBmb3IgY29tbWVudHMgcmVnYXJkaW5nIGxh dGVuY3kgbWVhc3VyZW1lbnRzPG86cD48L286cD48L3A+DQo8L2Rpdj4NCjwvZGl2Pg0KPHAgY2xh c3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPiZuYnNwOzxvOnA+PC9vOnA+ PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPkJhY2tn cm91bmQ6PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2lu LWxlZnQ6LjVpbiI+QmRldi5jIGN1cnJlbnRseSBtYWludGFpbnMgYSBjb3VudCBvZiByZWFkcy93 cml0ZXMvYnl0ZXNfcmVhZC9ieXRlc193cml0dGVuIGluIGNoYW5uZWwtJmd0O3N0YXQuIFRoaXMg aW5mb3JtYXRpb24gaXMgcmV0cmlldmVkIChhbmQgcmVzZXQpIHZpYSBzcGRrX2JkZXZfZ2V0X2lv X3N0YXQuPG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2lu LWxlZnQ6LjVpbiI+Jm5ic3A7PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBz dHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+UHJvcG9zYWw6PG86cD48L286cD48L3A+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+QWRkIGxhdGVuY3kgaW5mb3Jt YXRpb24gdG8gY2hhbm5lbC0mZ3Q7c3RhdCBhbmQgZW5hYmxlIHRoZSBvcHRpb24gdG8gcHJvdmlk ZSBhIGhpc3RvZ3JhbS48bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxl PSJtYXJnaW4tbGVmdDouNWluIj5XZSBjYW4gbWVhc3VyZSB0aGUgbGF0ZW5jeSBvZiBlYWNoIElP IGFuZCBrZWVwIGEgcnVubmluZyB0b3RhbCBvbiBhIHJlYWQvd3JpdGUgYmFzaXMuIFdlIGNhbiBh bHNvIHVzZSB0aGUgbWVhc3VyZWQgbGF0ZW5jeSB0byBrZWVwIGEgcnVubmluZyBjb3VudCBvZiBy ZWFkcy93cml0ZXMgaW4gdGhlaXIgYXNzb2NpYXRlZCBoaXN0b2dyYW0g4oCcYnVja2V0c+KAnS48 bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDou NWluIj5UaGUgcXVlc3Rpb24gaXMsIGhvdyBkbyB3ZSBtZWFzdXJlIGxhdGVuY3k/PG86cD48L286 cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+Jm5i c3A7PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxl ZnQ6LjVpbiI+T3B0aW9uIDE6PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBz dHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+TWVhc3VyZSBsYXRlbmN5IGF0IHRoZSBOVk1lIHRyYWNr ZXIgLjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1s ZWZ0Oi41aW4iPkN1cnJlbnRseSwgd2UgYWxyZWFkeSB0aW1lc3RhbXAgZXZlcnkgY29tbWFuZCBw bGFjZWQgb24gdGhlIHRyYWNrZXIgaWYgdGhlIGFib3J0IHRpbWVvdXQgY2FsbGJhY2sgaXMgcmVn aXN0ZXJlZC4gV2hlbiB3ZSByZW1vdmUgYSBjb21wbGV0ZWQgSU8gZnJvbSB0aGUgdHJhY2tlciwg d2UgY2FuIHRpbWVzdGFtcCBpdCBhZ2FpbiBhbmQgY2FsY3VsYXRlIHRoZSBsYXRlbmN5LjxvOnA+ PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4i PlRoZXJlIGFyZSBzZXZlcmFsIGlzc3VlcyBoZXJlIHRoYXQgbmVlZCB0byBiZSBjb25zaWRlcmVk LjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0 Oi41aW4iPldlIG5lZWQgdG8gZ2V0IHRoZSBsYXRlbmN5IGluZm9ybWF0aW9uIGJhY2sgdXAgdG8g dGhlIGJkZXYgbGF5ZXIsIG1vc3QgbGlrZWx5IHRocm91Z2ggYSBjYWxsYmFjayBhcmd1bWVudCwg YnV0IHRoaXMgd291bGQgcmVxdWlyZSBhIGNoYW5nZSB0aGUgTlZNZSBBUEkuIElmIHRoZSBiZGV2 IGJyZWFrcyBkb3duIGEgcmVxdWVzdCBpbnRvIHNtYWxsZXIgSU9zLCBpdCBjYW4NCiBhZGQgdXAg dGhlIGxhdGVuY2llcyBvZiB0aGUgY2hpbGQgSU9zIGZvciBpb19zdGF0IGFuZCBoaXN0b2dyYW0g cHVycG9zZXMuPG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFy Z2luLWxlZnQ6LjVpbiI+QWxzbywgdGhpcyBtZXRob2QgZG9lcyBub3QgdGFrZSBpbnRvIGFjY291 bnQgYW55IGxhdGVuY3kgYWRkZWQgYnkgdGhlIHNwZGsgYmRldi9udm1lIGxheWVycyAoZXhjZXB0 IHRoZSBwb2xsZXIuKSBJZiBhIHJlcXVlc3Qgd2FzIHF1ZXVlZCBiZWZvcmUgYmVpbmcgcGxhY2Vk IG9uIHRoZSB0cmFja2VyLCB0aGVuIHRoZSB0aW1lIGl0IHdhcyBxdWV1ZWQgaXMgbm90IGZhY3Rv cmVkDQogaW50byB0aGUgbGF0ZW5jeSBjYWxjdWxhdGlvbi48bzpwPjwvbzpwPjwvcD4NCjxwIGNs YXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj4mbmJzcDs8bzpwPjwvbzpw PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj5PcHRp b24gMjo8bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4t bGVmdDouNWluIj5NZWFzdXJlIGxhdGVuY3kgYXQgdGhlIGJkZXYgbGF5ZXIuPG86cD48L286cD48 L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+V2UgY2Fu IHRpbWVzdGFtcCBhdCBzdWJtaXQgYW5kIGFnYWluIGF0IGNvbXBsZXRpb24uIFRoaXMgd291bGQg a2VlcCBhbGwgaW9fc3RhdCBpbmZvcm1hdGlvbiBsb2NhbCB0byBiZGV2IGFuZCB3b3VsZCB0YWtl IGludG8gYWNjb3VudCB0aGUgb3ZlcmhlYWQgb2YgbW9zdCBxdWV1ZWQgb3BlcmF0aW9ucy4gQW55 IGFwcGxpY2F0aW9ucyB3cml0dGVuIGRpcmVjdGx5IHRvDQogdGhlIE5WTWUgbGF5ZXIgd291bGQg aGF2ZSB0byBjYWxjdWxhdGUgdGhlaXIgb3duIGxhdGVuY2llcywgYnV0IHRoYXQgaXMgY3VycmVu dGx5IHRydWUgZm9yIGFsbCBpb19zdGF0cy48bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29O b3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj4mbmJzcDs8bzpwPjwvbzpwPjwvcD4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj5J4oCZbSBzdXJlIHRo YXQgdGhlcmUgYXJlIG90aGVyIGlzc3VlcyBJIGFtIG1pc3NpbmcsIGJ1dCBJIHdvdWxkIGFwcHJl Y2lhdGUgYW55IGNvbW1lbnRzIG9uIGhvdyBiZXN0IHRvIG1vdmUgZm9yd2FyZCBvbiB0aGlzLjxv OnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41 aW4iPiZuYnNwOzxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1h cmdpbi1sZWZ0Oi41aW4iPlRoYW5rcyw8bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3Jt YWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj5QYXVsPG86cD48L286cD48L3A+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+Jm5ic3A7PG86cD48L286cD48 L3A+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K --===============3863136022233690908==--