From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============1521765728458107166==" MIME-Version: 1.0 From: Walker, Benjamin Subject: =?utf-8?q?Re=3A_=5BSPDK=5D_=E7=AD=94=E5=A4=8D=3A__SPDK_fio_benchmark_supp?= =?utf-8?q?ort?= Date: Thu, 16 Jun 2016 15:40:33 +0000 Message-ID: <1466091633.26925.41.camel@intel.com> In-Reply-To: PS1PR0401MB14994EACFB295CCE7091E0C9AA560@PS1PR0401MB1499.apcprd04.prod.outlook.com List-ID: To: spdk@lists.01.org --===============1521765728458107166== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable The perf tool allocates 1 queue pair per thread (worker_fn is the thread en= try point in the code) and there is 1 thread per core. Our findings on all = of the devices we've tested has been that using multiple queue pairs does n= ot increase the total performance of the SSD. The full quoted SSD hardware = performance can be achieved with a single queue pair. This finding, however= , is completely device specific. Different SSD controllers may have differe= nt behavior. Using multiple queue pairs is a big advantage for the host software (i.e. S= PDK) though. Using 1 queue pair per thread allows the host software to subm= it I/O in parallel with no locks. On Thu, 2016-06-16 at 14:24 +0000, Li Tianxiang wrote: Hi=EF=BC=8C I noticed that the perf app. only allocates one qpair. using : "ns_ctx->u.nvme.qpair =3D spdk_nvme_ctrlr_alloc_io_qpair(ns_ctx->entry->u.n= vme.ctrlr, 0);" Does more qpair help to the performance? And P3700 has about 30 submission queues default. ________________________________________ =E5=8F=91=E4=BB=B6=E4=BA=BA: SPDK > =E4=BB=A3=E8=A1=A8 Walker, Benjamin > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2016=E5=B9=B46=E6=9C=8815=E6=97=A5 23= :29 =E6=94=B6=E4=BB=B6=E4=BA=BA: Storage Performance Development Kit; sbradshaw= (a)micron.com; Robert.Cleveland(a)skhms.com<= mailto:Robert.Cleveland(a)skhms.com> =E4=B8=BB=E9=A2=98: Re: [SPDK] SPDK fio benchmark support The perf application should scale linearly with CPU cores - so if you get 1= .6 million IOPS with 2 SSDs on 1 core, you should get 3.2 million IOPS with= 4 SSDs on 2 cores. Since you aren't, there must be something else impactin= g the performance. Can you provide the following information: 1) The Xeon E5 SKU you are using 2) The number of CPU sockets and which socket the devices are attached to (= all 4 need to be attached to the same socket that you are running perf on). 3) The command line parameters you are passing to the perf tool. Thanks, Ben -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinikan= th) Pandurangan Sent: Wednesday, June 15, 2016 3:16 PM To: Storage Performance Development Kit >; sbradshaw(a)micron.com; Robe= rt.Cleveland(a)skhms.com Subject: Re: [SPDK] SPDK fio benchmark support Hello Ben, Thanks for the details. I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.6 = million) on 1 core. IOPS are same even when I had 4 SSDs on 1 core. But = I expect the IOPS to go up when I introduce extra cores with 4 SSDs and it = did not. Thanks, -Raj P -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjam= in Sent: Wednesday, June 15, 2016 1:33 PM To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Clev= eland(a)skhms.com Subject: Re: [SPDK] SPDK fio benchmark support The fio plugin for SPDK is new and was only tested for a single job. We'd w= elcome patches to make it span more cores when given more jobs if someone i= s interested in doing the work here. At some point, I'm sure we'll get to i= t ourselves, but it's lower priority than the other work we're doing right = now (NVMf target). The perf example we provide definitely does scale to man= y cores though, and it can optionally use libaio as its backend, so it is a= great alternative to using fio that works today with no extra effort. As far as expected performance scaling across cores with our perf tool, are= you seeing the maximum quoted IOPS for all of the device(s) on the system = with only a single core? If you are, adding more cores won't improve perfor= mance. If you are not getting the performance you expect from the hardware = I'd love to know the details of the devices and the platform. To provide a quick example from my own system (and none of the following sh= ould be treated as official benchmarking numbers for any reason), I have a = Haswell Xeon E7 with 8 Intel P3700 NVMe SSDs attached. Running the perf too= l on my system yields about 3 million 4k read IOPs on 1 core, which is the = equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hard= ware spec IOPS for 8 SSDs. If I only test against 4 SSDs I get the same per= formance (the maximum that the SSDs can provide) no matter how many cores I= give the tool. Thanks, Ben -----Original Message----- From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com] Sent: Wednesday, June 15, 2016 11:52 AM To: Walker, Benjamin >; sbradshaw(a)micron.com; spdk(a)= lists.01.org; Robert.Cleveland(a)skhms.com Cc: Raj (Rajinikanth) Pandurangan > Subject: RE: [SPDK] SPDK fio benchmark support Hello All, 1. I tried to run SPDK fio with multiple nvme drives. But for some reason,= I don't see more than 1 core being used even when I increased numjobs? Here is my config file. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D [global] ioengine=3D/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin group_reporting=3D1 direct=3D1 verify=3D0 time_based=3D1 ramp_time=3D0 runtime=3D10 [test1] iodepth=3D128 rw=3Drandread bs=3D4k #filename=3D0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1 filename=3D0000.09.00.0/1 numjobs=3D2 [test2] iodepth=3D128 rw=3Drandread bs=3D4k numjobs=3D1 filename=3D0000.08.00.0/1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Has anyone succeeded using multiple cores? 2. I have also tried with 'perf' and I noticed that even though multiple co= res were being used but performance numbers didn't scale. Seemed like simi= lar IOPS when I used single core. Any insights would really be appreciated. Thanks, -Raj P -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjam= in Sent: Friday, February 26, 2016 2:30 PM To: sbradshaw(a)micron.com; spdk(a)ml01.01.o= rg; Robert.Cleveland(a)skhms.com Subject: Re: [SPDK] SPDK fio benchmark support As a follow up to this, I pushed some improvements to SPDK's setup scripts = so that we now more easily support vfio. That makes it a lot easier to swit= ch back and forth between the unvme driver and the SPDK driver. We'd happil= y accept more patches to improve vfio if anyone has suggestions. Thanks all. On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote: Hi Robert, We built a fio ioengine plugin for benchmarking SPDK as well as our own use= rspace NVMe driver. Source and documentation are available here: https://= github.com/MicronSSD/unvme (SPDK fio plugin is ioengine/spdk_fio.c) If you have any questions on how to use it or interpret benchmark results, = feel free to ask. -Sam From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland Sent: Thursday, February 18, 2016 10:24 AM To: spdk(a)lists.01.org Subject: [SPDK] SPDK fio benchmark support Hello all, Has anyone done any work to make it easy to benchmark polling mode driver w= ith something like FIO? Thanks, Robert Cleveland The information contained in this e-mail is considered confidential of SK h= ynix memory solutions Inc. and intended only for the persons addressed or c= opied in this e-mail. Any unauthorized use, dissemination of the informatio= n, or copying of this message is strictly prohibited. If you are not the in= tended recipient, please contact the sender immediately and permanently del= ete the original and any copies of this email. _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk --===============1521765728458107166== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjwvaGVhZD4NCjxib2R5Pg0KPGRpdj5UaGUgcGVy ZiB0b29sIGFsbG9jYXRlcyAxIHF1ZXVlIHBhaXIgcGVyIHRocmVhZCAod29ya2VyX2ZuIGlzIHRo ZSB0aHJlYWQgZW50cnkgcG9pbnQgaW4gdGhlIGNvZGUpIGFuZCB0aGVyZSBpcyAxIHRocmVhZCBw ZXIgY29yZS4gT3VyIGZpbmRpbmdzIG9uIGFsbCBvZiB0aGUgZGV2aWNlcyB3ZSd2ZSB0ZXN0ZWQg aGFzIGJlZW4gdGhhdCB1c2luZyBtdWx0aXBsZSBxdWV1ZSBwYWlycyBkb2VzIG5vdCBpbmNyZWFz ZSB0aGUgdG90YWwgcGVyZm9ybWFuY2UNCiBvZiB0aGUgU1NELiBUaGUgZnVsbCBxdW90ZWQgU1NE IGhhcmR3YXJlIHBlcmZvcm1hbmNlIGNhbiBiZSBhY2hpZXZlZCB3aXRoIGEgc2luZ2xlIHF1ZXVl IHBhaXIuIFRoaXMgZmluZGluZywgaG93ZXZlciwgaXMgY29tcGxldGVseSBkZXZpY2Ugc3BlY2lm aWMuIERpZmZlcmVudCBTU0QgY29udHJvbGxlcnMgbWF5IGhhdmUgZGlmZmVyZW50IGJlaGF2aW9y LjwvZGl2Pg0KPGRpdj48YnI+DQo8L2Rpdj4NCjxkaXY+VXNpbmcgbXVsdGlwbGUgcXVldWUgcGFp cnMgaXMgYSBiaWcgYWR2YW50YWdlIGZvciB0aGUgaG9zdCBzb2Z0d2FyZSAoaS5lLiBTUERLKSB0 aG91Z2guIFVzaW5nIDEgcXVldWUgcGFpciBwZXIgdGhyZWFkIGFsbG93cyB0aGUgaG9zdCBzb2Z0 d2FyZSB0byBzdWJtaXQgSS9PIGluIHBhcmFsbGVsIHdpdGggbm8gbG9ja3MuPC9kaXY+DQo8ZGl2 Pjxicj4NCjwvZGl2Pg0KPGRpdj5PbiBUaHUsIDIwMTYtMDYtMTYgYXQgMTQ6MjQgJiM0MzswMDAw LCBMaSBUaWFueGlhbmcgd3JvdGU6PC9kaXY+DQo8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4NCjxw cmU+SGnvvIwNCg0KSSBub3RpY2VkIHRoYXQgdGhlIHBlcmYgIGFwcC4gIG9ubHkgIGFsbG9jYXRl cyBvbmUgcXBhaXIuDQoNCnVzaW5nIDogDQomcXVvdDtuc19jdHgtJmd0O3UubnZtZS5xcGFpciA9 IHNwZGtfbnZtZV9jdHJscl9hbGxvY19pb19xcGFpcihuc19jdHgtJmd0O2VudHJ5LSZndDt1Lm52 bWUuY3RybHIsIDApOyZxdW90Ow0KDQpEb2VzIG1vcmUgIHFwYWlyIGhlbHAgdG8gdGhlICBwZXJm b3JtYW5jZT8NCg0KQW5kIFAzNzAwICBoYXMgYWJvdXQgMzAgc3VibWlzc2lvbiAgcXVldWVzIGRl ZmF1bHQuDQoNCg0KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0K5Y+R 5Lu25Lq6OiBTUERLICZsdDs8YSBocmVmPSJtYWlsdG86c3Bkay1ib3VuY2VzQGxpc3RzLjAxLm9y ZyI+c3Bkay1ib3VuY2VzQGxpc3RzLjAxLm9yZzwvYT4mZ3Q7IOS7o+ihqCBXYWxrZXIsIEJlbmph bWluICZsdDs8YSBocmVmPSJtYWlsdG86YmVuamFtaW4ud2Fsa2VyQGludGVsLmNvbSI+YmVuamFt aW4ud2Fsa2VyQGludGVsLmNvbTwvYT4mZ3Q7DQrlj5HpgIHml7bpl7Q6IDIwMTblubQ25pyIMTXm l6UgMjM6MjkNCuaUtuS7tuS6ujogU3RvcmFnZSBQZXJmb3JtYW5jZSBEZXZlbG9wbWVudCBLaXQ7 IDxhIGhyZWY9Im1haWx0bzpzYnJhZHNoYXdAbWljcm9uLmNvbSI+c2JyYWRzaGF3QG1pY3Jvbi5j b208L2E+OyA8YSBocmVmPSJtYWlsdG86Um9iZXJ0LkNsZXZlbGFuZEBza2htcy5jb20iPlJvYmVy dC5DbGV2ZWxhbmRAc2tobXMuY29tPC9hPg0K5Li76aKYOiBSZTogW1NQREtdIFNQREsgZmlvIGJl bmNobWFyayBzdXBwb3J0DQoNClRoZSBwZXJmIGFwcGxpY2F0aW9uIHNob3VsZCBzY2FsZSBsaW5l YXJseSB3aXRoIENQVSBjb3JlcyAtIHNvIGlmIHlvdSBnZXQgMS42IG1pbGxpb24gSU9QUyB3aXRo IDIgU1NEcyBvbiAxIGNvcmUsIHlvdSBzaG91bGQgZ2V0IDMuMiBtaWxsaW9uIElPUFMgd2l0aCA0 IFNTRHMgb24gMiBjb3Jlcy4gU2luY2UgeW91IGFyZW4ndCwgdGhlcmUgbXVzdCBiZSBzb21ldGhp bmcgZWxzZSBpbXBhY3RpbmcgdGhlIHBlcmZvcm1hbmNlLiBDYW4geW91IHByb3ZpZGUgdGhlIGZv bGxvd2luZyBpbmZvcm1hdGlvbjoNCg0KMSkgVGhlIFhlb24gRTUgU0tVIHlvdSBhcmUgdXNpbmcN CjIpIFRoZSBudW1iZXIgb2YgQ1BVIHNvY2tldHMgYW5kIHdoaWNoIHNvY2tldCB0aGUgZGV2aWNl cyBhcmUgYXR0YWNoZWQgdG8gKGFsbCA0IG5lZWQgdG8gYmUgYXR0YWNoZWQgdG8gdGhlIHNhbWUg c29ja2V0IHRoYXQgeW91IGFyZSBydW5uaW5nIHBlcmYgb24pLg0KMykgVGhlIGNvbW1hbmQgbGlu ZSBwYXJhbWV0ZXJzIHlvdSBhcmUgcGFzc2luZyB0byB0aGUgcGVyZiB0b29sLg0KDQpUaGFua3Ms DQpCZW4NCg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IFNQREsgWzxhIGhyZWY9 Im1haWx0bzpzcGRrLWJvdW5jZXNAbGlzdHMuMDEub3JnIj5tYWlsdG86c3Bkay1ib3VuY2VzQGxp c3RzLjAxLm9yZzwvYT5dIE9uIEJlaGFsZiBPZiBSYWogKFJhamluaWthbnRoKSBQYW5kdXJhbmdh bg0KU2VudDogV2VkbmVzZGF5LCBKdW5lIDE1LCAyMDE2IDM6MTYgUE0NClRvOiBTdG9yYWdlIFBl cmZvcm1hbmNlIERldmVsb3BtZW50IEtpdCAmbHQ7PGEgaHJlZj0ibWFpbHRvOnNwZGtAbGlzdHMu MDEub3JnIj5zcGRrQGxpc3RzLjAxLm9yZzwvYT4mZ3Q7OyA8YSBocmVmPSJtYWlsdG86c2JyYWRz aGF3QG1pY3Jvbi5jb20iPnNicmFkc2hhd0BtaWNyb24uY29tPC9hPjsgPGEgaHJlZj0ibWFpbHRv OlJvYmVydC5DbGV2ZWxhbmRAc2tobXMuY29tIj5Sb2JlcnQuQ2xldmVsYW5kQHNraG1zLmNvbTwv YT4NClN1YmplY3Q6IFJlOiBbU1BES10gU1BESyBmaW8gYmVuY2htYXJrIHN1cHBvcnQNCg0KSGVs bG8gQmVuLA0KDQpUaGFua3MgZm9yIHRoZSBkZXRhaWxzLg0KDQpJIGhhdmUgSW50ZWwgWGVvbiBF NSBzeXN0ZW0gYW5kIGFibGUgdG8gc2F0dXJhdGUgMiBTYW1zdW5nIE5WTWUgU1NEcyAofjEuNiBt aWxsaW9uKSBvbiAxIGNvcmUuICAgSU9QUyBhcmUgc2FtZSBldmVuIHdoZW4gSSBoYWQgNCBTU0Rz IG9uIDEgY29yZS4gIEJ1dCBJIGV4cGVjdCB0aGUgSU9QUyB0byBnbyB1cCB3aGVuIEkgaW50cm9k dWNlIGV4dHJhIGNvcmVzIHdpdGggNCBTU0RzIGFuZCBpdCBkaWQgbm90Lg0KDQpUaGFua3MsDQot UmFqIFANCg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IFNQREsgWzxhIGhyZWY9 Im1haWx0bzpzcGRrLWJvdW5jZXNAbGlzdHMuMDEub3JnIj5tYWlsdG86c3Bkay1ib3VuY2VzQGxp c3RzLjAxLm9yZzwvYT5dIE9uIEJlaGFsZiBPZiBXYWxrZXIsIEJlbmphbWluDQpTZW50OiBXZWRu ZXNkYXksIEp1bmUgMTUsIDIwMTYgMTozMyBQTQ0KVG86IFJhaiAoUmFqaW5pa2FudGgpIFBhbmR1 cmFuZ2FuOyA8YSBocmVmPSJtYWlsdG86c2JyYWRzaGF3QG1pY3Jvbi5jb20iPnNicmFkc2hhd0Bt aWNyb24uY29tPC9hPjsgPGEgaHJlZj0ibWFpbHRvOnNwZGtAbGlzdHMuMDEub3JnIj5zcGRrQGxp c3RzLjAxLm9yZzwvYT47IDxhIGhyZWY9Im1haWx0bzpSb2JlcnQuQ2xldmVsYW5kQHNraG1zLmNv bSI+Um9iZXJ0LkNsZXZlbGFuZEBza2htcy5jb208L2E+DQpTdWJqZWN0OiBSZTogW1NQREtdIFNQ REsgZmlvIGJlbmNobWFyayBzdXBwb3J0DQoNClRoZSBmaW8gcGx1Z2luIGZvciBTUERLIGlzIG5l dyBhbmQgd2FzIG9ubHkgdGVzdGVkIGZvciBhIHNpbmdsZSBqb2IuIFdlJ2Qgd2VsY29tZSBwYXRj aGVzIHRvIG1ha2UgaXQgc3BhbiBtb3JlIGNvcmVzIHdoZW4gZ2l2ZW4gbW9yZSBqb2JzIGlmIHNv bWVvbmUgaXMgaW50ZXJlc3RlZCBpbiBkb2luZyB0aGUgd29yayBoZXJlLiBBdCBzb21lIHBvaW50 LCBJJ20gc3VyZSB3ZSdsbCBnZXQgdG8gaXQgb3Vyc2VsdmVzLCBidXQgaXQncyBsb3dlciBwcmlv cml0eSB0aGFuIHRoZSBvdGhlciB3b3JrIHdlJ3JlIGRvaW5nIHJpZ2h0IG5vdyAoTlZNZiB0YXJn ZXQpLiBUaGUgcGVyZiBleGFtcGxlIHdlIHByb3ZpZGUgZGVmaW5pdGVseSBkb2VzIHNjYWxlIHRv IG1hbnkgY29yZXMgdGhvdWdoLCBhbmQgaXQgY2FuIG9wdGlvbmFsbHkgdXNlIGxpYmFpbyBhcyBp dHMgYmFja2VuZCwgc28gaXQgaXMgYSBncmVhdCBhbHRlcm5hdGl2ZSB0byB1c2luZyBmaW8gdGhh dCB3b3JrcyB0b2RheSB3aXRoIG5vIGV4dHJhIGVmZm9ydC4NCg0KQXMgZmFyIGFzIGV4cGVjdGVk IHBlcmZvcm1hbmNlIHNjYWxpbmcgYWNyb3NzIGNvcmVzIHdpdGggb3VyIHBlcmYgdG9vbCwgYXJl IHlvdSBzZWVpbmcgdGhlIG1heGltdW0gcXVvdGVkIElPUFMgZm9yIGFsbCBvZiB0aGUgZGV2aWNl KHMpIG9uIHRoZSBzeXN0ZW0gd2l0aCBvbmx5IGEgc2luZ2xlIGNvcmU/IElmIHlvdSBhcmUsIGFk ZGluZyBtb3JlIGNvcmVzIHdvbid0IGltcHJvdmUgcGVyZm9ybWFuY2UuIElmIHlvdSBhcmUgbm90 IGdldHRpbmcgdGhlIHBlcmZvcm1hbmNlIHlvdSBleHBlY3QgZnJvbSB0aGUgaGFyZHdhcmUgSSdk IGxvdmUgdG8ga25vdyB0aGUgZGV0YWlscyBvZiB0aGUgZGV2aWNlcyBhbmQgdGhlIHBsYXRmb3Jt Lg0KDQpUbyBwcm92aWRlIGEgcXVpY2sgZXhhbXBsZSBmcm9tIG15IG93biBzeXN0ZW0gKGFuZCBu b25lIG9mIHRoZSBmb2xsb3dpbmcgc2hvdWxkIGJlIHRyZWF0ZWQgYXMgb2ZmaWNpYWwgYmVuY2ht YXJraW5nIG51bWJlcnMgZm9yIGFueSByZWFzb24pLCBJIGhhdmUgYSBIYXN3ZWxsIFhlb24gRTcg d2l0aCA4IEludGVsIFAzNzAwIE5WTWUgU1NEcyBhdHRhY2hlZC4gUnVubmluZyB0aGUgcGVyZiB0 b29sIG9uIG15IHN5c3RlbSB5aWVsZHMgYWJvdXQgMyBtaWxsaW9uIDRrIHJlYWQgSU9QcyBvbiAx IGNvcmUsIHdoaWNoIGlzIHRoZSBlcXVpdmFsZW50IG9mIDYgb3IgNyBvZiBteSBTU0RzLiBNb3Zp bmcgdG8gdHdvIGNvcmVzIGdpdmVzIG1lIHRoZSBmdWxsIGhhcmR3YXJlIHNwZWMgSU9QUyBmb3Ig OCBTU0RzLiBJZiBJIG9ubHkgdGVzdCBhZ2FpbnN0IDQgU1NEcyBJIGdldCB0aGUgc2FtZSBwZXJm b3JtYW5jZSAodGhlIG1heGltdW0gdGhhdCB0aGUgU1NEcyBjYW4gcHJvdmlkZSkgbm8gbWF0dGVy IGhvdyBtYW55IGNvcmVzIEkgZ2l2ZSB0aGUgdG9vbC4NCg0KVGhhbmtzLA0KQmVuDQoNCi0tLS0t T3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBSYWogKFJhamluaWthbnRoKSBQYW5kdXJhbmdh biBbPGEgaHJlZj0ibWFpbHRvOnJhamluaS5wYW5kdUBzYW1zdW5nLmNvbSI+bWFpbHRvOnJhamlu aS5wYW5kdUBzYW1zdW5nLmNvbTwvYT5dDQpTZW50OiBXZWRuZXNkYXksIEp1bmUgMTUsIDIwMTYg MTE6NTIgQU0NClRvOiBXYWxrZXIsIEJlbmphbWluICZsdDs8YSBocmVmPSJtYWlsdG86YmVuamFt aW4ud2Fsa2VyQGludGVsLmNvbSI+YmVuamFtaW4ud2Fsa2VyQGludGVsLmNvbTwvYT4mZ3Q7OyA8 YSBocmVmPSJtYWlsdG86c2JyYWRzaGF3QG1pY3Jvbi5jb20iPnNicmFkc2hhd0BtaWNyb24uY29t PC9hPjsgPGEgaHJlZj0ibWFpbHRvOnNwZGtAbGlzdHMuMDEub3JnIj5zcGRrQGxpc3RzLjAxLm9y ZzwvYT47IDxhIGhyZWY9Im1haWx0bzpSb2JlcnQuQ2xldmVsYW5kQHNraG1zLmNvbSI+Um9iZXJ0 LkNsZXZlbGFuZEBza2htcy5jb208L2E+DQpDYzogUmFqIChSYWppbmlrYW50aCkgUGFuZHVyYW5n YW4gJmx0OzxhIGhyZWY9Im1haWx0bzpyYWppbmkucGFuZHVAc2Ftc3VuZy5jb20iPnJhamluaS5w YW5kdUBzYW1zdW5nLmNvbTwvYT4mZ3Q7DQpTdWJqZWN0OiBSRTogW1NQREtdIFNQREsgZmlvIGJl bmNobWFyayBzdXBwb3J0DQoNCkhlbGxvIEFsbCwNCg0KMS4gSSB0cmllZCB0byBydW4gU1BESyBm aW8gd2l0aCBtdWx0aXBsZSBudm1lIGRyaXZlcy4gIEJ1dCBmb3Igc29tZSByZWFzb24sIEkgZG9u J3Qgc2VlIG1vcmUgdGhhbiAxIGNvcmUgYmVpbmcgdXNlZCBldmVuIHdoZW4gSSBpbmNyZWFzZWQg bnVtam9icz8NCg0KSGVyZSBpcyBteSBjb25maWcgZmlsZS4NCj09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0NCg0KW2dsb2JhbF0NCmlv ZW5naW5lPS9zcGRrLW1hc3Rlcl90ZW1wX2V4cGVyaW1lbnQvZXhhbXBsZXMvbnZtZS9maW9fcGx1 Z2luL2Zpb19wbHVnaW4NCmdyb3VwX3JlcG9ydGluZz0xDQpkaXJlY3Q9MQ0KdmVyaWZ5PTANCnRp bWVfYmFzZWQ9MQ0KcmFtcF90aW1lPTANCnJ1bnRpbWU9MTANCg0KDQpbdGVzdDFdDQppb2RlcHRo PTEyOA0Kcnc9cmFuZHJlYWQNCmJzPTRrDQojZmlsZW5hbWU9MDAwMC4wOS4wMC4wLzE6MDAwMC4w Ny4wMC4wLzE6MDAwMC4wYS4wMC4wLzENCmZpbGVuYW1lPTAwMDAuMDkuMDAuMC8xDQpudW1qb2Jz PTINCg0KW3Rlc3QyXQ0KaW9kZXB0aD0xMjgNCnJ3PXJhbmRyZWFkDQpicz00aw0KbnVtam9icz0x DQpmaWxlbmFtZT0wMDAwLjA4LjAwLjAvMQ0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KDQpIYXMgYW55b25lIHN1Y2NlZWRl ZCB1c2luZyBtdWx0aXBsZSBjb3Jlcz8NCg0KMi4gSSBoYXZlIGFsc28gdHJpZWQgd2l0aCAncGVy ZicgYW5kIEkgbm90aWNlZCB0aGF0IGV2ZW4gdGhvdWdoIG11bHRpcGxlIGNvcmVzIHdlcmUgYmVp bmcgdXNlZCBidXQgcGVyZm9ybWFuY2UgbnVtYmVycyBkaWRuJ3Qgc2NhbGUuICBTZWVtZWQgbGlr ZSBzaW1pbGFyIElPUFMgd2hlbiBJIHVzZWQgc2luZ2xlIGNvcmUuDQoNCkFueSBpbnNpZ2h0cyB3 b3VsZCByZWFsbHkgYmUgYXBwcmVjaWF0ZWQuDQoNClRoYW5rcywNCi1SYWogUA0KDQotLS0tLU9y aWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogU1BESyBbPGEgaHJlZj0ibWFpbHRvOnNwZGstYm91 bmNlc0BsaXN0cy4wMS5vcmciPm1haWx0bzpzcGRrLWJvdW5jZXNAbGlzdHMuMDEub3JnPC9hPl0g T24gQmVoYWxmIE9mIFdhbGtlciwgQmVuamFtaW4NClNlbnQ6IEZyaWRheSwgRmVicnVhcnkgMjYs IDIwMTYgMjozMCBQTQ0KVG86IDxhIGhyZWY9Im1haWx0bzpzYnJhZHNoYXdAbWljcm9uLmNvbSI+ c2JyYWRzaGF3QG1pY3Jvbi5jb208L2E+OyA8YSBocmVmPSJtYWlsdG86c3Bka0BtbDAxLjAxLm9y ZyI+c3Bka0BtbDAxLjAxLm9yZzwvYT47IDxhIGhyZWY9Im1haWx0bzpSb2JlcnQuQ2xldmVsYW5k QHNraG1zLmNvbSI+Um9iZXJ0LkNsZXZlbGFuZEBza2htcy5jb208L2E+DQpTdWJqZWN0OiBSZTog W1NQREtdIFNQREsgZmlvIGJlbmNobWFyayBzdXBwb3J0DQoNCkFzIGEgZm9sbG93IHVwIHRvIHRo aXMsIEkgcHVzaGVkIHNvbWUgaW1wcm92ZW1lbnRzIHRvIFNQREsncyBzZXR1cCBzY3JpcHRzIHNv IHRoYXQgd2Ugbm93IG1vcmUgZWFzaWx5IHN1cHBvcnQgdmZpby4gVGhhdCBtYWtlcyBpdCBhIGxv dCBlYXNpZXIgdG8gc3dpdGNoIGJhY2sgYW5kIGZvcnRoIGJldHdlZW4gdGhlIHVudm1lIGRyaXZl ciBhbmQgdGhlIFNQREsgZHJpdmVyLiBXZSdkIGhhcHBpbHkgYWNjZXB0IG1vcmUgcGF0Y2hlcyB0 byBpbXByb3ZlIHZmaW8gaWYgYW55b25lIGhhcyBzdWdnZXN0aW9ucy4NCg0KVGhhbmtzIGFsbC4N Cg0KT24gVGh1LCAyMDE2LTAyLTE4IGF0IDE4OjU5ICYjNDM7MDAwMCwgU2FtIEJyYWRzaGF3IChz YnJhZHNoYXcpIHdyb3RlOg0KSGkgUm9iZXJ0LA0KDQpXZSBidWlsdCBhIGZpbyBpb2VuZ2luZSBw bHVnaW4gZm9yIGJlbmNobWFya2luZyBTUERLIGFzIHdlbGwgYXMgb3VyIG93biB1c2Vyc3BhY2Ug TlZNZSBkcml2ZXIuICBTb3VyY2UgYW5kIGRvY3VtZW50YXRpb24gYXJlIGF2YWlsYWJsZSBoZXJl OiAgPGEgaHJlZj0iaHR0cHM6Ly9naXRodWIuY29tL01pY3JvblNTRC91bnZtZSI+aHR0cHM6Ly9n aXRodWIuY29tL01pY3JvblNTRC91bnZtZTwvYT4NCg0KKFNQREsgZmlvIHBsdWdpbiBpcyBpb2Vu Z2luZS9zcGRrX2Zpby5jKQ0KDQpJZiB5b3UgaGF2ZSBhbnkgcXVlc3Rpb25zIG9uIGhvdyB0byB1 c2UgaXQgb3IgaW50ZXJwcmV0IGJlbmNobWFyayByZXN1bHRzLCBmZWVsIGZyZWUgdG8gYXNrLg0K DQotU2FtDQoNCkZyb206IFNQREsgWzxhIGhyZWY9Im1haWx0bzpzcGRrLWJvdW5jZXNAbWwwMS4w MS5vcmciPm1haWx0bzpzcGRrLWJvdW5jZXNAbWwwMS4wMS5vcmc8L2E+XSBPbiBCZWhhbGYgT2Yg Um9iZXJ0IENsZXZlbGFuZA0KU2VudDogVGh1cnNkYXksIEZlYnJ1YXJ5IDE4LCAyMDE2IDEwOjI0 IEFNDQpUbzogPGEgaHJlZj0ibWFpbHRvOnNwZGtAbGlzdHMuMDEub3JnIj5zcGRrQGxpc3RzLjAx Lm9yZzwvYT4NClN1YmplY3Q6IFtTUERLXSBTUERLIGZpbyBiZW5jaG1hcmsgc3VwcG9ydA0KDQpI ZWxsbyBhbGwsDQoNCkhhcyBhbnlvbmUgZG9uZSBhbnkgd29yayB0byBtYWtlIGl0IGVhc3kgdG8g YmVuY2htYXJrIHBvbGxpbmcgbW9kZSBkcml2ZXIgd2l0aCBzb21ldGhpbmcgbGlrZSBGSU8/DQoN ClRoYW5rcywNClJvYmVydCBDbGV2ZWxhbmQNClRoZSBpbmZvcm1hdGlvbiBjb250YWluZWQgaW4g dGhpcyBlLW1haWwgaXMgY29uc2lkZXJlZCBjb25maWRlbnRpYWwgb2YgU0sgaHluaXggbWVtb3J5 IHNvbHV0aW9ucyBJbmMuIGFuZCBpbnRlbmRlZCBvbmx5IGZvciB0aGUgcGVyc29ucyBhZGRyZXNz ZWQgb3IgY29waWVkIGluIHRoaXMgZS1tYWlsLiBBbnkgdW5hdXRob3JpemVkIHVzZSwgZGlzc2Vt aW5hdGlvbiBvZiB0aGUgaW5mb3JtYXRpb24sIG9yIGNvcHlpbmcgb2YgdGhpcyBtZXNzYWdlIGlz IHN0cmljdGx5IHByb2hpYml0ZWQuIElmIHlvdSBhcmUgbm90IHRoZSBpbnRlbmRlZCByZWNpcGll bnQsIHBsZWFzZSBjb250YWN0IHRoZSBzZW5kZXIgaW1tZWRpYXRlbHkgYW5kIHBlcm1hbmVudGx5 IGRlbGV0ZSB0aGUgb3JpZ2luYWwgYW5kIGFueSBjb3BpZXMgb2YgdGhpcyBlbWFpbC4NCg0KX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NClNQREsgbWFpbGlu ZyBsaXN0DQo8YSBocmVmPSJtYWlsdG86U1BES0BsaXN0cy4wMS5vcmciPlNQREtAbGlzdHMuMDEu b3JnPC9hPiZsdDs8YSBocmVmPSJtYWlsdG86U1BES0BsaXN0cy4wMS5vcmciPm1haWx0bzpTUERL QGxpc3RzLjAxLm9yZzwvYT4mZ3Q7DQo8YSBocmVmPSJodHRwczovL2xpc3RzLjAxLm9yZy9tYWls bWFuL2xpc3RpbmZvL3NwZGsiPmh0dHBzOi8vbGlzdHMuMDEub3JnL21haWxtYW4vbGlzdGluZm8v c3BkazwvYT4NCg0KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X18NClNQREsgbWFpbGluZyBsaXN0DQo8YSBocmVmPSJtYWlsdG86U1BES0BsaXN0cy4wMS5vcmci PlNQREtAbGlzdHMuMDEub3JnPC9hPg0KPGEgaHJlZj0iaHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFp bG1hbi9saXN0aW5mby9zcGRrIj5odHRwczovL2xpc3RzLjAxLm9yZy9tYWlsbWFuL2xpc3RpbmZv L3NwZGs8L2E+DQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f Xw0KU1BESyBtYWlsaW5nIGxpc3QNCjxhIGhyZWY9Im1haWx0bzpTUERLQGxpc3RzLjAxLm9yZyI+ U1BES0BsaXN0cy4wMS5vcmc8L2E+DQo8YSBocmVmPSJodHRwczovL2xpc3RzLjAxLm9yZy9tYWls bWFuL2xpc3RpbmZvL3NwZGsiPmh0dHBzOi8vbGlzdHMuMDEub3JnL21haWxtYW4vbGlzdGluZm8v c3BkazwvYT4NCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f DQpTUERLIG1haWxpbmcgbGlzdA0KPGEgaHJlZj0ibWFpbHRvOlNQREtAbGlzdHMuMDEub3JnIj5T UERLQGxpc3RzLjAxLm9yZzwvYT4NCjxhIGhyZWY9Imh0dHBzOi8vbGlzdHMuMDEub3JnL21haWxt YW4vbGlzdGluZm8vc3BkayI+aHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFpbG1hbi9saXN0aW5mby9z cGRrPC9hPg0KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18N ClNQREsgbWFpbGluZyBsaXN0DQo8YSBocmVmPSJtYWlsdG86U1BES0BsaXN0cy4wMS5vcmciPlNQ REtAbGlzdHMuMDEub3JnPC9hPg0KPGEgaHJlZj0iaHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFpbG1h bi9saXN0aW5mby9zcGRrIj5odHRwczovL2xpc3RzLjAxLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3Nw ZGs8L2E+DQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0K U1BESyBtYWlsaW5nIGxpc3QNCjxhIGhyZWY9Im1haWx0bzpTUERLQGxpc3RzLjAxLm9yZyI+U1BE S0BsaXN0cy4wMS5vcmc8L2E+DQo8YSBocmVmPSJodHRwczovL2xpc3RzLjAxLm9yZy9tYWlsbWFu L2xpc3RpbmZvL3NwZGsiPmh0dHBzOi8vbGlzdHMuMDEub3JnL21haWxtYW4vbGlzdGluZm8vc3Bk azwvYT4NCjwvcHJlPg0KPC9ibG9ja3F1b3RlPg0KPC9ib2R5Pg0KPC9odG1sPg0K --===============1521765728458107166==--