From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============4271903889175358297==" MIME-Version: 1.0 From: Li Wang Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth Date: Mon, 01 Jan 2018 21:47:58 +0800 Message-ID: <09391948-BE8D-4B9B-B9E5-1B44688F4DC4@gmail.com> In-Reply-To: F933FE54-2741-4D96-AB01-79C8BDF0B91B@gmail.com List-ID: To: spdk@lists.01.org --===============4271903889175358297== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi All, I run fio with kernel nvme. The IOPS with queue depth =3D 1 is 25K, two ord= ers of magnitude higher than with spdk. The results are as follows: $ fio --filename=3D/dev/nvme0n1 --name=3Drandwrite --ioengine=3Dlibaio --io= depth=3D1 --rw=3Drandread --bs=3D4k --direct=3D1 --numjobs=3D1 --runtime=3D= 10 =E2=80=94group_reporting read : io=3D1002.8MB, bw=3D102665KB/s, iops=3D25666, runt=3D 10001msec I write a test which create 1 thread that performs a large number of random= writes to nvme in a loop using spdk with queue depth =3D1. In each round o= f the loop, the program submits a random write command to the queue pairs a= nd then call repeatedly call spdk_nvme_qpair_process_completions() until th= e submitted IO is processed. I found that when the depth is 1, the delay be= tween the calling of spdk_nvme_ns_cmd_write()/spdk_nvme_ns_cmd_write() and = the calling of spdk_nvme_qpair_process_completions() has huge effects on th= e IOPS. More preciously, I found that if I call spdk_nvme_qpair_process_com= pletions() immediately after the submission of write command (via spdk_nvme= _ns_cmd_write()), the IOPS will be extremely low, e.g., ~1KB. However, if t= he program sleeps for several microseconds, a much higher IOPS, say 10KB, c= an be achieved. My feeling is that calling spdk_nvme_qpair_process_completi= ons() too soon and too often could severely increase the processing time of= an IO command. = I also found that "perf -q 1 -s 512 -w write -t 1 -c 3=E2=80=9D achieved ov= er 50K IOPS, but "perf -q 1 -s 512 -w write -t 1 -c 1=E2=80=9D achieved onl= y ~1K IOPS. Do you know the reason behind this wired performance? Also, I=E2=80=99d lik= e to know if the wired performance is highly related to EC2 instance. I wil= l appreciate if you can share the results of the same commands on a hardwar= e rather than EC2 i3 instance? Thank you all for your time and your feedback is highly appreciated. Sincerely, Li Wang > On Dec 31, 2017, at 12:56 PM, Li Wang wrote: > = > Dear Brijesh, > = > Thanks for your step by step guidance. I will let you know when I get the= results. > = > Regards, > Li Wang > = >> On Dec 30, 2017, at 10:47 PM, Brijesh Singh wrote: >> = >> Hi Li, >> = >> Try following steps: >> - Switch from SPDK to NVMe kernel module. >> #rmmod uio_pci_generic && modprobe nvme && echo 1 > /sys/bus/pci/rescan >> - Install fio >> # yum install fio >> - Run fio with queue depth 1 >> # fio --filename=3D/dev/nvme0n1 --name=3Drandread --ioengine=3Dlibaio >> --iodepth=3D1 --rw=3Drandread --bs=3D4k --direct=3D1 --numjobs=3D1 --run= time=3D10 >> --group_reporting >> = >> When all tests are done, run following command to switch back to SPDK st= ack. >> # rmmod nvme && modprobe uio_pci_generic && echo 1 > /sys/bus/pci/rescan >> = >> Regards, >> Brijesh >> = >> = >> On Sat, Dec 30, 2017 at 1:53 AM, Li Wang wrote: >>> Hi Brijesh, >>> = >>> Thanks for your suggestion. May I know how I can run fio with kernel nv= me >>> driver? >>> = >>> Thanks, >>> Li >>> = >>> On 30 December 2017 at 12:06, Brijesh Singh >>> wrote: >>>> = >>>> Hi Li, >>>> = >>>> Do you see similar results when you run fio with kernel nvme driver? >>>> = >>>> Regards, >>>> Brijesh >>>> = >>>> On Fri, Dec 29, 2017 at 7:39 PM, Li Wang wrot= e: >>>>> Dear Crane, >>>>> = >>>>> Following you advice, I run a variety of random-read and random-write >>>>> workloads, with -s ranging from 512 to 51200. The IOPS with depth =3D= 1 is >>>>> consistently around 1K under various workloads and settings. >>>>> = >>>>> Thanks, >>>>> Li >>>>> = >>>>> On 30 December 2017 at 11:32, Crane Chu wrote: >>>>>> = >>>>>> Hi, Li, >>>>>> = >>>>>> Maybe you can try 4K random read to simplify the test. 512byte seq-r= ead >>>>>> has too much possibilties and surprises in ssd firmware. :) >>>>>> = >>>>>> -Crane >>>>>> = >>>>>> 2017=E5=B9=B412=E6=9C=8830=E6=97=A5 =E4=B8=8A=E5=8D=889:57=EF=BC=8C"= Li Wang" =E5=86=99=E9=81=93=EF=BC=9A >>>>>>> = >>>>>>> Dear Benjamin, >>>>>>> = >>>>>>> Thanks for you help. >>>>>>> = >>>>>>> I just allocate a dedicated host and rerun the test, but I got the >>>>>>> similar results. By creating a instance on the dedicated host, I am >>>>>>> sure >>>>>>> that the NVMe device I connected is backed by a local NVMe device. >>>>>>> It's not >>>>>>> an EBS volume, although I don't whether the device is emulated. >>>>>>> = >>>>>>> Thanks, >>>>>>> Li >>>>>>> = >>>>>>> On 30 December 2017 at 04:42, Walker, Benjamin >>>>>>> wrote: >>>>>>>> = >>>>>>>> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote: >>>>>>>>> Dear all, >>>>>>>>> = >>>>>>>>> I am running nvme/perf with master branch on a EC2 instance >>>>>>>>> (Redhat, 8 >>>>>>>>> CPU >>>>>>>>> cores). I noticed that the IOPS is around 1K when the queue depth >>>>>>>>> is >>>>>>>>> 1. By >>>>>>>>> simply increase the queue length to 2, the IOPS grows to over 60K. >>>>>>>>> May >>>>>>>>> I ask >>>>>>>>> if this is normal? >>>>>>>> = >>>>>>>> Is this an EC2 dedicated server (i.e. physical hardware) or a virt= ual >>>>>>>> machine? >>>>>>>> Is the NVMe device you're connected to backed by a local NVMe devi= ce, >>>>>>>> a >>>>>>>> remote >>>>>>>> NVMe device, or an EBS volume? Amazon presents basically all stora= ge >>>>>>>> as >>>>>>>> NVMe >>>>>>>> devices these days, but the actual backing storage can perform >>>>>>>> differently >>>>>>>> depending on the circumstances. >>>>>>>> = >>>>>>>> In all cases I'm aware of, the NVMe device presented is not the >>>>>>>> actual >>>>>>>> NVMe SSD. >>>>>>>> It's either a virtual NVMe device emulated in software, or it's >>>>>>>> implemented on a >>>>>>>> special card that Amazon puts in between the platform and the real >>>>>>>> SSD. >>>>>>>> SPDK >>>>>>>> hasn't been tested thoroughly with Amazon's emulated NVMe devices,= so >>>>>>>> there >>>>>>>> could be some issues there. Most NVMe devices historically were not >>>>>>>> validated >>>>>>>> against polled mode, ultra high performance drivers so when SPDK w= as >>>>>>>> first >>>>>>>> released we found a number of firmware bugs in existing SSDs that >>>>>>>> were >>>>>>>> only >>>>>>>> exposed because SPDK was so fast. We're seeing less and less of th= at >>>>>>>> now, but >>>>>>>> you could be hitting something like that here. >>>>>>>> _______________________________________________ >>>>>>>> SPDK mailing list >>>>>>>> SPDK(a)lists.01.org >>>>>>>> https://lists.01.org/mailman/listinfo/spdk >>>>>>> = >>>>>>> = >>>>>>> = >>>>>>> _______________________________________________ >>>>>>> SPDK mailing list >>>>>>> SPDK(a)lists.01.org >>>>>>> https://lists.01.org/mailman/listinfo/spdk >>>>>>> = >>>>>> = >>>>>> _______________________________________________ >>>>>> SPDK mailing list >>>>>> SPDK(a)lists.01.org >>>>>> https://lists.01.org/mailman/listinfo/spdk >>>>>> = >>>>> = >>>>> = >>>>> _______________________________________________ >>>>> SPDK mailing list >>>>> SPDK(a)lists.01.org >>>>> https://lists.01.org/mailman/listinfo/spdk >>>>> = >>>> _______________________________________________ >>>> SPDK mailing list >>>> SPDK(a)lists.01.org >>>> https://lists.01.org/mailman/listinfo/spdk >>> = >>> = >>> = >>> _______________________________________________ >>> SPDK mailing list >>> SPDK(a)lists.01.org >>> https://lists.01.org/mailman/listinfo/spdk >>> = >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk > = --===============4271903889175358297== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWw+PGhlYWQ+PG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0 L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPjwvaGVhZD48Ym9keSBzdHlsZT0id29yZC13cmFwOiBicmVh ay13b3JkOyAtd2Via2l0LW5ic3AtbW9kZTogc3BhY2U7IGxpbmUtYnJlYWs6IGFmdGVyLXdoaXRl LXNwYWNlOyIgY2xhc3M9IiI+PGRpdiBjbGFzcz0iIj5IaSBBbGwsPC9kaXY+PGRpdiBjbGFzcz0i Ij48YnIgY2xhc3M9IiI+PC9kaXY+PGRpdiBjbGFzcz0iIj5JIHJ1biBmaW8gd2l0aCBrZXJuZWwg bnZtZS4gVGhlIElPUFMgd2l0aCBxdWV1ZSBkZXB0aCA9IDEgaXMgMjVLLCB0d28gb3JkZXJzIG9m IG1hZ25pdHVkZSBoaWdoZXIgdGhhbiB3aXRoIHNwZGsuIFRoZSByZXN1bHRzIGFyZSBhcyBmb2xs b3dzOjwvZGl2PjxkaXYgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPjwvZGl2PjxkaXYgY2xhc3M9IiI+ PGRpdiBzdHlsZT0ibWFyZ2luOiAwcHg7IGZvbnQtc3RyZXRjaDogbm9ybWFsOyBmb250LXNpemU6 IDExcHg7IGxpbmUtaGVpZ2h0OiBub3JtYWw7IGZvbnQtZmFtaWx5OiBNZW5sbzsgYmFja2dyb3Vu ZC1jb2xvcjogcmdiKDI1NSwgMjU1LCAyNTUpOyIgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImZvbnQt dmFyaWFudC1saWdhdHVyZXM6IG5vLWNvbW1vbi1saWdhdHVyZXMiIGNsYXNzPSIiPiQgZmlvIC0t ZmlsZW5hbWU9L2Rldi9udm1lMG4xIC0tbmFtZT1yYW5kd3JpdGUgLS1pb2VuZ2luZT1saWJhaW8g LS1pb2RlcHRoPTEgLS1ydz1yYW5kcmVhZCAtLWJzPTRrIC0tZGlyZWN0PTEgLS1udW1qb2JzPTEg LS1ydW50aW1lPTEwIOKAlGdyb3VwX3JlcG9ydGluZzwvc3Bhbj48L2Rpdj48L2Rpdj48ZGl2IHN0 eWxlPSJtYXJnaW46IDBweDsgZm9udC1zdHJldGNoOiBub3JtYWw7IGZvbnQtc2l6ZTogMTFweDsg bGluZS1oZWlnaHQ6IG5vcm1hbDsgZm9udC1mYW1pbHk6IE1lbmxvOyBiYWNrZ3JvdW5kLWNvbG9y OiByZ2IoMjU1LCAyNTUsIDI1NSk7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iZm9udC12YXJpYW50 LWxpZ2F0dXJlczogbm8tY29tbW9uLWxpZ2F0dXJlcyIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPjwv c3Bhbj48L2Rpdj48ZGl2IHN0eWxlPSJtYXJnaW46IDBweDsgZm9udC1zdHJldGNoOiBub3JtYWw7 IGZvbnQtc2l6ZTogMTFweDsgbGluZS1oZWlnaHQ6IG5vcm1hbDsgZm9udC1mYW1pbHk6IE1lbmxv OyBiYWNrZ3JvdW5kLWNvbG9yOiByZ2IoMjU1LCAyNTUsIDI1NSk7IiBjbGFzcz0iIj48c3BhbiBz dHlsZT0iZm9udC12YXJpYW50LWxpZ2F0dXJlczogbm8tY29tbW9uLWxpZ2F0dXJlcyIgY2xhc3M9 IiI+cmVhZCA6IGlvPTEwMDIuOE1CLCBidz0xMDI2NjVLQi9zLCA8YiBjbGFzcz0iIj5pb3BzPC9i Pj0yNTY2NiwgcnVudD0gMTAwMDFtc2VjPC9zcGFuPjwvZGl2PjxkaXYgY2xhc3M9IiI+PGJyIGNs YXNzPSIiPjwvZGl2PjxkaXYgY2xhc3M9IiI+SSB3cml0ZSBhIHRlc3Qgd2hpY2ggY3JlYXRlIDEg dGhyZWFkIHRoYXQgcGVyZm9ybXMgYSBsYXJnZSBudW1iZXIgb2YgcmFuZG9tIHdyaXRlcyB0byBu dm1lIGluIGEgbG9vcCB1c2luZyBzcGRrIHdpdGggcXVldWUgZGVwdGggPTEuIEluIGVhY2ggcm91 bmQgb2YgdGhlIGxvb3AsIHRoZSBwcm9ncmFtIHN1Ym1pdHMgYSByYW5kb20gd3JpdGUgY29tbWFu ZCB0byB0aGUgcXVldWUgcGFpcnMgYW5kIHRoZW4gY2FsbCByZXBlYXRlZGx5IGNhbGwgc3Bka19u dm1lX3FwYWlyX3Byb2Nlc3NfY29tcGxldGlvbnMoKSB1bnRpbCB0aGUgc3VibWl0dGVkIElPIGlz IHByb2Nlc3NlZC4gSSBmb3VuZCB0aGF0IHdoZW4gdGhlIGRlcHRoIGlzIDEsIHRoZSBkZWxheSBi ZXR3ZWVuIHRoZSBjYWxsaW5nIG9mIHNwZGtfbnZtZV9uc19jbWRfd3JpdGUoKS9zcGRrX252bWVf bnNfY21kX3dyaXRlKCkgYW5kIHRoZSBjYWxsaW5nIG9mIHNwZGtfbnZtZV9xcGFpcl9wcm9jZXNz X2NvbXBsZXRpb25zKCkgaGFzIGh1Z2UgZWZmZWN0cyBvbiB0aGUgSU9QUy4gTW9yZSBwcmVjaW91 c2x5LCBJIGZvdW5kIHRoYXQgaWYgSSBjYWxsIHNwZGtfbnZtZV9xcGFpcl9wcm9jZXNzX2NvbXBs ZXRpb25zKCkgaW1tZWRpYXRlbHkgYWZ0ZXIgdGhlIHN1Ym1pc3Npb24gb2Ygd3JpdGUgY29tbWFu ZCAodmlhIHNwZGtfbnZtZV9uc19jbWRfd3JpdGUoKSksIHRoZSBJT1BTIHdpbGwgYmUgZXh0cmVt ZWx5IGxvdywgZS5nLiwgfjFLQi4gSG93ZXZlciwgaWYgdGhlIHByb2dyYW0gc2xlZXBzIGZvciBz ZXZlcmFsIG1pY3Jvc2Vjb25kcywgYSBtdWNoIGhpZ2hlciBJT1BTLCBzYXkgMTBLQiwgY2FuIGJl IGFjaGlldmVkLiZuYnNwOzxzcGFuIHN0eWxlPSJiYWNrZ3JvdW5kLWNvbG9yOiByZ2IoMjU1LCAy NTUsIDI1NSk7IiBjbGFzcz0iIj48Zm9udCBmYWNlPSJNZW5sbyIgY2xhc3M9IiI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZTogMTFweDsiIGNsYXNzPSIiPk15IGZlZWxpbmcgaXMgdGhhdCBjYWxsaW5n Jm5ic3A7PC9zcGFuPjwvZm9udD48L3NwYW4+c3Bka19udm1lX3FwYWlyX3Byb2Nlc3NfY29tcGxl dGlvbnMoKSB0b28gc29vbiBhbmQgdG9vIG9mdGVuIGNvdWxkIHNldmVyZWx5IGluY3JlYXNlIHRo ZSBwcm9jZXNzaW5nIHRpbWUgb2YgYW4gSU8gY29tbWFuZC4mbmJzcDs8L2Rpdj48ZGl2IGNsYXNz PSIiPjxiciBjbGFzcz0iIj48L2Rpdj48ZGl2IGNsYXNzPSIiPkkgYWxzbyBmb3VuZCB0aGF0ICI8 c3BhbiBzdHlsZT0iYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1NSwgMjU1LCAyNTUpOyIgY2xhc3M9 IiI+PGZvbnQgZmFjZT0iTWVubG8iIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6IDEx cHg7IiBjbGFzcz0iIj5wZXJmIC1xIDEgLXMgNTEyIC13IHdyaXRlIC10IDEgLWMgM+KAnSBhY2hp ZXZlZCBvdmVyIDUwSyBJT1BTLCBidXQmbmJzcDs8L3NwYW4+PC9mb250Pjwvc3Bhbj4iPHNwYW4g c3R5bGU9ImJhY2tncm91bmQtY29sb3I6IHJnYigyNTUsIDI1NSwgMjU1KTsiIGNsYXNzPSIiPjxm b250IGZhY2U9Ik1lbmxvIiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMXB4OyIg Y2xhc3M9IiI+cGVyZiAtcSAxIC1zIDUxMiAtdyB3cml0ZSAtdCAxIC1jIDHigJ0gYWNoaWV2ZWQg b25seSB+MUsgSU9QUy48L3NwYW4+PC9mb250Pjwvc3Bhbj48L2Rpdj48ZGl2IGNsYXNzPSIiPjxz cGFuIHN0eWxlPSJiYWNrZ3JvdW5kLWNvbG9yOiByZ2IoMjU1LCAyNTUsIDI1NSk7IiBjbGFzcz0i Ij48Zm9udCBmYWNlPSJNZW5sbyIgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTFw eDsiIGNsYXNzPSIiPjxiciBjbGFzcz0iIj48L3NwYW4+PC9mb250Pjwvc3Bhbj48L2Rpdj48ZGl2 IGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJiYWNrZ3JvdW5kLWNvbG9yOiByZ2IoMjU1LCAyNTUsIDI1 NSk7IiBjbGFzcz0iIj48Zm9udCBmYWNlPSJNZW5sbyIgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZTogMTFweDsiIGNsYXNzPSIiPkRvIHlvdSBrbm93IHRoZSByZWFzb24gYmVoaW5kIHRo aXMgd2lyZWQgcGVyZm9ybWFuY2U/IEFsc28sIEnigJlkIGxpa2UgdG8ga25vdyBpZiB0aGUgd2ly ZWQgcGVyZm9ybWFuY2UgaXMgaGlnaGx5IHJlbGF0ZWQgdG8gRUMyIGluc3RhbmNlLiBJIHdpbGwg YXBwcmVjaWF0ZSBpZiB5b3UgY2FuIHNoYXJlIHRoZSByZXN1bHRzIG9mIHRoZSBzYW1lIGNvbW1h bmRzIG9uIGEgaGFyZHdhcmUgcmF0aGVyIHRoYW4gRUMyIGkzIGluc3RhbmNlPzwvc3Bhbj48L2Zv bnQ+PC9zcGFuPjwvZGl2PjxkaXYgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImJhY2tncm91bmQtY29s b3I6IHJnYigyNTUsIDI1NSwgMjU1KTsiIGNsYXNzPSIiPjxmb250IGZhY2U9Ik1lbmxvIiBjbGFz cz0iIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMXB4OyIgY2xhc3M9IiI+PGJyIGNsYXNzPSIi Pjwvc3Bhbj48L2ZvbnQ+PC9zcGFuPjwvZGl2PjxkaXYgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPjwv ZGl2PjxkaXYgY2xhc3M9IiI+VGhhbmsgeW91IGFsbCBmb3IgeW91ciB0aW1lIGFuZCB5b3VyIGZl ZWRiYWNrIGlzIGhpZ2hseSBhcHByZWNpYXRlZC48L2Rpdj48ZGl2IGNsYXNzPSIiPjxiciBjbGFz cz0iIj48L2Rpdj48ZGl2IGNsYXNzPSIiPlNpbmNlcmVseSw8L2Rpdj48ZGl2IGNsYXNzPSIiPkxp IFdhbmc8L2Rpdj48ZGl2IGNsYXNzPSIiPjxiciBjbGFzcz0iIj48L2Rpdj48ZGl2IGNsYXNzPSIi PjxiciBjbGFzcz0iIj48L2Rpdj48ZGl2IGNsYXNzPSIiPjxiciBjbGFzcz0iIj48L2Rpdj48ZGl2 PjxiciBjbGFzcz0iIj48YmxvY2txdW90ZSB0eXBlPSJjaXRlIiBjbGFzcz0iIj48ZGl2IGNsYXNz PSIiPk9uIERlYyAzMSwgMjAxNywgYXQgMTI6NTYgUE0sIExpIFdhbmcgJmx0OzxhIGhyZWY9Im1h aWx0bzp3YW5nbGkxNDI2QGdtYWlsLmNvbSIgY2xhc3M9IiI+d2FuZ2xpMTQyNkBnbWFpbC5jb208 L2E+Jmd0OyB3cm90ZTo8L2Rpdj48YnIgY2xhc3M9IkFwcGxlLWludGVyY2hhbmdlLW5ld2xpbmUi PjxkaXYgY2xhc3M9IiI+PGRpdiBjbGFzcz0iIj5EZWFyIEJyaWplc2gsPGJyIGNsYXNzPSIiPjxi ciBjbGFzcz0iIj5UaGFua3MgZm9yIHlvdXIgc3RlcCBieSBzdGVwIGd1aWRhbmNlLiBJIHdpbGwg bGV0IHlvdSBrbm93IHdoZW4gSSBnZXQgdGhlIHJlc3VsdHMuPGJyIGNsYXNzPSIiPjxiciBjbGFz cz0iIj5SZWdhcmRzLDxiciBjbGFzcz0iIj5MaSBXYW5nPGJyIGNsYXNzPSIiPjxiciBjbGFzcz0i Ij48YmxvY2txdW90ZSB0eXBlPSJjaXRlIiBjbGFzcz0iIj5PbiBEZWMgMzAsIDIwMTcsIGF0IDEw OjQ3IFBNLCBCcmlqZXNoIFNpbmdoICZsdDs8YSBocmVmPSJtYWlsdG86YnJpamVzaC5zLnNpbmdo QGdtYWlsLmNvbSIgY2xhc3M9IiI+YnJpamVzaC5zLnNpbmdoQGdtYWlsLmNvbTwvYT4mZ3Q7IHdy b3RlOjxiciBjbGFzcz0iIj48YnIgY2xhc3M9IiI+SGkgTGksPGJyIGNsYXNzPSIiPjxiciBjbGFz cz0iIj5UcnkgZm9sbG93aW5nIHN0ZXBzOjxiciBjbGFzcz0iIj4tIFN3aXRjaCBmcm9tIFNQREsg dG8gTlZNZSBrZXJuZWwgbW9kdWxlLjxiciBjbGFzcz0iIj4gJm5ic3A7Jm5ic3A7I3JtbW9kIHVp b19wY2lfZ2VuZXJpYyAmYW1wOyZhbXA7IG1vZHByb2JlIG52bWUgJmFtcDsmYW1wOyBlY2hvIDEg Jmd0OyAvc3lzL2J1cy9wY2kvcmVzY2FuPGJyIGNsYXNzPSIiPi0gSW5zdGFsbCBmaW88YnIgY2xh c3M9IiI+ICZuYnNwOyZuYnNwOyMgeXVtIGluc3RhbGwgZmlvPGJyIGNsYXNzPSIiPi0gUnVuIGZp byB3aXRoIHF1ZXVlIGRlcHRoIDE8YnIgY2xhc3M9IiI+ICZuYnNwOyMgZmlvIC0tZmlsZW5hbWU9 L2Rldi9udm1lMG4xIC0tbmFtZT1yYW5kcmVhZCAtLWlvZW5naW5lPWxpYmFpbzxiciBjbGFzcz0i Ij4tLWlvZGVwdGg9MSAtLXJ3PXJhbmRyZWFkIC0tYnM9NGsgLS1kaXJlY3Q9MSAtLW51bWpvYnM9 MSAtLXJ1bnRpbWU9MTA8YnIgY2xhc3M9IiI+LS1ncm91cF9yZXBvcnRpbmc8YnIgY2xhc3M9IiI+ PGJyIGNsYXNzPSIiPldoZW4gYWxsIHRlc3RzIGFyZSBkb25lLCBydW4gZm9sbG93aW5nIGNvbW1h bmQgdG8gc3dpdGNoIGJhY2sgdG8gU1BESyBzdGFjay48YnIgY2xhc3M9IiI+ICMgcm1tb2QgbnZt ZSAmYW1wOyZhbXA7IG1vZHByb2JlIHVpb19wY2lfZ2VuZXJpYyAmYW1wOyZhbXA7IGVjaG8gMSAm Z3Q7IC9zeXMvYnVzL3BjaS9yZXNjYW48YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPlJlZ2FyZHMs PGJyIGNsYXNzPSIiPkJyaWplc2g8YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPjxiciBjbGFzcz0i Ij5PbiBTYXQsIERlYyAzMCwgMjAxNyBhdCAxOjUzIEFNLCBMaSBXYW5nICZsdDs8YSBocmVmPSJt YWlsdG86d2FuZ2xpMTQyNkBnbWFpbC5jb20iIGNsYXNzPSIiPndhbmdsaTE0MjZAZ21haWwuY29t PC9hPiZndDsgd3JvdGU6PGJyIGNsYXNzPSIiPjxibG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNz PSIiPkhpIEJyaWplc2gsPGJyIGNsYXNzPSIiPjxiciBjbGFzcz0iIj5UaGFua3MgZm9yIHlvdXIg c3VnZ2VzdGlvbi4gTWF5IEkga25vdyBob3cgSSBjYW4gcnVuIGZpbyB3aXRoIGtlcm5lbCBudm1l PGJyIGNsYXNzPSIiPmRyaXZlcj88YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPlRoYW5rcyw8YnIg Y2xhc3M9IiI+TGk8YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPk9uIDMwIERlY2VtYmVyIDIwMTcg YXQgMTI6MDYsIEJyaWplc2ggU2luZ2ggJmx0OzxhIGhyZWY9Im1haWx0bzpicmlqZXNoLnMuc2lu Z2hAZ21haWwuY29tIiBjbGFzcz0iIj5icmlqZXNoLnMuc2luZ2hAZ21haWwuY29tPC9hPiZndDs8 YnIgY2xhc3M9IiI+d3JvdGU6PGJyIGNsYXNzPSIiPjxibG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNs YXNzPSIiPjxiciBjbGFzcz0iIj5IaSBMaSw8YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPkRvIHlv dSBzZWUgc2ltaWxhciByZXN1bHRzIHdoZW4geW91IHJ1biBmaW8gd2l0aCBrZXJuZWwgbnZtZSBk cml2ZXI/PGJyIGNsYXNzPSIiPjxiciBjbGFzcz0iIj5SZWdhcmRzLDxiciBjbGFzcz0iIj5Ccmlq ZXNoPGJyIGNsYXNzPSIiPjxiciBjbGFzcz0iIj5PbiBGcmksIERlYyAyOSwgMjAxNyBhdCA3OjM5 IFBNLCBMaSBXYW5nICZsdDs8YSBocmVmPSJtYWlsdG86d2FuZ2xpMTQyNkBnbWFpbC5jb20iIGNs YXNzPSIiPndhbmdsaTE0MjZAZ21haWwuY29tPC9hPiZndDsgd3JvdGU6PGJyIGNsYXNzPSIiPjxi bG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNzPSIiPkRlYXIgQ3JhbmUsPGJyIGNsYXNzPSIiPjxi ciBjbGFzcz0iIj5Gb2xsb3dpbmcgeW91IGFkdmljZSwgSSBydW4gYSB2YXJpZXR5IG9mIHJhbmRv bS1yZWFkIGFuZCByYW5kb20td3JpdGU8YnIgY2xhc3M9IiI+d29ya2xvYWRzLCB3aXRoIC1zIHJh bmdpbmcgZnJvbSA1MTIgdG8gNTEyMDAuIFRoZSBJT1BTIHdpdGggZGVwdGggPSAxIGlzPGJyIGNs YXNzPSIiPmNvbnNpc3RlbnRseSBhcm91bmQgMUsgdW5kZXIgdmFyaW91cyB3b3JrbG9hZHMgYW5k IHNldHRpbmdzLjxiciBjbGFzcz0iIj48YnIgY2xhc3M9IiI+VGhhbmtzLDxiciBjbGFzcz0iIj5M aTxiciBjbGFzcz0iIj48YnIgY2xhc3M9IiI+T24gMzAgRGVjZW1iZXIgMjAxNyBhdCAxMTozMiwg Q3JhbmUgQ2h1ICZsdDs8YSBocmVmPSJtYWlsdG86Y3JhbmVjaHVAZ21haWwuY29tIiBjbGFzcz0i Ij5jcmFuZWNodUBnbWFpbC5jb208L2E+Jmd0OyB3cm90ZTo8YnIgY2xhc3M9IiI+PGJsb2NrcXVv dGUgdHlwZT0iY2l0ZSIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPkhpLCBMaSw8YnIgY2xhc3M9IiI+ PGJyIGNsYXNzPSIiPk1heWJlIHlvdSBjYW4gdHJ5IDRLIHJhbmRvbSByZWFkIHRvIHNpbXBsaWZ5 IHRoZSB0ZXN0LiA1MTJieXRlIHNlcS1yZWFkPGJyIGNsYXNzPSIiPmhhcyB0b28gbXVjaCBwb3Nz aWJpbHRpZXMgYW5kIHN1cnByaXNlcyBpbiBzc2QgZmlybXdhcmUuIDopPGJyIGNsYXNzPSIiPjxi ciBjbGFzcz0iIj4tQ3JhbmU8YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPjIwMTflubQxMuaciDMw 5pelIOS4iuWNiDk6NTfvvIwiTGkgV2FuZyIgJmx0OzxhIGhyZWY9Im1haWx0bzp3YW5nbGkxNDI2 QGdtYWlsLmNvbSIgY2xhc3M9IiI+d2FuZ2xpMTQyNkBnbWFpbC5jb208L2E+Jmd0O+WGmemBk++8 mjxiciBjbGFzcz0iIj48YmxvY2txdW90ZSB0eXBlPSJjaXRlIiBjbGFzcz0iIj48YnIgY2xhc3M9 IiI+RGVhciBCZW5qYW1pbiw8YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPlRoYW5rcyBmb3IgeW91 IGhlbHAuPGJyIGNsYXNzPSIiPjxiciBjbGFzcz0iIj5JIGp1c3QgYWxsb2NhdGUgYSBkZWRpY2F0 ZWQgaG9zdCBhbmQgcmVydW4gdGhlIHRlc3QsIGJ1dCBJIGdvdCB0aGU8YnIgY2xhc3M9IiI+c2lt aWxhciByZXN1bHRzLiBCeSBjcmVhdGluZyBhIGluc3RhbmNlIG9uIHRoZSBkZWRpY2F0ZWQgaG9z dCwgSSBhbTxiciBjbGFzcz0iIj5zdXJlPGJyIGNsYXNzPSIiPnRoYXQgdGhlIE5WTWUgZGV2aWNl IEkgY29ubmVjdGVkIGlzIGJhY2tlZCBieSBhIGxvY2FsIE5WTWUgZGV2aWNlLjxiciBjbGFzcz0i Ij5JdCdzIG5vdDxiciBjbGFzcz0iIj5hbiBFQlMgdm9sdW1lLCBhbHRob3VnaCBJIGRvbid0IHdo ZXRoZXIgdGhlIGRldmljZSBpcyBlbXVsYXRlZC48YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPlRo YW5rcyw8YnIgY2xhc3M9IiI+TGk8YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPk9uIDMwIERlY2Vt YmVyIDIwMTcgYXQgMDQ6NDIsIFdhbGtlciwgQmVuamFtaW48YnIgY2xhc3M9IiI+Jmx0OzxhIGhy ZWY9Im1haWx0bzpiZW5qYW1pbi53YWxrZXJAaW50ZWwuY29tIiBjbGFzcz0iIj5iZW5qYW1pbi53 YWxrZXJAaW50ZWwuY29tPC9hPiZndDsgd3JvdGU6PGJyIGNsYXNzPSIiPjxibG9ja3F1b3RlIHR5 cGU9ImNpdGUiIGNsYXNzPSIiPjxiciBjbGFzcz0iIj5PbiBGcmksIDIwMTctMTItMjkgYXQgMjM6 NDggKzA4MDAsIExpIFdhbmcgd3JvdGU6PGJyIGNsYXNzPSIiPjxibG9ja3F1b3RlIHR5cGU9ImNp dGUiIGNsYXNzPSIiPkRlYXIgYWxsLDxiciBjbGFzcz0iIj48YnIgY2xhc3M9IiI+SSBhbSBydW5u aW5nIG52bWUvcGVyZiB3aXRoIG1hc3RlciBicmFuY2ggb24gYSBFQzIgaW5zdGFuY2U8YnIgY2xh c3M9IiI+KFJlZGhhdCwgODxiciBjbGFzcz0iIj5DUFU8YnIgY2xhc3M9IiI+Y29yZXMpLiBJIG5v dGljZWQgdGhhdCB0aGUgSU9QUyBpcyBhcm91bmQgMUsgd2hlbiB0aGUgcXVldWUgZGVwdGg8YnIg Y2xhc3M9IiI+aXM8YnIgY2xhc3M9IiI+MS4gQnk8YnIgY2xhc3M9IiI+c2ltcGx5IGluY3JlYXNl IHRoZSBxdWV1ZSBsZW5ndGggdG8gMiwgdGhlIElPUFMgZ3Jvd3MgdG8gb3ZlciA2MEsuPGJyIGNs YXNzPSIiPk1heTxiciBjbGFzcz0iIj5JIGFzazxiciBjbGFzcz0iIj5pZiB0aGlzIGlzIG5vcm1h bD88YnIgY2xhc3M9IiI+PC9ibG9ja3F1b3RlPjxiciBjbGFzcz0iIj5JcyB0aGlzIGFuIEVDMiBk ZWRpY2F0ZWQgc2VydmVyIChpLmUuIHBoeXNpY2FsIGhhcmR3YXJlKSBvciBhIHZpcnR1YWw8YnIg Y2xhc3M9IiI+bWFjaGluZT88YnIgY2xhc3M9IiI+SXMgdGhlIE5WTWUgZGV2aWNlIHlvdSdyZSBj b25uZWN0ZWQgdG8gYmFja2VkIGJ5IGEgbG9jYWwgTlZNZSBkZXZpY2UsPGJyIGNsYXNzPSIiPmE8 YnIgY2xhc3M9IiI+cmVtb3RlPGJyIGNsYXNzPSIiPk5WTWUgZGV2aWNlLCBvciBhbiBFQlMgdm9s dW1lPyBBbWF6b24gcHJlc2VudHMgYmFzaWNhbGx5IGFsbCBzdG9yYWdlPGJyIGNsYXNzPSIiPmFz PGJyIGNsYXNzPSIiPk5WTWU8YnIgY2xhc3M9IiI+ZGV2aWNlcyB0aGVzZSBkYXlzLCBidXQgdGhl IGFjdHVhbCBiYWNraW5nIHN0b3JhZ2UgY2FuIHBlcmZvcm08YnIgY2xhc3M9IiI+ZGlmZmVyZW50 bHk8YnIgY2xhc3M9IiI+ZGVwZW5kaW5nIG9uIHRoZSBjaXJjdW1zdGFuY2VzLjxiciBjbGFzcz0i Ij48YnIgY2xhc3M9IiI+SW4gYWxsIGNhc2VzIEknbSBhd2FyZSBvZiwgdGhlIE5WTWUgZGV2aWNl IHByZXNlbnRlZCBpcyBub3QgdGhlPGJyIGNsYXNzPSIiPmFjdHVhbDxiciBjbGFzcz0iIj5OVk1l IFNTRC48YnIgY2xhc3M9IiI+SXQncyBlaXRoZXIgYSB2aXJ0dWFsIE5WTWUgZGV2aWNlIGVtdWxh dGVkIGluIHNvZnR3YXJlLCBvciBpdCdzPGJyIGNsYXNzPSIiPmltcGxlbWVudGVkIG9uIGE8YnIg Y2xhc3M9IiI+c3BlY2lhbCBjYXJkIHRoYXQgQW1hem9uIHB1dHMgaW4gYmV0d2VlbiB0aGUgcGxh dGZvcm0gYW5kIHRoZSByZWFsPGJyIGNsYXNzPSIiPlNTRC48YnIgY2xhc3M9IiI+U1BESzxiciBj bGFzcz0iIj5oYXNuJ3QgYmVlbiB0ZXN0ZWQgdGhvcm91Z2hseSB3aXRoIEFtYXpvbidzIGVtdWxh dGVkIE5WTWUgZGV2aWNlcywgc288YnIgY2xhc3M9IiI+dGhlcmU8YnIgY2xhc3M9IiI+Y291bGQg YmUgc29tZSBpc3N1ZXMgdGhlcmUuIE1vc3QgTlZNZSBkZXZpY2VzIGhpc3RvcmljYWxseSB3ZXJl IG5vdDxiciBjbGFzcz0iIj52YWxpZGF0ZWQ8YnIgY2xhc3M9IiI+YWdhaW5zdCBwb2xsZWQgbW9k ZSwgdWx0cmEgaGlnaCBwZXJmb3JtYW5jZSBkcml2ZXJzIHNvIHdoZW4gU1BESyB3YXM8YnIgY2xh c3M9IiI+Zmlyc3Q8YnIgY2xhc3M9IiI+cmVsZWFzZWQgd2UgZm91bmQgYSBudW1iZXIgb2YgZmly bXdhcmUgYnVncyBpbiBleGlzdGluZyBTU0RzIHRoYXQ8YnIgY2xhc3M9IiI+d2VyZTxiciBjbGFz cz0iIj5vbmx5PGJyIGNsYXNzPSIiPmV4cG9zZWQgYmVjYXVzZSBTUERLIHdhcyBzbyBmYXN0LiBX ZSdyZSBzZWVpbmcgbGVzcyBhbmQgbGVzcyBvZiB0aGF0PGJyIGNsYXNzPSIiPm5vdywgYnV0PGJy IGNsYXNzPSIiPnlvdSBjb3VsZCBiZSBoaXR0aW5nIHNvbWV0aGluZyBsaWtlIHRoYXQgaGVyZS48 YnIgY2xhc3M9IiI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X188YnIgY2xhc3M9IiI+U1BESyBtYWlsaW5nIGxpc3Q8YnIgY2xhc3M9IiI+PGEgaHJlZj0ibWFp bHRvOlNQREtAbGlzdHMuMDEub3JnIiBjbGFzcz0iIj5TUERLQGxpc3RzLjAxLm9yZzwvYT48YnIg Y2xhc3M9IiI+aHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFpbG1hbi9saXN0aW5mby9zcGRrPGJyIGNs YXNzPSIiPjwvYmxvY2txdW90ZT48YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPjxiciBjbGFzcz0i Ij5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxiciBjbGFz cz0iIj5TUERLIG1haWxpbmcgbGlzdDxiciBjbGFzcz0iIj48YSBocmVmPSJtYWlsdG86U1BES0Bs aXN0cy4wMS5vcmciIGNsYXNzPSIiPlNQREtAbGlzdHMuMDEub3JnPC9hPjxiciBjbGFzcz0iIj5o dHRwczovL2xpc3RzLjAxLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3NwZGs8YnIgY2xhc3M9IiI+PGJy IGNsYXNzPSIiPjwvYmxvY2txdW90ZT48YnIgY2xhc3M9IiI+X19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX188YnIgY2xhc3M9IiI+U1BESyBtYWlsaW5nIGxpc3Q8 YnIgY2xhc3M9IiI+PGEgaHJlZj0ibWFpbHRvOlNQREtAbGlzdHMuMDEub3JnIiBjbGFzcz0iIj5T UERLQGxpc3RzLjAxLm9yZzwvYT48YnIgY2xhc3M9IiI+aHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFp bG1hbi9saXN0aW5mby9zcGRrPGJyIGNsYXNzPSIiPjxiciBjbGFzcz0iIj48L2Jsb2NrcXVvdGU+ PGJyIGNsYXNzPSIiPjxiciBjbGFzcz0iIj5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fXzxiciBjbGFzcz0iIj5TUERLIG1haWxpbmcgbGlzdDxiciBjbGFzcz0i Ij48YSBocmVmPSJtYWlsdG86U1BES0BsaXN0cy4wMS5vcmciIGNsYXNzPSIiPlNQREtAbGlzdHMu MDEub3JnPC9hPjxiciBjbGFzcz0iIj5odHRwczovL2xpc3RzLjAxLm9yZy9tYWlsbWFuL2xpc3Rp bmZvL3NwZGs8YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPjwvYmxvY2txdW90ZT5fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxiciBjbGFzcz0iIj5TUERLIG1h aWxpbmcgbGlzdDxiciBjbGFzcz0iIj48YSBocmVmPSJtYWlsdG86U1BES0BsaXN0cy4wMS5vcmci IGNsYXNzPSIiPlNQREtAbGlzdHMuMDEub3JnPC9hPjxiciBjbGFzcz0iIj5odHRwczovL2xpc3Rz LjAxLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3NwZGs8YnIgY2xhc3M9IiI+PC9ibG9ja3F1b3RlPjxi ciBjbGFzcz0iIj48YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPl9fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fPGJyIGNsYXNzPSIiPlNQREsgbWFpbGluZyBsaXN0 PGJyIGNsYXNzPSIiPjxhIGhyZWY9Im1haWx0bzpTUERLQGxpc3RzLjAxLm9yZyIgY2xhc3M9IiI+ U1BES0BsaXN0cy4wMS5vcmc8L2E+PGJyIGNsYXNzPSIiPmh0dHBzOi8vbGlzdHMuMDEub3JnL21h aWxtYW4vbGlzdGluZm8vc3BkazxiciBjbGFzcz0iIj48YnIgY2xhc3M9IiI+PC9ibG9ja3F1b3Rl Pl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPGJyIGNsYXNz PSIiPlNQREsgbWFpbGluZyBsaXN0PGJyIGNsYXNzPSIiPjxhIGhyZWY9Im1haWx0bzpTUERLQGxp c3RzLjAxLm9yZyIgY2xhc3M9IiI+U1BES0BsaXN0cy4wMS5vcmc8L2E+PGJyIGNsYXNzPSIiPmh0 dHBzOi8vbGlzdHMuMDEub3JnL21haWxtYW4vbGlzdGluZm8vc3BkazxiciBjbGFzcz0iIj48L2Js b2NrcXVvdGU+PGJyIGNsYXNzPSIiPjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48YnIg Y2xhc3M9IiI+PC9ib2R5PjwvaHRtbD4= --===============4271903889175358297==--