From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2818005018292825104==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O Threads Date: Wed, 31 Jan 2018 18:09:23 +0000 Message-ID: <3AE75A2D-B58F-496E-8AC5-BED6F58F7D39@intel.com> In-Reply-To: CALOt4ifK34j3QAkJTSA4HLrW=ixqRftFCL4vnRRQvAsdZTN6rA@mail.gmail.com List-ID: To: spdk@lists.01.org --===============2818005018292825104== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Fenggang, The max IOPs number is per-device =E2=80=93 not per-queue. The observed la= tency for each I/O - from submission to completion - will be the same wheth= er the 128 I/O are submitted on one queue or across four queues. Spreading= the I/O across four queues instead of one just means that the device will = process =C2=BC the rate of I/O from each of the four queues compared to if = it was submitted on a single queue. For BlobFS, spreading the I/O across multiple NVMe queues would not normall= y help with latency. There are NVMe features such as Weighted Round Robin = (WRR), which provide different priorities to different queues. With WRR, m= ultiple NVMe queues could be used to separate high priority I/O (i.e. WAL w= rites) from lower priority I/O (i.e. background compaction I/O). Most NVMe= devices today do not support WRR however and even then it=E2=80=99s still = questionable whether WRR alone would be sufficient or if additional softwar= e queuing would be required. Thanks, -Jim From: SPDK on behalf of Fenggang Wu Reply-To: Storage Performance Development Kit Date: Wednesday, January 31, 2018 at 10:49 AM To: Storage Performance Development Kit Cc: "wuxx0835(a)umn.edu" Subject: [SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O Threa= ds Hi All, I read from the SPDK doc "NVMe Driver Design -- Scaling Performance" (here<= http://www.spdk.io/doc/nvme.html#nvme_design>), which saids: " For example, if a device claims to be capable of 450,000 I/O per second a= t queue depth 128, in practice it does not matter if the driver is using 4 = queue pairs each with queue depth 32, or a single queue pair with queue dep= th 128." Does this consider the queuing latency? I am guessing the latency in the tw= o cases will be different ( in qp/qd =3D 4/32 and in qp/qd =3D 1/128). In t= he 4 threads case, the latency will be 1/4 of the 1 thread case. Do I get i= t right? If so, then I got confused as the document also says: "In order to take full advantage of this scaling, applications should consi= der organizing their internal data structures such that data is assigned ex= clusively to a single thread." Please correct me if I get it wrong. I understand that if the dedicate I/O = thread has the total ownership of the I/O data structures, there is no lock= contention to slow down the I/O. I believe that BlobFS is also designed in= this philosophy in that only one thread is doing I/O. But considering the RocksDB case, if the shared data structure has already = been largely taken care of by the RocksDB logic via locking (which is inevi= table anyway), the I/O requests each RocksDB thread sends to the BlobFS cou= ld also has its own queue pair to do I/O. More I/O threads means shorter qu= eue depth and smaller queuing delay. Even if there is some FS metadata operations that may require some locking,= but I would guest such metadata operation takes only a small portion. Therefore, is it a viable idea to have more I/O threads in the BlobFS to se= rve the multi-threaded RocksDB for a smaller delay? What will be the pitfal= ls, or challenges? Any thoughts/comments are appreciated. Thank you very much! Best! -Fenggang --===============2818005018292825104== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4 bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l dGEgbmFtZT0iVGl0bGUiIGNvbnRlbnQ9IiI+DQo8bWV0YSBuYW1lPSJLZXl3b3JkcyIgY29udGVu dD0iIj4NCjxtZXRhIG5hbWU9IkdlbmVyYXRvciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUg KGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxlPjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8N CkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0 IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OkRlbmdYaWFuOw0K CXBhbm9zZS0xOjIgMSA2IDAgMyAxIDEgMSAxIDE7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWls eTpDYWxpYnJpOw0KCXBhbm9zZS0xOjIgMTUgNSAyIDIgMiA0IDMgMiA0O30NCi8qIFN0eWxlIERl ZmluaXRpb25zICovDQpwLk1zb05vcm1hbCwgbGkuTXNvTm9ybWFsLCBkaXYuTXNvTm9ybWFsDQoJ e21hcmdpbjowaW47DQoJbWFyZ2luLWJvdHRvbTouMDAwMXB0Ow0KCWZvbnQtc2l6ZToxMi4wcHQ7 DQoJZm9udC1mYW1pbHk6IlRpbWVzIE5ldyBSb21hbiI7fQ0KYTpsaW5rLCBzcGFuLk1zb0h5cGVy bGluaw0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6Ymx1ZTsNCgl0ZXh0LWRlY29y YXRpb246dW5kZXJsaW5lO30NCmE6dmlzaXRlZCwgc3Bhbi5Nc29IeXBlcmxpbmtGb2xsb3dlZA0K CXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6cHVycGxlOw0KCXRleHQtZGVjb3JhdGlv bjp1bmRlcmxpbmU7fQ0Kc3Bhbi5FbWFpbFN0eWxlMTcNCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29u YWwtcmVwbHk7DQoJZm9udC1mYW1pbHk6Q2FsaWJyaTsNCgljb2xvcjp3aW5kb3d0ZXh0O30NCnNw YW4ubXNvSW5zDQoJe21zby1zdHlsZS10eXBlOmV4cG9ydC1vbmx5Ow0KCW1zby1zdHlsZS1uYW1l OiIiOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7DQoJY29sb3I6dGVhbDt9DQouTXNvQ2hw RGVmYXVsdA0KCXttc28tc3R5bGUtdHlwZTpleHBvcnQtb25seTsNCglmb250LXNpemU6MTAuMHB0 O30NCkBwYWdlIFdvcmRTZWN0aW9uMQ0KCXtzaXplOjguNWluIDExLjBpbjsNCgltYXJnaW46MS4w aW4gMS4waW4gMS4waW4gMS4waW47fQ0KZGl2LldvcmRTZWN0aW9uMQ0KCXtwYWdlOldvcmRTZWN0 aW9uMTt9DQotLT48L3N0eWxlPg0KPC9oZWFkPg0KPGJvZHkgYmdjb2xvcj0id2hpdGUiIGxhbmc9 IkVOLVVTIiBsaW5rPSJibHVlIiB2bGluaz0icHVycGxlIj4NCjxkaXYgY2xhc3M9IldvcmRTZWN0 aW9uMSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBw dDtmb250LWZhbWlseTpDYWxpYnJpIj5IaSBGZW5nZ2FuZyw8bzpwPjwvbzpwPjwvc3Bhbj48L3A+ DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250 LWZhbWlseTpDYWxpYnJpIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxp YnJpIj5UaGUgbWF4IElPUHMgbnVtYmVyIGlzIHBlci1kZXZpY2Ug4oCTIG5vdCBwZXItcXVldWUu Jm5ic3A7IFRoZSBvYnNlcnZlZCBsYXRlbmN5IGZvciBlYWNoIEkvTyAtIGZyb20gc3VibWlzc2lv biB0byBjb21wbGV0aW9uIC0gd2lsbCBiZSB0aGUgc2FtZSB3aGV0aGVyIHRoZSAxMjggSS9PIGFy ZSBzdWJtaXR0ZWQgb24gb25lIHF1ZXVlIG9yDQogYWNyb3NzIGZvdXIgcXVldWVzLiZuYnNwOyBT cHJlYWRpbmcgdGhlIEkvTyBhY3Jvc3MgZm91ciBxdWV1ZXMgaW5zdGVhZCBvZiBvbmUganVzdCBt ZWFucyB0aGF0IHRoZSBkZXZpY2Ugd2lsbCBwcm9jZXNzIMK8IHRoZSByYXRlIG9mIEkvTyBmcm9t IGVhY2ggb2YgdGhlIGZvdXIgcXVldWVzIGNvbXBhcmVkIHRvIGlmIGl0IHdhcyBzdWJtaXR0ZWQg b24gYSBzaW5nbGUgcXVldWUuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05v cm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaSI+ PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4g c3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaSI+Rm9yIEJsb2JGUywg c3ByZWFkaW5nIHRoZSBJL08gYWNyb3NzIG11bHRpcGxlIE5WTWUgcXVldWVzIHdvdWxkIG5vdCBu b3JtYWxseSBoZWxwIHdpdGggbGF0ZW5jeS4mbmJzcDsgVGhlcmUgYXJlIE5WTWUgZmVhdHVyZXMg c3VjaCBhcyBXZWlnaHRlZCBSb3VuZCBSb2JpbiAoV1JSKSwgd2hpY2ggcHJvdmlkZSBkaWZmZXJl bnQgcHJpb3JpdGllcw0KIHRvIGRpZmZlcmVudCBxdWV1ZXMuJm5ic3A7IFdpdGggV1JSLCBtdWx0 aXBsZSBOVk1lIHF1ZXVlcyBjb3VsZCBiZSB1c2VkIHRvIHNlcGFyYXRlIGhpZ2ggcHJpb3JpdHkg SS9PIChpLmUuIFdBTCB3cml0ZXMpIGZyb20gbG93ZXIgcHJpb3JpdHkgSS9PIChpLmUuIGJhY2tn cm91bmQgY29tcGFjdGlvbiBJL08pLiZuYnNwOyBNb3N0IE5WTWUgZGV2aWNlcyB0b2RheSBkbyBu b3Qgc3VwcG9ydCBXUlIgaG93ZXZlciBhbmQgZXZlbiB0aGVuIGl04oCZcyBzdGlsbCBxdWVzdGlv bmFibGUNCiB3aGV0aGVyIFdSUiBhbG9uZSB3b3VsZCBiZSBzdWZmaWNpZW50IG9yIGlmIGFkZGl0 aW9uYWwgc29mdHdhcmUgcXVldWluZyB3b3VsZCBiZSByZXF1aXJlZC48bzpwPjwvbzpwPjwvc3Bh bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBw dDtmb250LWZhbWlseTpDYWxpYnJpIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWls eTpDYWxpYnJpIj5UaGFua3MsPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05v cm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaSI+ PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaSI+LUppbTxvOnA+PC9vOnA+PC9z cGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEu MHB0O2ZvbnQtZmFtaWx5OkNhbGlicmkiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFt aWx5OkNhbGlicmkiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29O b3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmki PjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu IHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmkiPjxvOnA+Jm5ic3A7 PC9vOnA+PC9zcGFuPjwvcD4NCjxkaXYgc3R5bGU9ImJvcmRlcjpub25lO2JvcmRlci10b3A6c29s aWQgI0I1QzRERiAxLjBwdDtwYWRkaW5nOjMuMHB0IDBpbiAwaW4gMGluIj4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj48Yj48c3BhbiBzdHlsZT0iZm9udC1m YW1pbHk6Q2FsaWJyaTtjb2xvcjpibGFjayI+RnJvbToNCjwvc3Bhbj48L2I+PHNwYW4gc3R5bGU9 ImZvbnQtZmFtaWx5OkNhbGlicmk7Y29sb3I6YmxhY2siPlNQREsgJmx0O3NwZGstYm91bmNlc0Bs aXN0cy4wMS5vcmcmZ3Q7IG9uIGJlaGFsZiBvZiBGZW5nZ2FuZyBXdSAmbHQ7ZmVuZ2dhbmdAY3Mu dW1uLmVkdSZndDs8YnI+DQo8Yj5SZXBseS1UbzogPC9iPlN0b3JhZ2UgUGVyZm9ybWFuY2UgRGV2 ZWxvcG1lbnQgS2l0ICZsdDtzcGRrQGxpc3RzLjAxLm9yZyZndDs8YnI+DQo8Yj5EYXRlOiA8L2I+ V2VkbmVzZGF5LCBKYW51YXJ5IDMxLCAyMDE4IGF0IDEwOjQ5IEFNPGJyPg0KPGI+VG86IDwvYj5T dG9yYWdlIFBlcmZvcm1hbmNlIERldmVsb3BtZW50IEtpdCAmbHQ7c3Bka0BsaXN0cy4wMS5vcmcm Z3Q7PGJyPg0KPGI+Q2M6IDwvYj4mcXVvdDt3dXh4MDgzNUB1bW4uZWR1JnF1b3Q7ICZsdDt3dXh4 MDgzNUB1bW4uZWR1Jmd0Ozxicj4NCjxiPlN1YmplY3Q6IDwvYj5bU1BES10gUGVyZm9ybWFuY2Ug U2NhbGluZyBpbiBCbG9iRlMvUm9ja3NEQiBieSBNdWx0aXBsZSBJL08gVGhyZWFkczxvOnA+PC9v OnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxl PSJtYXJnaW4tbGVmdDouNWluIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj5IaSBBbGwsJm5i c3A7IDxvOnA+PC9vOnA+PC9wPg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJt YXJnaW4tbGVmdDouNWluIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj5JIHJlYWQgZnJvbSB0 aGUgU1BESyBkb2MgJnF1b3Q7TlZNZSBEcml2ZXIgRGVzaWduIC0tIFNjYWxpbmcgUGVyZm9ybWFu Y2UmcXVvdDsgKDxhIGhyZWY9Imh0dHA6Ly93d3cuc3Bkay5pby9kb2MvbnZtZS5odG1sI252bWVf ZGVzaWduIj5oZXJlPC9hPiksIHdoaWNoIHNhaWRzOjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8 ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPjxvOnA+ Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5 bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPiZxdW90OyZuYnNwO0ZvciBleGFtcGxlLCBpZiBhIGRldmlj ZSBjbGFpbXMgdG8gYmUgY2FwYWJsZSBvZiA0NTAsMDAwIEkvTyBwZXIgc2Vjb25kIGF0IHF1ZXVl IGRlcHRoIDEyOCwgaW4gcHJhY3RpY2UgaXQgZG9lcyBub3QgbWF0dGVyIGlmIHRoZSBkcml2ZXIg aXMgdXNpbmcgNCBxdWV1ZSBwYWlycyBlYWNoIHdpdGggcXVldWUgZGVwdGggMzIsIG9yIGEgc2lu Z2xlIHF1ZXVlIHBhaXINCiB3aXRoIHF1ZXVlIGRlcHRoIDEyOC4mcXVvdDs8bzpwPjwvbzpwPjwv cD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVm dDouNWluIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj5Eb2VzIHRoaXMgY29uc2lkZXIgdGhl IHF1ZXVpbmcgbGF0ZW5jeT8gSSBhbSBndWVzc2luZyB0aGUgbGF0ZW5jeSBpbiB0aGUgdHdvIGNh c2VzIHdpbGwgYmUgZGlmZmVyZW50ICggaW4gcXAvcWQgPSA0LzMyIGFuZCBpbiBxcC9xZCA9IDEv MTI4KS4gSW4gdGhlIDQgdGhyZWFkcyBjYXNlLCB0aGUgbGF0ZW5jeSB3aWxsIGJlIDEvNCZuYnNw O29mIHRoZSAxIHRocmVhZCBjYXNlLg0KIERvIEkgZ2V0IGl0IHJpZ2h0PzxvOnA+PC9vOnA+PC9w Pg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0 Oi41aW4iPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1z b05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPklmIHNvLCB0aGVuIEkgZ290IGNvbmZ1 c2VkIGFzIHRoZSBkb2N1bWVudCBhbHNvIHNheXM6PG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxk aXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+PG86cD4m bmJzcDs8L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibWFyZ2luLWxlZnQ6LjVpbiI+JnF1b3Q7SW4gb3JkZXIgdG8gdGFrZSBmdWxsIGFkdmFudGFn ZSBvZiB0aGlzIHNjYWxpbmcsIGFwcGxpY2F0aW9ucyBzaG91bGQgY29uc2lkZXIgb3JnYW5pemlu ZyB0aGVpciBpbnRlcm5hbCBkYXRhIHN0cnVjdHVyZXMgc3VjaCB0aGF0IGRhdGEgaXMgYXNzaWdu ZWQgZXhjbHVzaXZlbHkgdG8gYSBzaW5nbGUgdGhyZWFkLiZxdW90OzxvOnA+PC9vOnA+PC9wPg0K PC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41 aW4iPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05v cm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPlBsZWFzZSBjb3JyZWN0IG1lIGlmIEkgZ2V0 IGl0IHdyb25nLiBJIHVuZGVyc3RhbmQgdGhhdCBpZiB0aGUgZGVkaWNhdGUgSS9PIHRocmVhZCBo YXMgdGhlIHRvdGFsIG93bmVyc2hpcCBvZiB0aGUgSS9PIGRhdGEgc3RydWN0dXJlcywgdGhlcmUg aXMgbm8gbG9jayBjb250ZW50aW9uIHRvIHNsb3cgZG93biB0aGUgSS9PLiBJIGJlbGlldmUgdGhh dCBCbG9iRlMgaXMgYWxzbw0KIGRlc2lnbmVkIGluIHRoaXMgcGhpbG9zb3BoeSBpbiB0aGF0IG9u bHkgb25lIHRocmVhZCBpcyBkb2luZyBJL08uJm5ic3A7PG86cD48L286cD48L3A+DQo8L2Rpdj4N CjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+PG86 cD4mbmJzcDs8L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBz dHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+QnV0IGNvbnNpZGVyaW5nIHRoZSBSb2Nrc0RCIGNhc2Us IGlmIHRoZSBzaGFyZWQgZGF0YSBzdHJ1Y3R1cmUgaGFzIGFscmVhZHkgYmVlbiBsYXJnZWx5IHRh a2VuIGNhcmUgb2YgYnkgdGhlIFJvY2tzREIgbG9naWMgdmlhIGxvY2tpbmcgKHdoaWNoIGlzIGlu ZXZpdGFibGUgYW55d2F5KSwgdGhlIEkvTyByZXF1ZXN0cyBlYWNoIFJvY2tzREIgdGhyZWFkIHNl bmRzIHRvDQogdGhlIEJsb2JGUyBjb3VsZCBhbHNvIGhhcyBpdHMgb3duIHF1ZXVlIHBhaXIgdG8g ZG8gSS9PLiBNb3JlIEkvTyB0aHJlYWRzIG1lYW5zIHNob3J0ZXIgcXVldWUgZGVwdGggYW5kIHNt YWxsZXIgcXVldWluZyBkZWxheS4mbmJzcDs8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj48bzpwPiZuYnNw OzwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJt YXJnaW4tbGVmdDouNWluIj5FdmVuIGlmIHRoZXJlIGlzIHNvbWUgRlMgbWV0YWRhdGEgb3BlcmF0 aW9ucyB0aGF0IG1heSByZXF1aXJlIHNvbWUgbG9ja2luZywgYnV0IEkgd291bGQgZ3Vlc3Qgc3Vj aCBtZXRhZGF0YSBvcGVyYXRpb24gdGFrZXMgb25seSBhIHNtYWxsIHBvcnRpb24uPG86cD48L286 cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2lu LWxlZnQ6LjVpbiI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+VGhlcmVmb3JlLCBpcyBpdCBh IHZpYWJsZSBpZGVhIHRvIGhhdmUgbW9yZSBJL08gdGhyZWFkcyBpbiB0aGUgQmxvYkZTIHRvIHNl cnZlIHRoZSBtdWx0aS10aHJlYWRlZCBSb2Nrc0RCIGZvciBhIHNtYWxsZXIgZGVsYXk/IFdoYXQg d2lsbCBiZSB0aGUgcGl0ZmFsbHMsIG9yIGNoYWxsZW5nZXM/PG86cD48L286cD48L3A+DQo8L2Rp dj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+ PG86cD4mbmJzcDs8L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFs IiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+QW55IHRob3VnaHRzL2NvbW1lbnRzIGFyZSBhcHBy ZWNpYXRlZC4gVGhhbmsgeW91IHZlcnkgbXVjaCE8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRp dj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj48bzpwPiZu YnNwOzwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxl PSJtYXJnaW4tbGVmdDouNWluIj5CZXN0ITxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0K PHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPi1GZW5nZ2FuZzxv OnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K --===============2818005018292825104==--