From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============1319673921932971709==" MIME-Version: 1.0 From: =?utf-8?q?=E5=82=A8_=3Ccjj25233_at_163=2Ecom=3E?= Subject: Re: [SPDK] A issue about maximums of write latency when we access the same block consecutively. Date: Fri, 11 Aug 2017 21:27:23 +0800 Message-ID: <22a59da1.abd0.15dd17a5a85.Coremail.cjj25233@163.com> In-Reply-To: CAPtQi4Q=X+Ekg3+4-VphwQiJCue5QAUiX_3iEas7FSNGJiuQUQ@mail.gmail.com List-ID: To: spdk@lists.01.org --===============1319673921932971709== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi, Crane, Thanks for your enlightening suggestions, 1. we are afraid that we can not do secure erasing currently...The device i= s shared and we need coordinating with other fellows before erasing. = 2. we did not considerate the influence of the device state deeply, and we = think you make sense. = We will provide a new test report for the FOB state to verify your sugg= estion. 3. our test demo is based on the spdk/examples/nvme/hello_world/hello_world= .c=EF=BC=8Cand the granularity of I/O is set as 4K. 4. thanks for your knowledge sharing^_^ Best wishes, Jiajia Chu At 2017-08-09 13:34:17, "Crane Chu" wrote: Hi, Jiajia, = Here are some of my suggestion: try Secure Erase before the test. It will eliminate most of the FTL backgro= und operations, and put SSD into so-called FOB state. = most of the enterprise-grade SSD provides Quality of Service in speications= . For example, P3608 ensures 99.99% 4K write IO complete in 4.7 ms when QD= =3D1. Spec also provides another write latency value like 20us. You can con= sider them as worst case and best case respectively. But the distribution b= etween them is hard to predict in non-FOB state, due to the diversity of fi= rmware, host working load pattern, and NAND Flash characteristics. = mapping granularity is 4K in most of the SSD firmware design. So, if LBA si= ze is 512B in your test, it will make scenario more complex. Try to test wi= th 4K aligned IO. http://codecapsule.com/2014/02/12/coding-for-ssds-part-1-introduction-and-t= able-of-contents/ Thanks, = -Crane Chu On Mon, Aug 7, 2017 at 2:40 PM, =E5=82=A8 wrote: Hi, Ben We truly appreciate your concrete analysis and help in resolving the proble= m. As you say, Different devices have their different and complex mechanisms. = We do not know how the hardware processes each request actually, but we thi= nk you make sense. Maybe it will be relieved with the hardware upgrade and development. We hope that we can touch other devices and verify whether this problem exi= sts in the future. We also expect other fellows using other devices could share the test resul= ts about this issue. Thanks a lot. = At 2017-08-04 07:36:07, "Walker, Benjamin" wr= ote: >On Wed, 2017-08-02 at 19:13 +0800, =E5=82=A8 wrote: >> Answers: >> (1) "access" =3D write. We experiment read and write operations respect= ively, = >> but only find the strange phenomenon in the writing experiments. >> The comparison of experiments can be seen in accessories. >> (2) We use a NAND based SSD, Intel P3608. >> (3) The result presented in accessories is produced with no delay. = >> We try to set "sleep(1)" between the two operations, but it seems = does >> not work. >> = >> At 2017-08-02 07:49:13, "Walker, Benjamin" = wrote: >> > Hi Jiajia, >> > = >> > I have a bunch of questions that will help me figure out what you are >> > seeing. = >> > = >> > 1) When you say "access", do you mean read or write? The behavior of t= hese >> > two operations is quite different. >> > 2) Are you using a NAND based or 3D XPoint based SSD? These again work >> > entirely differently. >> > 3) When you access the same block repeatedly, what's the delay between= each >> > access? None? > >I was able to verify the behavior you are seeing. I'm afraid I'm not going= to be >able to give you an exact answer for your particular device - I don't have >insight into the specifics of how each SSD is implemented. I brainstormed = with a >few of my colleagues though, so what I can do is give you some idea of wha= t is >happening inside of the device that will make it clear why writing to the = same >block over and over may cause performance problems. > >A good mental model for an SSD is basically a log of (LBA, data) pairs. Wh= en you >write to any LBA, it just appends to the end of the log and updates an int= ernal >map of the location of that LBA. It does this appending by buffering sever= al >writes into RAM located on the SSD, then it sends that batch of data to th= e NAND >all at once. The other important understanding is that the SSD is composed= of a >large number of physical NAND dies, with some number of entirely parallel = NAND >channels that can handle writes. Writing to the log sends the batched data= to >each channel more or less round-robin. The final thing to remember is that= this >whole process is implemented in hardware, not software, so adding things l= ike >coordination between parallel operations is not as simple as just adding a= lock. > >When you write the same LBA over and over, a few things could happen insid= e the >SSD (I don't know how your SSD specifically works). = > >One possibility is that the SSD could see that the LBA is already buffered= in >memory from a previous write and it could just update that memory. However= , that >doesn't actually work in general. The data in that memory buffer may be >currently in use as part of a write to actual NAND, or may even be current= ly >being read. So the only option is to append to the end of the log for each= new >write to the LBA. This could probably be coordinated with locking in softw= are, >but remember that the SSD controller is implemented in hardware. If handli= ng >this case makes the design far more complex, it may not be possible given = power, >latency, and other budgets. > >Another possibility is that the data is appended to the log for each write= just >like any other I/O. However, it is still more complicated than the case wh= ere >random LBAs are being written to. Once one buffer is filled up, a write to= NAND >is issued. When that write completes, it has to update the map for the loc= ation >of the LBA. If, while that write is outstanding, another buffer fills up w= ith >new writes to the same LBA, the device has to figure out what to do. If it >submits the second NAND write to a new channel, it's then effectively raci= ng >against the first write. If they complete out of order, the user will end = up >with stale data. This case could also probably be handled by better coordi= nation > on the completion side, but again there is a complexity trade off when >implementing this in actual hardware. = > >The easiest solution is probably to just detect if a NAND write is active = for an >LBA in a given buffer, and then just queue up the next write until the one >before it finishes. That adds potentially a lot of latency, but it simplif= ies >the hardware design considerably. > >Ultimately, I have no idea what that SSD is actually doing, but you can se= e that >it's fairly complex to handle this case. It is certainly more complex than >handling random I/O. > >I hope that helps, >Ben = _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk --===============1319673921932971709== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBzdHlsZT0ibGluZS1oZWlnaHQ6MS43O2NvbG9yOiMwMDAwMDA7Zm9udC1zaXplOjE0cHg7 Zm9udC1mYW1pbHk6QXJpYWwiPjxkaXY+SGksIENyYW5lLDwvZGl2PjxkaXY+PGJyPjwvZGl2Pjxk aXY+VGhhbmtzIGZvciB5b3VyIGVubGlnaHRlbmluZyBzdWdnZXN0aW9ucyw8L2Rpdj48ZGl2PjEu IHdlIGFyZSBhZnJhaWQgdGhhdCB3ZSBjYW4gbm90IGRvIHNlY3VyZSBlcmFzaW5nIGN1cnJlbnRs eS4uLlRoZSBkZXZpY2UgaXMgc2hhcmVkIGFuZCB3ZSBuZWVkIGNvb3JkaW5hdGluZyB3aXRoIG90 aGVyIGZlbGxvd3MgYmVmb3JlIGVyYXNpbmcuJm5ic3A7PC9kaXY+PGRpdj4yLiB3ZSBkaWQgbm90 IGNvbnNpZGVyYXRlIHRoZSBpbmZsdWVuY2Ugb2YgdGhlIGRldmljZSBzdGF0ZSBkZWVwbHksIGFu ZCB3ZSB0aGluayB5b3UgbWFrZSBzZW5zZS4gJm5ic3A7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7 IFdlIHdpbGwgcHJvdmlkZSBhIG5ldyB0ZXN0IHJlcG9ydCBmb3IgdGhlIEZPQiBzdGF0ZSB0byB2 ZXJpZnkgeW91ciBzdWdnZXN0aW9uLjwvZGl2PjxkaXY+My4gb3VyIHRlc3QgZGVtbyBpcyBiYXNl ZCBvbiB0aGUmbmJzcDtzcGRrL2V4YW1wbGVzL252bWUvaGVsbG9fd29ybGQvaGVsbG9fd29ybGQu Y6OsYW5kIHRoZSBncmFudWxhcml0eSBvZiBJL08gaXMgc2V0IGFzIDRLLjwvZGl2PjxkaXY+NC4g dGhhbmtzIGZvciB5b3VyIGtub3dsZWRnZSBzaGFyaW5nXl9ePC9kaXY+PGRpdj48YnI+PC9kaXY+ PGRpdj5CZXN0IHdpc2hlcyw8L2Rpdj48ZGl2PiZuYnNwO0ppYWppYSBDaHU8L2Rpdj48YnI+PGRp diBzdHlsZT0icG9zaXRpb246cmVsYXRpdmU7em9vbToxIj48L2Rpdj48ZGl2IGlkPSJkaXZOZXRl YXNlTWFpbENhcmQiPjwvZGl2Pjxicj5BdCAyMDE3LTA4LTA5IDEzOjM0OjE3LCAiQ3JhbmUgQ2h1 IiAmbHQ7Y3JhbmVjaHVAZ21haWwuY29tJmd0OyB3cm90ZTo8YnI+IDxibG9ja3F1b3RlIGlkPSJp c1JlcGx5Q29udGVudCIgc3R5bGU9IlBBRERJTkctTEVGVDogMWV4OyBNQVJHSU46IDBweCAwcHgg MHB4IDAuOGV4OyBCT1JERVItTEVGVDogI2NjYyAxcHggc29saWQiPjxkaXYgZGlyPSJsdHIiPkhp LCBKaWFqaWEsJm5ic3A7PGRpdj48YnI+PC9kaXY+PGRpdj5IZXJlIGFyZSBzb21lIG9mIG15IHN1 Z2dlc3Rpb246PC9kaXY+PGRpdj48b2w+PGxpPnRyeSBTZWN1cmUgRXJhc2UgYmVmb3JlIHRoZSB0 ZXN0LiBJdCB3aWxsIGVsaW1pbmF0ZSBtb3N0IG9mIHRoZSBGVEwgYmFja2dyb3VuZCBvcGVyYXRp b25zLCBhbmQgcHV0IFNTRCBpbnRvIHNvLWNhbGxlZCBGT0Igc3RhdGUuJm5ic3A7PC9saT48bGk+ bW9zdCBvZiB0aGUgZW50ZXJwcmlzZS1ncmFkZSBTU0QgcHJvdmlkZXMgUXVhbGl0eSBvZiBTZXJ2 aWNlIGluIHNwZWljYXRpb25zLiBGb3IgZXhhbXBsZSwgUDM2MDggZW5zdXJlcyA5OS45OSUgNEsg d3JpdGUgSU8gY29tcGxldGUgaW4gNC43IG1zIHdoZW4gUUQ9MS4gU3BlYyBhbHNvIHByb3ZpZGVz IGFub3RoZXIgd3JpdGUgbGF0ZW5jeSB2YWx1ZSBsaWtlIDIwdXMuIFlvdSBjYW4gY29uc2lkZXIg dGhlbSBhcyB3b3JzdCBjYXNlIGFuZCBiZXN0IGNhc2UgcmVzcGVjdGl2ZWx5LiBCdXQgdGhlIGRp c3RyaWJ1dGlvbiBiZXR3ZWVuIHRoZW0gaXMgaGFyZCB0byBwcmVkaWN0IGluIG5vbi1GT0Igc3Rh dGUsIGR1ZSB0byB0aGUgZGl2ZXJzaXR5IG9mIGZpcm13YXJlLCBob3N0IHdvcmtpbmcgbG9hZCBw YXR0ZXJuLCBhbmQgTkFORCBGbGFzaCBjaGFyYWN0ZXJpc3RpY3MuJm5ic3A7PC9saT48bGk+bWFw cGluZyBncmFudWxhcml0eSBpcyA0SyBpbiBtb3N0IG9mIHRoZSBTU0QgZmlybXdhcmUgZGVzaWdu LiBTbywgaWYgTEJBIHNpemUgaXMgNTEyQiBpbiB5b3VyIHRlc3QsIGl0IHdpbGwgbWFrZSBzY2Vu YXJpbyBtb3JlIGNvbXBsZXguIFRyeSB0byB0ZXN0IHdpdGggNEsgYWxpZ25lZCBJTy48L2xpPjxs aT48YSBocmVmPSJodHRwOi8vY29kZWNhcHN1bGUuY29tLzIwMTQvMDIvMTIvY29kaW5nLWZvci1z c2RzLXBhcnQtMS1pbnRyb2R1Y3Rpb24tYW5kLXRhYmxlLW9mLWNvbnRlbnRzLyI+aHR0cDovL2Nv ZGVjYXBzdWxlLmNvbS8yMDE0LzAyLzEyL2NvZGluZy1mb3Itc3Nkcy1wYXJ0LTEtaW50cm9kdWN0 aW9uLWFuZC10YWJsZS1vZi1jb250ZW50cy88L2E+PGJyPjwvbGk+PC9vbD48ZGl2PlRoYW5rcywm bmJzcDs8L2Rpdj48L2Rpdj48ZGl2Pi1DcmFuZSBDaHU8L2Rpdj48L2Rpdj48ZGl2IGNsYXNzPSJn bWFpbF9leHRyYSI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5PbiBNb24sIEF1ZyA3LCAy MDE3IGF0IDI6NDAgUE0sILSiIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOmNq ajI1MjMzQDE2My5jb20iIHRhcmdldD0iX2JsYW5rIj5jamoyNTIzM0AxNjMuY29tPC9hPiZndDs8 L3NwYW4+IHdyb3RlOjxicj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJt YXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6 MWV4Ij48ZGl2IHN0eWxlPSJsaW5lLWhlaWdodDoxLjc7Y29sb3I6IzAwMDAwMDtmb250LXNpemU6 MTRweDtmb250LWZhbWlseTpBcmlhbCI+SGksIEJlbjxicj48ZGl2PldlIHRydWx5Jm5ic3A7YXBw cmVjaWF0ZSB5b3VyIGNvbmNyZXRlIGFuYWx5c2lzIGFuZCBoZWxwIGluIHJlc29sdmluZyB0aGUg cHJvYmxlbS48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkFzIHlvdSBzYXksIERpZmZlcmVudCBk ZXZpY2VzIGhhdmUgdGhlaXIgZGlmZmVyZW50IGFuZCBjb21wbGV4IG1lY2hhbmlzbXMuJm5ic3A7 PC9kaXY+PGRpdj5XZSBkbyBub3Qga25vdyBob3cgdGhlIGhhcmR3YXJlIHByb2Nlc3NlcyBlYWNo IHJlcXVlc3QgYWN0dWFsbHksIGJ1dCB3ZSB0aGluayB5b3UgbWFrZSBzZW5zZS48L2Rpdj48ZGl2 Pk1heWJlIGl0IHdpbGwgYmUgcmVsaWV2ZWQgd2l0aCB0aGUgaGFyZHdhcmUgdXBncmFkZSBhbmQg ZGV2ZWxvcG1lbnQuPC9kaXY+PGRpdj5XZSBob3BlIHRoYXQgd2UgY2FuIHRvdWNoIG90aGVyIGRl dmljZXMgYW5kIHZlcmlmeSB3aGV0aGVyIHRoaXMgcHJvYmxlbSBleGlzdHMgaW4gdGhlIGZ1dHVy ZS48L2Rpdj48ZGl2PldlIGFsc28gZXhwZWN0IG90aGVyIGZlbGxvd3MgdXNpbmcgb3RoZXIgZGV2 aWNlcyBjb3VsZCBzaGFyZSB0aGUgdGVzdCByZXN1bHRzIGFib3V0IHRoaXMgaXNzdWUuPC9kaXY+ PGRpdj5UaGFua3MgYSBsb3QuICZuYnNwOyZuYnNwOzwvZGl2PjxkaXY+PGRpdiBjbGFzcz0iaDUi PjxwcmU+QXQgMjAxNy0wOC0wNCAwNzozNjowNywgIldhbGtlciwgQmVuamFtaW4iICZsdDs8YSBo cmVmPSJtYWlsdG86YmVuamFtaW4ud2Fsa2VyQGludGVsLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmJl bmphbWluLndhbGtlckBpbnRlbC5jb208L2E+Jmd0OyB3cm90ZToKJmd0O09uIFdlZCwgMjAxNy0w OC0wMiBhdCAxOToxMyArMDgwMCwgtKIgd3JvdGU6CiZndDsmZ3Q7IEFuc3dlcnM6CiZndDsmZ3Q7 ICgxKSAiYWNjZXNzIiA9IHdyaXRlLiBXZSBleHBlcmltZW50IHJlYWQgYW5kIHdyaXRlIG9wZXJh dGlvbnMgIHJlc3BlY3RpdmVseSwgCiZndDsmZ3Q7ICAgICAgIGJ1dCBvbmx5IGZpbmQgdGhlIHN0 cmFuZ2UgcGhlbm9tZW5vbiBpbiB0aGUgd3JpdGluZyBleHBlcmltZW50cy4KJmd0OyZndDsgICAg ICAgVGhlIGNvbXBhcmlzb24gb2YgZXhwZXJpbWVudHMgY2FuIGJlIHNlZW4gaW4gYWNjZXNzb3Jp ZXMuCiZndDsmZ3Q7ICgyKSBXZSB1c2UgYSBOQU5EIGJhc2VkIFNTRCwgSW50ZWwgUDM2MDguCiZn dDsmZ3Q7ICgzKSBUaGUgcmVzdWx0IHByZXNlbnRlZCBpbiBhY2Nlc3NvcmllcyBpcyBwcm9kdWNl ZCB3aXRoIG5vIGRlbGF5LiAKJmd0OyZndDsgICAgICAgV2UgdHJ5IHRvIHNldCAic2xlZXAoMSki IGJldHdlZW4gdGhlIHR3byBvcGVyYXRpb25zLCBidXQgaXQgc2VlbXMgZG9lcwomZ3Q7Jmd0OyBu b3Qgd29yay4KJmd0OyZndDsgCiZndDsmZ3Q7IEF0IDIwMTctMDgtMDIgMDc6NDk6MTMsICJXYWxr ZXIsIEJlbmphbWluIiAmbHQ7PGEgaHJlZj0ibWFpbHRvOmJlbmphbWluLndhbGtlckBpbnRlbC5j b20iIHRhcmdldD0iX2JsYW5rIj5iZW5qYW1pbi53YWxrZXJAaW50ZWwuY29tPC9hPiZndDsgd3Jv dGU6CiZndDsmZ3Q7ICZndDsgSGkgSmlhamlhLAomZ3Q7Jmd0OyAmZ3Q7IAomZ3Q7Jmd0OyAmZ3Q7 IEkgaGF2ZSBhIGJ1bmNoIG9mIHF1ZXN0aW9ucyB0aGF0IHdpbGwgaGVscCBtZSBmaWd1cmUgb3V0 IHdoYXQgeW91IGFyZQomZ3Q7Jmd0OyAmZ3Q7IHNlZWluZy4gCiZndDsmZ3Q7ICZndDsgCiZndDsm Z3Q7ICZndDsgMSkgV2hlbiB5b3Ugc2F5ICJhY2Nlc3MiLCBkbyB5b3UgbWVhbiByZWFkIG9yIHdy aXRlPyBUaGUgYmVoYXZpb3Igb2YgdGhlc2UKJmd0OyZndDsgJmd0OyB0d28gb3BlcmF0aW9ucyBp cyBxdWl0ZSBkaWZmZXJlbnQuCiZndDsmZ3Q7ICZndDsgMikgQXJlIHlvdSB1c2luZyBhIE5BTkQg YmFzZWQgb3IgM0QgWFBvaW50IGJhc2VkIFNTRD8gVGhlc2UgYWdhaW4gd29yawomZ3Q7Jmd0OyAm Z3Q7IGVudGlyZWx5IGRpZmZlcmVudGx5LgomZ3Q7Jmd0OyAmZ3Q7IDMpIFdoZW4geW91IGFjY2Vz cyB0aGUgc2FtZSBibG9jayByZXBlYXRlZGx5LCB3aGF0J3MgdGhlIGRlbGF5IGJldHdlZW4gZWFj aAomZ3Q7Jmd0OyAmZ3Q7IGFjY2Vzcz8gTm9uZT8KJmd0OwomZ3Q7SSB3YXMgYWJsZSB0byB2ZXJp ZnkgdGhlIGJlaGF2aW9yIHlvdSBhcmUgc2VlaW5nLiBJJ20gYWZyYWlkIEknbSBub3QgZ29pbmcg dG8gYmUKJmd0O2FibGUgdG8gZ2l2ZSB5b3UgYW4gZXhhY3QgYW5zd2VyIGZvciB5b3VyIHBhcnRp Y3VsYXIgZGV2aWNlIC0gSSBkb24ndCBoYXZlCiZndDtpbnNpZ2h0IGludG8gdGhlIHNwZWNpZmlj cyBvZiBob3cgZWFjaCBTU0QgaXMgaW1wbGVtZW50ZWQuIEkgYnJhaW5zdG9ybWVkIHdpdGggYQom Z3Q7ZmV3IG9mIG15IGNvbGxlYWd1ZXMgdGhvdWdoLCBzbyB3aGF0IEkgY2FuIGRvIGlzIGdpdmUg eW91IHNvbWUgaWRlYSBvZiB3aGF0IGlzCiZndDtoYXBwZW5pbmcgaW5zaWRlIG9mIHRoZSBkZXZp Y2UgdGhhdCB3aWxsIG1ha2UgaXQgY2xlYXIgd2h5IHdyaXRpbmcgdG8gdGhlIHNhbWUKJmd0O2Js b2NrIG92ZXIgYW5kIG92ZXIgbWF5IGNhdXNlIHBlcmZvcm1hbmNlIHByb2JsZW1zLgomZ3Q7CiZn dDtBIGdvb2QgbWVudGFsIG1vZGVsIGZvciBhbiBTU0QgaXMgYmFzaWNhbGx5IGEgbG9nIG9mIChM QkEsIGRhdGEpIHBhaXJzLiBXaGVuIHlvdQomZ3Q7d3JpdGUgdG8gYW55IExCQSwgaXQganVzdCBh cHBlbmRzIHRvIHRoZSBlbmQgb2YgdGhlIGxvZyBhbmQgdXBkYXRlcyBhbiBpbnRlcm5hbAomZ3Q7 bWFwIG9mIHRoZSBsb2NhdGlvbiBvZiB0aGF0IExCQS4gSXQgZG9lcyB0aGlzIGFwcGVuZGluZyBi eSBidWZmZXJpbmcgc2V2ZXJhbAomZ3Q7d3JpdGVzIGludG8gUkFNIGxvY2F0ZWQgb24gdGhlIFNT RCwgdGhlbiBpdCBzZW5kcyB0aGF0IGJhdGNoIG9mIGRhdGEgdG8gdGhlIE5BTkQKJmd0O2FsbCBh dCBvbmNlLiBUaGUgb3RoZXIgaW1wb3J0YW50IHVuZGVyc3RhbmRpbmcgaXMgdGhhdCB0aGUgU1NE IGlzIGNvbXBvc2VkIG9mIGEKJmd0O2xhcmdlIG51bWJlciBvZiBwaHlzaWNhbCBOQU5EIGRpZXMs IHdpdGggc29tZSBudW1iZXIgb2YgZW50aXJlbHkgcGFyYWxsZWwgTkFORAomZ3Q7Y2hhbm5lbHMg dGhhdCBjYW4gaGFuZGxlIHdyaXRlcy4gV3JpdGluZyB0byB0aGUgbG9nIHNlbmRzIHRoZSBiYXRj aGVkIGRhdGEgdG8KJmd0O2VhY2ggY2hhbm5lbCBtb3JlIG9yIGxlc3Mgcm91bmQtcm9iaW4uIFRo ZSBmaW5hbCB0aGluZyB0byByZW1lbWJlciBpcyB0aGF0IHRoaXMKJmd0O3dob2xlIHByb2Nlc3Mg aXMgaW1wbGVtZW50ZWQgaW4gaGFyZHdhcmUsIG5vdCBzb2Z0d2FyZSwgc28gYWRkaW5nIHRoaW5n cyBsaWtlCiZndDtjb29yZGluYXRpb24gYmV0d2VlbiBwYXJhbGxlbCBvcGVyYXRpb25zIGlzIG5v dCBhcyBzaW1wbGUgYXMganVzdCBhZGRpbmcgYSBsb2NrLgomZ3Q7CiZndDtXaGVuIHlvdSB3cml0 ZSB0aGUgc2FtZSBMQkEgb3ZlciBhbmQgb3ZlciwgYSBmZXcgdGhpbmdzIGNvdWxkIGhhcHBlbiBp bnNpZGUgdGhlCiZndDtTU0QgKEkgZG9uJ3Qga25vdyBob3cgeW91ciBTU0Qgc3BlY2lmaWNhbGx5 IHdvcmtzKS4gCiZndDsKJmd0O09uZSBwb3NzaWJpbGl0eSBpcyB0aGF0IHRoZSBTU0QgY291bGQg c2VlIHRoYXQgdGhlIExCQSBpcyBhbHJlYWR5IGJ1ZmZlcmVkIGluCiZndDttZW1vcnkgZnJvbSBh IHByZXZpb3VzIHdyaXRlIGFuZCBpdCBjb3VsZCBqdXN0IHVwZGF0ZSB0aGF0IG1lbW9yeS4gSG93 ZXZlciwgdGhhdAomZ3Q7ZG9lc24ndCBhY3R1YWxseSB3b3JrIGluIGdlbmVyYWwuIFRoZSBkYXRh IGluIHRoYXQgbWVtb3J5IGJ1ZmZlciBtYXkgYmUKJmd0O2N1cnJlbnRseSBpbiB1c2UgYXMgcGFy dCBvZiBhIHdyaXRlIHRvIGFjdHVhbCBOQU5ELCBvciBtYXkgZXZlbiBiZSBjdXJyZW50bHkKJmd0 O2JlaW5nIHJlYWQuIFNvIHRoZSBvbmx5IG9wdGlvbiBpcyB0byBhcHBlbmQgdG8gdGhlIGVuZCBv ZiB0aGUgbG9nIGZvciBlYWNoIG5ldwomZ3Q7d3JpdGUgdG8gdGhlIExCQS4gVGhpcyBjb3VsZCBw cm9iYWJseSBiZSBjb29yZGluYXRlZCB3aXRoIGxvY2tpbmcgaW4gc29mdHdhcmUsCiZndDtidXQg cmVtZW1iZXIgdGhhdCB0aGUgU1NEIGNvbnRyb2xsZXIgaXMgaW1wbGVtZW50ZWQgaW4gaGFyZHdh cmUuIElmIGhhbmRsaW5nCiZndDt0aGlzIGNhc2UgbWFrZXMgdGhlIGRlc2lnbiBmYXIgbW9yZSBj b21wbGV4LCBpdCBtYXkgbm90IGJlIHBvc3NpYmxlIGdpdmVuIHBvd2VyLAomZ3Q7bGF0ZW5jeSwg YW5kIG90aGVyIGJ1ZGdldHMuCiZndDsKJmd0O0Fub3RoZXIgcG9zc2liaWxpdHkgaXMgdGhhdCB0 aGUgZGF0YSBpcyBhcHBlbmRlZCB0byB0aGUgbG9nIGZvciBlYWNoIHdyaXRlIGp1c3QKJmd0O2xp a2UgYW55IG90aGVyIEkvTy4gSG93ZXZlciwgaXQgaXMgc3RpbGwgbW9yZSBjb21wbGljYXRlZCB0 aGFuIHRoZSBjYXNlIHdoZXJlCiZndDtyYW5kb20gTEJBcyBhcmUgYmVpbmcgd3JpdHRlbiB0by4g T25jZSBvbmUgYnVmZmVyIGlzIGZpbGxlZCB1cCwgYSB3cml0ZSB0byBOQU5ECiZndDtpcyBpc3N1 ZWQuIFdoZW4gdGhhdCB3cml0ZSBjb21wbGV0ZXMsIGl0IGhhcyB0byB1cGRhdGUgdGhlIG1hcCBm b3IgdGhlIGxvY2F0aW9uCiZndDtvZiB0aGUgTEJBLiBJZiwgd2hpbGUgdGhhdCB3cml0ZSBpcyBv dXRzdGFuZGluZywgYW5vdGhlciBidWZmZXIgZmlsbHMgdXAgd2l0aAomZ3Q7bmV3IHdyaXRlcyB0 byB0aGUgc2FtZSBMQkEsIHRoZSBkZXZpY2UgaGFzIHRvIGZpZ3VyZSBvdXQgd2hhdCB0byBkby4g SWYgaXQKJmd0O3N1Ym1pdHMgdGhlIHNlY29uZCBOQU5EIHdyaXRlIHRvIGEgbmV3IGNoYW5uZWws IGl0J3MgdGhlbiBlZmZlY3RpdmVseSByYWNpbmcKJmd0O2FnYWluc3QgdGhlIGZpcnN0IHdyaXRl LiBJZiB0aGV5IGNvbXBsZXRlIG91dCBvZiBvcmRlciwgdGhlIHVzZXIgd2lsbCBlbmQgdXAKJmd0 O3dpdGggc3RhbGUgZGF0YS4gVGhpcyBjYXNlIGNvdWxkIGFsc28gcHJvYmFibHkgYmUgaGFuZGxl ZCBieSBiZXR0ZXIgY29vcmRpbmF0aW9uCiZndDsgb24gdGhlIGNvbXBsZXRpb24gc2lkZSwgYnV0 IGFnYWluIHRoZXJlIGlzIGEgY29tcGxleGl0eSB0cmFkZSBvZmYgd2hlbgomZ3Q7aW1wbGVtZW50 aW5nIHRoaXMgaW4gYWN0dWFsIGhhcmR3YXJlLiAKJmd0OwomZ3Q7VGhlIGVhc2llc3Qgc29sdXRp b24gaXMgcHJvYmFibHkgdG8ganVzdCBkZXRlY3QgaWYgYSBOQU5EIHdyaXRlIGlzIGFjdGl2ZSBm b3IgYW4KJmd0O0xCQSBpbiBhIGdpdmVuIGJ1ZmZlciwgYW5kIHRoZW4ganVzdCBxdWV1ZSB1cCB0 aGUgbmV4dCB3cml0ZSB1bnRpbCB0aGUgb25lCiZndDtiZWZvcmUgaXQgZmluaXNoZXMuIFRoYXQg YWRkcyBwb3RlbnRpYWxseSBhIGxvdCBvZiBsYXRlbmN5LCBidXQgaXQgc2ltcGxpZmllcwomZ3Q7 dGhlIGhhcmR3YXJlIGRlc2lnbiBjb25zaWRlcmFibHkuCiZndDsKJmd0O1VsdGltYXRlbHksIEkg aGF2ZSBubyBpZGVhIHdoYXQgdGhhdCBTU0QgaXMgYWN0dWFsbHkgZG9pbmcsIGJ1dCB5b3UgY2Fu IHNlZSB0aGF0CiZndDtpdCdzIGZhaXJseSBjb21wbGV4IHRvIGhhbmRsZSB0aGlzIGNhc2UuIEl0 IGlzIGNlcnRhaW5seSBtb3JlIGNvbXBsZXggdGhhbgomZ3Q7aGFuZGxpbmcgcmFuZG9tIEkvTy4K Jmd0OwomZ3Q7SSBob3BlIHRoYXQgaGVscHMsCiZndDtCZW4KPC9wcmU+PC9kaXY+PC9kaXY+PC9k aXY+PGJyPjxicj48c3BhbiB0aXRsZT0ibmV0ZWFzZWZvb3RlciI+PHA+Jm5ic3A7PC9wPjwvc3Bh bj48YnI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPHdicj48L3dicj5fX19fX19fX19f X19fX19fXzxicj4KU1BESyBtYWlsaW5nIGxpc3Q8YnI+CjxhIGhyZWY9Im1haWx0bzpTUERLQGxp c3RzLjAxLm9yZyI+U1BES0BsaXN0cy4wMS5vcmc8L2E+PGJyPgo8YSBocmVmPSJodHRwczovL2xp c3RzLjAxLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3NwZGsiIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0 PSJfYmxhbmsiPmh0dHBzOi8vbGlzdHMuMDEub3JnL21haWxtYW4vPHdicj48L3dicj5saXN0aW5m by9zcGRrPC9hPjxicj4KPGJyPjwvYmxvY2txdW90ZT48L2Rpdj48YnI+PC9kaXY+CjwvYmxvY2tx dW90ZT48L2Rpdj48YnI+PGJyPjxzcGFuIHRpdGxlPSJuZXRlYXNlZm9vdGVyIj48cD4mbmJzcDs8 L3A+PC9zcGFuPg== --===============1319673921932971709==--