From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============1467021888500899230==" MIME-Version: 1.0 From: Sreeni (Sreenivasa) Busam (Stellus) Subject: Re: [SPDK] Request for more details on SPDK driver API. Date: Mon, 13 Nov 2017 06:36:55 +0000 Message-ID: <0fa061d0031f4705b96e62d928e27988@stellus.com> In-Reply-To: 82C9F782B054C94B9FC04A331649C77A9D424551@FMSMSX108.amr.corp.intel.com List-ID: To: spdk@lists.01.org --===============1467021888500899230== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Paul, Thank you very much for taking the time to write a clear and lengthy reply = addressing my questions. I have got a fair understanding about the bdev lay= er and the use of it. I looked at the hello_world and fio example code already and tested the cod= e, and they were working well with a local NVMe SSD device connected. I nee= d to check out the blob hello_world example though. I will take a look at the presentations and follow-up if I have any other q= uestions. I have to join the #spdk channel, and will contact you if I need = any help there. I have a few questions regarding how the failures of read and write request= s are handled in the lower layer and controller level, if they are handled = and other questions. Sreeni From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E Sent: Saturday, November 11, 2017 6:41 AM To: Storage Performance Development Kit Subject: Re: [SPDK] Request for more details on SPDK driver API. Hi Sreeni, Welcome! Before I give you the start of an answer (others will chime in as= well I'm sure) I should mention that we're also on IRC at freenode in the = #spdk channel. You can generally find an expert there that can help via re= al-time or slightly delayed discussions :) Also, if you haven't checked out the presentations from our summit earlier = this year they are at http://www.spdk.io/news/2017/05/03/summit_presentatio= ns/ and there's some good ones in there that might help you. In most of those decks you'll find an architecture diagram that might help = pull all of this together for you. The key point for understanding your que= stions is to realize that SPDK is a collection of layered modules, some opt= ional, that all work together to form a storage stack in user space. So if= you think of the top of the stack being front end protocol modules like iS= CSI and NVMeoF and then the next big layer down being a generic block layer= followed by a lower device driver layer it should start to make more sense= . The generic block layer is a lightweight block abstraction at the top an= d then device specific modules at the bottom to interface with device drive= rs. So, anywhere you see an API with "bdev" in it, like the 2 you mention, = you're in the generic block layer. If the application you've constructed i= s based on NVMe then the next layer down will be the NVMe driver where you'= ll find the API that you listed in (1) below. As I mentioned though, a lot= of the layers are optional including the bdev layer. If you wanted to wri= te an application that didn't use bdevs (you knew it was only ever going to= be NVMe and didn't want the minimal overhead of bdev) you could use those = NVMe driver APIs directly in your application. Hope that makes sense, again I'm sure others will add more clarifying bits = of information as well. Here are some examples: https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is an ap= plication that doesn't use bdev, talks directly to NVMe https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is a sim= ple blobstore example (I didn't mention blobstore above but its another opt= ional layer) that does use a bdev. In this example it uses a ram disk (mall= oc) back end but could also easily use an NVMe back end directly and there'= s an open patch that shows how that works at https://review.gerrithub.io/#/= c/375460/ - both of these might not be the best example for your question b= ecause blobstore is thrown in the mix but you can see the bdev portion in t= here as part of blobstore in a subdir called bdev under /lib/blob Anyway, hope that helps more than it confuses :) Feel free to keep asking q= uestions until it makes sense... Thx Paul PS: there's also a bunch of existing bdev back end modules here https://git= hub.com/spdk/spdk/tree/master/lib/bdev that may help clarify the concepts. = Note that you can also stack bdevs in order to intercept IO coming and goin= g so you can do value added things easily, we call those "virtual bdevs" an= d you'll see some examples in the dir I just mentioned (ie split, lvol) From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreeni= vasa) Busam (Stellus) Sent: Friday, November 10, 2017 6:25 PM To: spdk(a)lists.01.org Subject: [SPDK] Request for more details on SPDK driver API. I am new to SPDK driver and trying to understand the various API's availabl= e to do I/O on NVMe local device and using NVMeoF protocol on a target devi= ce. I see different interfaces for the read and write I/O operations. It seems there are two types of interfaces available. 1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related functio= n. This is used for PCIe devices. It seems to apply to devices attached loc= ally. 2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API's i= n lib/bdev. I have been looking at the code and trying to understand how this API and r= elated APIs for the module has been used, the API's in 2. are used for blob= fs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem and = for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*. Is this a correct assumption? Is the lib/bdev layer developed to address on= ly these cases? Can the lib/bdev API be used on a local NVMe device? If so, is there a test= program to understand how to create a bdev device and do the I/O on NVMe S= SD block device? Please let me know if there is an example to understand how to use the lib/= nvmf interfaces. I want to test a device on remote target using NVMeoF prot= ocol. Thanks very much for any help in the lib/bdev API. --===============1467021888500899230== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv VFIvUkVDLWh0bWw0MCI+CjxoZWFkPgo8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNv bnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11cy1hc2NpaSI+CjxtZXRhIG5hbWU9IkdlbmVyYXRv ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1bSkiPgo8c3R5bGU+ PCEtLQovKiBGb250IERlZmluaXRpb25zICovCkBmb250LWZhY2UKCXtmb250LWZhbWlseTpXaW5n ZGluZ3M7CglwYW5vc2UtMTo1IDAgMCAwIDAgMCAwIDAgMCAwO30KQGZvbnQtZmFjZQoJe2ZvbnQt ZmFtaWx5OiJDYW1icmlhIE1hdGgiOwoJcGFub3NlLTE6MiA0IDUgMyA1IDQgNiAzIDIgNDt9CkBm b250LWZhY2UKCXtmb250LWZhbWlseToiTWFsZ3VuIEdvdGhpYyI7CglwYW5vc2UtMToyIDExIDUg MyAyIDAgMCAyIDAgNDt9CkBmb250LWZhY2UKCXtmb250LWZhbWlseTpDYWxpYnJpOwoJcGFub3Nl LTE6MiAxNSA1IDIgMiAyIDQgMyAyIDQ7fQpAZm9udC1mYWNlCgl7Zm9udC1mYW1pbHk6IlxATWFs Z3VuIEdvdGhpYyI7CglwYW5vc2UtMToyIDExIDUgMyAyIDAgMCAyIDAgNDt9Ci8qIFN0eWxlIERl ZmluaXRpb25zICovCnAuTXNvTm9ybWFsLCBsaS5Nc29Ob3JtYWwsIGRpdi5Nc29Ob3JtYWwKCXtt YXJnaW46MGluOwoJbWFyZ2luLWJvdHRvbTouMDAwMXB0OwoJZm9udC1zaXplOjExLjBwdDsKCWZv bnQtZmFtaWx5OiJDYWxpYnJpIixzYW5zLXNlcmlmO30KYTpsaW5rLCBzcGFuLk1zb0h5cGVybGlu awoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsKCWNvbG9yOiMwNTYzQzE7Cgl0ZXh0LWRlY29yYXRp b246dW5kZXJsaW5lO30KYTp2aXNpdGVkLCBzcGFuLk1zb0h5cGVybGlua0ZvbGxvd2VkCgl7bXNv LXN0eWxlLXByaW9yaXR5Ojk5OwoJY29sb3I6Izk1NEY3MjsKCXRleHQtZGVjb3JhdGlvbjp1bmRl cmxpbmU7fQpwLk1zb0xpc3RQYXJhZ3JhcGgsIGxpLk1zb0xpc3RQYXJhZ3JhcGgsIGRpdi5Nc29M aXN0UGFyYWdyYXBoCgl7bXNvLXN0eWxlLXByaW9yaXR5OjM0OwoJbWFyZ2luLXRvcDowaW47Cglt YXJnaW4tcmlnaHQ6MGluOwoJbWFyZ2luLWJvdHRvbTowaW47CgltYXJnaW4tbGVmdDouNWluOwoJ bWFyZ2luLWJvdHRvbTouMDAwMXB0OwoJZm9udC1zaXplOjExLjBwdDsKCWZvbnQtZmFtaWx5OiJD YWxpYnJpIixzYW5zLXNlcmlmO30Kc3Bhbi5FbWFpbFN0eWxlMTgKCXttc28tc3R5bGUtdHlwZTpw ZXJzb25hbDsKCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixzYW5zLXNlcmlmOwoJY29sb3I6d2luZG93 dGV4dDt9CnNwYW4uRW1haWxTdHlsZTE5Cgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWw7Cglmb250 LWZhbWlseToiQ2FsaWJyaSIsc2Fucy1zZXJpZjsKCWNvbG9yOiMxRjQ5N0Q7fQpzcGFuLkVtYWls U3R5bGUyMAoJe21zby1zdHlsZS10eXBlOnBlcnNvbmFsLXJlcGx5OwoJZm9udC1mYW1pbHk6IkNh bGlicmkiLHNhbnMtc2VyaWY7Cgljb2xvcjojMUY0OTdEO30KLk1zb0NocERlZmF1bHQKCXttc28t c3R5bGUtdHlwZTpleHBvcnQtb25seTsKCWZvbnQtc2l6ZToxMC4wcHQ7fQpAcGFnZSBXb3JkU2Vj dGlvbjEKCXtzaXplOjguNWluIDExLjBpbjsKCW1hcmdpbjoxLjBpbiAxLjBpbiAxLjBpbiAxLjBp bjt9CmRpdi5Xb3JkU2VjdGlvbjEKCXtwYWdlOldvcmRTZWN0aW9uMTt9Ci0tPjwvc3R5bGU+PCEt LVtpZiBndGUgbXNvIDldPjx4bWw+CjxvOnNoYXBlZGVmYXVsdHMgdjpleHQ9ImVkaXQiIHNwaWRt YXg9IjEwMjYiIC8+CjwveG1sPjwhW2VuZGlmXS0tPjwhLS1baWYgZ3RlIG1zbyA5XT48eG1sPgo8 bzpzaGFwZWxheW91dCB2OmV4dD0iZWRpdCI+CjxvOmlkbWFwIHY6ZXh0PSJlZGl0IiBkYXRhPSIx IiAvPgo8L286c2hhcGVsYXlvdXQ+PC94bWw+PCFbZW5kaWZdLS0+CjwvaGVhZD4KPGJvZHkgbGFu Zz0iRU4tVVMiIGxpbms9IiMwNTYzQzEiIHZsaW5rPSIjOTU0RjcyIj4KPGRpdiBjbGFzcz0iV29y ZFNlY3Rpb24xIj4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImNvbG9yOiMxRjQ5 N0QiPkhpIFBhdWwsPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPgo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+VGhhbmsgeW91 IHZlcnkgbXVjaCBmb3IgdGFraW5nIHRoZSB0aW1lIHRvIHdyaXRlIGEgY2xlYXIgYW5kIGxlbmd0 aHkgcmVwbHkgYWRkcmVzc2luZyBteSBxdWVzdGlvbnMuIEkgaGF2ZSBnb3QgYSBmYWlyIHVuZGVy c3RhbmRpbmcgYWJvdXQgdGhlIGJkZXYgbGF5ZXIgYW5kIHRoZSB1c2Ugb2YgaXQuPG86cD48L286 cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFG NDk3RCI+SSBsb29rZWQgYXQgdGhlIGhlbGxvX3dvcmxkIGFuZCBmaW8gZXhhbXBsZSBjb2RlIGFs cmVhZHkgYW5kIHRlc3RlZCB0aGUgY29kZSwgYW5kIHRoZXkgd2VyZSB3b3JraW5nIHdlbGwgd2l0 aCBhIGxvY2FsIE5WTWUgU1NEIGRldmljZSBjb25uZWN0ZWQuIEkgbmVlZCB0byBjaGVjayBvdXQg dGhlIGJsb2IgaGVsbG9fd29ybGQgZXhhbXBsZSB0aG91Z2guPG86cD48L286cD48L3NwYW4+PC9w Pgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+SSB3aWxs IHRha2UgYSBsb29rIGF0IHRoZSBwcmVzZW50YXRpb25zIGFuZCBmb2xsb3ctdXAgaWYgSSBoYXZl IGFueSBvdGhlciBxdWVzdGlvbnMuIEkgaGF2ZSB0byBqb2luIHRoZSAjc3BkayBjaGFubmVsLCBh bmQgd2lsbCBjb250YWN0IHlvdSBpZiBJIG5lZWQgYW55IGhlbHAgdGhlcmUuCjxvOnA+PC9vOnA+ PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImNvbG9yOiMxRjQ5 N0QiPkkgaGF2ZSBhIGZldyBxdWVzdGlvbnMgcmVnYXJkaW5nIGhvdyB0aGUgZmFpbHVyZXMgb2Yg cmVhZCBhbmQgd3JpdGUgcmVxdWVzdHMgYXJlIGhhbmRsZWQgaW4gdGhlIGxvd2VyIGxheWVyIGFu ZCBjb250cm9sbGVyIGxldmVsLCBpZiB0aGV5IGFyZSBoYW5kbGVkIGFuZCBvdGhlciBxdWVzdGlv bnMuPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHls ZT0iY29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0i TXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+U3JlZW5pPG86cD48L286cD48 L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3 RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBzdHlsZT0iY29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPgo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8 L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48YSBuYW1lPSJfTWFpbEVuZENv bXBvc2UiPjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bh bj48L2E+PC9wPgo8ZGl2Pgo8ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItdG9wOnNvbGlk ICNFMUUxRTEgMS4wcHQ7cGFkZGluZzozLjBwdCAwaW4gMGluIDBpbiI+CjxwIGNsYXNzPSJNc29O b3JtYWwiPjxiPkZyb206PC9iPiBTUERLIFttYWlsdG86c3Bkay1ib3VuY2VzQGxpc3RzLjAxLm9y Z10gPGI+T24gQmVoYWxmIE9mCjwvYj5MdXNlLCBQYXVsIEU8YnI+CjxiPlNlbnQ6PC9iPiBTYXR1 cmRheSwgTm92ZW1iZXIgMTEsIDIwMTcgNjo0MSBBTTxicj4KPGI+VG86PC9iPiBTdG9yYWdlIFBl cmZvcm1hbmNlIERldmVsb3BtZW50IEtpdCAmbHQ7c3Bka0BsaXN0cy4wMS5vcmcmZ3Q7PGJyPgo8 Yj5TdWJqZWN0OjwvYj4gUmU6IFtTUERLXSBSZXF1ZXN0IGZvciBtb3JlIGRldGFpbHMgb24gU1BE SyBkcml2ZXIgQVBJLjxvOnA+PC9vOnA+PC9wPgo8L2Rpdj4KPC9kaXY+CjxwIGNsYXNzPSJNc29O b3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBz dHlsZT0iY29sb3I6IzFGNDk3RCI+SGkgU3JlZW5pLDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImNvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7 PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImNvbG9y OiMxRjQ5N0QiPldlbGNvbWUhJm5ic3A7IEJlZm9yZSBJIGdpdmUgeW91IHRoZSBzdGFydCBvZiBh biBhbnN3ZXIgKG90aGVycyB3aWxsIGNoaW1lIGluIGFzIHdlbGwgSSYjODIxNzttIHN1cmUpIEkg c2hvdWxkIG1lbnRpb24gdGhhdCB3ZSYjODIxNztyZSBhbHNvIG9uIElSQyBhdCBmcmVlbm9kZSBp biB0aGUgI3NwZGsgY2hhbm5lbC4mbmJzcDsgWW91IGNhbiBnZW5lcmFsbHkgZmluZCBhbiBleHBl cnQgdGhlcmUgdGhhdAogY2FuIGhlbHAgdmlhIHJlYWwtdGltZSBvciBzbGlnaHRseSBkZWxheWVk IGRpc2N1c3Npb25zIDwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6V2luZ2RpbmdzO2Nv bG9yOiMxRjQ5N0QiPko8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOiMxRjQ5N0QiPjxvOnA+PC9v OnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImNvbG9yOiMx RjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PHNwYW4gc3R5bGU9ImNvbG9yOiMxRjQ5N0QiPkFsc28sIGlmIHlvdSBoYXZlbiYjODIxNzt0IGNo ZWNrZWQgb3V0IHRoZSBwcmVzZW50YXRpb25zIGZyb20gb3VyIHN1bW1pdCBlYXJsaWVyIHRoaXMg eWVhciB0aGV5IGFyZSBhdAo8YSBocmVmPSJodHRwOi8vd3d3LnNwZGsuaW8vbmV3cy8yMDE3LzA1 LzAzL3N1bW1pdF9wcmVzZW50YXRpb25zLyI+aHR0cDovL3d3dy5zcGRrLmlvL25ld3MvMjAxNy8w NS8wMy9zdW1taXRfcHJlc2VudGF0aW9ucy88L2E+IGFuZCB0aGVyZSYjODIxNztzIHNvbWUgZ29v ZCBvbmVzIGluIHRoZXJlIHRoYXQgbWlnaHQgaGVscCB5b3UuPG86cD48L286cD48L3NwYW4+PC9w Pgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+PG86cD4m bmJzcDs8L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0i Y29sb3I6IzFGNDk3RCI+SW4gbW9zdCBvZiB0aG9zZSBkZWNrcyB5b3UmIzgyMTc7bGwgZmluZCBh biBhcmNoaXRlY3R1cmUgZGlhZ3JhbSB0aGF0IG1pZ2h0IGhlbHAgcHVsbCBhbGwgb2YgdGhpcyB0 b2dldGhlciBmb3IgeW91LiBUaGUga2V5IHBvaW50IGZvciB1bmRlcnN0YW5kaW5nIHlvdXIgcXVl c3Rpb25zIGlzIHRvIHJlYWxpemUgdGhhdCBTUERLIGlzIGEgY29sbGVjdGlvbiBvZiBsYXllcmVk CiBtb2R1bGVzLCBzb21lIG9wdGlvbmFsLCB0aGF0IGFsbCB3b3JrIHRvZ2V0aGVyIHRvIGZvcm0g YSBzdG9yYWdlIHN0YWNrIGluIHVzZXIgc3BhY2UuJm5ic3A7IFNvIGlmIHlvdSB0aGluayBvZiB0 aGUgdG9wIG9mIHRoZSBzdGFjayBiZWluZyBmcm9udCBlbmQgcHJvdG9jb2wgbW9kdWxlcyBsaWtl IGlTQ1NJIGFuZCBOVk1lb0YgYW5kIHRoZW4gdGhlIG5leHQgYmlnIGxheWVyIGRvd24gYmVpbmcg YSBnZW5lcmljIGJsb2NrIGxheWVyIGZvbGxvd2VkIGJ5IGEKIGxvd2VyIGRldmljZSBkcml2ZXIg bGF5ZXIgaXQgc2hvdWxkIHN0YXJ0IHRvIG1ha2UgbW9yZSBzZW5zZS4mbmJzcDsgVGhlIGdlbmVy aWMgYmxvY2sgbGF5ZXIgaXMgYSBsaWdodHdlaWdodCBibG9jayBhYnN0cmFjdGlvbiBhdCB0aGUg dG9wIGFuZCB0aGVuIGRldmljZSBzcGVjaWZpYyBtb2R1bGVzIGF0IHRoZSBib3R0b20gdG8gaW50 ZXJmYWNlIHdpdGggZGV2aWNlIGRyaXZlcnMuIFNvLCBhbnl3aGVyZSB5b3Ugc2VlIGFuIEFQSSB3 aXRoICYjODIyMDtiZGV2JiM4MjIxOyBpbgogaXQsIGxpa2UgdGhlIDIgeW91IG1lbnRpb24sIHlv dSYjODIxNztyZSBpbiB0aGUgZ2VuZXJpYyBibG9jayBsYXllci4mbmJzcDsgSWYgdGhlIGFwcGxp Y2F0aW9uIHlvdSYjODIxNzt2ZSBjb25zdHJ1Y3RlZCBpcyBiYXNlZCBvbiBOVk1lIHRoZW4gdGhl IG5leHQgbGF5ZXIgZG93biB3aWxsIGJlIHRoZSBOVk1lIGRyaXZlciB3aGVyZSB5b3UmIzgyMTc7 bGwgZmluZCB0aGUgQVBJIHRoYXQgeW91IGxpc3RlZCBpbiAoMSkgYmVsb3cuJm5ic3A7IEFzIEkg bWVudGlvbmVkIHRob3VnaCwgYSBsb3Qgb2YKIHRoZSBsYXllcnMgYXJlIG9wdGlvbmFsIGluY2x1 ZGluZyB0aGUgYmRldiBsYXllci4mbmJzcDsgSWYgeW91IHdhbnRlZCB0byB3cml0ZSBhbiBhcHBs aWNhdGlvbiB0aGF0IGRpZG4mIzgyMTc7dCB1c2UgYmRldnMgKHlvdSBrbmV3IGl0IHdhcyBvbmx5 IGV2ZXIgZ29pbmcgdG8gYmUgTlZNZSBhbmQgZGlkbiYjODIxNzt0IHdhbnQgdGhlIG1pbmltYWwg b3ZlcmhlYWQgb2YgYmRldikgeW91IGNvdWxkIHVzZSB0aG9zZSBOVk1lIGRyaXZlciBBUElzIGRp cmVjdGx5IGluIHlvdXIgYXBwbGljYXRpb24uPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFz cz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286 cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFG NDk3RCI+SG9wZSB0aGF0IG1ha2VzIHNlbnNlLCBhZ2FpbiBJJiM4MjE3O20gc3VyZSBvdGhlcnMg d2lsbCBhZGQgbW9yZSBjbGFyaWZ5aW5nIGJpdHMgb2YgaW5mb3JtYXRpb24gYXMgd2VsbC4mbmJz cDsgSGVyZSBhcmUgc29tZSBleGFtcGxlczo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNz PSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj48bzpwPiZuYnNwOzwvbzpw Pjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0 OTdEIj48YSBocmVmPSJodHRwczovL2dpdGh1Yi5jb20vc3Bkay9zcGRrL3RyZWUvbWFzdGVyL2V4 YW1wbGVzL252bWUvaGVsbG9fd29ybGQiPmh0dHBzOi8vZ2l0aHViLmNvbS9zcGRrL3NwZGsvdHJl ZS9tYXN0ZXIvZXhhbXBsZXMvbnZtZS9oZWxsb193b3JsZDwvYT4gaXMgYW4gYXBwbGljYXRpb24g dGhhdCBkb2VzbiYjODIxNzt0IHVzZSBiZGV2LCB0YWxrcyBkaXJlY3RseSB0byBOVk1lPG86cD48 L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6 IzFGNDk3RCI+PGEgaHJlZj0iaHR0cHM6Ly9naXRodWIuY29tL3NwZGsvc3Bkay90cmVlL21hc3Rl ci9leGFtcGxlcy9ibG9iL2hlbGxvX3dvcmxkIj5odHRwczovL2dpdGh1Yi5jb20vc3Bkay9zcGRr L3RyZWUvbWFzdGVyL2V4YW1wbGVzL2Jsb2IvaGVsbG9fd29ybGQ8L2E+IGlzIGEgc2ltcGxlIGJs b2JzdG9yZSBleGFtcGxlIChJIGRpZG4mIzgyMTc7dCBtZW50aW9uIGJsb2JzdG9yZSBhYm92ZQog YnV0IGl0cyBhbm90aGVyIG9wdGlvbmFsIGxheWVyKSB0aGF0IGRvZXMgdXNlIGEgYmRldi4gSW4g dGhpcyBleGFtcGxlIGl0IHVzZXMgYSByYW0gZGlzayAobWFsbG9jKSBiYWNrIGVuZCBidXQgY291 bGQgYWxzbyBlYXNpbHkgdXNlIGFuIE5WTWUgYmFjayBlbmQgZGlyZWN0bHkgYW5kIHRoZXJlJiM4 MjE3O3MgYW4gb3BlbiBwYXRjaCB0aGF0IHNob3dzIGhvdyB0aGF0IHdvcmtzIGF0CjxhIGhyZWY9 Imh0dHBzOi8vcmV2aWV3LmdlcnJpdGh1Yi5pby8jL2MvMzc1NDYwLyI+aHR0cHM6Ly9yZXZpZXcu Z2Vycml0aHViLmlvLyMvYy8zNzU0NjAvPC9hPiAtIGJvdGggb2YgdGhlc2UgbWlnaHQgbm90IGJl IHRoZSBiZXN0IGV4YW1wbGUgZm9yIHlvdXIgcXVlc3Rpb24gYmVjYXVzZSBibG9ic3RvcmUgaXMg dGhyb3duIGluIHRoZSBtaXggYnV0IHlvdSBjYW4gc2VlIHRoZSBiZGV2IHBvcnRpb24gaW4gdGhl cmUgYXMgcGFydCBvZiBibG9ic3RvcmUKIGluIGEgc3ViZGlyIGNhbGxlZCBiZGV2IHVuZGVyIC9s aWIvYmxvYjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4g c3R5bGU9ImNvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xh c3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImNvbG9yOiMxRjQ5N0QiPkFueXdheSwgaG9wZSB0 aGF0IGhlbHBzIG1vcmUgdGhhbiBpdCBjb25mdXNlcwo8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQt ZmFtaWx5OldpbmdkaW5ncztjb2xvcjojMUY0OTdEIj5KPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xv cjojMUY0OTdEIj4gRmVlbCBmcmVlIHRvIGtlZXAgYXNraW5nIHF1ZXN0aW9ucyB1bnRpbCBpdCBt YWtlcyBzZW5zZSYjODIzMDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3Jt YWwiPjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48 L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj5UaHg8 bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJj b2xvcjojMUY0OTdEIj5QYXVsPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9y bWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+ PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+UFM6 IHRoZXJlJiM4MjE3O3MgYWxzbyBhIGJ1bmNoIG9mIGV4aXN0aW5nIGJkZXYgYmFjayBlbmQgbW9k dWxlcyBoZXJlCjxhIGhyZWY9Imh0dHBzOi8vZ2l0aHViLmNvbS9zcGRrL3NwZGsvdHJlZS9tYXN0 ZXIvbGliL2JkZXYiPmh0dHBzOi8vZ2l0aHViLmNvbS9zcGRrL3NwZGsvdHJlZS9tYXN0ZXIvbGli L2JkZXY8L2E+IHRoYXQgbWF5IGhlbHAgY2xhcmlmeSB0aGUgY29uY2VwdHMuIE5vdGUgdGhhdCB5 b3UgY2FuIGFsc28gc3RhY2sgYmRldnMgaW4gb3JkZXIgdG8gaW50ZXJjZXB0IElPIGNvbWluZyBh bmQgZ29pbmcgc28geW91IGNhbiBkbyB2YWx1ZSBhZGRlZCB0aGluZ3MKIGVhc2lseSwgd2UgY2Fs bCB0aG9zZSAmIzgyMjA7dmlydHVhbCBiZGV2cyYjODIyMTsgYW5kIHlvdSYjODIxNztsbCBzZWUg c29tZSBleGFtcGxlcyBpbiB0aGUgZGlyIEkganVzdCBtZW50aW9uZWQgKGllIHNwbGl0LCBsdm9s KTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9 ImNvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1z b05vcm1hbCI+PHNwYW4gc3R5bGU9ImNvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9z cGFuPjwvcD4KPGRpdj4KPGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLXRvcDpzb2xpZCAj RTFFMUUxIDEuMHB0O3BhZGRpbmc6My4wcHQgMGluIDBpbiAwaW4iPgo8cCBjbGFzcz0iTXNvTm9y bWFsIj48YSBuYW1lPSJfX19fX3JlcGx5c2VwYXJhdG9yIj48L2E+PGI+RnJvbTo8L2I+IFNQREsg WzxhIGhyZWY9Im1haWx0bzpzcGRrLWJvdW5jZXNAbGlzdHMuMDEub3JnIj5tYWlsdG86c3Bkay1i b3VuY2VzQGxpc3RzLjAxLm9yZzwvYT5dCjxiPk9uIEJlaGFsZiBPZiA8L2I+U3JlZW5pIChTcmVl bml2YXNhKSBCdXNhbSAoU3RlbGx1cyk8YnI+CjxiPlNlbnQ6PC9iPiBGcmlkYXksIE5vdmVtYmVy IDEwLCAyMDE3IDY6MjUgUE08YnI+CjxiPlRvOjwvYj4gPGEgaHJlZj0ibWFpbHRvOnNwZGtAbGlz dHMuMDEub3JnIj5zcGRrQGxpc3RzLjAxLm9yZzwvYT48YnI+CjxiPlN1YmplY3Q6PC9iPiBbU1BE S10gUmVxdWVzdCBmb3IgbW9yZSBkZXRhaWxzIG9uIFNQREsgZHJpdmVyIEFQSS48bzpwPjwvbzpw PjwvcD4KPC9kaXY+CjwvZGl2Pgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpw PjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+SSBhbSBuZXcgdG8gU1BESyBkcml2ZXIgYW5kIHRy eWluZyB0byB1bmRlcnN0YW5kIHRoZSB2YXJpb3VzIEFQSSYjODIxNztzIGF2YWlsYWJsZSB0byBk byBJL08gb24gTlZNZSBsb2NhbCBkZXZpY2UgYW5kIHVzaW5nIE5WTWVvRiBwcm90b2NvbCBvbiBh IHRhcmdldCBkZXZpY2UuCjxvOnA+PC9vOnA+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj5JIHNl ZSBkaWZmZXJlbnQgaW50ZXJmYWNlcyBmb3IgdGhlIHJlYWQgYW5kIHdyaXRlIEkvTyBvcGVyYXRp b25zLjxvOnA+PC9vOnA+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj5JdCBzZWVtcyB0aGVyZSBh cmUgdHdvIHR5cGVzIG9mIGludGVyZmFjZXMgYXZhaWxhYmxlLjxvOnA+PC9vOnA+PC9wPgo8cCBj bGFzcz0iTXNvTm9ybWFsIj4xLiBzcGRrX252bWVfbnNfY21kX3JlYWQoKSBhbmQgc3Bka19udm1l X25zX2NtZF93cml0ZSgpIGFuZCByZWxhdGVkIGZ1bmN0aW9uLiBUaGlzIGlzIHVzZWQgZm9yIFBD SWUgZGV2aWNlcy4gSXQgc2VlbXMgdG8gYXBwbHkgdG8gZGV2aWNlcyBhdHRhY2hlZCBsb2NhbGx5 LjxvOnA+PC9vOnA+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj4yLiBzcGRrX2JkZXZfcmVhZF9i bG9ja3MoKSBhbmQgc3Bka19iZGV2X3dyaXRlX2Jsb2NrcygpIGFuZCByZWxhdGVkIEFQSSYjODIx NztzIGluIGxpYi9iZGV2LjxvOnA+PC9vOnA+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj5JIGhh dmUgYmVlbiBsb29raW5nIGF0IHRoZSBjb2RlIGFuZCB0cnlpbmcgdG8gdW5kZXJzdGFuZCBob3cg dGhpcyBBUEkgYW5kIHJlbGF0ZWQgQVBJcyBmb3IgdGhlIG1vZHVsZSBoYXMgYmVlbiB1c2VkLCB0 aGUgQVBJJiM4MjE3O3MgaW4gMi4gYXJlIHVzZWQgZm9yIGJsb2JmcyBJL08gb3BlcmF0aW9uIHNw ZGsvbGliL2Jsb2IvYmRldi8qLCB3aGVuIHRoZXJlIGlzIGEgYmxvYiBmaWxlc3lzdGVtIGFuZCBm b3IgSS9PIG9wZXJhdGlvbgogb24gTlZNZSBkZXZpY2VzIHVzaW5nIE5WTWVvRiBwcm90b2NvbCBz cGRrL2xpYi9udm1mLyouPG86cD48L286cD48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPklzIHRo aXMgYSBjb3JyZWN0IGFzc3VtcHRpb24/IElzIHRoZSBsaWIvYmRldiBsYXllciBkZXZlbG9wZWQg dG8gYWRkcmVzcyBvbmx5IHRoZXNlIGNhc2VzPzxvOnA+PC9vOnA+PC9wPgo8cCBjbGFzcz0iTXNv Tm9ybWFsIj5DYW4gdGhlIGxpYi9iZGV2IEFQSSBiZSB1c2VkIG9uIGEgbG9jYWwgTlZNZSBkZXZp Y2U/IElmIHNvLCBpcyB0aGVyZSBhIHRlc3QgcHJvZ3JhbSB0byB1bmRlcnN0YW5kIGhvdyB0byBj cmVhdGUgYSBiZGV2IGRldmljZSBhbmQgZG8gdGhlIEkvTyBvbiBOVk1lIFNTRCBibG9jayBkZXZp Y2U/PG86cD48L286cD48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPlBsZWFzZSBsZXQgbWUga25v dyBpZiB0aGVyZSBpcyBhbiBleGFtcGxlIHRvIHVuZGVyc3RhbmQgaG93IHRvIHVzZSB0aGUgbGli L252bWYgaW50ZXJmYWNlcy4gSSB3YW50IHRvIHRlc3QgYSBkZXZpY2Ugb24gcmVtb3RlIHRhcmdl dCB1c2luZyBOVk1lb0YgcHJvdG9jb2wuPG86cD48L286cD48L3A+CjxwIGNsYXNzPSJNc29Ob3Jt YWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj5UaGFua3MgdmVy eSBtdWNoIGZvciBhbnkgaGVscCBpbiB0aGUgbGliL2JkZXYgQVBJLjxvOnA+PC9vOnA+PC9wPgo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4KPHAgY2xhc3M9Ik1zb05v cm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+CjwvZGl2Pgo8L2JvZHk+CjwvaHRtbD4K --===============1467021888500899230==--