From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============1568716594075540538==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] Debugging /sys driver_override affecting driver binding Date: Mon, 02 Jul 2018 16:54:38 +0000 Message-ID: <0576F8C5-AB9F-4F50-BD77-B99EB4AF800A@intel.com> In-Reply-To: 28D485C5-DBA2-4CBE-8008-F385195FE39F@oracle.com List-ID: To: spdk@lists.01.org --===============1568716594075540538== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Lance, I haven=E2=80=99t seen this problem before. Does this system have multiple= NVMe devices, and only one (or a subset) of them has driver_override set? I notice that the two examples you gave were both for PCI BDF 40:00.0 =E2= =80=93 I assume that means this was collected on two different systems? https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci seems to= indicate that something must be writing =E2=80=9Cnvme=E2=80=9D to that fil= e =E2=80=93 i.e. it=E2=80=99s not done internally by the kernel. Maybe our= setup.sh script should at least check for this =E2=80=93 i.e. don=E2=80=99= t reset driver_override to (null) but print a warning message if it finds i= t set to =E2=80=9Cnvme=E2=80=9D. -Jim From: SPDK on behalf of Lance Hartmann ORACLE= Reply-To: Storage Performance Development Kit Date: Tuesday, June 26, 2018 at 10:17 AM To: Storage Performance Development Kit Subject: [SPDK] Debugging /sys driver_override affecting driver binding During my experimentation of unbinding NVMe controllers from the Linux nvme= driver and then binding them to vfio-pci for use with SPDK, I encountered = unusual behavior with one of the controllers. For some, initially inexpli= cable, reason, one of the NVMe controllers did get unbound as desired from = the nvme driver, but it refused to bind to vfio-pci, whereas all the other = NVMe controllers had no trouble at all binding to vfio-pci. Inspection of = the kernel log (dmesg) didn=E2=80=99t help. And so after a bunch of debug= ging I uncovered the culprit: the /sys driver attribute, driver_override.= By default, all of my NVMe controller=E2=80=99s appeared to have that at= tribute empty/null, e.g.: # cat /sys/bus/pci/devices/0000:40:00.0/driver_override (null) However, I discovered that for the NVMe controller that refused to bind to = vfio-pci, its driver_override attribute contained the string =E2=80=9Cnvme= =E2=80=9D: # cat /sys/bus/pci/devices/0000:40:00.0/driver_override nvme Per Linux kernel documentation, ABI/testing/sysfs-bus-pci: This file allows the driver for a device to be specified which will override standard static and dynamic ID matching. When specified, only a driver with a name matching the value written to driver_override will have an opportunity to bind to the device. =E2=80=A6 Eureka! So, that explains why I had a particular NVMe device that refused= to bind to vfio-pci. I wanted to share this discovery with other folks i= n case that run into a similar issue. Now, the mystery that remains: h= ow and why did this particular NVMe controller get its driver_override attr= ibute set to =E2=80=9Cnvme=E2=80=9D? It=E2=80=99s not being used as boot = device, I=E2=80=99ve never attempted to use it with LVM (Linux Logical Volu= me Management), nor built any file systems on it, or any such thing. I gr= ep=E2=80=99d through both my real rootfs=E2=80=99s /etc and searched throug= h my initramfs as well, but I=E2=80=99ve yet to discover what=E2=80=99s res= ponsible for setting that particular NVMe controller=E2=80=99s driver_overr= ide. Anyone have some ideas? thanks, -- Lance Hartmann lance.hartmann(a)oracle.com --===============1568716594075540538== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4 bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l dGEgbmFtZT0iVGl0bGUiIGNvbnRlbnQ9IiI+DQo8bWV0YSBuYW1lPSJLZXl3b3JkcyIgY29udGVu dD0iIj4NCjxtZXRhIG5hbWU9IkdlbmVyYXRvciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUg KGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxlPjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8N CkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6QXJpYWw7DQoJcGFub3NlLTE6MiAxMSA2IDQgMiAy IDIgMiAyIDQ7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWlseToiQ291cmllciBOZXciOw0KCXBh bm9zZS0xOjIgNyAzIDkgMiAyIDUgMiA0IDQ7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWlseToi Q2FtYnJpYSBNYXRoIjsNCglwYW5vc2UtMToyIDQgNSAzIDUgNCA2IDMgMiA0O30NCkBmb250LWZh Y2UNCgl7Zm9udC1mYW1pbHk6RGVuZ1hpYW47DQoJcGFub3NlLTE6MiAxIDYgMCAzIDEgMSAxIDEg MTt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3NlLTE6MiAxNSA1 IDIgMiAyIDQgMyAyIDQ7fQ0KLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8NCnAuTXNvTm9ybWFsLCBs aS5Nc29Ob3JtYWwsIGRpdi5Nc29Ob3JtYWwNCgl7bWFyZ2luOjBpbjsNCgltYXJnaW4tYm90dG9t Oi4wMDAxcHQ7DQoJZm9udC1zaXplOjEyLjBwdDsNCglmb250LWZhbWlseToiVGltZXMgTmV3IFJv bWFuIjt9DQphOmxpbmssIHNwYW4uTXNvSHlwZXJsaW5rDQoJe21zby1zdHlsZS1wcmlvcml0eTo5 OTsNCgljb2xvcjpibHVlOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7fQ0KYTp2aXNpdGVk LCBzcGFuLk1zb0h5cGVybGlua0ZvbGxvd2VkDQoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsNCglj b2xvcjpwdXJwbGU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQpzcGFuLkVtYWlsU3R5 bGUxNw0KCXttc28tc3R5bGUtdHlwZTpwZXJzb25hbC1yZXBseTsNCglmb250LWZhbWlseTpDYWxp YnJpOw0KCWNvbG9yOndpbmRvd3RleHQ7fQ0Kc3Bhbi5tc29JbnMNCgl7bXNvLXN0eWxlLXR5cGU6 ZXhwb3J0LW9ubHk7DQoJbXNvLXN0eWxlLW5hbWU6IiI7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVy bGluZTsNCgljb2xvcjp0ZWFsO30NCi5Nc29DaHBEZWZhdWx0DQoJe21zby1zdHlsZS10eXBlOmV4 cG9ydC1vbmx5Ow0KCWZvbnQtc2l6ZToxMC4wcHQ7fQ0KQHBhZ2UgV29yZFNlY3Rpb24xDQoJe3Np emU6OC41aW4gMTEuMGluOw0KCW1hcmdpbjoxLjBpbiAxLjBpbiAxLjBpbiAxLjBpbjt9DQpkaXYu V29yZFNlY3Rpb24xDQoJe3BhZ2U6V29yZFNlY3Rpb24xO30NCi0tPjwvc3R5bGU+DQo8L2hlYWQ+ DQo8Ym9keSBiZ2NvbG9yPSJ3aGl0ZSIgbGFuZz0iRU4tVVMiIGxpbms9ImJsdWUiIHZsaW5rPSJw dXJwbGUiPg0KPGRpdiBjbGFzcz0iV29yZFNlY3Rpb24xIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmkiPkhpIExh bmNlLDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0 eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmkiPjxvOnA+Jm5ic3A7PC9v OnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNp emU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmkiPkkgaGF2ZW7igJl0IHNlZW4gdGhpcyBwcm9i bGVtIGJlZm9yZS4mbmJzcDsgRG9lcyB0aGlzIHN5c3RlbSBoYXZlIG11bHRpcGxlIE5WTWUgZGV2 aWNlcywgYW5kIG9ubHkgb25lIChvciBhIHN1YnNldCkgb2YgdGhlbSBoYXMgZHJpdmVyX292ZXJy aWRlIHNldD88bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpIj48bzpwPiZuYnNw OzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9u dC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpIj5JIG5vdGljZSB0aGF0IHRoZSB0d28g ZXhhbXBsZXMgeW91IGdhdmUgd2VyZSBib3RoIGZvciBQQ0kgQkRGIDQwOjAwLjAg4oCTIEkgYXNz dW1lIHRoYXQgbWVhbnMgdGhpcyB3YXMgY29sbGVjdGVkIG9uIHR3byBkaWZmZXJlbnQgc3lzdGVt cz88bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHls ZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpIj48bzpwPiZuYnNwOzwvbzpw Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXpl OjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpIj48YSBocmVmPSJodHRwczovL3d3dy5rZXJuZWwu b3JnL2RvYy9Eb2N1bWVudGF0aW9uL0FCSS90ZXN0aW5nL3N5c2ZzLWJ1cy1wY2kiPmh0dHBzOi8v d3d3Lmtlcm5lbC5vcmcvZG9jL0RvY3VtZW50YXRpb24vQUJJL3Rlc3Rpbmcvc3lzZnMtYnVzLXBj aTwvYT4gc2VlbXMgdG8gaW5kaWNhdGUgdGhhdCBzb21ldGhpbmcgbXVzdA0KIGJlIHdyaXRpbmcg 4oCcbnZtZeKAnSB0byB0aGF0IGZpbGUg4oCTIGkuZS4gaXTigJlzIG5vdCBkb25lIGludGVybmFs bHkgYnkgdGhlIGtlcm5lbC4mbmJzcDsgTWF5YmUgb3VyIHNldHVwLnNoIHNjcmlwdCBzaG91bGQg YXQgbGVhc3QgY2hlY2sgZm9yIHRoaXMg4oCTIGkuZS4gZG9u4oCZdCByZXNldCBkcml2ZXJfb3Zl cnJpZGUgdG8gKG51bGwpIGJ1dCBwcmludCBhIHdhcm5pbmcgbWVzc2FnZSBpZiBpdCBmaW5kcyBp dCBzZXQgdG8g4oCcbnZtZeKAnS48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJp Ij48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpIj4tSmltPG86cD48 L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQt c2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaSI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+ PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7 Zm9udC1mYW1pbHk6Q2FsaWJyaSI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPGRpdiBz dHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLXRvcDpzb2xpZCAjQjVDNERGIDEuMHB0O3BhZGRpbmc6 My4wcHQgMGluIDBpbiAwaW4iPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1s ZWZ0Oi41aW4iPjxiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTpDYWxpYnJpO2NvbG9yOmJsYWNr Ij5Gcm9tOg0KPC9zcGFuPjwvYj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6Q2FsaWJyaTtjb2xv cjpibGFjayI+U1BESyAmbHQ7c3Bkay1ib3VuY2VzQGxpc3RzLjAxLm9yZyZndDsgb24gYmVoYWxm IG9mIExhbmNlIEhhcnRtYW5uIE9SQUNMRSAmbHQ7bGFuY2UuaGFydG1hbm5Ab3JhY2xlLmNvbSZn dDs8YnI+DQo8Yj5SZXBseS1UbzogPC9iPlN0b3JhZ2UgUGVyZm9ybWFuY2UgRGV2ZWxvcG1lbnQg S2l0ICZsdDtzcGRrQGxpc3RzLjAxLm9yZyZndDs8YnI+DQo8Yj5EYXRlOiA8L2I+VHVlc2RheSwg SnVuZSAyNiwgMjAxOCBhdCAxMDoxNyBBTTxicj4NCjxiPlRvOiA8L2I+U3RvcmFnZSBQZXJmb3Jt YW5jZSBEZXZlbG9wbWVudCBLaXQgJmx0O3NwZGtAbGlzdHMuMDEub3JnJmd0Ozxicj4NCjxiPlN1 YmplY3Q6IDwvYj5bU1BES10gRGVidWdnaW5nIC9zeXMgZHJpdmVyX292ZXJyaWRlIGFmZmVjdGlu ZyBkcml2ZXIgYmluZGluZzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj48bzpwPiZuYnNwOzwv bzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1h bCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTMuMHB0 O2ZvbnQtZmFtaWx5OkFyaWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4N CjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6QXJpYWwiPkR1cmluZyBteSBl eHBlcmltZW50YXRpb24gb2YgdW5iaW5kaW5nIE5WTWUgY29udHJvbGxlcnMgZnJvbSB0aGUgTGlu dXggbnZtZSBkcml2ZXIgYW5kIHRoZW4gYmluZGluZyB0aGVtIHRvIHZmaW8tcGNpIGZvciB1c2Ug d2l0aCBTUERLLCBJIGVuY291bnRlcmVkIHVudXN1YWwgYmVoYXZpb3INCiB3aXRoIG9uZSBvZiB0 aGUgY29udHJvbGxlcnMuICZuYnNwOyBGb3Igc29tZSwgaW5pdGlhbGx5IGluZXhwbGljYWJsZSwg cmVhc29uLCBvbmUgb2YgdGhlIE5WTWUgY29udHJvbGxlcnMgZGlkIGdldCB1bmJvdW5kIGFzIGRl c2lyZWQgZnJvbSB0aGUgbnZtZSBkcml2ZXIsIGJ1dCBpdCByZWZ1c2VkIHRvIGJpbmQgdG8gdmZp by1wY2ksIHdoZXJlYXMgYWxsIHRoZSBvdGhlciBOVk1lIGNvbnRyb2xsZXJzIGhhZCBubyB0cm91 YmxlIGF0IGFsbCBiaW5kaW5nIHRvDQogdmZpby1wY2kuICZuYnNwO0luc3BlY3Rpb24gb2YgdGhl IGtlcm5lbCBsb2cgKGRtZXNnKSBkaWRu4oCZdCBoZWxwLiAmbmJzcDsgQW5kIHNvIGFmdGVyIGEg YnVuY2ggb2YgZGVidWdnaW5nIEkgdW5jb3ZlcmVkIHRoZSBjdWxwcml0OiAmbmJzcDsgdGhlIC9z eXMgZHJpdmVyIGF0dHJpYnV0ZSwNCjwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEzLjBw dDtmb250LWZhbWlseTomcXVvdDtDb3VyaWVyIE5ldyZxdW90OyI+ZHJpdmVyX292ZXJyaWRlPC9z cGFuPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTMuMHB0O2ZvbnQtZmFtaWx5OkFyaWFsIj4uICZu YnNwOyBCeSBkZWZhdWx0LCBhbGwgb2YgbXkgTlZNZSBjb250cm9sbGVy4oCZcyBhcHBlYXJlZCB0 byBoYXZlIHRoYXQgYXR0cmlidXRlIGVtcHR5L251bGwsIGUuZy46PG86cD48L286cD48L3NwYW4+ PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1s ZWZ0Oi41aW4iPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTMuMHB0O2ZvbnQtZmFtaWx5OkFyaWFs Ij48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxibG9ja3F1b3RlIHN0eWxl PSJtYXJnaW4tbGVmdDozMC4wcHQ7bWFyZ2luLXJpZ2h0OjBpbiI+DQo8ZGl2Pg0KPGRpdj4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj48c3BhbiBzdHlsZT0i Zm9udC1zaXplOjEzLjBwdDtmb250LWZhbWlseTomcXVvdDtDb3VyaWVyIE5ldyZxdW90OyI+IyBj YXQgL3N5cy9idXMvcGNpL2RldmljZXMvMDAwMDo0MDowMC4wL2RyaXZlcl9vdmVycmlkZTwvc3Bh bj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEzLjBwdDtmb250LWZhbWlseTpBcmlhbCI+PG86cD48 L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8L2Rpdj4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPjxzcGFuIHN0eWxlPSJmb250LXNp emU6MTMuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NvdXJpZXIgTmV3JnF1b3Q7Ij4obnVsbCk8L3Nw YW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6QXJpYWwiPjxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPC9kaXY+DQo8L2Jsb2NrcXVvdGU+DQo8ZGl2Pg0K PHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPjxzcGFuIHN0eWxl PSJmb250LXNpemU6MTMuMHB0O2ZvbnQtZmFtaWx5OkFyaWFsIj48bzpwPiZuYnNwOzwvbzpwPjwv c3Bhbj48L3A+DQo8L2Rpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVm dDouNWluIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEzLjBwdDtmb250LWZhbWlseTpBcmlhbCI+ SG93ZXZlciwgSSBkaXNjb3ZlcmVkIHRoYXQgZm9yIHRoZSBOVk1lIGNvbnRyb2xsZXIgdGhhdCBy ZWZ1c2VkIHRvIGJpbmQgdG8gdmZpby1wY2ksIGl0cw0KPC9zcGFuPjxzcGFuIHN0eWxlPSJmb250 LXNpemU6MTMuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NvdXJpZXIgTmV3JnF1b3Q7Ij5kcml2ZXJf b3ZlcnJpZGU8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6 QXJpYWwiPiBhdHRyaWJ1dGUgY29udGFpbmVkIHRoZSBzdHJpbmcg4oCcbnZtZeKAnToNCjxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFy Z2luLWxlZnQ6LjVpbiI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6 QXJpYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGJsb2NrcXVvdGUg c3R5bGU9Im1hcmdpbi1sZWZ0OjMwLjBwdDttYXJnaW4tcmlnaHQ6MGluIj4NCjxkaXY+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q291cmllciBOZXcmcXVvdDsiPiMgY2F0 IC9zeXMvYnVzL3BjaS9kZXZpY2VzLzAwMDA6NDA6MDAuMC9kcml2ZXJfb3ZlcnJpZGU8L3NwYW4+ PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6QXJpYWwiPjxvOnA+PC9v OnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxl PSJtYXJnaW4tbGVmdDouNWluIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEzLjBwdDtmb250LWZh bWlseTomcXVvdDtDb3VyaWVyIE5ldyZxdW90OyI+bnZtZTwvc3Bhbj48c3BhbiBzdHlsZT0iZm9u dC1zaXplOjEzLjBwdDtmb250LWZhbWlseTpBcmlhbCI+PG86cD48L286cD48L3NwYW4+PC9wPg0K PC9kaXY+DQo8L2Jsb2NrcXVvdGU+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9 Im1hcmdpbi1sZWZ0Oi41aW4iPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTMuMHB0O2ZvbnQtZmFt aWx5OkFyaWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6QXJpYWwiPlBlciBMaW51eCBrZXJuZWwgZG9j dW1lbnRhdGlvbiwgQUJJL3Rlc3Rpbmcvc3lzZnMtYnVzLXBjaTo8bzpwPjwvbzpwPjwvc3Bhbj48 L3A+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4i PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTMuMHB0O2ZvbnQtZmFtaWx5OkFyaWFsIj48bzpwPiZu YnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW4t bGVmdDozMC4wcHQ7bWFyZ2luLXJpZ2h0OjBpbiI+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1h bCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPjxpPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTMu MHB0O2ZvbnQtZmFtaWx5OkFyaWFsIj5UaGlzIGZpbGUgYWxsb3dzIHRoZSBkcml2ZXIgZm9yIGEg ZGV2aWNlIHRvIGJlIHNwZWNpZmllZCB3aGljaDwvc3Bhbj48L2k+PHNwYW4gc3R5bGU9ImZvbnQt c2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6QXJpYWwiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwv ZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWlu Ij48aT48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEzLjBwdDtmb250LWZhbWlseTpBcmlhbCI+d2ls bCBvdmVycmlkZSBzdGFuZGFyZCBzdGF0aWMgYW5kIGR5bmFtaWMgSUQgbWF0Y2hpbmcuICZuYnNw O1doZW48L3NwYW4+PC9pPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTMuMHB0O2ZvbnQtZmFtaWx5 OkFyaWFsIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+PGk+PHNwYW4gc3R5bGU9ImZvbnQt c2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6QXJpYWwiPnNwZWNpZmllZCwgb25seSBhIGRyaXZlciB3 aXRoIGEgbmFtZSBtYXRjaGluZyB0aGUgdmFsdWUgd3JpdHRlbjwvc3Bhbj48L2k+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6QXJpYWwiPjxvOnA+PC9vOnA+PC9zcGFu PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4t bGVmdDouNWluIj48aT48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEzLjBwdDtmb250LWZhbWlseTpB cmlhbCI+dG8gZHJpdmVyX292ZXJyaWRlIHdpbGwgaGF2ZSBhbiBvcHBvcnR1bml0eSB0byBiaW5k IHRvIHRoZTwvc3Bhbj48L2k+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1p bHk6QXJpYWwiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWluIj48aT48c3BhbiBzdHlsZT0iZm9u dC1zaXplOjEzLjBwdDtmb250LWZhbWlseTpBcmlhbCI+ZGV2aWNlLjwvc3Bhbj48L2k+PHNwYW4g c3R5bGU9ImZvbnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6QXJpYWwiPjxvOnA+PC9vOnA+PC9z cGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJn aW4tbGVmdDouNWluIj48aT48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEzLjBwdDtmb250LWZhbWls eTpBcmlhbCI+4oCmPC9zcGFuPjwvaT48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEzLjBwdDtmb250 LWZhbWlseTpBcmlhbCI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8L2Jsb2NrcXVv dGU+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4i PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTMuMHB0O2ZvbnQtZmFtaWx5OkFyaWFsIj48bzpwPiZu YnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFs IiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMy4wcHQ7 Zm9udC1mYW1pbHk6QXJpYWwiPkV1cmVrYSEgJm5ic3A7IFNvLCB0aGF0IGV4cGxhaW5zIHdoeSBJ IGhhZCBhIHBhcnRpY3VsYXIgTlZNZSBkZXZpY2UgdGhhdCByZWZ1c2VkIHRvIGJpbmQgdG8gdmZp by1wY2kuICZuYnNwOyBJIHdhbnRlZCB0byBzaGFyZSB0aGlzIGRpc2NvdmVyeSB3aXRoIG90aGVy IGZvbGtzIGluIGNhc2UgdGhhdA0KIHJ1biBpbnRvIGEgc2ltaWxhciBpc3N1ZS4gJm5ic3A7IE5v dywgdGhlIG15c3RlcnkgdGhhdCByZW1haW5zOiAmbmJzcDsgJm5ic3A7aG93IGFuZCB3aHkgZGlk IHRoaXMgcGFydGljdWxhciBOVk1lIGNvbnRyb2xsZXIgZ2V0IGl0cw0KPC9zcGFuPjxzcGFuIHN0 eWxlPSJmb250LXNpemU6MTMuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NvdXJpZXIgTmV3JnF1b3Q7 Ij5kcml2ZXJfb3ZlcnJpZGU8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMy4wcHQ7Zm9u dC1mYW1pbHk6QXJpYWwiPiBhdHRyaWJ1dGUgc2V0IHRvIOKAnG52bWXigJ0/ICZuYnNwOyBJdOKA mXMgbm90IGJlaW5nIHVzZWQgYXMgYm9vdCBkZXZpY2UsIEnigJl2ZSBuZXZlciBhdHRlbXB0ZWQg dG8gdXNlIGl0IHdpdGggTFZNIChMaW51eCBMb2dpY2FsIFZvbHVtZQ0KIE1hbmFnZW1lbnQpLCBu b3IgYnVpbHQgYW55IGZpbGUgc3lzdGVtcyBvbiBpdCwgb3IgYW55IHN1Y2ggdGhpbmcuICZuYnNw OyBJIGdyZXDigJlkIHRocm91Z2ggYm90aCBteSByZWFsIHJvb3Rmc+KAmXMNCjwvc3Bhbj48c3Bh biBzdHlsZT0iZm9udC1zaXplOjEzLjBwdDtmb250LWZhbWlseTomcXVvdDtDb3VyaWVyIE5ldyZx dW90OyI+L2V0Yzwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEzLjBwdDtmb250LWZhbWls eTpBcmlhbCI+IGFuZCBzZWFyY2hlZCB0aHJvdWdoIG15IGluaXRyYW1mcyBhcyB3ZWxsLCBidXQg SeKAmXZlIHlldCB0byBkaXNjb3ZlciB3aGF04oCZcyByZXNwb25zaWJsZSBmb3Igc2V0dGluZyB0 aGF0IHBhcnRpY3VsYXIgTlZNZSBjb250cm9sbGVy4oCZcw0KPC9zcGFuPjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTMuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NvdXJpZXIgTmV3JnF1b3Q7Ij5kcml2 ZXJfb3ZlcnJpZGU8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1p bHk6QXJpYWwiPi4gJm5ic3A7ICZuYnNwO0FueW9uZSBoYXZlIHNvbWUgaWRlYXM/PG86cD48L286 cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9 Im1hcmdpbi1sZWZ0Oi41aW4iPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTMuMHB0O2ZvbnQtZmFt aWx5OkFyaWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMy4wcHQ7Zm9udC1mYW1pbHk6QXJpYWwiPnRoYW5rcyw8bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNv LW1hcmdpbi10b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG87bWFyZ2luLWxl ZnQ6LjVpbiI+DQo8c3BhbiBzdHlsZT0iZm9udC1zaXplOjEzLjBwdDtmb250LWZhbWlseTpBcmlh bCI+LS08YnI+DQpMYW5jZSBIYXJ0bWFubjxicj4NCjxhIGhyZWY9Im1haWx0bzpsYW5jZS5oYXJ0 bWFubkBvcmFjbGUuY29tIj5sYW5jZS5oYXJ0bWFubkBvcmFjbGUuY29tPC9hPiA8bzpwPjwvbzpw Pjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4t bGVmdDouNWluIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEzLjBwdDtmb250LWZhbWlseTpBcmlh bCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0K PC9kaXY+DQo8L2JvZHk+DQo8L2h0bWw+DQo= --===============1568716594075540538==--