From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0034968843711121044==" MIME-Version: 1.0 From: Lance Hartmann ORACLE Subject: [SPDK] Debugging /sys driver_override affecting driver binding Date: Tue, 26 Jun 2018 12:17:24 -0500 Message-ID: <28D485C5-DBA2-4CBE-8008-F385195FE39F@oracle.com> List-ID: To: spdk@lists.01.org --===============0034968843711121044== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable During my experimentation of unbinding NVMe controllers from the Linux nvme= driver and then binding them to vfio-pci for use with SPDK, I encountered = unusual behavior with one of the controllers. For some, initially inexpli= cable, reason, one of the NVMe controllers did get unbound as desired from = the nvme driver, but it refused to bind to vfio-pci, whereas all the other = NVMe controllers had no trouble at all binding to vfio-pci. Inspection of = the kernel log (dmesg) didn=E2=80=99t help. And so after a bunch of debug= ging I uncovered the culprit: the /sys driver attribute, driver_override.= By default, all of my NVMe controller=E2=80=99s appeared to have that at= tribute empty/null, e.g.: # cat /sys/bus/pci/devices/0000:40:00.0/driver_override (null) However, I discovered that for the NVMe controller that refused to bind to = vfio-pci, its driver_override attribute contained the string =E2=80=9Cnvme= =E2=80=9D: # cat /sys/bus/pci/devices/0000:40:00.0/driver_override nvme Per Linux kernel documentation, ABI/testing/sysfs-bus-pci: This file allows the driver for a device to be specified which will override standard static and dynamic ID matching. When specified, only a driver with a name matching the value written to driver_override will have an opportunity to bind to the device. =E2=80=A6 Eureka! So, that explains why I had a particular NVMe device that refused= to bind to vfio-pci. I wanted to share this discovery with other folks i= n case that run into a similar issue. Now, the mystery that remains: h= ow and why did this particular NVMe controller get its driver_override attr= ibute set to =E2=80=9Cnvme=E2=80=9D? It=E2=80=99s not being used as boot = device, I=E2=80=99ve never attempted to use it with LVM (Linux Logical Volu= me Management), nor built any file systems on it, or any such thing. I gr= ep=E2=80=99d through both my real rootfs=E2=80=99s /etc and searched throug= h my initramfs as well, but I=E2=80=99ve yet to discover what=E2=80=99s res= ponsible for setting that particular NVMe controller=E2=80=99s driver_overr= ide. Anyone have some ideas? thanks, -- Lance Hartmann lance.hartmann(a)oracle.com --===============0034968843711121044== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWw+PGhlYWQ+PC9oZWFkPjxib2R5PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiAnQXJpYWxN VCcsICdBcmlhbCc7IGZvbnQtc2l6ZTogMTNwdDsgY29sb3I6IHJnYmEoMCwgMCwgMCwgMS4wKTsi PjxkaXYgc3R5bGU9IndvcmQtd3JhcDogYnJlYWstd29yZDsgLXdlYmtpdC1uYnNwLW1vZGU6IHNw YWNlOyBsaW5lLWJyZWFrOiBhZnRlci13aGl0ZS1zcGFjZTsiIGNsYXNzPSIiPjxkaXYgY2xhc3M9 IiI+PGJyIGNsYXNzPSIiPjwvZGl2PjxkaXYgY2xhc3M9IiI+RHVyaW5nIG15IGV4cGVyaW1lbnRh dGlvbiBvZiB1bmJpbmRpbmcgTlZNZSBjb250cm9sbGVycyBmcm9tIHRoZSBMaW51eCBudm1lIGRy aXZlciBhbmQgdGhlbiBiaW5kaW5nIHRoZW0gdG8gdmZpby1wY2kgZm9yIHVzZSB3aXRoIFNQREss IEkgZW5jb3VudGVyZWQgdW51c3VhbCBiZWhhdmlvciB3aXRoIG9uZSBvZiB0aGUgY29udHJvbGxl cnMuICZuYnNwOyBGb3Igc29tZSwgaW5pdGlhbGx5IGluZXhwbGljYWJsZSwgcmVhc29uLCBvbmUg b2YgdGhlIE5WTWUgY29udHJvbGxlcnMgZGlkIGdldCB1bmJvdW5kIGFzIGRlc2lyZWQgZnJvbSB0 aGUgbnZtZSBkcml2ZXIsIGJ1dCBpdCByZWZ1c2VkIHRvIGJpbmQgdG8gdmZpby1wY2ksIHdoZXJl YXMgYWxsIHRoZSBvdGhlciBOVk1lIGNvbnRyb2xsZXJzIGhhZCBubyB0cm91YmxlIGF0IGFsbCBi aW5kaW5nIHRvIHZmaW8tcGNpLiAmbmJzcDtJbnNwZWN0aW9uIG9mIHRoZSBrZXJuZWwgbG9nIChk bWVzZykgZGlkbuKAmXQgaGVscC4gJm5ic3A7IEFuZCBzbyBhZnRlciBhIGJ1bmNoIG9mIGRlYnVn Z2luZyBJIHVuY292ZXJlZCB0aGUgY3VscHJpdDogJm5ic3A7IHRoZSAvc3lzIGRyaXZlciBhdHRy aWJ1dGUsIDxmb250IGZhY2U9IkNvdXJpZXIgTmV3IiBjbGFzcz0iIj5kcml2ZXJfb3ZlcnJpZGU8 L2ZvbnQ+LiAmbmJzcDsgQnkgZGVmYXVsdCwgYWxsIG9mIG15IE5WTWUgY29udHJvbGxlcuKAmXMg YXBwZWFyZWQgdG8gaGF2ZSB0aGF0IGF0dHJpYnV0ZSBlbXB0eS9udWxsLCBlLmcuOjwvZGl2Pjxk aXYgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPjwvZGl2PjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46 IDAgMCAwIDQwcHg7IGJvcmRlcjogbm9uZTsgcGFkZGluZzogMHB4OyIgY2xhc3M9IiI+PGRpdiBj bGFzcz0iIj48ZGl2IGNsYXNzPSIiPjxmb250IGZhY2U9IkNvdXJpZXIgTmV3IiBjbGFzcz0iIj4j IGNhdCAvc3lzL2J1cy9wY2kvZGV2aWNlcy8wMDAwOjQwOjAwLjAvZHJpdmVyX292ZXJyaWRlPC9m b250PjwvZGl2PjwvZGl2PjxkaXYgY2xhc3M9IiI+PGRpdiBjbGFzcz0iIj48Zm9udCBmYWNlPSJD b3VyaWVyIE5ldyIgY2xhc3M9IiI+KG51bGwpPC9mb250PjwvZGl2PjwvZGl2PjwvYmxvY2txdW90 ZT48ZGl2IGNsYXNzPSIiPjxiciBjbGFzcz0iIj48L2Rpdj5Ib3dldmVyLCBJIGRpc2NvdmVyZWQg dGhhdCBmb3IgdGhlIE5WTWUgY29udHJvbGxlciB0aGF0IHJlZnVzZWQgdG8gYmluZCB0byB2Zmlv LXBjaSwgaXRzIDxmb250IGZhY2U9IkNvdXJpZXIgTmV3IiBjbGFzcz0iIj5kcml2ZXJfb3ZlcnJp ZGU8L2ZvbnQ+IGF0dHJpYnV0ZSBjb250YWluZWQgdGhlIHN0cmluZyDigJxudm1l4oCdOjxkaXYg Y2xhc3M9IiI+PGJyIGNsYXNzPSIiPjwvZGl2PjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46IDAg MCAwIDQwcHg7IGJvcmRlcjogbm9uZTsgcGFkZGluZzogMHB4OyIgY2xhc3M9IiI+PGRpdiBjbGFz cz0iIj48Zm9udCBmYWNlPSJDb3VyaWVyIE5ldyIgY2xhc3M9IiI+IyBjYXQgL3N5cy9idXMvcGNp L2RldmljZXMvMDAwMDo0MDowMC4wL2RyaXZlcl9vdmVycmlkZTwvZm9udD48L2Rpdj48ZGl2IGNs YXNzPSIiPjxmb250IGZhY2U9IkNvdXJpZXIgTmV3IiBjbGFzcz0iIj5udm1lPC9mb250PjwvZGl2 PjwvYmxvY2txdW90ZT48ZGl2IGNsYXNzPSIiPjxiciBjbGFzcz0iIj48L2Rpdj48ZGl2IGNsYXNz PSIiPlBlciBMaW51eCBrZXJuZWwgZG9jdW1lbnRhdGlvbiwgQUJJL3Rlc3Rpbmcvc3lzZnMtYnVz LXBjaTo8YnIgY2xhc3M9IiI+PGRpdiBjbGFzcz0iIj48YnIgY2xhc3M9IndlYmtpdC1ibG9jay1w bGFjZWhvbGRlciI+PC9kaXY+PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbjogMCAwIDAgNDBweDsg Ym9yZGVyOiBub25lOyBwYWRkaW5nOiAwcHg7IiBjbGFzcz0iIj48ZGl2IGNsYXNzPSIiPjxpIGNs YXNzPSIiPlRoaXMgZmlsZSBhbGxvd3MgdGhlIGRyaXZlciBmb3IgYSBkZXZpY2UgdG8gYmUgc3Bl Y2lmaWVkIHdoaWNoPC9pPjwvZGl2PjxkaXYgY2xhc3M9IiI+PGkgY2xhc3M9IiI+d2lsbCBvdmVy cmlkZSBzdGFuZGFyZCBzdGF0aWMgYW5kIGR5bmFtaWMgSUQgbWF0Y2hpbmcuICZuYnNwO1doZW48 L2k+PC9kaXY+PGRpdiBjbGFzcz0iIj48aSBjbGFzcz0iIj5zcGVjaWZpZWQsIG9ubHkgYSBkcml2 ZXIgd2l0aCBhIG5hbWUgbWF0Y2hpbmcgdGhlIHZhbHVlIHdyaXR0ZW48L2k+PC9kaXY+PGRpdiBj bGFzcz0iIj48aSBjbGFzcz0iIj50byBkcml2ZXJfb3ZlcnJpZGUgd2lsbCBoYXZlIGFuIG9wcG9y dHVuaXR5IHRvIGJpbmQgdG8gdGhlPC9pPjwvZGl2PjxkaXYgY2xhc3M9IiI+PGkgY2xhc3M9IiI+ ZGV2aWNlLjwvaT48L2Rpdj48ZGl2IGNsYXNzPSIiPjxpIGNsYXNzPSIiPuKApjwvaT48L2Rpdj48 L2Jsb2NrcXVvdGU+PGRpdiBjbGFzcz0iIj48YnIgY2xhc3M9IiI+PC9kaXY+PGRpdiBjbGFzcz0i Ij5FdXJla2EhICZuYnNwOyBTbywgdGhhdCBleHBsYWlucyB3aHkgSSBoYWQgYSBwYXJ0aWN1bGFy IE5WTWUgZGV2aWNlIHRoYXQgcmVmdXNlZCB0byBiaW5kIHRvIHZmaW8tcGNpLiAmbmJzcDsgSSB3 YW50ZWQgdG8gc2hhcmUgdGhpcyBkaXNjb3Zlcnkgd2l0aCBvdGhlciBmb2xrcyBpbiBjYXNlIHRo YXQgcnVuIGludG8gYSBzaW1pbGFyIGlzc3VlLiAmbmJzcDsgTm93LCB0aGUgbXlzdGVyeSB0aGF0 IHJlbWFpbnM6ICZuYnNwOyAmbmJzcDtob3cgYW5kIHdoeSBkaWQgdGhpcyBwYXJ0aWN1bGFyIE5W TWUgY29udHJvbGxlciBnZXQgaXRzIDxmb250IGZhY2U9IkNvdXJpZXIgTmV3IiBjbGFzcz0iIj5k cml2ZXJfb3ZlcnJpZGU8L2ZvbnQ+IGF0dHJpYnV0ZSBzZXQgdG8g4oCcbnZtZeKAnT8gJm5ic3A7 IEl04oCZcyBub3QgYmVpbmcgdXNlZCBhcyBib290IGRldmljZSwgSeKAmXZlIG5ldmVyIGF0dGVt cHRlZCB0byB1c2UgaXQgd2l0aCBMVk0gKExpbnV4IExvZ2ljYWwgVm9sdW1lIE1hbmFnZW1lbnQp LCBub3IgYnVpbHQgYW55IGZpbGUgc3lzdGVtcyBvbiBpdCwgb3IgYW55IHN1Y2ggdGhpbmcuICZu YnNwOyBJIGdyZXDigJlkIHRocm91Z2ggYm90aCBteSByZWFsIHJvb3Rmc+KAmXMgPGZvbnQgZmFj ZT0iQ291cmllciBOZXciIGNsYXNzPSIiPi9ldGM8L2ZvbnQ+IGFuZCBzZWFyY2hlZCB0aHJvdWdo IG15IGluaXRyYW1mcyBhcyB3ZWxsLCBidXQgSeKAmXZlIHlldCB0byBkaXNjb3ZlciB3aGF04oCZ cyByZXNwb25zaWJsZSBmb3Igc2V0dGluZyB0aGF0IHBhcnRpY3VsYXIgTlZNZSBjb250cm9sbGVy 4oCZcyA8Zm9udCBmYWNlPSJDb3VyaWVyIE5ldyIgY2xhc3M9IiI+ZHJpdmVyX292ZXJyaWRlPC9m b250Pi4gJm5ic3A7ICZuYnNwO0FueW9uZSBoYXZlIHNvbWUgaWRlYXM/PC9kaXY+PGRpdiBjbGFz cz0iIj48YnIgY2xhc3M9IiI+PC9kaXY+PGRpdiBjbGFzcz0iIj50aGFua3MsPC9kaXY+PGRpdiBj bGFzcz0iIj48c3BhbiBpZD0idW1fc2lnbmF0dXJlX21hcmtlcl9iZWdpbiIgY2xhc3M9IiI+Cjxw IHN0eWxlPSJmb250LXN0eWxlOm5vcm1hbCIgY2xhc3M9IiI+Ci0tPGJyIGNsYXNzPSIiPgpMYW5j ZSBIYXJ0bWFubjxiciBjbGFzcz0iIj4KPGEgaHJlZj0ibWFpbHRvOmxhbmNlLmhhcnRtYW5uQG9y YWNsZS5jb20iIGNsYXNzPSIiPmxhbmNlLmhhcnRtYW5uQG9yYWNsZS5jb208L2E+CjwvcD4KPHNw YW4gaWQ9InVtX3NpZ25hdHVyZV9tYXJrZXJfZW5kIiBjbGFzcz0iIj48L3NwYW4+PC9zcGFuPjwv ZGl2Pgo8YnIgY2xhc3M9IiI+PC9kaXY+PC9kaXY+PC9kaXY+PC9ib2R5PjwvaHRtbD4K --===============0034968843711121044==--