Re: Permanently binding NIC ports with DPDK drivers

[Panu Matilainen <pmatilai@redhat.com>]

On 11/11/2015 06:28 PM, Bruce Richardson wrote:
> On Wed, Nov 11, 2015 at 04:13:01PM +0000, Montorsi, Francesco wrote:
>> Hi,
>> Is there a way to permanently (i.e., have the configuration automatically applied after reboot) bind a NIC port to DPDK?
>>
>> In case there's none, I'm thinking to save in my software a list of the NIC ports chosen by the user for use with DPDK and then, upon software startup to just do
>>      for (int i=0; i < ...; i++)
>>       system("dpdk_nic_bind.py --bind=igb_uio " + PCI_device_chosen[i]);
>> Do you see any problem with that?
>>
>> Thanks!
>> Francesco Montorsi
>>
>
> Hi Francesco,
>
> I'm not aware of any way to make the bindings permanent across reboots. What you
> have suggested will work, but there are probably better ways to do the same thing.
> For example, a couple of lines in an rc.local script can reapply the bindings at
> boot for you. I'm sure others can suggest other ways of having the same effect,
> for example, there may be a way to automatically do this using udev or systemd
> or some such package.

I've been looking into this recently, here's what I have so far:
http://laiskiainen.org/git/?p=driverctl.git

For the impatient, "make rpm" should produce something usable for recent 
Fedora/RHEL systems, usage looks somewhat like this:

Find devices currently driven by ixgbe driver:
# driverctl -v list-devices | grep ixgbe
0000:01:00.0 ixgbe (Ethernet 10G 4P X520/I350 rNDC)
0000:01:00.1 ixgbe (Ethernet 10G 4P X520/I350 rNDC)

Change them to use the vfio-pci driver permanently:
# driverctl set-override 0000:01:00.0 vfio-pci
# driverctl set-override 0000:01:00.1 vfio-pci

Find devices with driver overrides:
[root@wsfd-netdev32 ~]# driverctl -v list-devices|grep \*
0000:01:00.0 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)
0000:01:00.1 vfio-pci [*] (Ethernet 10G 4P X520/I350 rNDC)

Remove the permanent driver override for device 0000:01:00.1:
# driverctl unset-override 0000:01:00.1

In addition it has udev rules to export vfio and uio devices on systemd 
level, eg the above looks like this with normal drivers:

# systemctl |grep 0000:01:00
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.0-net-em1.device 
                  loaded active plugged   Ethernet 10G 4P X520/I350 rNDC
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.1-net-em2.device 
                  loaded active plugged   Ethernet 10G 4P X520/I350 rNDC

When changed to vfio, with upstream systemd/udev rules they would just 
disappear entirely, but with the driverctl rules they become:

# systemctl |grep 0000:01:00
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.0-vfio.device 
                  loaded active plugged 
/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.0/vfio
sys-devices-pci0000:00-0000:00:03.0-0000:01:00.1-vfio.device 
                  loaded active plugged 
/sys/devices/pci0000:00/0000:00:03.0/0000:01:00.1/vfio

	- Panu -