Any ideas how to stop DPDK from banning me from the box.

Any ideas how to stop DPDK from banning me from the box.

[Dmitry Vyal]

Greetings!

I've been playing with dpdk on my desktop for some time and decided to 
finally test in on a server. It has a plenty of nics:

dev@box:~$ lspci |grep Ether
01:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
05:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
0c:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network 
Connection (rev 01)
0c:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network 
Connection (rev 01)
0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network 
Connection (rev 01)
0e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network 
Connection (rev 01)
10:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network 
Connection (rev 01)
10:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network 
Connection (rev 01)
12:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network 
Connection
82:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
82:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
88:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
88:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
8a:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
8a:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)

I connect to the server using ssh through one of gigabit ports. The 
problem is that no matter which cards I blacklist with -b option then 
starting my dpdk application it takes away all the interfaces from the 
kernel and later returns blacklisted ones. This disrupts my ssh session 
and I can no longer connect to the server.

Any ideas how to overcome this?

Best regards,
Dmitry Vyal.

Re: Any ideas how to stop DPDK from banning me from the box.

[Patrick Mahan]

Dmitry,

Please post the exact blacklist option.  I use the '-b' option all the time with my test box (Dell T620).

Thanks,

Patrick

Sent from my iPad

On Jul 18, 2013, at 8:42 AM, Dmitry Vyal <dmitryvyal-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Greetings!
> 
> I've been playing with dpdk on my desktop for some time and decided to finally test in on a server. It has a plenty of nics:
> 
> dev@box:~$ lspci |grep Ether
> 01:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 01:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 05:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 05:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 0c:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> 0c:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> 0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> 0e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> 10:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> 10:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> 12:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
> 82:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 82:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 88:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 88:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 8a:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 8a:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> 
> I connect to the server using ssh through one of gigabit ports. The problem is that no matter which cards I blacklist with -b option then starting my dpdk application it takes away all the interfaces from the kernel and later returns blacklisted ones. This disrupts my ssh session and I can no longer connect to the server.
> 
> Any ideas how to overcome this?
> 
> Best regards,
> Dmitry Vyal.

Re: Any ideas how to stop DPDK from banning me from the box.

[Marco Chiappero]

Il 18/07/2013 17:42, Dmitry Vyal ha scritto:
> Greetings!

Hi

> [...]
> I connect to the server using ssh through one of gigabit ports.

I do the very same thing and I have no such problem.

> The
> problem is that no matter which cards I blacklist with -b option then
> starting my dpdk application it takes away all the interfaces from the
> kernel and later returns blacklisted ones. This disrupts my ssh session
> and I can no longer connect to the server.

Maybe you are using wrong IDs, verify that.

Here a working example:


pino:~/dpdk-1.2.3r4 ~> lspci | grep -i ethernet
02:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 
10GigE, PCIe 2.0 5GT/s] (rev b0)
03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
04:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network 
Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network 
Connection (rev 01)


pino:~/dpdk-1.2.3r4 ~> sudo build/app/testpmd -c 0xf -n 3 -r 1 -b 
0000:05:00.0 -b 0000:05:00.1
EAL: coremask set to f
...
Initializing port 0... done:  Link Down
Initializing port 1... done:  Link Down
Initializing port 2... done:  Link Up - speed 10000 Mbps - full-duplex
Initializing port 3... done:  Link Up - speed 10000 Mbps - full-duplex
No commandline core given, start packet forwarding
   io packet forwarding - CRC stripping disabled - packets/burst=16
   nb forwarding cores=1 - nb forwarding ports=4
   RX queues=1 - RX desc=128 - RX free threshold=0
   RX threshold registers: pthresh=8 hthresh=8 wthresh=4
   TX queues=1 - TX desc=512 - TX free threshold=0
   TX threshold registers: pthresh=36 hthresh=0 wthresh=0
   TX RS bit threshold=0
Press enter to exit

[ssh is still up]


Regards,
Marco

Re: Any ideas how to stop DPDK from banning me from the box.

[Dmitry Vyal]

Hi Marco,

thanks for posting the example I carefully composed mine using your as a 
basis and, guess, all works I don't have the exact cmdline I issued 
yesterday. Looks like bash didn't have a chance to save history But I 
guess I put all the -b options after the -- delimiter so they weren't 
parsed by rte_eal_init. You saved me a day!
>
> pino:~/dpdk-1.2.3r4 ~> sudo build/app/testpmd -c 0xf -n 3 -r 1 -b 
> 0000:05:00.0 -b 0000:05:00.1
> EAL: coremask set to f
> ...
> Initializing port 0... done:  Link Down
> Initializing port 1... done:  Link Down
> Initializing port 2... done:  Link Up - speed 10000 Mbps - full-duplex
> Initializing port 3... done:  Link Up - speed 10000 Mbps - full-duplex
> No commandline core given, start packet forwarding
>   io packet forwarding - CRC stripping disabled - packets/burst=16
>   nb forwarding cores=1 - nb forwarding ports=4
>   RX queues=1 - RX desc=128 - RX free threshold=0
>   RX threshold registers: pthresh=8 hthresh=8 wthresh=4
>   TX queues=1 - TX desc=512 - TX free threshold=0
>   TX threshold registers: pthresh=36 hthresh=0 wthresh=0
>   TX RS bit threshold=0
> Press enter to exit
>
> [ssh is still up]
>
>
> Regards,
> Marco

Re: Any ideas how to stop DPDK from banning me from the box.

[Marco Chiappero]

Il 19/07/2013 09:11, Dmitry Vyal ha scritto:
> Hi Marco,
>
> thanks for posting the example I carefully composed mine using your as a
> basis and, guess, all works I don't have the exact cmdline I issued
> yesterday. Looks like bash didn't have a chance to save history But I
> guess I put all the -b options after the -- delimiter so they weren't
> parsed by rte_eal_init. You saved me a day!


Hi Dmitry,

I'm glad I could help!
By the way, you can also avoid using the PMD igb driver entirely if you 
don't need it (I don't as I just care about 10Gbit interfaces, but since 
you have lots of 82576 NICs I'm not that sure); just change your compile 
configuration, either the default one or a custom one (depending on what 
you are using):

pino:~/dpdk-1.2.3r4 -> grep IGB config/defconfig_x86_64-marco-linuxapp-gcc
# Compile burst-oriented IGB PMD driver
CONFIG_RTE_LIBRTE_IGB_PMD=n
CONFIG_RTE_LIBRTE_IGB_DEBUG_INIT=n
CONFIG_RTE_LIBRTE_IGB_DEBUG_RX=n
CONFIG_RTE_LIBRTE_IGB_DEBUG_TX=n
CONFIG_RTE_LIBRTE_IGB_DEBUG_TX_FREE=n
CONFIG_RTE_LIBRTE_IGB_DEBUG_DRIVER=n


As you can see here I keep using the kernel space igb driver:

pino:~/dpdk-1.2.3r4 -> lspci -kD | grep -A 2 -i ether
0000:02:00.0 Ethernet controller: Mellanox Technologies MT26448 
[ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
	Subsystem: Mellanox Technologies Device 0016
	Kernel driver in use: mlx4_core
0000:03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
	Kernel driver in use: igb_uio
0000:03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
	Kernel driver in use: igb_uio
0000:04:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
	Kernel driver in use: igb_uio
0000:04:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
	Kernel driver in use: igb_uio
0000:05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit 
Network Connection (rev 01)
	Subsystem: Super Micro Computer Inc Device 10c9
	Kernel driver in use: igb
0000:05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit 
Network Connection (rev 01)
	Subsystem: Super Micro Computer Inc Device 10c9
	Kernel driver in use: igb

Regards,
Marco

Re: Any ideas how to stop DPDK from banning me from the box.

[Antti Kantee]

Semi-related: why does DPDK take a blacklist instead of a list of 
devices that it should use (perhaps with a special value "all" in case 
someone really wants that behavior)?  Historical reasons?

(yes, I've also bricked my ssh connections several times)

Re: Any ideas how to stop DPDK from banning me from the box.

[Vincent JARDIN]

On 19/07/2013 16:30, Antti Kantee wrote:
> Semi-related: why does DPDK take a blacklist instead of a list of
> devices that it should use (perhaps with a special value "all" in case
> someone really wants that behavior)?  Historical reasons?
>
> (yes, I've also bricked my ssh connections several times)

Antti, you are right. Should you need to make it better with a device 
name, your patch can be included.

About such ssh connection, we did not feel the pain at 6WIND since we 
have a stack that manages such issues so it does not break, but your 
request is valid.

Looking for any patch that would help,
   Vincent