[PATCH] doc: update mlx4 usage and dependencies

[PATCH] doc: update mlx4 usage and dependencies

[Adrien Mazarguil]

- libmlx4 and libibverbs dependencies distributed with Mellanox OFED are now
  also available on DPDK.org to make installation easier.
- Document Mellanox OFED and firmware versions to use.
- Add links to Mellanox and its community websites.
- Add kernel modules parameters section.

Signed-off-by: Adrien Mazarguil <adrien.mazarguil-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>
---
 doc/guides/nics/mlx4.rst | 85 ++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 78 insertions(+), 7 deletions(-)

diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst
index b26c219..93a7f57 100644
--- a/doc/guides/nics/mlx4.rst
+++ b/doc/guides/nics/mlx4.rst
@@ -31,8 +31,15 @@ MLX4 poll mode driver library
 =============================
 
 The MLX4 poll mode driver library (**librte_pmd_mlx4**) implements support
-for **Mellanox ConnectX-3** 10/40 Gbps adapters (EN 40, EN 10, Pro EN 40) as
-well as their virtual functions (VF) in SR-IOV context.
+for **Mellanox ConnectX-3 EN** 10/40 Gbps adapters as well as their virtual
+functions (VF) in SR-IOV context.
+
+Information and documentation about this family of adapters can be found on
+the `Mellanox website <http://www.mellanox.com>`_. Help is also provided by
+the `Mellanox community <http://community.mellanox.com/welcome>`_.
+
+There is also a `section dedicated to this poll mode driver
+<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`_.
 
 .. note::
 
@@ -78,7 +85,7 @@ Features and limitations
 - Multiple MAC addresses (unicast, multicast) can be configured.
 - Scattered packets are supported for TX and RX.
 
-..
+.. break
 
 - RSS hash key cannot be modified.
 - Hardware counters are not implemented (they are software counters).
@@ -90,6 +97,8 @@ Configuration
 Compilation options
 ~~~~~~~~~~~~~~~~~~~
 
+These options can be modified in the ``.config`` file.
+
 - ``CONFIG_RTE_LIBRTE_MLX4_PMD`` (default **n**)
 
   Toggle compilation of librte_pmd_mlx4 itself.
@@ -146,6 +155,30 @@ Run-time configuration
 
 - **ethtool** operations on related kernel interfaces also affect the PMD.
 
+Kernel module parameters
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The **mlx4_core** kernel module has several parameters that affect the
+behavior and/or the performance of librte_pmd_mlx4. Some of them are described
+below.
+
+- **num_vfs** (integer or triplet, optionally prefixed by device address
+  strings)
+
+  Create the given number of VFs on the specified devices.
+
+- **log_num_mgm_entry_size** (integer)
+
+  Device-managed flow steering (DMFS) is required by DPDK applications. It is
+  enabled by using a negative value, the last four bits of which have a
+  special meaning.
+
+  - **-1**: force device-managed flow steering (DMFS).
+  - **-7**: configure optimized steering mode to improve performance with the
+    following limitation: Ethernet frames with the port MAC address as the
+    destination cannot be received, even in promiscuous mode. Additional MAC
+    addresses can still be set by ``rte_eth_dev_mac_addr_addr()``.
+
 Prerequisites
 -------------
 
@@ -185,6 +218,26 @@ DPDK and must be installed separately:
   - mlx4_ib: InifiniBand device driver.
   - ib_uverbs: user space driver for verbs (entry point for libibverbs).
 
+- **Firmware update**
+
+  Mellanox OFED releases include firmware updates for ConnectX-3 adapters.
+
+  Because each release provides new features, these updates must be applied to
+  match the kernel modules and libraries they come with.
+
+.. note::
+
+   Both libraries are BSD and GPL licensed. Linux kernel modules are GPL
+   licensed.
+
+Currently supported by DPDK:
+
+- Mellanox OFED **2.4-1**.
+- Firmware version **2.33.5000** and higher.
+
+Getting Mellanox OFED
+~~~~~~~~~~~~~~~~~~~~~
+
 While these libraries and kernel modules are available on OpenFabrics
 Aliance's `website <https://www.openfabrics.org/>`_ and provided by package
 managers on most distributions, this PMD requires Ethernet extensions that
@@ -193,13 +246,31 @@ may not be supported at the moment (this is a work in progress).
 `Mellanox OFED
 <http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>`_
 includes the necessary support and should be used in the meantime. For DPDK,
-only libibverbs, libmlx4 and mlnx-ofed-kernel packages are required from
-that distribution.
+only libibverbs, libmlx4, mlnx-ofed-kernel packages and firmware updates are
+required from that distribution.
 
 .. note::
 
-   Both libraries are BSD and GPL licensed. Linux kernel modules are GPL
-   licensed.
+   Several versions of Mellanox OFED are available. Installing the version
+   this DPDK release was developped and tested against is strongly
+   recommended. Please check the `prerequisites`_.
+
+Getting libibverbs and libmlx4 from DPDK.org
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Based on Mellanox OFED, optimized libibverbs and libmlx4 versions can be
+optionally downloaded from DPDK.org:
+
+`<http://www.dpdk.org/download/mlx4>`_
+
+Some enhancements are done for better performance with DPDK applications and
+are not merged upstream yet.
+
+Since it is partly achieved by tuning compilation options to disable features
+not needed by DPDK, linking these libraries statically and avoid system-wide
+installation is the preferred method.
+
+Installation documentation is available from the above link.
 
 Usage example
 -------------
-- 
2.1.0

Re: [PATCH] doc: update mlx4 usage and dependencies

[Thomas Monjalon]

2015-04-02 11:33, Adrien Mazarguil:
> - libmlx4 and libibverbs dependencies distributed with Mellanox OFED are now
>   also available on DPDK.org to make installation easier.
> - Document Mellanox OFED and firmware versions to use.
> - Add links to Mellanox and its community websites.
> - Add kernel modules parameters section.
> 
> Signed-off-by: Adrien Mazarguil <adrien.mazarguil-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>

Applied, thanks

Re: [PATCH] doc: update mlx4 usage and dependencies

[Raghav Sethi]

Hi Adrien,

I've copied dev@dpdk.

I re-installed Mellanox OFED, and then re-compiled DPDK without the flags
for the static library. I'm running Ubuntu 14.04. However, I still get the
same error message (below).

I then tried re-installing the Mellanox libraries on dpdk.org, and compiled
another copy of DPDK with them statically linked. This appears to work
perfectly. I no longer get the permission issues I was getting before,
possibly that was unrelated/fixed by OFED. Now both OFED, and OFED+dpdk.org
libs statically linked have the same issue:

EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
/usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, version
IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
/usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
reference
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs0
PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
EAL: No probed ethernet devices

The OFED install was successful (and it apparently removes/overwrites old
versions of the libraries). The installer also checks the firmware
versions, and it says I'm on the latest. Loading the kernel modules is also
successful, so I can't for the life of me figure out what this issue is now.

Again, just to be clear, the full list of steps was:
1. Install OFED, run /etc/init.d/openibd restart
2. Compile DPDK with the CONFIG_RTE_LIBRTE_MLX4_PMD=y
3. Set up hugepages
4. Load kernel modules
5. Run testpmd

Best,
Raghav

On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <adrien.mazarguil-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>
wrote:

> Hi Raghav,
>
> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
> > Hi Adrien,
> >
> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
> I'm
> > a grad student at Princeton working on a research project.
>
> It's fine, but let's discuss on the ML as much as possible next time so the
> community can benefit from it. Since very few people use this driver at the
> moment (DPDK 2.0 is not even out yet), I have my own questions for you.
>
> > Given your patch (which was very helpful, thanks), here is the series of
> > all steps that I'm taking to run testpmd on a ConnectX-3 EN 40G
> >
> > 1. Compile and install the mellanox-specific libraries listed here:
> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>
> This is fine, but did you use the provided Makefile to compile these
> libraries statically? If so, no issues?
>
> Also you did not install Mellanox OFED? You should, because updated kernel
> modules come from that distribution. It won't work properly (or at all)
> without them.
>
> The libraries from DPDK.org are actually optional once you've installed
> Mellanox OFED. They only improve performance. Remember to update firmwares
> also.
>
> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS
> > 3. Create a new configuration which includes
> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
> > 4. Compile DPDK 2.0 with the new configuration using setup.sh
> > 5. Set up hugepages using setup.sh
> > 6. Load the kernel modules ib_uverbs mlx4_en mlx4_core mlx4_ib
> > 7. Run testpmd with the right PCI bus address
> >
> > However, I consistently get:
> > sudo: unable to execute ./testpmd: Permission denied
>
> Now that doesn't make sense, isn't that file missing permissions or
> something? Are you running RHEL with a restrictive SELinux policy?
> Is sudo configured to allow you to run testpmd?
>
> In any case, I don't think it's related to librte_pmd_mlx4.
>
> > I tried taking the interface down with ifconfig because
> > (./tools/dpdk_nic_bind.py --status shows the interface as active), but
> that
> > didn't change anything.
>
> No, you should leave the interfaces up, it's not supposed to disappear, the
> PMD will bring them back up anyway. Actually the Linux kernel interface
> remains fully functional even while DPDK applications are running. You can
> send packets out of it but as long as both the kernel and DPDK applications
> share the same MAC address, only DPDK applications will receive replies. If
> you change the MAC address on the kernel interface however, you'll be able
> to use it normally.
>
> > I tried installing the mellanox libraries globally, but then I see these
> > error messages:
> >
> > -------
> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> version
> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
> > version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> > reference
> > libibverbs: Warning: no userspace device-specific driver found for
> > /sys/class/infiniband_verbs/uverbs0
> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> > EAL: No probed ethernet devices
> > Interactive-mode selected
> > EAL: Error - exiting with code: 1
> >   Cause: Creation of mbuf pool for socket 0 failed
> > --------
>
> That's because the libibverbs version you're using is not compatible with
> the userspace drivers installed on your system (libibverbs uses dlopen() to
> load liblmx4). Even though you've installed both from DPDK.org, seems you
> have another version installed by your distribution packages.
>
> If you intend to use dynamic libraries, you should uninstall libibverbs,
> libmlx4 and related packages first.
>
> Regards,
>
> --
> Adrien Mazarguil
> 6WIND
>

Re: [PATCH] doc: update mlx4 usage and dependencies

[Raghav Sethi]

Hi Adrien,

Quick update: So I just tried the setup again on a new machine with the
exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a
charm.

Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
>From the error message it appears that the libraries are incorrectly
installed/configured. However, the OFED uninstall script removes the
libraries in question
(libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,  libibverbs.so), I
checked! In fact, I diffed the libibverbs.so from both machines, and
they're exactly identical. So it seems either the OFED
installation/uninstallation is somehow flawed, or the error message from
testpmd is misleading.

Best,
Raghav

On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs-sXtZGUMCxtn536EW/eD9aw@public.gmane.org>
wrote:

> Hi Adrien,
>
> I've copied dev@dpdk.
>
> I re-installed Mellanox OFED, and then re-compiled DPDK without the flags
> for the static library. I'm running Ubuntu 14.04. However, I still get the
> same error message (below).
>
> I then tried re-installing the Mellanox libraries on dpdk.org, and
> compiled another copy of DPDK with them statically linked. This appears to
> work perfectly. I no longer get the permission issues I was getting before,
> possibly that was unrelated/fixed by OFED. Now both OFED, and OFED+
> dpdk.org libs statically linked have the same issue:
>
> EAL: PCI device 0000:06:00.0 on NUMA socket 0
> EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, version
> IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
> version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> reference
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0
> PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> EAL: No probed ethernet devices
>
> The OFED install was successful (and it apparently removes/overwrites old
> versions of the libraries). The installer also checks the firmware
> versions, and it says I'm on the latest. Loading the kernel modules is also
> successful, so I can't for the life of me figure out what this issue is now.
>
> Again, just to be clear, the full list of steps was:
> 1. Install OFED, run /etc/init.d/openibd restart
> 2. Compile DPDK with the CONFIG_RTE_LIBRTE_MLX4_PMD=y
> 3. Set up hugepages
> 4. Load kernel modules
> 5. Run testpmd
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> adrien.mazarguil-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> wrote:
>
>> Hi Raghav,
>>
>> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> > Hi Adrien,
>> >
>> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
>> I'm
>> > a grad student at Princeton working on a research project.
>>
>> It's fine, but let's discuss on the ML as much as possible next time so
>> the
>> community can benefit from it. Since very few people use this driver at
>> the
>> moment (DPDK 2.0 is not even out yet), I have my own questions for you.
>>
>> > Given your patch (which was very helpful, thanks), here is the series of
>> > all steps that I'm taking to run testpmd on a ConnectX-3 EN 40G
>> >
>> > 1. Compile and install the mellanox-specific libraries listed here:
>> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>>
>> This is fine, but did you use the provided Makefile to compile these
>> libraries statically? If so, no issues?
>>
>> Also you did not install Mellanox OFED? You should, because updated kernel
>> modules come from that distribution. It won't work properly (or at all)
>> without them.
>>
>> The libraries from DPDK.org are actually optional once you've installed
>> Mellanox OFED. They only improve performance. Remember to update firmwares
>> also.
>>
>> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS
>> > 3. Create a new configuration which includes
>> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> > 4. Compile DPDK 2.0 with the new configuration using setup.sh
>> > 5. Set up hugepages using setup.sh
>> > 6. Load the kernel modules ib_uverbs mlx4_en mlx4_core mlx4_ib
>> > 7. Run testpmd with the right PCI bus address
>> >
>> > However, I consistently get:
>> > sudo: unable to execute ./testpmd: Permission denied
>>
>> Now that doesn't make sense, isn't that file missing permissions or
>> something? Are you running RHEL with a restrictive SELinux policy?
>> Is sudo configured to allow you to run testpmd?
>>
>> In any case, I don't think it's related to librte_pmd_mlx4.
>>
>> > I tried taking the interface down with ifconfig because
>> > (./tools/dpdk_nic_bind.py --status shows the interface as active), but
>> that
>> > didn't change anything.
>>
>> No, you should leave the interfaces up, it's not supposed to disappear,
>> the
>> PMD will bring them back up anyway. Actually the Linux kernel interface
>> remains fully functional even while DPDK applications are running. You can
>> send packets out of it but as long as both the kernel and DPDK
>> applications
>> share the same MAC address, only DPDK applications will receive replies.
>> If
>> you change the MAC address on the kernel interface however, you'll be able
>> to use it normally.
>>
>> > I tried installing the mellanox libraries globally, but then I see these
>> > error messages:
>> >
>> > -------
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
>> > version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> > reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> > Interactive-mode selected
>> > EAL: Error - exiting with code: 1
>> >   Cause: Creation of mbuf pool for socket 0 failed
>> > --------
>>
>> That's because the libibverbs version you're using is not compatible with
>> the userspace drivers installed on your system (libibverbs uses dlopen()
>> to
>> load liblmx4). Even though you've installed both from DPDK.org, seems you
>> have another version installed by your distribution packages.
>>
>> If you intend to use dynamic libraries, you should uninstall libibverbs,
>> libmlx4 and related packages first.
>>
>> Regards,
>>
>> --
>> Adrien Mazarguil
>> 6WIND
>>
>

Re: [PATCH] doc: update mlx4 usage and dependencies

[Olga Shern]

Hi Raghav, 

What OFED version did you install?

There are 2 ways to compile mlx4:
1. You can compile it dynamically with the libibverbs  that are coming with Mellanox_OFED 2.4
2. To get better performance: compile it statically with libibverbs and libmlx4 that can be downloaded from http://dpdk.org/download, the README inside librte_pmd_mlx4 explains how to do it
     If you compile the pmd statically then MLNX_OFED is needed only for kernel modules. 
 
Best Regards,
Olga


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Raghav Sethi
Sent: Friday, April 03, 2015 9:21 PM
To: Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies

Hi Adrien,

Quick update: So I just tried the setup again on a new machine with the exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a charm.

Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
From the error message it appears that the libraries are incorrectly installed/configured. However, the OFED uninstall script removes the libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,  libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both machines, and they're exactly identical. So it seems either the OFED installation/uninstallation is somehow flawed, or the error message from testpmd is misleading.

Best,
Raghav

On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs@cs.princeton.edu>
wrote:

> Hi Adrien,
>
> I've copied dev@dpdk.
>
> I re-installed Mellanox OFED, and then re-compiled DPDK without the 
> flags for the static library. I'm running Ubuntu 14.04. However, I 
> still get the same error message (below).
>
> I then tried re-installing the Mellanox libraries on dpdk.org, and 
> compiled another copy of DPDK with them statically linked. This 
> appears to work perfectly. I no longer get the permission issues I was 
> getting before, possibly that was unrelated/fixed by OFED. Now both 
> OFED, and OFED+ dpdk.org libs statically linked have the same issue:
>
> EAL: PCI device 0000:06:00.0 on NUMA socket 0
> EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, 
> version
> IBVERBS_1.1 not defined in file libibverbs.so.1 with link time 
> reference
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol 
> ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file 
> libibverbs.so.1 with link time reference
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0
> PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> EAL: No probed ethernet devices
>
> The OFED install was successful (and it apparently removes/overwrites 
> old versions of the libraries). The installer also checks the firmware 
> versions, and it says I'm on the latest. Loading the kernel modules is 
> also successful, so I can't for the life of me figure out what this issue is now.
>
> Again, just to be clear, the full list of steps was:
> 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with 
> the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel 
> modules 5. Run testpmd
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil < 
> adrien.mazarguil@6wind.com> wrote:
>
>> Hi Raghav,
>>
>> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> > Hi Adrien,
>> >
>> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
>> I'm
>> > a grad student at Princeton working on a research project.
>>
>> It's fine, but let's discuss on the ML as much as possible next time 
>> so the community can benefit from it. Since very few people use this 
>> driver at the moment (DPDK 2.0 is not even out yet), I have my own 
>> questions for you.
>>
>> > Given your patch (which was very helpful, thanks), here is the 
>> > series of all steps that I'm taking to run testpmd on a ConnectX-3 
>> > EN 40G
>> >
>> > 1. Compile and install the mellanox-specific libraries listed here:
>> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>>
>> This is fine, but did you use the provided Makefile to compile these 
>> libraries statically? If so, no issues?
>>
>> Also you did not install Mellanox OFED? You should, because updated 
>> kernel modules come from that distribution. It won't work properly 
>> (or at all) without them.
>>
>> The libraries from DPDK.org are actually optional once you've 
>> installed Mellanox OFED. They only improve performance. Remember to 
>> update firmwares also.
>>
>> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new 
>> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5. 
>> > Set up hugepages using setup.sh 6. Load the kernel modules 
>> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right 
>> > PCI bus address
>> >
>> > However, I consistently get:
>> > sudo: unable to execute ./testpmd: Permission denied
>>
>> Now that doesn't make sense, isn't that file missing permissions or 
>> something? Are you running RHEL with a restrictive SELinux policy?
>> Is sudo configured to allow you to run testpmd?
>>
>> In any case, I don't think it's related to librte_pmd_mlx4.
>>
>> > I tried taking the interface down with ifconfig because 
>> > (./tools/dpdk_nic_bind.py --status shows the interface as active), 
>> > but
>> that
>> > didn't change anything.
>>
>> No, you should leave the interfaces up, it's not supposed to 
>> disappear, the PMD will bring them back up anyway. Actually the Linux 
>> kernel interface remains fully functional even while DPDK 
>> applications are running. You can send packets out of it but as long 
>> as both the kernel and DPDK applications share the same MAC address, 
>> only DPDK applications will receive replies.
>> If
>> you change the MAC address on the kernel interface however, you'll be 
>> able to use it normally.
>>
>> > I tried installing the mellanox libraries globally, but then I see 
>> > these error messages:
>> >
>> > -------
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time 
>> > reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol 
>> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file 
>> > libibverbs.so.1 with link time reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> > Interactive-mode selected
>> > EAL: Error - exiting with code: 1
>> >   Cause: Creation of mbuf pool for socket 0 failed
>> > --------
>>
>> That's because the libibverbs version you're using is not compatible 
>> with the userspace drivers installed on your system (libibverbs uses 
>> dlopen() to load liblmx4). Even though you've installed both from 
>> DPDK.org, seems you have another version installed by your 
>> distribution packages.
>>
>> If you intend to use dynamic libraries, you should uninstall 
>> libibverbs,
>> libmlx4 and related packages first.
>>
>> Regards,
>>
>> --
>> Adrien Mazarguil
>> 6WIND
>>
>

Re: [PATCH] doc: update mlx4 usage and dependencies

[Raghav Sethi]

Hi Olga,

Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64

I tried both variants:
- Compiling with dynamic linking against OFED libraries
- Compiling with static linking against dpdk.org libraries

Both have the same issue.

Best,
Raghav

On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:

> Hi Raghav,
>
> What OFED version did you install?
>
> There are 2 ways to compile mlx4:
> 1. You can compile it dynamically with the libibverbs  that are coming
> with Mellanox_OFED 2.4
> 2. To get better performance: compile it statically with libibverbs and
> libmlx4 that can be downloaded from http://dpdk.org/download, the README
> inside librte_pmd_mlx4 explains how to do it
>      If you compile the pmd statically then MLNX_OFED is needed only for
> kernel modules.
>
> Best Regards,
> Olga
>
>
> -----Original Message-----
> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Raghav Sethi
> Sent: Friday, April 03, 2015 9:21 PM
> To: Adrien Mazarguil; dev-VfR2kkLFssw@public.gmane.org; Xiaozhou Li
> Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>
> Hi Adrien,
>
> Quick update: So I just tried the setup again on a new machine with the
> exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a
> charm.
>
> Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
> From the error message it appears that the libraries are incorrectly
> installed/configured. However, the OFED uninstall script removes the
> libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,
> libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both
> machines, and they're exactly identical. So it seems either the OFED
> installation/uninstallation is somehow flawed, or the error message from
> testpmd is misleading.
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs-sXtZGUMCxtn536EW/eD9aw@public.gmane.org>
> wrote:
>
> > Hi Adrien,
> >
> > I've copied dev@dpdk.
> >
> > I re-installed Mellanox OFED, and then re-compiled DPDK without the
> > flags for the static library. I'm running Ubuntu 14.04. However, I
> > still get the same error message (below).
> >
> > I then tried re-installing the Mellanox libraries on dpdk.org, and
> > compiled another copy of DPDK with them statically linked. This
> > appears to work perfectly. I no longer get the permission issues I was
> > getting before, possibly that was unrelated/fixed by OFED. Now both
> > OFED, and OFED+ dpdk.org libs statically linked have the same issue:
> >
> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> > version
> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> > reference
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> > libibverbs.so.1 with link time reference
> > libibverbs: Warning: no userspace device-specific driver found for
> > /sys/class/infiniband_verbs/uverbs0
> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> > EAL: No probed ethernet devices
> >
> > The OFED install was successful (and it apparently removes/overwrites
> > old versions of the libraries). The installer also checks the firmware
> > versions, and it says I'm on the latest. Loading the kernel modules is
> > also successful, so I can't for the life of me figure out what this
> issue is now.
> >
> > Again, just to be clear, the full list of steps was:
> > 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
> > modules 5. Run testpmd
> >
> > Best,
> > Raghav
> >
> > On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> > adrien.mazarguil-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> wrote:
> >
> >> Hi Raghav,
> >>
> >> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
> >> > Hi Adrien,
> >> >
> >> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
> >> I'm
> >> > a grad student at Princeton working on a research project.
> >>
> >> It's fine, but let's discuss on the ML as much as possible next time
> >> so the community can benefit from it. Since very few people use this
> >> driver at the moment (DPDK 2.0 is not even out yet), I have my own
> >> questions for you.
> >>
> >> > Given your patch (which was very helpful, thanks), here is the
> >> > series of all steps that I'm taking to run testpmd on a ConnectX-3
> >> > EN 40G
> >> >
> >> > 1. Compile and install the mellanox-specific libraries listed here:
> >> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
> >>
> >> This is fine, but did you use the provided Makefile to compile these
> >> libraries statically? If so, no issues?
> >>
> >> Also you did not install Mellanox OFED? You should, because updated
> >> kernel modules come from that distribution. It won't work properly
> >> (or at all) without them.
> >>
> >> The libraries from DPDK.org are actually optional once you've
> >> installed Mellanox OFED. They only improve performance. Remember to
> >> update firmwares also.
> >>
> >> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
> >> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
> >> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
> >> > Set up hugepages using setup.sh 6. Load the kernel modules
> >> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
> >> > PCI bus address
> >> >
> >> > However, I consistently get:
> >> > sudo: unable to execute ./testpmd: Permission denied
> >>
> >> Now that doesn't make sense, isn't that file missing permissions or
> >> something? Are you running RHEL with a restrictive SELinux policy?
> >> Is sudo configured to allow you to run testpmd?
> >>
> >> In any case, I don't think it's related to librte_pmd_mlx4.
> >>
> >> > I tried taking the interface down with ifconfig because
> >> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
> >> > but
> >> that
> >> > didn't change anything.
> >>
> >> No, you should leave the interfaces up, it's not supposed to
> >> disappear, the PMD will bring them back up anyway. Actually the Linux
> >> kernel interface remains fully functional even while DPDK
> >> applications are running. You can send packets out of it but as long
> >> as both the kernel and DPDK applications share the same MAC address,
> >> only DPDK applications will receive replies.
> >> If
> >> you change the MAC address on the kernel interface however, you'll be
> >> able to use it normally.
> >>
> >> > I tried installing the mellanox libraries globally, but then I see
> >> > these error messages:
> >> >
> >> > -------
> >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> >> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> >> > libibverbs: Warning: couldn't load driver
> '/usr/lib/libibverbs/libmlx5':
> >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> >> version
> >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> >> > reference
> >> > libibverbs: Warning: couldn't load driver
> '/usr/lib/libibverbs/libmlx4':
> >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> >> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> >> > libibverbs.so.1 with link time reference
> >> > libibverbs: Warning: no userspace device-specific driver found for
> >> > /sys/class/infiniband_verbs/uverbs0
> >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> >> > EAL: No probed ethernet devices
> >> > Interactive-mode selected
> >> > EAL: Error - exiting with code: 1
> >> >   Cause: Creation of mbuf pool for socket 0 failed
> >> > --------
> >>
> >> That's because the libibverbs version you're using is not compatible
> >> with the userspace drivers installed on your system (libibverbs uses
> >> dlopen() to load liblmx4). Even though you've installed both from
> >> DPDK.org, seems you have another version installed by your
> >> distribution packages.
> >>
> >> If you intend to use dynamic libraries, you should uninstall
> >> libibverbs,
> >> libmlx4 and related packages first.
> >>
> >> Regards,
> >>
> >> --
> >> Adrien Mazarguil
> >> 6WIND
> >>
> >
>

Re: [PATCH] doc: update mlx4 usage and dependencies

[Olga Shern]

Hi Raghav,

After running /etc/init.d/openibd restart you don’t need to load the modules, they are loaded by the openibd script.
Can you try to run :
/etc/init.d/openibd stop
Verify that modules are not loaded
/etc/init.d/openib start

If it won’t help:
What Mellanox NIC do you have on the “problematic” machine?
What command line do you use to run the application?

Please send output of:

1.       ”lspci | grep Mellanox”

2.       ibdev2netdev

3.       ldd <your application>

4.       lsmod

Thanks
Olga


From: Raghav Sethi [mailto:raghavs@CS.Princeton.EDU]
Sent: Saturday, April 04, 2015 11:54 PM
To: Olga Shern; Adrien Mazarguil; dev@dpdk.org; Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies

Hi Olga,

Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64

I tried both variants:
- Compiling with dynamic linking against OFED libraries
- Compiling with static linking against dpdk.org<http://dpdk.org> libraries

Both have the same issue.

Best,
Raghav

On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas@mellanox.com<mailto:olgas@mellanox.com>> wrote:
Hi Raghav,

What OFED version did you install?

There are 2 ways to compile mlx4:
1. You can compile it dynamically with the libibverbs  that are coming with Mellanox_OFED 2.4
2. To get better performance: compile it statically with libibverbs and libmlx4 that can be downloaded from http://dpdk.org/download, the README inside librte_pmd_mlx4 explains how to do it
     If you compile the pmd statically then MLNX_OFED is needed only for kernel modules.

Best Regards,
Olga


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>] On Behalf Of Raghav Sethi
Sent: Friday, April 03, 2015 9:21 PM
To: Adrien Mazarguil; dev@dpdk.org<mailto:dev@dpdk.org>; Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies

Hi Adrien,

Quick update: So I just tried the setup again on a new machine with the exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a charm.

Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
From the error message it appears that the libraries are incorrectly installed/configured. However, the OFED uninstall script removes the libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,  libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both machines, and they're exactly identical. So it seems either the OFED installation/uninstallation is somehow flawed, or the error message from testpmd is misleading.

Best,
Raghav

On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs@cs.princeton.edu<mailto:raghavs@cs.princeton.edu>>
wrote:

> Hi Adrien,
>
> I've copied dev@dpdk.
>
> I re-installed Mellanox OFED, and then re-compiled DPDK without the
> flags for the static library. I'm running Ubuntu 14.04. However, I
> still get the same error message (below).
>
> I then tried re-installing the Mellanox libraries on dpdk.org<http://dpdk.org>, and
> compiled another copy of DPDK with them statically linked. This
> appears to work perfectly. I no longer get the permission issues I was
> getting before, possibly that was unrelated/fixed by OFED. Now both
> OFED, and OFED+ dpdk.org<http://dpdk.org> libs statically linked have the same issue:
>
> EAL: PCI device 0000:06:00.0 on NUMA socket 0
> EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> version
> IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> reference
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> libibverbs.so.1 with link time reference
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0
> PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> EAL: No probed ethernet devices
>
> The OFED install was successful (and it apparently removes/overwrites
> old versions of the libraries). The installer also checks the firmware
> versions, and it says I'm on the latest. Loading the kernel modules is
> also successful, so I can't for the life of me figure out what this issue is now.
>
> Again, just to be clear, the full list of steps was:
> 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
> the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
> modules 5. Run testpmd
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> adrien.mazarguil@6wind.com<mailto:adrien.mazarguil@6wind.com>> wrote:
>
>> Hi Raghav,
>>
>> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> > Hi Adrien,
>> >
>> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
>> I'm
>> > a grad student at Princeton working on a research project.
>>
>> It's fine, but let's discuss on the ML as much as possible next time
>> so the community can benefit from it. Since very few people use this
>> driver at the moment (DPDK 2.0 is not even out yet), I have my own
>> questions for you.
>>
>> > Given your patch (which was very helpful, thanks), here is the
>> > series of all steps that I'm taking to run testpmd on a ConnectX-3
>> > EN 40G
>> >
>> > 1. Compile and install the mellanox-specific libraries listed here:
>> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>>
>> This is fine, but did you use the provided Makefile to compile these
>> libraries statically? If so, no issues?
>>
>> Also you did not install Mellanox OFED? You should, because updated
>> kernel modules come from that distribution. It won't work properly
>> (or at all) without them.
>>
>> The libraries from DPDK.org are actually optional once you've
>> installed Mellanox OFED. They only improve performance. Remember to
>> update firmwares also.
>>
>> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
>> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
>> > Set up hugepages using setup.sh 6. Load the kernel modules
>> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
>> > PCI bus address
>> >
>> > However, I consistently get:
>> > sudo: unable to execute ./testpmd: Permission denied
>>
>> Now that doesn't make sense, isn't that file missing permissions or
>> something? Are you running RHEL with a restrictive SELinux policy?
>> Is sudo configured to allow you to run testpmd?
>>
>> In any case, I don't think it's related to librte_pmd_mlx4.
>>
>> > I tried taking the interface down with ifconfig because
>> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
>> > but
>> that
>> > didn't change anything.
>>
>> No, you should leave the interfaces up, it's not supposed to
>> disappear, the PMD will bring them back up anyway. Actually the Linux
>> kernel interface remains fully functional even while DPDK
>> applications are running. You can send packets out of it but as long
>> as both the kernel and DPDK applications share the same MAC address,
>> only DPDK applications will receive replies.
>> If
>> you change the MAC address on the kernel interface however, you'll be
>> able to use it normally.
>>
>> > I tried installing the mellanox libraries globally, but then I see
>> > these error messages:
>> >
>> > -------
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> > reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
>> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
>> > libibverbs.so.1 with link time reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> > Interactive-mode selected
>> > EAL: Error - exiting with code: 1
>> >   Cause: Creation of mbuf pool for socket 0 failed
>> > --------
>>
>> That's because the libibverbs version you're using is not compatible
>> with the userspace drivers installed on your system (libibverbs uses
>> dlopen() to load liblmx4). Even though you've installed both from
>> DPDK.org, seems you have another version installed by your
>> distribution packages.
>>
>> If you intend to use dynamic libraries, you should uninstall
>> libibverbs,
>> libmlx4 and related packages first.
>>
>> Regards,
>>
>> --
>> Adrien Mazarguil
>> 6WIND
>>
>

Re: [PATCH] doc: update mlx4 usage and dependencies

[Raghav Sethi]

Hi Olga,

Thanks very much for helping.

I tried stopping and starting, after stop lsmod | grep mlx had no matches.
After start, it had the output listed below. The device is the same as on
the rest of the machines - Mellanox ConnectX3-EN 40G single port. The
application parameters are the same as the ones listed on the docs (except
of course PCI address is different):

sudo ./testpmd -c 0xff -n 4 -w 0000:06:00.0 -- --rxq=2 --txq=2 -i

Outputs you requested:

1. lspci | grep Mellanox
06:00.0 Ethernet controller: Mellanox Technologies MT27500 Family
[ConnectX-3]

2. ibdev2netdev
mlx4_0 port 1 ==> p3p1 (Up)

3. ldd testpmd
        linux-vdso.so.1 =>  (0x00007fff7c1dc000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1a7dc07000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1a7d9ff000)
        libibverbs.so.1 => /usr/local/lib/libibverbs.so.1
(0x00007f1a7d7eb000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1a7d5e7000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f1a7d3c9000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1a7d003000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1a7df19000)

4. lsmod | grep mlx
mlx5_ib               138046  0
mlx5_core             108376  1 mlx5_ib
mlx4_ib               199880  0
ib_sa                  43732  5 rdma_cm,ib_cm,mlx4_ib,rdma_ucm,ib_ipoib
mlx4_en               137047  0
ib_mad                 51544  4 ib_cm,ib_sa,mlx4_ib,ib_umad
vxlan                  37619  1 mlx4_en
ib_core               130498  12
rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib
mlx4_core             332375  2 mlx4_en,mlx4_ib
compat                 13738  17
rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_en,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_addr,ib_core,ib_umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib
ptp                    18933  3 igb,ixgbe,mlx4_en

Best regards,
Raghav

On Sat, Apr 4, 2015 at 5:04 PM Olga Shern <olgas-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:

>  Hi Raghav,
>
>
>
> After running /etc/init.d/openibd restart you don’t need to load the
> modules, they are loaded by the openibd script.
>
> Can you try to run :
>
> /etc/init.d/openibd stop
>
> Verify that modules are not loaded
>
> /etc/init.d/openib start
>
>
>
> If it won’t help:
>
> What Mellanox NIC do you have on the “problematic” machine?
>
> What command line do you use to run the application?
>
>
>
> Please send output of:
>
> 1.       ”lspci | grep Mellanox”
>
> 2.       ibdev2netdev
>
> 3.       ldd <your application>
>
> 4.       lsmod
>
>
>
> Thanks
>
> Olga
>
>
>
>
>
> *From:* Raghav Sethi [mailto:raghavs-Mr68eyIdVKO79RPNQ/giaA@public.gmane.org]
> *Sent:* Saturday, April 04, 2015 11:54 PM
> *To:* Olga Shern; Adrien Mazarguil; dev-VfR2kkLFssw@public.gmane.org; Xiaozhou Li
>
>
> *Subject:* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>
>
>
> Hi Olga,
>
> Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64
>
> I tried both variants:
>
> - Compiling with dynamic linking against OFED libraries
>
> - Compiling with static linking against dpdk.org libraries
>
>
>
> Both have the same issue.
>
>
>
> Best,
>
> Raghav
>
>
>
> On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>
> Hi Raghav,
>
> What OFED version did you install?
>
> There are 2 ways to compile mlx4:
> 1. You can compile it dynamically with the libibverbs  that are coming
> with Mellanox_OFED 2.4
> 2. To get better performance: compile it statically with libibverbs and
> libmlx4 that can be downloaded from http://dpdk.org/download, the README
> inside librte_pmd_mlx4 explains how to do it
>      If you compile the pmd statically then MLNX_OFED is needed only for
> kernel modules.
>
> Best Regards,
> Olga
>
>
> -----Original Message-----
> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Raghav Sethi
> Sent: Friday, April 03, 2015 9:21 PM
> To: Adrien Mazarguil; dev-VfR2kkLFssw@public.gmane.org; Xiaozhou Li
> Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>
> Hi Adrien,
>
> Quick update: So I just tried the setup again on a new machine with the
> exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a
> charm.
>
> Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
> From the error message it appears that the libraries are incorrectly
> installed/configured. However, the OFED uninstall script removes the
> libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,
> libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both
> machines, and they're exactly identical. So it seems either the OFED
> installation/uninstallation is somehow flawed, or the error message from
> testpmd is misleading.
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs-sXtZGUMCxtn536EW/eD9aw@public.gmane.org>
> wrote:
>
> > Hi Adrien,
> >
> > I've copied dev@dpdk.
> >
> > I re-installed Mellanox OFED, and then re-compiled DPDK without the
> > flags for the static library. I'm running Ubuntu 14.04. However, I
> > still get the same error message (below).
> >
> > I then tried re-installing the Mellanox libraries on dpdk.org, and
> > compiled another copy of DPDK with them statically linked. This
> > appears to work perfectly. I no longer get the permission issues I was
> > getting before, possibly that was unrelated/fixed by OFED. Now both
> > OFED, and OFED+ dpdk.org libs statically linked have the same issue:
> >
> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> > version
> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> > reference
> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> > libibverbs.so.1 with link time reference
> > libibverbs: Warning: no userspace device-specific driver found for
> > /sys/class/infiniband_verbs/uverbs0
> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> > EAL: No probed ethernet devices
> >
> > The OFED install was successful (and it apparently removes/overwrites
> > old versions of the libraries). The installer also checks the firmware
> > versions, and it says I'm on the latest. Loading the kernel modules is
> > also successful, so I can't for the life of me figure out what this
> issue is now.
> >
> > Again, just to be clear, the full list of steps was:
> > 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
> > modules 5. Run testpmd
> >
> > Best,
> > Raghav
> >
> > On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> > adrien.mazarguil-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> wrote:
> >
> >> Hi Raghav,
> >>
> >> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
> >> > Hi Adrien,
> >> >
> >> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
> >> I'm
> >> > a grad student at Princeton working on a research project.
> >>
> >> It's fine, but let's discuss on the ML as much as possible next time
> >> so the community can benefit from it. Since very few people use this
> >> driver at the moment (DPDK 2.0 is not even out yet), I have my own
> >> questions for you.
> >>
> >> > Given your patch (which was very helpful, thanks), here is the
> >> > series of all steps that I'm taking to run testpmd on a ConnectX-3
> >> > EN 40G
> >> >
> >> > 1. Compile and install the mellanox-specific libraries listed here:
> >> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
> >>
> >> This is fine, but did you use the provided Makefile to compile these
> >> libraries statically? If so, no issues?
> >>
> >> Also you did not install Mellanox OFED? You should, because updated
> >> kernel modules come from that distribution. It won't work properly
> >> (or at all) without them.
> >>
> >> The libraries from DPDK.org are actually optional once you've
> >> installed Mellanox OFED. They only improve performance. Remember to
> >> update firmwares also.
> >>
> >> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
> >> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
> >> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
> >> > Set up hugepages using setup.sh 6. Load the kernel modules
> >> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
> >> > PCI bus address
> >> >
> >> > However, I consistently get:
> >> > sudo: unable to execute ./testpmd: Permission denied
> >>
> >> Now that doesn't make sense, isn't that file missing permissions or
> >> something? Are you running RHEL with a restrictive SELinux policy?
> >> Is sudo configured to allow you to run testpmd?
> >>
> >> In any case, I don't think it's related to librte_pmd_mlx4.
> >>
> >> > I tried taking the interface down with ifconfig because
> >> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
> >> > but
> >> that
> >> > didn't change anything.
> >>
> >> No, you should leave the interfaces up, it's not supposed to
> >> disappear, the PMD will bring them back up anyway. Actually the Linux
> >> kernel interface remains fully functional even while DPDK
> >> applications are running. You can send packets out of it but as long
> >> as both the kernel and DPDK applications share the same MAC address,
> >> only DPDK applications will receive replies.
> >> If
> >> you change the MAC address on the kernel interface however, you'll be
> >> able to use it normally.
> >>
> >> > I tried installing the mellanox libraries globally, but then I see
> >> > these error messages:
> >> >
> >> > -------
> >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
> >> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> >> > libibverbs: Warning: couldn't load driver
> '/usr/lib/libibverbs/libmlx5':
> >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> >> version
> >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> >> > reference
> >> > libibverbs: Warning: couldn't load driver
> '/usr/lib/libibverbs/libmlx4':
> >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> >> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> >> > libibverbs.so.1 with link time reference
> >> > libibverbs: Warning: no userspace device-specific driver found for
> >> > /sys/class/infiniband_verbs/uverbs0
> >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> >> > EAL: No probed ethernet devices
> >> > Interactive-mode selected
> >> > EAL: Error - exiting with code: 1
> >> >   Cause: Creation of mbuf pool for socket 0 failed
> >> > --------
> >>
> >> That's because the libibverbs version you're using is not compatible
> >> with the userspace drivers installed on your system (libibverbs uses
> >> dlopen() to load liblmx4). Even though you've installed both from
> >> DPDK.org, seems you have another version installed by your
> >> distribution packages.
> >>
> >> If you intend to use dynamic libraries, you should uninstall
> >> libibverbs,
> >> libmlx4 and related packages first.
> >>
> >> Regards,
> >>
> >> --
> >> Adrien Mazarguil
> >> 6WIND
> >>
> >
>
>

Re: [PATCH] doc: update mlx4 usage and dependencies

[Olga Shern]

If you compiled pmd statically with libibverbs and libmlx4 you shouldn't see it in ldd.
Seems that something went wrong.
In any case, ofed doesn't install libraries under /usr/local
Please remove all mlx related libraries under /usr/local/ like libmlx4 and libibverbs





-------- Original message --------
From: Raghav Sethi
Date:05/04/2015 00:29 (GMT+02:00)
To: Olga Shern ,Adrien Mazarguil ,dev-VfR2kkLFssw@public.gmane.org,Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies


Hi Olga,

Thanks very much for helping.

I tried stopping and starting, after stop lsmod | grep mlx had no matches. After start, it had the output listed below. The device is the same as on the rest of the machines - Mellanox ConnectX3-EN 40G single port. The application parameters are the same as the ones listed on the docs (except of course PCI address is different):

sudo ./testpmd -c 0xff -n 4 -w 0000:06:00.0 -- --rxq=2 --txq=2 -i

Outputs you requested:

1. lspci | grep Mellanox
06:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

2. ibdev2netdev
mlx4_0 port 1 ==> p3p1 (Up)

3. ldd testpmd
        linux-vdso.so.1 =>  (0x00007fff7c1dc000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1a7dc07000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1a7d9ff000)
        libibverbs.so.1 => /usr/local/lib/libibverbs.so.1 (0x00007f1a7d7eb000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1a7d5e7000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1a7d3c9000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1a7d003000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1a7df19000)

4. lsmod | grep mlx
mlx5_ib               138046  0
mlx5_core             108376  1 mlx5_ib
mlx4_ib               199880  0
ib_sa                  43732  5 rdma_cm,ib_cm,mlx4_ib,rdma_ucm,ib_ipoib
mlx4_en               137047  0
ib_mad                 51544  4 ib_cm,ib_sa,mlx4_ib,ib_umad
vxlan                  37619  1 mlx4_en
ib_core               130498  12 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib
mlx4_core             332375  2 mlx4_en,mlx4_ib
compat                 13738  17 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_en,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_addr,ib_core,ib_umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib
ptp                    18933  3 igb,ixgbe,mlx4_en

Best regards,
Raghav

On Sat, Apr 4, 2015 at 5:04 PM Olga Shern <olgas-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org<mailto:olgas@mellanox.com>> wrote:
Hi Raghav,

After running /etc/init.d/openibd restart you don’t need to load the modules, they are loaded by the openibd script.
Can you try to run :
/etc/init.d/openibd stop
Verify that modules are not loaded
/etc/init.d/openib start

If it won’t help:
What Mellanox NIC do you have on the “problematic” machine?
What command line do you use to run the application?

Please send output of:

1.       ”lspci | grep Mellanox”

2.       ibdev2netdev

3.       ldd <your application>

4.       lsmod

Thanks
Olga


From: Raghav Sethi [mailto:raghavs-Mr68eyIdVKO79RPNQ/giaA@public.gmane.org<mailto:raghavs-Mr68eyIdVKM@public.gmane.orgeton.EDU>]
Sent: Saturday, April 04, 2015 11:54 PM
To: Olga Shern; Adrien Mazarguil; dev-VfR2kkLFssw@public.gmane.org<mailto:dev-VfR2kkLFssw@public.gmane.org>; Xiaozhou Li

Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies

Hi Olga,

Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64

I tried both variants:
- Compiling with dynamic linking against OFED libraries
- Compiling with static linking against dpdk.org<http://dpdk.org> libraries

Both have the same issue.

Best,
Raghav

On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org<mailto:olgas@mellanox.com>> wrote:
Hi Raghav,

What OFED version did you install?

There are 2 ways to compile mlx4:
1. You can compile it dynamically with the libibverbs  that are coming with Mellanox_OFED 2.4
2. To get better performance: compile it statically with libibverbs and libmlx4 that can be downloaded from http://dpdk.org/download, the README inside librte_pmd_mlx4 explains how to do it
     If you compile the pmd statically then MLNX_OFED is needed only for kernel modules.

Best Regards,
Olga


-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org<mailto:dev-bounces-VfR2kkLFssw@public.gmane.org>] On Behalf Of Raghav Sethi
Sent: Friday, April 03, 2015 9:21 PM
To: Adrien Mazarguil; dev-VfR2kkLFssw@public.gmane.org<mailto:dev-VfR2kkLFssw@public.gmane.org>; Xiaozhou Li
Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies

Hi Adrien,

Quick update: So I just tried the setup again on a new machine with the exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a charm.

Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
>From the error message it appears that the libraries are incorrectly installed/configured. However, the OFED uninstall script removes the libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,  libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both machines, and they're exactly identical. So it seems either the OFED installation/uninstallation is somehow flawed, or the error message from testpmd is misleading.

Best,
Raghav

On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs-sXtZGUMCxtn536EW/eD9aw@public.gmane.org<mailto:raghavs-sXtZGUMCxtn536EW/eD9aw@public.gmane.org>>
wrote:

> Hi Adrien,
>
> I've copied dev@dpdk.
>
> I re-installed Mellanox OFED, and then re-compiled DPDK without the
> flags for the static library. I'm running Ubuntu 14.04. However, I
> still get the same error message (below).
>
> I then tried re-installing the Mellanox libraries on dpdk.org<http://dpdk.org>, and
> compiled another copy of DPDK with them statically linked. This
> appears to work perfectly. I no longer get the permission issues I was
> getting before, possibly that was unrelated/fixed by OFED. Now both
> OFED, and OFED+ dpdk.org<http://dpdk.org> libs statically linked have the same issue:
>
> EAL: PCI device 0000:06:00.0 on NUMA socket 0
> EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
> version
> IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> reference
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
> ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
> libibverbs.so.1 with link time reference
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0
> PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> EAL: No probed ethernet devices
>
> The OFED install was successful (and it apparently removes/overwrites
> old versions of the libraries). The installer also checks the firmware
> versions, and it says I'm on the latest. Loading the kernel modules is
> also successful, so I can't for the life of me figure out what this issue is now.
>
> Again, just to be clear, the full list of steps was:
> 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
> the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
> modules 5. Run testpmd
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> adrien.mazarguil-pdR9zngts4EAvxtiuMwx3w@public.gmane.org<mailto:adrien.mazarguil-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>> wrote:
>
>> Hi Raghav,
>>
>> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> > Hi Adrien,
>> >
>> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
>> I'm
>> > a grad student at Princeton working on a research project.
>>
>> It's fine, but let's discuss on the ML as much as possible next time
>> so the community can benefit from it. Since very few people use this
>> driver at the moment (DPDK 2.0 is not even out yet), I have my own
>> questions for you.
>>
>> > Given your patch (which was very helpful, thanks), here is the
>> > series of all steps that I'm taking to run testpmd on a ConnectX-3
>> > EN 40G
>> >
>> > 1. Compile and install the mellanox-specific libraries listed here:
>> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>>
>> This is fine, but did you use the provided Makefile to compile these
>> libraries statically? If so, no issues?
>>
>> Also you did not install Mellanox OFED? You should, because updated
>> kernel modules come from that distribution. It won't work properly
>> (or at all) without them.
>>
>> The libraries from DPDK.org are actually optional once you've
>> installed Mellanox OFED. They only improve performance. Remember to
>> update firmwares also.
>>
>> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
>> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
>> > Set up hugepages using setup.sh 6. Load the kernel modules
>> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
>> > PCI bus address
>> >
>> > However, I consistently get:
>> > sudo: unable to execute ./testpmd: Permission denied
>>
>> Now that doesn't make sense, isn't that file missing permissions or
>> something? Are you running RHEL with a restrictive SELinux policy?
>> Is sudo configured to allow you to run testpmd?
>>
>> In any case, I don't think it's related to librte_pmd_mlx4.
>>
>> > I tried taking the interface down with ifconfig because
>> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
>> > but
>> that
>> > didn't change anything.
>>
>> No, you should leave the interfaces up, it's not supposed to
>> disappear, the PMD will bring them back up anyway. Actually the Linux
>> kernel interface remains fully functional even while DPDK
>> applications are running. You can send packets out of it but as long
>> as both the kernel and DPDK applications share the same MAC address,
>> only DPDK applications will receive replies.
>> If
>> you change the MAC address on the kernel interface however, you'll be
>> able to use it normally.
>>
>> > I tried installing the mellanox libraries globally, but then I see
>> > these error messages:
>> >
>> > -------
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> > reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
>> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
>> > libibverbs.so.1 with link time reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> > Interactive-mode selected
>> > EAL: Error - exiting with code: 1
>> >   Cause: Creation of mbuf pool for socket 0 failed
>> > --------
>>
>> That's because the libibverbs version you're using is not compatible
>> with the userspace drivers installed on your system (libibverbs uses
>> dlopen() to load liblmx4). Even though you've installed both from
>> DPDK.org, seems you have another version installed by your
>> distribution packages.
>>
>> If you intend to use dynamic libraries, you should uninstall
>> libibverbs,
>> libmlx4 and related packages first.
>>
>> Regards,
>>
>> --
>> Adrien Mazarguil
>> 6WIND
>>
>

Re: [PATCH] doc: update mlx4 usage and dependencies

[Raghav Sethi]

Hi Olga,

Thanks so much! I had sent you the output for the case with dynamic
linking. Removing the libraries from /usr/local and re-installing OFED
solved the library issue. However, there is a new problem (below)

An earlier install in the wrong place must have caused the problem. The
OFED uninstaller does not remove the libraries from /usr/local, only
usr/lib apparently. Also, the installer apparently doesn't install them in
/usr/lib if they are present in /usr/local.

Here is the output for testpmd now:

EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
PMD: librte_pmd_mlx4: PCI information matches, using device "mlx4_0" (VF:
false)
PMD: librte_pmd_mlx4: 1 port(s) detected
PMD: librte_pmd_mlx4: experimental ibv_exp_query_device
EAL: No probed ethernet devices

Any idea what the issue is in this case?

Best,
Raghav

On Sat, Apr 4, 2015 at 6:04 PM Olga Shern <olgas-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:

>   If you compiled pmd statically with libibverbs and libmlx4 you
> shouldn't see it in ldd.
> Seems that something went wrong.
> In any case, ofed doesn't install libraries under /usr/local
> Please remove all mlx related libraries under /usr/local/ like libmlx4 and
> libibverbs
>
>
>
>
>
> -------- Original message --------
> From: Raghav Sethi
> Date:05/04/2015 00:29 (GMT+02:00)
> To: Olga Shern ,Adrien Mazarguil ,dev-VfR2kkLFssw@public.gmane.org,Xiaozhou Li
> Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>
>
>  Hi Olga,
>
> Thanks very much for helping.
>
>  I tried stopping and starting, after stop lsmod | grep mlx had no
> matches. After start, it had the output listed below. The device is the
> same as on the rest of the machines - Mellanox ConnectX3-EN 40G single
> port. The application parameters are the same as the ones listed on the
> docs (except of course PCI address is different):
>
>  sudo ./testpmd -c 0xff -n 4 -w 0000:06:00.0 -- --rxq=2 --txq=2 -i
>
>  Outputs you requested:
>
>  1. lspci | grep Mellanox
> 06:00.0 Ethernet controller: Mellanox Technologies MT27500 Family
> [ConnectX-3]
>
>  2. ibdev2netdev
> mlx4_0 port 1 ==> p3p1 (Up)
>
>  3. ldd testpmd
>         linux-vdso.so.1 =>  (0x00007fff7c1dc000)
>         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1a7dc07000)
>         librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
> (0x00007f1a7d9ff000)
>         libibverbs.so.1 => /usr/local/lib/libibverbs.so.1
> (0x00007f1a7d7eb000)
>         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> (0x00007f1a7d5e7000)
>         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007f1a7d3c9000)
>         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1a7d003000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007f1a7df19000)
>
>  4. lsmod | grep mlx
> mlx5_ib               138046  0
> mlx5_core             108376  1 mlx5_ib
> mlx4_ib               199880  0
> ib_sa                  43732  5 rdma_cm,ib_cm,mlx4_ib,rdma_ucm,ib_ipoib
> mlx4_en               137047  0
> ib_mad                 51544  4 ib_cm,ib_sa,mlx4_ib,ib_umad
> vxlan                  37619  1 mlx4_en
> ib_core               130498  12 rdma_cm,ib_cm,ib_sa,iw_cm,
> mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib
> mlx4_core             332375  2 mlx4_en,mlx4_ib
> compat                 13738  17 rdma_cm,ib_cm,ib_sa,iw_cm,
> mlx4_en,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_addr,ib_core,ib_
> umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib
> ptp                    18933  3 igb,ixgbe,mlx4_en
>
>  Best regards,
> Raghav
>
> On Sat, Apr 4, 2015 at 5:04 PM Olga Shern <olgas-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>
>>  Hi Raghav,
>>
>>
>>
>> After running /etc/init.d/openibd restart you don’t need to load the
>> modules, they are loaded by the openibd script.
>>
>> Can you try to run :
>>
>> /etc/init.d/openibd stop
>>
>> Verify that modules are not loaded
>>
>> /etc/init.d/openib start
>>
>>
>>
>> If it won’t help:
>>
>> What Mellanox NIC do you have on the “problematic” machine?
>>
>> What command line do you use to run the application?
>>
>>
>>
>> Please send output of:
>>
>> 1.       ”lspci | grep Mellanox”
>>
>> 2.       ibdev2netdev
>>
>> 3.       ldd <your application>
>>
>> 4.       lsmod
>>
>>
>>
>> Thanks
>>
>> Olga
>>
>>
>>
>>
>>
>> *From:* Raghav Sethi [mailto:raghavs-Mr68eyIdVKO79RPNQ/giaA@public.gmane.org]
>> *Sent:* Saturday, April 04, 2015 11:54 PM
>> *To:* Olga Shern; Adrien Mazarguil; dev-VfR2kkLFssw@public.gmane.org; Xiaozhou Li
>>
>>
>> *Subject:* Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>>
>>
>>
>> Hi Olga,
>>
>> Thanks for your reply. OFED version was 2.4-1.0.4 for ubuntu14.04-x86_64
>>
>> I tried both variants:
>>
>> - Compiling with dynamic linking against OFED libraries
>>
>> - Compiling with static linking against dpdk.org libraries
>>
>>
>>
>> Both have the same issue.
>>
>>
>>
>> Best,
>>
>> Raghav
>>
>>
>>
>> On Sat, Apr 4, 2015 at 4:49 PM Olga Shern <olgas-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>>
>> Hi Raghav,
>>
>> What OFED version did you install?
>>
>> There are 2 ways to compile mlx4:
>> 1. You can compile it dynamically with the libibverbs  that are coming
>> with Mellanox_OFED 2.4
>> 2. To get better performance: compile it statically with libibverbs and
>> libmlx4 that can be downloaded from http://dpdk.org/download, the README
>> inside librte_pmd_mlx4 explains how to do it
>>      If you compile the pmd statically then MLNX_OFED is needed only for
>> kernel modules.
>>
>> Best Regards,
>> Olga
>>
>>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Raghav Sethi
>> Sent: Friday, April 03, 2015 9:21 PM
>> To: Adrien Mazarguil; dev-VfR2kkLFssw@public.gmane.org; Xiaozhou Li
>> Subject: Re: [dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies
>>
>> Hi Adrien,
>>
>> Quick update: So I just tried the setup again on a new machine with the
>> exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a
>> charm.
>>
>> Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
>> From the error message it appears that the libraries are incorrectly
>> installed/configured. However, the OFED uninstall script removes the
>> libraries in question (libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,
>> libibverbs.so), I checked! In fact, I diffed the libibverbs.so from both
>> machines, and they're exactly identical. So it seems either the OFED
>> installation/uninstallation is somehow flawed, or the error message from
>> testpmd is misleading.
>>
>> Best,
>> Raghav
>>
>> On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs-sXtZGUMCxtn536EW/eD9aw@public.gmane.org>
>> wrote:
>>
>> > Hi Adrien,
>> >
>> > I've copied dev@dpdk.
>> >
>> > I re-installed Mellanox OFED, and then re-compiled DPDK without the
>> > flags for the static library. I'm running Ubuntu 14.04. However, I
>> > still get the same error message (below).
>> >
>> > I then tried re-installing the Mellanox libraries on dpdk.org, and
>> > compiled another copy of DPDK with them statically linked. This
>> > appears to work perfectly. I no longer get the permission issues I was
>> > getting before, possibly that was unrelated/fixed by OFED. Now both
>> > OFED, and OFED+ dpdk.org libs statically linked have the same issue:
>> >
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> > version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> > reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
>> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
>> > libibverbs.so.1 with link time reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> >
>> > The OFED install was successful (and it apparently removes/overwrites
>> > old versions of the libraries). The installer also checks the firmware
>> > versions, and it says I'm on the latest. Loading the kernel modules is
>> > also successful, so I can't for the life of me figure out what this
>> issue is now.
>> >
>> > Again, just to be clear, the full list of steps was:
>> > 1. Install OFED, run /etc/init.d/openibd restart 2. Compile DPDK with
>> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y 3. Set up hugepages 4. Load kernel
>> > modules 5. Run testpmd
>> >
>> > Best,
>> > Raghav
>> >
>> > On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
>> > adrien.mazarguil-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> wrote:
>> >
>> >> Hi Raghav,
>> >>
>> >> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> >> > Hi Adrien,
>> >> >
>> >> > Thanks for your reply - I hope it's OK that I'm emailing you
>> off-list.
>> >> I'm
>> >> > a grad student at Princeton working on a research project.
>> >>
>> >> It's fine, but let's discuss on the ML as much as possible next time
>> >> so the community can benefit from it. Since very few people use this
>> >> driver at the moment (DPDK 2.0 is not even out yet), I have my own
>> >> questions for you.
>> >>
>> >> > Given your patch (which was very helpful, thanks), here is the
>> >> > series of all steps that I'm taking to run testpmd on a ConnectX-3
>> >> > EN 40G
>> >> >
>> >> > 1. Compile and install the mellanox-specific libraries listed here:
>> >> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>> >>
>> >> This is fine, but did you use the provided Makefile to compile these
>> >> libraries statically? If so, no issues?
>> >>
>> >> Also you did not install Mellanox OFED? You should, because updated
>> >> kernel modules come from that distribution. It won't work properly
>> >> (or at all) without them.
>> >>
>> >> The libraries from DPDK.org are actually optional once you've
>> >> installed Mellanox OFED. They only improve performance. Remember to
>> >> update firmwares also.
>> >>
>> >> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS 3. Create a new
>> >> > configuration which includes the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> >> > 4. Compile DPDK 2.0 with the new configuration using setup.sh 5.
>> >> > Set up hugepages using setup.sh 6. Load the kernel modules
>> >> > ib_uverbs mlx4_en mlx4_core mlx4_ib 7. Run testpmd with the right
>> >> > PCI bus address
>> >> >
>> >> > However, I consistently get:
>> >> > sudo: unable to execute ./testpmd: Permission denied
>> >>
>> >> Now that doesn't make sense, isn't that file missing permissions or
>> >> something? Are you running RHEL with a restrictive SELinux policy?
>> >> Is sudo configured to allow you to run testpmd?
>> >>
>> >> In any case, I don't think it's related to librte_pmd_mlx4.
>> >>
>> >> > I tried taking the interface down with ifconfig because
>> >> > (./tools/dpdk_nic_bind.py --status shows the interface as active),
>> >> > but
>> >> that
>> >> > didn't change anything.
>> >>
>> >> No, you should leave the interfaces up, it's not supposed to
>> >> disappear, the PMD will bring them back up anyway. Actually the Linux
>> >> kernel interface remains fully functional even while DPDK
>> >> applications are running. You can send packets out of it but as long
>> >> as both the kernel and DPDK applications share the same MAC address,
>> >> only DPDK applications will receive replies.
>> >> If
>> >> you change the MAC address on the kernel interface however, you'll be
>> >> able to use it normally.
>> >>
>> >> > I tried installing the mellanox libraries globally, but then I see
>> >> > these error messages:
>> >> >
>> >> > -------
>> >> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> >> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
>> >> > libibverbs: Warning: couldn't load driver
>> '/usr/lib/libibverbs/libmlx5':
>> >> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> >> version
>> >> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> >> > reference
>> >> > libibverbs: Warning: couldn't load driver
>> '/usr/lib/libibverbs/libmlx4':
>> >> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol
>> >> > ibv_exp_cmd_destroy_flow, version IBVERBS_1.1 not defined in file
>> >> > libibverbs.so.1 with link time reference
>> >> > libibverbs: Warning: no userspace device-specific driver found for
>> >> > /sys/class/infiniband_verbs/uverbs0
>> >> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> >> > EAL: No probed ethernet devices
>> >> > Interactive-mode selected
>> >> > EAL: Error - exiting with code: 1
>> >> >   Cause: Creation of mbuf pool for socket 0 failed
>> >> > --------
>> >>
>> >> That's because the libibverbs version you're using is not compatible
>> >> with the userspace drivers installed on your system (libibverbs uses
>> >> dlopen() to load liblmx4). Even though you've installed both from
>> >> DPDK.org, seems you have another version installed by your
>> >> distribution packages.
>> >>
>> >> If you intend to use dynamic libraries, you should uninstall
>> >> libibverbs,
>> >> libmlx4 and related packages first.
>> >>
>> >> Regards,
>> >>
>> >> --
>> >> Adrien Mazarguil
>> >> 6WIND
>> >>
>> >
>>
>>