Re: [PATCH] mk: fix application compilation with lmnl and mlx5

[Adrien Mazarguil <adrien.mazarguil@6wind.com>]

On Tue, Jul 24, 2018 at 01:03:28PM +0000, Shahaf Shuler wrote:
> Tuesday, July 24, 2018 3:56 PM, Adrien Mazarguil:
> > Subject: Re: [PATCH] mk: fix application compilation with lmnl and mlx5
> > 
> > On Tue, Jul 24, 2018 at 11:21:52AM +0000, Shahaf Shuler wrote:
> > > Tuesday, July 24, 2018 12:29 PM, Nelio Laranjeiro:
> > > > Subject: [PATCH] mk: fix application compilation with lmnl and mlx5
> > > >
> > > > When Mellanox MLX5 PMD is compiled with
> > > > CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS=y, the external dependency
> > on
> > > > libmln is missing.
> > > >
> > > > Fixes: 4d5cce06231a ("net/mlx5: lay groundwork for switch offloads")
> > > > Cc: adrien.mazarguil@6wind.com
> > > >
> > > > Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
> > > > ---
> > > >  mk/rte.app.mk | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk index
> > > > f4d28c2da..ff39d37aa
> > > > 100644
> > > > --- a/mk/rte.app.mk
> > > > +++ b/mk/rte.app.mk
> > > > @@ -149,7 +149,7 @@ else
> > > >  _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD)       += -lrte_pmd_mlx4 -
> > > > libverbs -lmlx4
> > > >  endif
> > > >  ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS),y)
> > > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD)       += -lrte_pmd_mlx5 -ldl
> > > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD)       += -lrte_pmd_mlx5 -ldl
> > -
> > > > lmnl
> > >
> > > This issue raise some more basic question.
> > > The DLOPEN mode was introduced to run in systems which don't have
> > verbs/mlx5 libs installed, because those were the only dependencies for the
> > PMD back then.
> > > Now we have the libmnl, which is external dependency just like rdma-core,
> > and following your fix, hard linked also in case of DLOPEN option.
> > > It means the whole DPDK binary/lib will be depended on libmnl and this is
> > not what we want with DLOPEN.
> > >
> > > Can we consider different options:
> > > 1. always statically link libmnl
> > > 2. dlopen libmnl just like we do for verbs/mlx5
> > 
> > Regarding 2, unlike rdma-core/MLNX_OFED, libmnl should be available pretty
> > much everywhere iproute2 can be found. The minimal version supported
> > (1.0.3) was released in 2012.
> > 
> > Using the glue approach for such a small library seems overkill; should we
> > choose this path, we must also consider to get rid of it entirely since doing so
> > would require more glue code than what mlx5 needs from this library.
> > 
> > So with the current approach, either the application or the PMD inherits a
> > dependency to libmnl, depending on whether
> > CONFIG_RTE_BUILD_SHARED_LIB is respectively disabled or enabled.
> > 
> > If disabled, applications that want static linkage can specify -static as part of
> > their compilation flags to let the compiler automatically look for libmnl.a as
> > needed. 
> 
> Can you confirm static compilation w/ libmnl is working w/o any issues? 

Challenge accepted. Using -static is not something DPDK applications usually
do and needs a few tweaks to compile successfully though (unless you meant
something else by "static compilation"?)

First you need to force rdma-core to generate static versions of
libmlx4/libmlx5 and libiberbs as it's not the default:

 $ cmake -DENABLE_STATIC=1 .

Next, patch DPDK to remove references to -lgcc_s, -fPIC and -export-dynamic:

 $ sed -i 's/[[:space:]]*-lgcc_s//' $(git grep -l -- -lgcc_s)
 $ sed -i 's/[[:space:]]*-fPIC//' $(git grep -l -- -fPIC)
 $ sed -i 's/[[:space:]]*-export-dynamic//' $(git grep -l -- -export-dynamic)

Then append rdma-core dependencies to libmnl users (mlx5) in order to avoid
undefined references to libnl/libm stuff normally pulled by libibverbs:

 $ sed -i 's/-lmnl/-lmnl -lnl-route-3 -lnl-3 -lm/' $(git grep -l -- -lmnl)

Configure DPDK with mlx5 enabled:

 $ make config T=x86_64-native-linuxapp-gcc
 $ sed -i 's/^\(CONFIG_RTE_LIBRTE_MLX5_PMD\)=.*/\1=y/' build/.config

Finally add -static to $CC (the easiest method I could find) while compiling
DPDK:

 $ make CC='gcc -static'

You can ignore warnings about statically linking "getaddrinfo" and
friends. What matters is we now have a testpmd binary with no dependencies:

 $ readelf -d build/app/testpmd | grep NEEDED
 $

 $ ldd build/app/testpmd
         not a dynamic executable
 $

Now start testpmd with a mlx5 adapter and a couple of representors for fun:

 # ./build/app/testpmd -w 06:00.0,representor=[0-1] -n 4 -c 0x80 -- -i --rxq=1 --txq=1
 EAL: Detected 32 lcore(s)
 EAL: Detected 2 NUMA nodes
 EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
 EAL: No free hugepages reported in hugepages-1048576kB
 EAL: Probing VFIO support...
 EAL: PCI device 0000:06:00.0 on NUMA socket 0
 EAL:   probe driver: 15b3:1017 net_mlx5
 [...]
 Configuring Port 0 (socket 0)
 Port 0: 24:8A:07:8D:AE:F6
 Configuring Port 1 (socket 0)
 Port 1: A6:41:D9:89:DB:12
 Configuring Port 2 (socket 0)
 Port 2: FE:A8:15:80:DE:C0
 Checking link statuses...
 Done
 testpmd>

Phew!

> To put this in perspective, this also applies to all other dependencies
> > it will collect while compiling DPDK (libz, libdl, libpcap, libnuma to name a
> > few).
> > 
> > In my opinion, the purpose of *_DLOPEN_DEPS is to deal with large,
> > nonstandard libraries where versioning issues are commonplace. This doesn't
> > apply to libmnl, which shouldn't be a maintenance nightmare to package
> > maintainers. I suggest to leave things as is.

-- 
Adrien Mazarguil
6WIND