From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adrien Mazarguil Subject: Re: [PATCH] mk: fix application compilation with lmnl and mlx5 Date: Tue, 24 Jul 2018 17:13:50 +0200 Message-ID: <20180724151349.GT5211@6wind.com> References: <14690e825609ee181e3cd522302d4788ef436f35.1532424524.git.nelio.laranjeiro@6wind.com> <20180724125556.GS5211@6wind.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: =?utf-8?B?TsOpbGlv?= Laranjeiro , "dev@dpdk.org" , Yongseok Koh To: Shahaf Shuler Return-path: Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by dpdk.org (Postfix) with ESMTP id 72BFD1DA4 for ; Tue, 24 Jul 2018 17:14:07 +0200 (CEST) Received: by mail-wr1-f68.google.com with SMTP id h9-v6so4536416wro.3 for ; Tue, 24 Jul 2018 08:14:07 -0700 (PDT) Content-Disposition: inline In-Reply-To: List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Tue, Jul 24, 2018 at 01:03:28PM +0000, Shahaf Shuler wrote: > Tuesday, July 24, 2018 3:56 PM, Adrien Mazarguil: > > Subject: Re: [PATCH] mk: fix application compilation with lmnl and mlx5 > > > > On Tue, Jul 24, 2018 at 11:21:52AM +0000, Shahaf Shuler wrote: > > > Tuesday, July 24, 2018 12:29 PM, Nelio Laranjeiro: > > > > Subject: [PATCH] mk: fix application compilation with lmnl and mlx5 > > > > > > > > When Mellanox MLX5 PMD is compiled with > > > > CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS=y, the external dependency > > on > > > > libmln is missing. > > > > > > > > Fixes: 4d5cce06231a ("net/mlx5: lay groundwork for switch offloads") > > > > Cc: adrien.mazarguil@6wind.com > > > > > > > > Signed-off-by: Nelio Laranjeiro > > > > --- > > > > mk/rte.app.mk | 2 +- > > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk index > > > > f4d28c2da..ff39d37aa > > > > 100644 > > > > --- a/mk/rte.app.mk > > > > +++ b/mk/rte.app.mk > > > > @@ -149,7 +149,7 @@ else > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 - > > > > libverbs -lmlx4 > > > > endif > > > > ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS),y) > > > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 -ldl > > > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 -ldl > > - > > > > lmnl > > > > > > This issue raise some more basic question. > > > The DLOPEN mode was introduced to run in systems which don't have > > verbs/mlx5 libs installed, because those were the only dependencies for the > > PMD back then. > > > Now we have the libmnl, which is external dependency just like rdma-core, > > and following your fix, hard linked also in case of DLOPEN option. > > > It means the whole DPDK binary/lib will be depended on libmnl and this is > > not what we want with DLOPEN. > > > > > > Can we consider different options: > > > 1. always statically link libmnl > > > 2. dlopen libmnl just like we do for verbs/mlx5 > > > > Regarding 2, unlike rdma-core/MLNX_OFED, libmnl should be available pretty > > much everywhere iproute2 can be found. The minimal version supported > > (1.0.3) was released in 2012. > > > > Using the glue approach for such a small library seems overkill; should we > > choose this path, we must also consider to get rid of it entirely since doing so > > would require more glue code than what mlx5 needs from this library. > > > > So with the current approach, either the application or the PMD inherits a > > dependency to libmnl, depending on whether > > CONFIG_RTE_BUILD_SHARED_LIB is respectively disabled or enabled. > > > > If disabled, applications that want static linkage can specify -static as part of > > their compilation flags to let the compiler automatically look for libmnl.a as > > needed. > > Can you confirm static compilation w/ libmnl is working w/o any issues? Challenge accepted. Using -static is not something DPDK applications usually do and needs a few tweaks to compile successfully though (unless you meant something else by "static compilation"?) First you need to force rdma-core to generate static versions of libmlx4/libmlx5 and libiberbs as it's not the default: $ cmake -DENABLE_STATIC=1 . Next, patch DPDK to remove references to -lgcc_s, -fPIC and -export-dynamic: $ sed -i 's/[[:space:]]*-lgcc_s//' $(git grep -l -- -lgcc_s) $ sed -i 's/[[:space:]]*-fPIC//' $(git grep -l -- -fPIC) $ sed -i 's/[[:space:]]*-export-dynamic//' $(git grep -l -- -export-dynamic) Then append rdma-core dependencies to libmnl users (mlx5) in order to avoid undefined references to libnl/libm stuff normally pulled by libibverbs: $ sed -i 's/-lmnl/-lmnl -lnl-route-3 -lnl-3 -lm/' $(git grep -l -- -lmnl) Configure DPDK with mlx5 enabled: $ make config T=x86_64-native-linuxapp-gcc $ sed -i 's/^\(CONFIG_RTE_LIBRTE_MLX5_PMD\)=.*/\1=y/' build/.config Finally add -static to $CC (the easiest method I could find) while compiling DPDK: $ make CC='gcc -static' You can ignore warnings about statically linking "getaddrinfo" and friends. What matters is we now have a testpmd binary with no dependencies: $ readelf -d build/app/testpmd | grep NEEDED $ $ ldd build/app/testpmd not a dynamic executable $ Now start testpmd with a mlx5 adapter and a couple of representors for fun: # ./build/app/testpmd -w 06:00.0,representor=[0-1] -n 4 -c 0x80 -- -i --rxq=1 --txq=1 EAL: Detected 32 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 15b3:1017 net_mlx5 [...] Configuring Port 0 (socket 0) Port 0: 24:8A:07:8D:AE:F6 Configuring Port 1 (socket 0) Port 1: A6:41:D9:89:DB:12 Configuring Port 2 (socket 0) Port 2: FE:A8:15:80:DE:C0 Checking link statuses... Done testpmd> Phew! > To put this in perspective, this also applies to all other dependencies > > it will collect while compiling DPDK (libz, libdl, libpcap, libnuma to name a > > few). > > > > In my opinion, the purpose of *_DLOPEN_DEPS is to deal with large, > > nonstandard libraries where versioning issues are commonplace. This doesn't > > apply to libmnl, which shouldn't be a maintenance nightmare to package > > maintainers. I suggest to leave things as is. -- Adrien Mazarguil 6WIND