Re: [PATCH 1/1] eal: Don't fail secondary if primary is missing tailqs

[Luca Boccassi <bluca@debian.org>]

On Wed, 2018-11-14 at 17:40 +0000, Burdick, Cliff wrote:
> 
> -----Original Message-----
> From: Bruce Richardson [mailto:bruce.richardson@intel.com] 
> Sent: Wednesday, November 14, 2018 3:48 AM
> To: Burdick, Cliff
> Cc: Thomas Monjalon; Burakov, Anatoly; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH 1/1] eal: Don't fail secondary if
> primary is missing tailqs
> 
> On Tue, Nov 13, 2018 at 11:42:51PM +0000, Burdick, Cliff wrote:
> > > 
> > > ________________________________________
> > > From: Thomas Monjalon [thomas@monjalon.net]
> > > Sent: Tuesday, November 13, 2018 2:18 PM
> > > To: Burdick, Cliff
> > > Cc: Burakov, Anatoly; dev@dpdk.org; bruce.richardson@intel.com
> > > Subject: Re: [dpdk-dev] [PATCH 1/1] eal: Don't fail secondary if 
> > > primary is missing tailqs
> > > 
> > > 13/11/2018 23:08, Burdick, Cliff:
> > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > 13/11/2018 17:38, Burdick, Cliff:
> > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > 13/11/2018 16:45, Burdick, Cliff:
> > > > > > > From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
> > > > > > > > On 13-Nov-18 9:21 AM, Thomas Monjalon wrote:
> > > > > > > > > 13/11/2018 00:33, Burdick, Cliff:
> > > > > > > > > > This patch was submitted by Jean Tourrilhes over
> > > > > > > > > > two 
> > > > > > > > > > years ago, but didn't receive any responses. I hit
> > > > > > > > > > the 
> > > > > > > > > > same issue recently when trying to use cgo (Golang)
> > > > > > > > > > as a 
> > > > > > > > > > primary process linked to libdpdk.a against a C++ 
> > > > > > > > > > application linked against the same library.> > >
> > > > > > > > > 
> > > > > > > > > The question is to know why you don't have the same 
> > > > > > > > > constructors in primary and seconday?
> > > > > > > > 
> > > > > > > > I've hit similar things in the past. I believe it was
> > > > > > > > caused 
> > > > > > > > by our build system stripping out unused libraries
> > > > > > > > (such as 
> > > > > > > > rte_hash) from the binary and thus not calling the 
> > > > > > > > constructor in the primary, but doing so in the
> > > > > > > > secondary 
> > > > > > > > (or something to that effect). In any case, this is
> > > > > > > > caused 
> > > > > > > > by linking different number of libraries to primary
> > > > > > > > and 
> > > > > > > > secondary, and should probably be fixed in the build
> > > > > > > > system, 
> > > > > > > > not in the tailqs code (unless we specifically support 
> > > > > > > > having different linked libraries to primary and 
> > > > > > > > secondary?).> > > >
> > > > > > > 
> > > > > > > Right, I think the original author of the patch stated
> > > > > > > the 
> > > > > > > reasons in the link I provided. The build system seems
> > > > > > > like 
> > > > > > > the most appropriate place to fix it, but the patch got
> > > > > > > me 
> > > > > > > going quickly. I think the question is whether you want
> > > > > > > DPDK 
> > > > > > > to support these other ways of linking. I'm certainly not
> > > > > > > the 
> > > > > > > first to use cgo, since there's a virtual switch project
> > > > > > > doing the same:
> > > > > > > 
> > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__gith
> > > > > > > ub.co
> > > > > > > m_lago
> > > > > > > pu
> > > > > > > s_vsw&d=DwICAg&c=jcv3orpCsv7C4ly8-
> > > > > > > ubDoUfrxF5xIGWmptxGWP5vi5w&r
> > > > > > > =m1RLQ
> > > > > > > OG
> > > > > > > Okz9MauvVLZmiGtyWc5mA7DejbPFNE1IDj-
> > > > > > > 4&m=hQqVCNwW7eoEzB_hLFK97i8
> > > > > > > idS8FI qX 
> > > > > > > oPeclwsIZq7Y&s=BMoBlwkqljwWIBY3SE3pPMCfVnOUlDuZLrno4-
> > > > > > > SojKM&e=
> > > > > > > 
> > > > > > > They don't use primary/secondary processes, though, so
> > > > > > > the 
> > > > > > > issue is  never hit. I'm in a situation where using cgo
> > > > > > > seemed 
> > > > > > > like the easiest  path to accomplish what I'm doing since
> > > > > > > I 
> > > > > > > needed specialized  libraries for it that were not
> > > > > > > available in 
> > > > > > > C/C++. At some point I  bet someone would use Cython to
> > > > > > > start 
> > > > > > > linking against DPDK as well,  and we'd likely run into
> > > > > > > the 
> > > > > > > same issue.> > > > For sure, we want to support using
> > > > > > > DPDK with cgo or cython.
> > > > > > > But it is not clear what is the relation with not having
> > > > > > > the 
> > > > > > > same compilation for primary and secondary. Please could
> > > > > > > you 
> > > > > > > elaborate?> > >
> > > > > > 
> > > > > > Hi Thomas, I think Jean explained it well here:
> > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__dev.dp
> > > > > > dk.narkive.
> > > > > > com_ZM3a7QD1_dpdk-2Ddev-2Dbug-2Dstatic-2Dconstructors-
> > > > > > 2Dconsider
> > > > > > ed-2De 
> > > > > > vil&d=DwICAg&c=jcv3orpCsv7C4ly8-
> > > > > > ubDoUfrxF5xIGWmptxGWP5vi5w&r=m1R
> > > > > > LQOGOk 
> > > > > > z9MauvVLZmiGtyWc5mA7DejbPFNE1IDj-
> > > > > > 4&m=C69wDgrjDX8_oXp1M_3bnmWOOZd
> > > > > > GqwBBG 
> > > > > > 9vzkyGDWGQ&s=YYn2N7WrzJpH1ptNrLZad0nPAQdrUyqBckk2uFuWYPQ&e=
> > > > > > 
> > > > > > "The build system of the application does not have all the 
> > > > > > subtelties of the DPDK build system, and ends up including
> > > > > > *all* 
> > > > > > the constructors, wether they are used or not in the code. 
> > > > > > Moreover, they are included in a different order. Actually,
> > > > > > by 
> > > > > > default the builds include no constructors at all (which is
> > > > > > a 
> > > > > > big fail), so the library needs to be included with 
> > > > > > --whole-archive (see Snort DPDK instructions)."
> > > > > > 
> > > > > > I will get to the bottom of my exact case to understand
> > > > > > what's 
> > > > > > happening, but my primary application is a cgo application
> > > > > > that 
> > > > > > I'm linking to by using almost exactly the same flags that
> > > > > > are 
> > > > > > used in the DPDK build system to build examples. The DPDK 
> > > > > > libraries I'm linking against is a single location for
> > > > > > both 
> > > > > > primary and secondary; in other words, I don't build DPDK
> > > > > > twice.
> > > > > > 
> > > > > > OK I understand, thanks.
> > > > > > 
> > > > > > You had alluded to a pkg-config for DPDK in the 2015
> > > > > > thread, 
> > > > > > which cgo can use, but I don't know if that ever was 
> > > > > > implemented. Cgo can use pkg-config if it's available,
> > > > > > otherwise 
> > > > > > the only tools are specifying LDFLAGS and CFLAGS into their
> > > > > > build system.
> > > > > 
> > > > > Yes, the right answer is still pkg-config :) The good news is
> > > > > that it is now implemented thanks to the meson build system:
> > > > >     
> > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__git.dpdk.
> > > > > org_dp
> > > > > dk_tree_doc_build-2Dsdk-2Dmeson.txt-
> > > > > 23n182&d=DwICAg&c=jcv3orpCsv7C4
> > > > > ly8-
> > > > > ubDoUfrxF5xIGWmptxGWP5vi5w&r=m1RLQOGOkz9MauvVLZmiGtyWc5mA7Dej
> > > > > bP
> > > > > FNE1IDj-
> > > > > >4&m=C69wDgrjDX8_oXp1M_3bnmWOOZdGqwBBG9vzkyGDWGQ&s=oC86KD_R
> > > > > J1T6rfzi3x5zFT1Ri13ELpKmsyFqpgDbgFg&e=
> > > > 
> > > > Hi Thomas, are there instructions on how to use pkg-config to
> > > > build the mlx4/5 PMD? I noticed a patch was submitted in
> > > > September to add support for it, but that link you provided on
> > > > using meson doesn't say how to build specific drivers. It
> > > > appears to be disabled by default.
> > > > If the dependency is found, meson will enable the PMD when
> > > > building DPDK.
> > > 
> > > Do you know where exactly that is? I've been using mlx5 for a
> > > while on this system, and I can see that 18.11-rc2 meson
> > > build+ninja built the pmd, but it's not on the --libs listing for
> > > pkg-config. Does it tell me what I was missing?
> > > 
> > 
> > For dynamic linking of applications, the drivers are not normally
> > linked in. Instead, they should be loaded from the drivers
> > directory as .so files
> > - normally by default in EAL as the driver .so's should be copied
> > to the EAL_PMD_PATH where EAL finds them automatically. [This
> > applies to both meson and make builds, though only meson generates
> > the .pc file for pkg-config]
> > 
> > If you are doing a static build, then you need to explicitly link
> > in the drivers. You can get a list from pkg-config using the "
> > --static" flag in this case. A good example of how to use pkg-
> > config in this way can be found in the Makefiles for most examples,
> > e.g. examples/helloworld/Makefile, reproduced below.
> > 
> > Regards,
> > /Bruce
> > 
> > all: shared
> > .PHONY: shared static
> > shared: build/$(APP)-shared
> >         ln -sf $(APP)-shared build/$(APP)
> > static: build/$(APP)-static
> >         ln -sf $(APP)-static build/$(APP)
> > 
> > PC_FILE := $(shell pkg-config --path libdpdk) CFLAGS += -O3 $(shell
> > pkg-config --cflags libdpdk) LDFLAGS_SHARED = $(shell pkg-config --
> > libs libdpdk) LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --
> > static --libs libdpdk)
> > 
> > build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
> >         $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS)
> > $(LDFLAGS_SHARED)
> > 
> > build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
> >         $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS)
> > $(LDFLAGS_STATIC)
> > 
> > build:
> >         @mkdir -p $@
> 
> Thanks Bruce. I tried getting this to compile using cgo, and it's
> causing more pain than it's worth. I used the --static --libs
> options, and there were still numerous linker errors, some specific
> to mlx, and some not. Specifically libmlx5 and libmnl are both
> needed, but they're not part of the linker line from pkg-config. At
> this point I'll just break the Go application up into a separate C
> application that handles all the DPDK parts, and send messages
> between them.

Hi,

As far as I can see, both mnl and mlx5 (and all the other reverse
dependencies) are listed correctly in the libdpdk.pc Libs.private
entry, and pkg-config --libs --static libdpdk.pc does print them as
intended. This is with 18.11-rc3.
Are you sure everything is correct (pkg-config path is right, --static
is used, etc)?

-- 
Kind regards,
Luca Boccassi