[PATCH 0/5] add versioned symbols for recently stabilized APIs

[Dariusz Sosnowski <dsosnowski@nvidia.com>]

Main goal of this patchset is to address https://bugs.dpdk.org/show_bug.cgi?id=1957
but it also handles other recently stabilized symbols and has some minor fixes:

- Patch 1 - Fix RTE_VERSION_EXPERIMENTAL_SYMBOL macro on clang.
- Patch 2 - Allow function versioning inside drivers.
- Patch 3 - Version the function symbols stabilized in
  https://git.dpdk.org/dpdk/commit/?id=e8cab133645f5466ef75e511629add43b68a5027
- Patch 4 - Introduce versioning macros for global variable symbols.
- Patch 5 - Version the function and variable symbols stabilized in
  https://git.dpdk.org/dpdk/commit/?id=4ee2f5c1cedf9ee7f39afa667f71b07f4004ba5c

Issue is still not fully fixed for stabilized global variables:
rte_flow_dynf_metadata_offs and rte_flow_dynf_metadata_mask.
Patch 4 and 5 address the bug for these global variables,
by providing a single storage for both EXPERIMENTAL and
DPDK_26 variable symbol versions.
This is achieved through symbol aliasing.
But this solution is limited only to executables compiled with clang.

clang and gcc have a different default behavior regarding relocations
of global variables exposed by shared libraries.

With clang, R_X86_64_GLOB_DAT relocations are generated for executables:

   $ readelf -sW build-26.07/lib/librte_ethdev.so | grep rte_flow_dynf_metadata_offs
       113: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
       116: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL
       970: 00000000000ea4c0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_impl
      1212: 00000000000ea4c0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_v26
      1325: 00000000000ea4c0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_exp
      1415: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
      1705: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL

    $ readelf -rW build-26.07/drivers/librte_net_mlx5.so | grep rte_flow_dynf_metadata_offs
    0000000003ed5f18  0000001600000006 R_X86_64_GLOB_DAT      0000000000000000 rte_flow_dynf_metadata_offs@DPDK_26 + 0

    $ readelf -rW build-25.11/app/dpdk-testpmd | grep rte_flow_dynf_metadata_offs
--> 000000000028ef70  0000011300000006 R_X86_64_GLOB_DAT      0000000000000000 rte_flow_dynf_metadata_offs@EXPERIMENTAL + 0

With gcc, R_X86_64_COPY relocations are generated:

    $ readelf -sW build-26.07/lib/librte_ethdev.so | grep rte_flow_dynf_metadata_offs
       113: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
       116: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL
      1471: 00000000000e74e0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_impl
      2134: 00000000000e74e0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_v26
      2247: 00000000000e74e0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_exp
      2337: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
      2627: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL

    $ readelf -rW build-26.07/drivers/librte_net_mlx5.so | grep rte_flow_dynf_metadata_offs
    00000000046dbef0  0000001600000006 R_X86_64_GLOB_DAT      0000000000000000 rte_flow_dynf_metadata_offs@DPDK_26 + 0

    $ readelf -rW build-25.11/app/dpdk-testpmd | grep rte_flow_dynf_metadata_offs
--> 000000000029b540  000001d200000005 R_X86_64_COPY          000000000029b540 rte_flow_dynf_metadata_offs@EXPERIMENTAL + 0

With copy relocations (testpmd linked through gcc) the following happens:

- When variable symbol (with EXPERIMENTAL version) gets resolved inside executable,
  global variable gets copied from read-only data to executable's BSS section.
  Executable will access this variable through BSS.
- When variable symbol (with DPDK_26 version) gets resolved inside a library,
  global variable is accessed indirectly through GOT.
  It is stored inside BSS section of the shared library.

So executable and libraries refer to different storage,
eventually leading to inconsistent runtime behavior.
Problems only appears when executable and library require
different versions of global variable symbol.
If testpmd from 26.07 is used with libraries from 26.07,
GOT entry for these variables will point to copied variable.

Without copy relocations (testpmd linked through clang) both
executable and libraries access the global variable indirectly through GOT.
Runtime behavior is consistent, regardless of the mix of variable symbol versions.

The only other solution I could find was to use dlsym() inside libraries
to dynamically resolve the location rte_flow_dynf_metadata_offs and rte_flow_dynf_metadata_mask,
but this solution sounds like an overkill.
Essentially this would require moving to getter/setter functions for these variables
inside the library.

I would appreciate any feedback or suggestions if anybody had encountered a similar issue before.

Dariusz Sosnowski (5):
  eal: fix macro for versioned experimental symbol
  drivers: support function versioning
  net/mlx5: fix stabilized function versions
  eal: support aliases for versioned variable symbols
  ethdev: fix promoted flow metadata symbols

 buildtools/gen-version-map.py        | 11 ++++++++++
 drivers/meson.build                  |  8 +++++++
 drivers/net/mlx5/meson.build         |  2 ++
 drivers/net/mlx5/mlx5_driver_event.c | 22 ++++++++++++++-----
 drivers/net/mlx5/mlx5_flow.c         | 18 ++++++++++-----
 lib/eal/common/eal_export.h          | 24 +++++++++++++++++++-
 lib/ethdev/meson.build               |  2 ++
 lib/ethdev/rte_flow.c                | 33 ++++++++++++++++++----------
 8 files changed, 96 insertions(+), 24 deletions(-)

--
2.47.3